Why projects fail

The project management industry has matured and grown significantly over the past 15 years, but a large number of projects still fail. Why is this the case?

Project failures are more prevalent and more persistent than you might think. In fact, published project failures are only the tip of the iceberg! Human nature being what it is, a great deal of energy and ingenuity is spent trying to find ways to make failed projects appear successful.

The root of project failure

A key observation from our work across a wide range of projects is that having a methodology is usually better than not having one at all. However, slavishly following a generic or one-size-fits-all methodology is usually a recipe for well-organised failure.

Organisations commonly force the project to fit their standard methodologies rather than vice versa. Mostly this is because they don’t have tools to enable them to quantify the characteristics of the project and its level of complexity.

Projects that avoid failure often do so because the project manager is skilled enough to ‘work the system’ to do what is needed to make the project successful. Further to this, much of what successful project managers do is based on experience and intuition. This experience and intuition is not documented, and the project managers themselves often have difficulty consciously articulating what they do or why they do it.

The result of this is that being successful on one project doesn’t mean you will be successful on the next. In fact one of the main dangers for project managers is generalising from their own specific experience.

A more scientific way of putting this is that successful project managers rely on decision-making heuristics that are largely formed sub-consciously through experience and their own perceptual wiring or, as the saying goes, ‘experience is not what happens to you, it’s what you do with what happens to you’.

Where’s the research?

To date, there has been little in the way of useful quantitative research into these heuristics and where they do or don’t work. Research into project failures has been mostly case-based, that is, anecdotal or specific to a given situation, and after the fact in the brilliance of hindsight. Guidance from this type of research runs the risk of generalising from the specific or simply being misunderstood.

As a result there is very little in the way of systematic or scientific assistance available to project managers that can help them predict and prevent failure, and particularly not when they’re in the ‘heat of battle’. Most of the current tools that are supposed to do this are not empirically proven and usually default to being little more than compliance exercises (e.g. most risk logs). There’s comfort in following process in the absence of insight.

Key challenges of addressing project failure

There is no single right answer and certainly no silver bullet when addressing why projects fail. To know what works best in what situation, first you need a lot of data. Most of the data we need about what project managers actually do is not collected in project management systems or project documentation.

So the answer to the question “Why do a large number of projects still fail?” may be to do with the fact that while many organisations use best practice and ensure their processes conform to industry standards, they do so under the assumption that these approaches will work in their specific situation.

Every project is unique so it should be no surprise that when applying standard methodologies across different projects with different groups of stakeholders, outcomes can fall short—or projects fail.

More successful project managers usually have a deep understanding of what makes a particular project—and just as importantly the people involved with it—tick; they then adjust their management approaches accordingly. How they do this, however, is something of a mystery, even to the project managers themselves.

How data can address project failure

Shedding light on this ‘dark art’ means taking on the momentous task of surveying project management professionals to learn about the practices they employ, the people with whom they work, and their experiences within the project management industry.

By collecting survey data directly from the people that do the work (i.e. project professionals), it will be possible to develop a deeper understanding of what actually works within the ‘DNA’ of a particular organisation. It may even be possible, given enough data, to calibrate specific solutions for each organisation.

Collecting data is one thing but it’s what is done with the data that counts. A new research tool, My Project Panel, is an online initiative to gather data from which a variety of tools might be developed. A more scientific way of putting this is that by utilising collected data, it’s possible to ‘reverse engineer’ the heuristics used by successful project managers and the situations in which those heuristics are applicable, through statistical analysis.

Data collected through My Project Panel will enable a range of project management tools to be developed, validated and calibrated. Typically those tools will be aimed at determining what is ‘right practice’ in a given situation. This will enable organisations to better utilise their resources and to better direct their investments in developing project delivery capability.

Of course the more data collected the greater the accuracy and granularity (specificity) of the tools. The stretch target is to be able to calibrate tools to specific organisations and, in effect, provide the project management community with access to the best thinking in the industry on how to tackle a given situation.

So the hope is that My Project Panel will collect perspectives and experiences from practicing project managers so that it can be learnt what works when, and combine this with the ‘hard’ project data via a unique set of algorithms to identify the areas that need improvement.

The people behind My Project Panel are passionate about what they do, and they want to bring about change at an industry level. Specifically, they want to help individuals who work in projects to improve competencies and processes. In a broad sense this has the potential to impact people from all walks of life, with the real reward knowing that meaningful change has been brought to the industry that many people have dedicated their working lives to.

Time—and the volume of data My Project Panel is able to collect—will be the ultimate decider. Every project professional can add to the research at myprojectpanel.com to help bring about meaningful change.

By Jeroen Bolluijt and Brett Nan Tie, co-directors of My Project Panel