Measuring up to good project KPIs

Communication occurs at many levels and the result of any one communication is based on what the receiver perceives as important. Asking questions can be a very effective way of communicating and influencing stakeholders, but asking the wrong questions can easily be the cause of undesirable outcomes—exercise care!

One of the key communication channels in most projects is the request from PMOs and similar bodies to project management teams for information and the supply of the data requested. Quality guru W Edwards Deming once quipped: “In God we trust, all others bring data.” Combined with the modern management mantra ‘you cannot manage what you cannot measure’, this illustrates the importance of collecting data, though you also have to make sure you’re measuring and communicating the right things.

While exchanging data with the PMO may not look like a communication, and is very often structured within formal policies, the reality can be very different; if you ignore what’s actually being communicated, you can get very undesirable outcomes.

Bad KPIs encourage bad behaviour

A policy introduced to measure the performance of our local hospitals a few years ago offers a salient lesson. The Victorian Government decided to ‘incentivise’ its hospitals by rewarding good performance and fining them for poor performance, using a standard set of key performance indicators (KPIs).

One key measure was the time patients wait in the casualty/emergency area before being admitted to the hospital. To achieve a good KPI, some administrators were manipulating data to avoid fines and win bonuses, both adjusted against the hospital’s funding for the year so no one personally benefitted from the manipulation. But the problems only surfaced after an audit report found KPI-induced behaviours that in many cases were worse for the patients and worse for the hospital than if nothing had occurred.

One example was the practice of transferring patients from emergency care to the operating theatre waiting area to avoid a fine for failing to admit the patient within the prescribed maximum time. The consequences of this action included:

• The patient being removed from an area focused on emergency care to an area with little monitoring capability, which meant a reduction in care to the patient.
• The reduction in throughput in the operating suites due to overcrowding and skilled theatre staff having to spend time on patient care rather than operations.
• A net reduction in the overall delivery of service by the hospital but an improved statistical report to government.

This structure of the KPI-induced behaviour was in the interests of everyone except the people the government and hospital are supposed to serve, the public needing hospital care. Some of the vested interests included:

• The government’s desire to look good by ‘reducing’ patient waiting time.
• The hospitals desire to achieve the maximum budget income for the year.
• The administrators’ desire, both in the hospital and the government, to avoid ‘rocking the boat’.

The government had its data and the system responded to the stimulus of the KPI, but everyone forgot the key objective—enhanced patient care.

There are a number of important lessons in this story for all PMO managers, portfolio, program and project managers to consider when setting up project dashboards and the like:

• The KPIs you choose are communicating information to stakeholders on what you think is ‘most important’ but what is easy to measure is not necessarily important.
• What you choose to measure will change behaviours. Focus on things that matter, like value and benefits, not easy-to-measure statistics like time and cost.
• Make sure the data you use is validated.
• A KPI system can not solve ‘the problem’ but it can be a powerful facilitator of solutions if it is set to measure the right statistics and ask the right questions.

And remember, if you pick the wrong KPI you will get behaviour changes that are worse than before the KPI was introduced. This paradox operates like Gresham’s law with a twist: the easy-to-measure drives out the harder to quantify, even when the latter is more important.

How useful is your information?

Gresham’s law is an economic principle that states: ‘When a government compulsorily overvalues one type of money and undervalues another, the undervalued money will leave the country or disappear from circulation into hoards, while the overvalued money will flood into circulation.’

It is commonly stated as: ‘Bad money drives out good’ or, in our case, simplistic information drives out useful information. Is it really worth destroying thousands of dollars in value by de-scoping a project simply to avoid finishing two weeks later than planned? Or is it better to have an informed conversation with the key stakeholders over value and what really matters?

Simply identifying a problem and creating a KPI is not enough! You should work with the project team to make sure an effective solution is crafted, and then measure the effectiveness of the solution. This is far more challenging than simply processing monthly reports on easily accessed information such a schedule performance, but is also one that can really contribute to the overall performance of your organisation.

Project reports are part of the project communication framework and the purpose of communication is to achieve an effect; you just need to make sure what you are communicating will actually achieve the effect you desire. What messages are the KPIs in your monthly reports sending?

For a more comprehensive resource on communication models, see Communication Theory [PDF]