Evaluating a program management office

Risk/issue management
Risk and issue management is a critical aspect of any PgMO and any program or project, and your metrics should include these factors. Merely measuring the numbers of risks and issues is not an effective indicator. The number of issues escalated to the PgMO from the projects could be a useful indicator of either a poor interrelationship between projects managers and the program office or an understanding of risks and issues and interdependencies across projects. The PgMO is not designed to micro-manage project risks and issues, but metrics capturing, at the program level, the effective management of risks and issues at the project level, as well as those managed by the PgMO, should be considered.

When issues arise, having a means to manage, track and report is important. An advanced PgMO may consider as a metric the percentage of issues with identified root causes and actions to rectify them (and the progress of such actions).

Detailing the measurement processes
The measurement plan should include key definitions for collecting and reporting metrics data, including what is meant by each metric (e.g. the operational definition as well as any normalisation/modification required), source of the data, who is responsible for collecting and analysing it and to whom it must be reported.

The communication plan should also detail how the metrics will be delivered to the various stakeholders. Delivering metrics to a stakeholder in a way that isn’t properly understood, regardless of how positive it may be, can alienate the PgMO. Stakeholders can suffer from data overload, and lose the intended message associated with the measures.

Having the proper mix of metrics is important. For example, both outcomes/results as well as in-process metrics should be developed. The latter are useful predictors (leading indicators) of the results that will be achieved, and allow taking corrective action. Measurement should look at both effectiveness (meeting primary customer/stakeholder requirements) and efficiency (how well organisational resources were used in carrying out the program).

Example metrics

Having looked at the considerations of a measurement strategy, let’s now turn our attention to some specific metrics strategies based on the concept of PgMO maturity. As has been observed, there is no standard set of metrics that will work for every PgMO. When planning the metrics for the PgMO, do so with the understanding that the metrics needs to be insightful, strategically focused, and help to drive decisions rather than telling people what they already know.

The right metrics for a PgMO will enable decisions which facilitate business strategy, increasing the value of the PgMO. Collecting and reporting the wrong metrics will make demonstrating the true value of the PgMO a challenge. While the exact metrics vary depending on the type of PgMP and other factors, the following are some example categories stratified by the maturity of a PgMO.

For a newly formed (young) PgMO:

• The number of interactions between stakeholders, project managers, and other key players. Since these interactions are critical to effective program outcomes, the PgMO can help ensure that they occur, increasing the probability of success (a process metric).
• Since program outcomes are of course important, the rollup status of projects (e.g. variance in timelines and resource usage, projected outcomes) should allow early detection of critical interface problems (process metrics).
• All programs, and therefore PgMOs should track and measure benefits. The benefits should be captured and reported based on the strategic objectives of the organisation (outcome metrics).

For an experienced PgMO:

• More in-depth status of critical interfaces, such as the number of problems encountered and resolved, and estimates of reliability/risks of program outcomes (process metrics).
• Alignment of program to stakeholder interests, such as stakeholder feedback relative to concerns and satisfaction.
• Cost of operating the PgMO (outcome metric, but efficiency rather than effectiveness).

For a mature PgMO:

• Value of the PgMO, such as value added and costs avoided, divided by the cost of PgMO.
• Project, program, and/or system technology knowledge/skills developed and deployed across projects and the organisation through PgMO efforts (outcomes).
• Comparison of the PgMO to benchmark PgMOs (could be outcome or process metrics).
• Percent of issues for which root cause was determined, and the ongoing benefits resulting from resolving the root causes through changes in PgMO processes.

Determining the level of maturity of the PgMO is complex, as it may be related to the length of time the office has been in place, the level of standards and/or skills used by the office, the number or complexity of tools used for program and project management and the number of successful program outcomes. A measurement system based on audits of the PgMO office processes, guided by a maturity rubric/matrix, can be useful when determining maturity level.

Far too often, measurement systems used in project/program management focus on what has been done, rather than whether the strategic intent of an initiative is being or has been attained. A PgMO, if properly designed and aligned with your business strategy, should capture metrics that help the organisation understand where they are on this trajectory and what ongoing actions will deliver success. Developing the right measurement system, and obtaining agreement on that system from key stakeholders, is a critical part of the PgMO, since it drives the way in which it operates.