6 mistakes lean practitioners make

Companies of all ages, sizes and industries are embracing lean principles in an effort to become more innovative, more profitable and more relevant to their customers. Lean can help a business by putting the focus on the key activities that create value for customers. Backed by an endless number of case studies of successful projects, lean becomes a compelling proposition.

But for all the success stories there are also cautionary tales of efforts that have faltered along the way. When using lean techniques in environments of high uncertainty, like markets under disruption or for innovation, the most common cause of ineffectiveness comes from the inability or unwillingness to learn from past experiments. Here are six common reasons why learning can be so difficult and how to avoid them.

1. Having inconclusive outcomes

Eric Ries’ The Lean Startup (2011) has been a significant addition to the lean movement. One of the key principles espoused by Ries is the Build-Measure-Learn feedback loop when executing in environments of uncertainty. The idea is to derive knowledge from assumptions, risks and unknowns, and to use that knowledge to guide teams and companies through uncertain environments. The result is a more stable path through disruption or to innovation. For this reason, the key measure of success is the cycle time of learning, or how quickly a hunch is validated or invalidated.

But what happens when there is no learning? It’s not uncommon to go through the build and measure parts of a cycle, to reach the end of the experiment only to discover that you’ve arrived at an unexpected, or worse, inconclusive result where it’s unclear what actually happened. There is nothing to learn and no knowledge to be gained.

In an experiment, such an outcome is a major problem. The whole idea of Build-Measure-Learn is to turn key unknowns into knowledge. You don’t get credit for just building something; you have to have created knowledge.

One way to avoid this outcome is to insert an additional step in the cycle: Frame. Before you start to build, you first need to frame the problem and frame the experiment.

Framing the problem helps direct your experiment toward the key unknown at hand. It involves setting boundaries around what the experiment is meant to learn or solve. Your ability to do this will be intricately tied to your ability to visualise and understand customer needs.

Once you have a sense of the problem, you then need to detail how you will run the experiment. Consider the reasons that led you to this point. What do you want to learn and why? What information feeds into your work from the Frame step? Then define your expectations for the experiment. This is your hypothesis and it’s what you should repeatedly expect as a result. Once it is well-articulated, you should be in a position to avoid inconclusive results.

2. Failing to account for variables

Even with good framing, outcomes are not always going to lead to consistent results that allow you to learn. There are dependent and independent variables that will impact your experiment. If you fail to account for these, your results will be inconsistent and impossible to analyse. It is key to understand your control variables. These are the elements in the environment you need to hold constant as much as possible to ensure you can understand the relationship between your actions and the results.

3. Lacking discipline in documentation

If we don’t document the expected results of our experiments, it’s human nature to perceive that the results match our expectations. Learning comes from the tension between what we expected would happen and what actually happened.

Before you move on to the build cycle, document everything: why you’re running the experiment, what you expect will happen and how it’s safe to run and recover from the experiment. The discipline keeps you focused and honest.

When you come to the end of the experiment, review the expectations you started out with, compare them to the results and seek to learn where invalidation has occurred. These are the moments when learning creates knowledge.

4. Lacking data

You need a quantity of robust data to derive actionable information. And for that, you will require collection methods.

As you build your experiment, consider what else you may need to build to ensure you can collect the appropriate data during the experiment run. Remember, you want to build the smallest possible increment sufficient to validate or invalidate your hypothesis with data. You may find you only need a paper prototype or a landing page to collect the necessary data.

Once designed and built, it’s time to run your experiment and collect your data. This could be as simple as conducting an invalidation interview with your potential customers or as elaborate as shipping software. The key is to work in increments that maximise the number of learning cycles you can run.

5. Inadequate sharing of knowledge

You’ve run the experiment, collected and analysed your data, and looked at how the results relate to your hypothesis. It’s time to share the findings with the rest of your team. You also need to decide how you will preserve the knowledge and map out what the findings mean for your next experiment.

Make sure that you share your findings in a very transparent way. If you are part of a co-located team, post materials on a wall in your workspace. Otherwise, put your materials online in a shared space where others can see them. This way, your team can easily access the current business model, assumptions, and experiments.

6. Having a culture that doesn’t support experimentation

In some organisations it can be difficult to do work where the outcome can’t be predicted exactly. In these environments it’s typical to run multiple experiments and each time the expected results match the outcomes of the experiment. In this case you’re spending the extra overhead of running experiments to confirm knowledge you already have. Strive to be surprised by the results of experiments at least 30% to 50% of the time. An unexpected result is different than an experiment that ‘blows up’. It’s in the balance of expected and unexpected results where knowledge creation is maximised.

There you have it, six common reasons why lean experiments fail to deliver. Ultimately, every one of the reasons relates to that first essential step: the need to frame your experiment before you begin to build. Without context or boundaries in an uncertain environment, it is all too easy for experiments to deliver inconclusive results or no results at all.

Stop wasting time running experiments that don’t drive learning, add Frame to your next Build-Measure-Learn cycle.