Agile Fundamentals: What is Empirical Process Control?

(This is a script for a video that will be produced soon.)

What is empirical process control? Why is it so important for effective software development?

Although their history is murky, I really like the quotes “In God we trust, all others bring data” and “Without data you’re just another person with an opinion.” 

Empirical means “based on, concerned with, or verifiable by observation or experience rather than theory or pure logic.” This is a fancy way of saying: base your decisions based on data, information, and knowledge derived through thoughtful observation and analysis. You will always learn more as the project unfolds, so create the conditions to maximize this learning and your ability to leverage it.

Scrum is based on empirical process control theory. The three pillars of Scrum theory—transparency, inspection, and adaptation—help teams make decisions empirically.

Transparency is effectively the access to the raw data we need to inspect and adapt combined with a shared understanding. For instance, both the developers and the product owner must know and agree on the acceptance criteria and Definition of Done for each user story. Otherwise we cannot inspect whether the completed work was performed correctly. Another example is that we should share openly, honestly, and respectfully at Retrospectives so that we can resolve any potentially challenging team dynamics before they become more destructive.

Inspection is about taking advantage of the transparency we’ve created and gaining useful knowledge.

For instance, suppose a burndown chart shows that we have 40% of the sprint remaining to complete 90% of the sprints tasks. It was our transparency—or our Agile software—that allowed us to track and present the data in the first place.

It is inspection when we look at the graph at the Daily Scrum and talk about it.

Adaptation is about responding effectively if something isn’t running as smoothly as we’d like. Using that burndown chart example, if we discover through our transparency and inspection that we are unlikely to meet the sprint goal in time for the Sprint Review, we should adapt to this change in situation. If we focus as a team on the most important work, what are we likely to finish and not finish? This will help the product owner reset expectations and maintain trust with stakeholders by transparently delivering potentially disappointing news earlier.

This is quite different from the old “Waterfall” way of doing things. This is a bit of an extreme over-simplification, but the legacy approach was basically “do a bunch of research, come up with a plan based on that research, and if things don’t go according to plan, replan or blame the people doing the work.” This knee-jerk, uninformed reaction robs the company of an opportunity to learn.

The empirical response to this is “Look at the data and knowledge you have, obtain any data whose absence is glaringly obvious, come up with a reasonable plan, and since this reasonable plan won’t survive first contact with reality… work with your team to maximize learnings when things don’t go as expected to respond effectively.” In order to succeed, create safety so that development teams feel safe transparently sharing information. Leverage that information to learn from observations and experience, update forecasts for stakeholders, and change course when you discover that a new direction would be better.