I’ve written about the importance of being aware of the risks of software development before. One of the points I made was that whenever software developers do anything worth their time and effort they are roaming on uncharted territories. Whatever has been done before is simply not worth doing ever again. Recognizing this is vital for innovation to thrive at a company, because when we boldly go where no man has gone before, uncertainty and chaos will be part of the game. We need to understand uncertainty, deal with uncertainty, embrace it, breathe it and thrive on it every single day.
Innovation is not a stroll in the park with a map in your hand. It is a pioneers’ march through massive swamp lands, huge mountains and dense forests undisturbed by humans. There is no map that tells you which way to go, neither can you draw one without running into some dead ends along the way.
In this post I will use the term software development — for lack of a better word — in a much broader sense than people usually do. Software development in this blog post will include everything that goes into shaping the customer experience with the end product, including product management, UX, coding and much more.
For a software project the primary source of cost is uncertainty, because it leads to waste. It is not surprising with that in mind, that every successful methodology we have been using is designed to reduce waste by mitigating the effects of uncertainty.
It’s obvious that we need to validate the problem we are trying to solve, we need to research which UI solutions will work best for the user, but the uncertainty rarely stops at the doorstep of the product management team. Even with the perfect plan for a product in our hands we still have uncertainty about what technological solutions are viable if there is any viable option at all.
A genuinely innovative product will rely on technology that might have been unfeasible just a few weeks or months earlier, otherwise someone else has probably already created that product. When technology is not taken into account while planning a new product it’s all too easy to come up with something that is just not viable at the time to build.
Once the product is released and customers start using it, uncertainty isn’t over yet. As the load grows on the servers, bottlenecks appear. When a product succeeds, a death march begins during which the team will need to handle roadblock after roadblock to survive growth.
It is not realistic to handle performance issues in advance. Even if it was possible to anticipate them, it’s not worth putting in the effort before the product proves to be successful on the market.
Innovation is not a stroll in the park with a map in your hand, but it doesn’t have to be an unbearable struggle against all odds. It can also be an exciting journey worth the hardships.
Science comes to the rescue
Turns out software developers are not the first to venture beyond the limits of existing human knowledge. Scientists have been on this exciting multi-generational adventure centuries before the first computers came around. These honorable people have been rigorously following a recipe for iterative exploration called the scientific method. This method stood the test of time. Its simplicity and elegance makes it even more impressive.
In its simplest form the scientific method can be described as a three step process. (I will be omitting a lot of details, but for the purpose of the rest of this post a simplified version will serve us better.) The scientist starts by forming a hypothesis: a statement that we suspect to be true but is currently unknown. The statement should also be theoretically falsifiable. Next she comes up with a way to test the hypothesis. This is usually in the form of an experiment. The scientist is looking for some kind of consequence of her original hypothesis that can easily be tested with an experiment. Eventually she performs the experiment and looks at the result. She examines the results, and tries to draw a conclusion on how well her hypothesis modeled the world.
When all experiments match up with the predictions within an acceptable margin of error the hypothesis becomes a theory and we use it to build the things around us from tiny digital watches to huge buildings. If it weren’t for this simple yet rigorous method we wouldn’t have cars, planes, computers, iPhones, modern healthcare, not even simple things like glass or light bulbs.
The scientific method is used in tiny iterations. Usually when a hypothesis is formed it is not tested directly at first, but it is broken down into simpler statements that can be tested individually. Later, when enough knowledge is collected the dots can be connected to create an experiment that can test the more complex hypothesis.
Iterative processes
About 6 months ago I had one of those light bulb moments where I suddenly realized that this simplified version of the scientific method is at the heart of every single successful methodology we have created over the years for software development. Its omnipresence is almost shocking once you recognized the pattern.
Just to name a few things that can be seen as specialized versions of the scientific method: Test Driven Development, weekly retrospectives, Lean UX, Lean Startup, Theory of Constraints and the Kaizen culture are all variations of this simple yet elegant concept. This is by no means an exhaustive list and I encourage the reader to look for other examples.
What all of these methods have in common, is that they are iterative, they rely on formalizing your assumptions into falsifiable statements and running experiments in order to validate those assumptions.
Learning to embrace uncertainty
It’s hard! It’s hard to get rid of our preconceptions about how professional teams work. Over the years I have seen projects succeed and fail, but what taught me most about how to succeed with a software project was watching countless teams competing at Lean Poker events. Although Lean Poker is only an 8 hour long simulation of a project in a competitive environment, the way winning teams differ from other teams is the same as with larger projects.
The most important traits of successful Lean Poker teams were the ability to adapt to change, tiny iterations and humility. It takes tremendous courage to face the unknown and to revise your plans once you went down one path.
Culture has a tremendous inertia, and it’s far easier to fall prey to the false promise of certainty than to embrace the unknown unknowns. First we need to recognize the problem and that is already hard enough due to our own confirmation bias. I’d bet many of the people who read this far feel that they already work in iterative processes while failing to recognize the waterfall making a slow but steady comeback in their day to day life. We don’t like being criticized, we don’t like admitting to ourselves that we need to change.
The reason I created Lean Poker was to give people the opportunity to try different attitudes to software development in a short time frame and allow for comparison between them. For me Lean Poker gave a deep understanding of why iterative methods work well in competitive environments since as a facilitator I had a chance to see what worked for each team and what didn’t. For the hundreds of developers worldwide who took part in these events Lean Poker has been — and will continue to be — a space for exchanging ideas on how to build a winning team and experimenting with new methods. So if you would like to see how moving to an iterative process would affect your team’s performance I invite you to visit an event near you, or to organize your own event for your tribe.
This blog originally appeared on the Emarsys Craftlab blog.
