Up front I admit that this post is less about facts and more about a narrative. If I may paraphrase George Box: All stories are wrong, but some are useful.
The story starts — like many conversations I've had — with someone saying "I have an idea for an app".
Chapter 1
As it turned out the idea was good, and immediately I opened my laptop and started designing the prototype. In the beginning I was very productive. The code moved forward at close to typing speed (the theoretical limit until a more direct input method is perfected). I could do this because I knew the entire code base, and as the sole developer I could also make all the decisions.
As with virtually any software project it quickly became apparent how huge an undertaking it was. As the months went on parts of the code started escaping my memory, I learned things, changed my style and preferences slightly. Gradually it was like there were two programmers on the project; me and me from the past. In fact there were many "me of the pasts". Thus being the "sole developer" was an illusion. To completely kill the "sole developer"-idea I eventually partnered with two other developers, to speed up progress.
Working with other people turned out not only to be a benefit, but also a cost. Suddenly there were parts of the code I did not know, and had to spend time familiarizing myself with. It was not clear anymore how to make decisions. Having the other developers come to me with every question would not leave me any time for coding. Having them make their own decisions meant that different parts of the software would not fit together, resulting in a weird eclectic app at best, errors and bugs at worst.
Chapter 2
As a programmer my instinct is to try to solve every problem with more structure — another layer of indirection as the aphorism of David Wheeler goes. I decided that we should start by spending some time figuring out what product we wanted to make. We would talk with our future users; Analyzing the potential market. Then we would make as many decisions as we could. Have workshops with paper mockups, and make sure we had the perfect design. Once we had the design for the app we could start building it, safe in the knowledge that all the decisions had been made. Once implemented it would be a simple matter of testing it. And finally releasing it to the users with some teaching material and more workshops. Everything would be heading ever faster towards the end. Just like a waterfall.
Confident that this was the right way to develop software, we hired a user experience guy, and a documentation & workshop guy. Our plan was idiot proof. Or so we thought. Analyzing, and designing went by with no problems, and we had what we felt like was the perfect design. During implementation however was when problems started occurring.
Just like data structures are frozen algorithms, code is frozen understanding, a notion expressed by Dan North. This means that whenever two people understand something differently their code often end up conflicting. As the code base grew larger we had to spend more and more time making sure that we had not broken anything. There are two ways to do this: more communication and investigation before implementing something new, or more testing afterwards. The code we had just added could have broken something somewhere else, so we had to test everything. None of us knew the entire system, so it would be very difficult to test part of the system that we had not contributed to, let alone fixing bugs if we found them. We were grinding to a halt.
Chapter 3
We had to spend ever increasing time bug fixing, or have to spend a considerable effort communicating with everyone to make sure we had the same understanding. We had two major — and very different — issues: there was no common design for the code base, and we spent a lot of time bug fixing.
We looked again to my favorite thing: Structure. We immediately agreed that the code was too complex to completely plan from the beginning, but we felt like we could at least predict how it would look in two weeks. Thus we instituted bi-weekly planning meetings.
As for the quality, we turned our heads to my second favorite thing: Automation. We would have every programmer write a small piece of code that would test their part of the system. In the end we could just run this code to see if the system worked.
Being better safe than sorry we also wanted to increase collaboration, to make sure everybody had the same understanding, further increasing quality. We decided to have everybody pair with someone else, and they would be both responsible for their code. This meant that misunderstanding got caught immediately, and new the practices were more easily followed as there were two to remember them — and make sure they were not skipped.
Closer collaboration in the form of bi-weekly meetings, and pair programming, together with automated testing was outrageous. This was extreme programming, and it greatly reduced the time we spend communicating and debugging.
Having unclogged the drain, we finally finished the first (of three) part of our app. With champagne and party hats in our bags we proudly went to our users and presented what we had built. We had not needed bring the party hats, as what ensued felt less like a party and more like being lion tamers meeting a lion for the first time.
It turns out that while we had been designing and implementing our system, another app — let's call it AstroSoft — had been released meaning that most of our functionality had become irrelevant. "Isn't this the same as AstroSoft?", was how I learned how much a question could hurt, when we had to reply "Yes, but we currently have fewer features, and is probably harder to use".
Chapter 4
In our ignorance we had trusted in the plan. The design we had made were perfect for the users. Unfortunately, much like I changed during the first months of the project. So had our users changed, and the design no longer fit. Before starting part two, I was determined not to end up with another fiasco. I needed ideas. I needed solutions. Being a bit of a fantasy nerd; I needed the 12 knights of the round table... of software development. I contacted my 11 most talented friends and invited them to a cabin where we would stay until we had come up with a solution. In the end a few of them brought some good friends and we ended up being 17.
We started with a basic question: How could we design something for how the users would be in the future? In reality, how could we even know if our product would be relevant when we were done building it? The only thing we could say with certainty was that it was relevant now, and would probably be relevant at least in a month. We realized that this looked a lot like the situation we had had with regards to designing the code base. In that case we decided to have continuous planning meeting, were we planed a little bit. Maybe we could use the same solution here.
In the end we had a document with four values, and in lieu of 12 knights we had 12 principles. Three of the values were sort of derived from my teams previous experiences. Two from our focus on collaboration:
- Interaction over processes, and
- Customer collaboration over contracts
And one from testing:
- Working software over comprehensive documentation
But we also added one that completely invalidated our previous approach:
- Responding to change over following a plan
Most notably — and as stated in our first principle — we would early and continuously deliver valuable software to our users. The ability to adapt, having the code be able to change suddenly, being agile, was ground breaking. Everybody present had experienced similar hardships that we had, and both the values and principles seems self evident to us. "Surely these will never expire" said one of us "we should we should put it on a website; agilemanifesto.org is available". Having a grammar nerd present the response was swift and merciless "That is a contradiction in terms; A manifesto can by definition not be agile. It is technically a Manifesto for Agile Software Development". "Oops, I already bought the domain name". "Well, done is done, let's just hope people don't end up calling it the Agile Manifesto, because they would sound like illiterates".
Chapter 5
Confident like a flying Icarus, I returned to my team and handed them what I felt was "it"; the right way to develop software. If you know the Greek myth you will know what comes next. My team got all fired up, agreed with everything I presented, but when I was done there was a hand in the air. "That's all well and good, but what does it mean in practice? How are we going to do this?" I had been so focused on the big picture I had not thought about how to concretize it.
When you have to be very concrete, I find it useful to pretend I am making a game. A game has to have very specific and concrete rules. Together with one of my co-partners we sat out to make the agile software development game. Being creative can be difficult but another trick I've picked up is to start by coming up with a silly name, one that has no connotation, so you can think about it freely. In our case we came up with the completely absurd word "Scrum".
Remember our end goal was to "continuously deliver valuable software to our users". First we defined who 'the users' are; again with a new term we called them "product owners". The 'continuous' part we already did for planning the code, so we decided that we would release on the same schedule; every 2 weeks. We called the period in between a "Sprint", although looking back this makes it seen like you have to be sprinting all the time, which is not possible, had I a second chance I would have called it "stage". On the topic of silly words I always thought the word "dodo" was hilarious, so I really wanted to include it in some way. We decided to have it stand for "definition of done, overall", to signify that we had to agree on a universal definition of what "done" meant.
We realized that what we were doing was just renaming things. We needed something that stood out. Something colorful. Then it came to me: post-its. I had heard of something called a Kanban board, that they used at factories. What I remembered was there were colorful post-its and columns. Our "Scrum board" would have three: to do, doing, and done (going well with our 'dodo'). We were on a roll so there was no time to investigate the details of Kanban, although I later learned that the power of the columns was to limit what they call WIP (work in progress). We should have probably included a mechanism for that. Would that this desk were a time desk, that I might correct my past mistakes.
We incorporated our planning meeting this time under the name "Sprint Planning", and also added a few more meetings, one for showing our progress to the product owners ("Sprint Review") and one that should enable us sprint even faster next week ("Sprint Retrospective"). We also baked in some more collaboration (and made sure the name of the game was edged into history) in the form of "Daily Scrum", 15 minutes of chit-chat about what each person did yesterday and what they are going to do today. Finally, any game needs a judge, which we called a "Scrum Master". All other players would just be "the Development Team". In order to avoid later disputes we ended up limiting the number of product owners to one. We also cut the "overall" from dodo because it seemed unnecessary.
Chapter 6
Armed with these new wonderful tools, these artifacts, we started building the planned part two of the system. Starting with the most important few features. We made sure that after each two weeks we showed our selected product owner what we had built. After she had approved the features we would have a few people deploy it to all the users. This small subteam was also responsible for keeping the servers running, and generally that the system was operational.
These were the golden days. Being a developer (or dev for short), was glorious. We got to do all these cool features all the time, and did not have to worry about all the pains of deployed nor maintaining them. It was awesome. Apparently though the operations guys (or ops for short) were not quite as happy. As it turned out nor were the users, because the stability was not always what we had dreamed of. In general actually the ops had trouble keeping up. We had also had some pretty bad scares with security so we brought on a security guy to go in and fix it. That was the straw that made our house of cards spill over.
Because neither maintainability nor security had been built into the software at a fundamental level it just was not possible to after-fit them. Although we managed to deliver part two of the system, it was clear that we could not continue like this. We had to combine the goals, and thus the teams. We needed the devs and the ops to work together, so that we might all thrive, we needed DevOps.
As we had matured and got used to this new way of working, we discovered that we could actually deliver even more often. Soon we started doing it every week, then every day, then multiple times a day. Slowly Scrum did not fit anymore. Luckily I had had time to sit down and study Kanban, and what it came from, namely 'Lean'. I realized that there was a simpler way to express what we had put in the manifesto all those years ago. I found that I could encompass everything in three practices, I coined them "the three ways":
- Flow — make sure things are flowing from left to right smoothly and continuously
- Feedback — get early and continuous feedback
- Learning — we need to continuously grow; be better and faster
As was clear, the silver bullet had been the word "continuous" all the time (pun intended). As Anthony Panozzo said "If something is hard, do it more often", the limit of "more and more often" being "continuously".
Chapter 7
Having formed a DevOps team, and following the three ways we very pretty well set. We delivered quickly, reliably, and the quality was better than ever. Very content I remember thinking to myself one morning "This is it. This is the right way to make software". And so i thought for a while. Until I noticed a small wrinkle in the scheme. As it turns out "learning continuously" is not as easy as it sounds. Being stumped again I wrote a letter (yes, he lives in an area with no cell phone service, or even phones, or electricity) to one of my heroes; Isaac.
He replied to me, that when I observe something I should form a hypothesis of what the underlying truth was. I should then proceed to device an experiment that would determine whether my hypothesis was false or not. Perform the experiment and based on the observations I made during it repeat the process.
This was a very elegant way to put it. I did fear that it would still be a bit too vague for some of my colleagues. I decided to reuse an old idea and make another board, this time the columns would be the stages of an experiment. I also fleshed it out a bit to add a bit more of my favorite thing:
- Problems and observations — here we put a post-it every time something makes us roll our eyes
- Options — here we brainstorm what could possibly have an effect on a problem
- Possible experiments — when we have at least 3 options we can promote one or more of them, making them a little more concrete
- Committed — when we have a few possible experiments we concretize them even further, and prepare to actually perform them
- Ongoing — what we are experimenting with right now
- Review — evaluate how the experiment went
- Next — where do we go from here.
With this we have been able to perform 5-10 experiments per week, constantly adapting and improving. It seems to me now the of the three ways the third is by far the most powerful. Maybe we would have even come up with the other two if we had had the third first. It is impossible to say for sure. But to underline this observation I wanted to call this method "Flow", like the first of the ways. And notice that when you take the first letter of every column it spells Popcorn, thus "Popcorn flow". We can make a strong case that evolution is the greatest force of nature, who would have thought that the same would apply to software? But I do feel confident when I say: This is it. This is the right way to make software.
Epilogue
I have tried to include a lot of facts in the story, such as quotes, explanation of fundamental ideas, and the whole chronology of the methods. However most of it is complete fiction. Most notably: no matter how much I wish I could have been one of the signatories of the manifesto for agile software development as the story suggests, this is simply not the case. Nor did I contribute in any way to Scrum, or any of the other methods discussed above.
What strikes me about the story is that most of the chapters (if not all) are "something that works somewhere else" applied to software development.
Waterfall was came from engineering buildings, Scrum came from Kanban which like DevOps came from Lean which itself came from the Toyota Production System, and Popcorn flow is the scientific method — with extra steps.
Why has it taken us so long to (re)discover things that were already invented? Did we really have to suffer through all the hardship? My guess is that there is a tendency for arrogance in our field. We like to develop our own libraries when we can, because ours will be better. We don't generally like to try new things, because we can make it work without. Even when we hear an interesting idea it is often met with "it doesn't apply to our type of organization" or "our type of software", or "our customers". Certainly I am guilty of arrogance myself sometimes; No code is as beautiful as my own. And that is also why I made the protagonist in the story very arrogant.
This is where, I wish I was smart enough, to apply this insight, to realize: what is the next old idea that will catapult us forward?
