As we near the end of the year, a majority of us start looking to the next and imagining what we would like to change or improve upon to achieve a goal. I’ll explain how mixing a technique that was pioneered by Toyota for lean manufacturing, adapts very well to DevOps transformations as well as personal goal setting and achievement.

What is Kaizen?

The Japanese words “kai-” which means “change” and “-zen” which means “good.”

Kaizen is a process created by the Toyota company to focus on continual improvement of their manufacturing processes. Possibly creating the granddaddy and literal analogue to what we in the tech industry understand as the general concept of Agile Software development. Kaizen as a strategy requires that there is involvement from all levels of a company. It doesn’t work as effectively if only one business unit attempts to use this method, and that is because for an organization of any size, the change comes from all places. The planning comes from identifying inefficient processes and then rapidly iterating on solutions until a proper one is found. For that to be most effective, you’ll need more than just a single team.

Read more on Kaizen: here, here, and here

How do I use Kaizen?

During new years it is customary in many parts of the world, to set new years resolutions. These are often big monumental changes, and as often joked about, falter after a few weeks. The focus of Kaizen is on small incremental improvements that can be done now, get those all complete, then re-evaluate, the next small incremental improvement to get closer to the overall goal.

Kaizen applied to real life

I gained too much weight over quarantine and need to lose 30 pounds.

If you immediately run to the gym, jump on a treadmill, and begin running, good for you. Keep at it. but what happens to most of us, is life. Simply said, things get in our way, we skip a day, and the next time you have a chance to blink, that goal is three months behind and maybe you’ve gone in the opposite direction and need to lose more weight than when you began.

Instead of setting one goal that is far away and hard to imagine how many days, how many miles, how many meals are between it and you now. Set a weekly goal. I’ll reduce calories and workout enough that I lose 2 lbs a week.

This way at the end of each week, you have mental check-ins and can adapt much more often to changes in your schedule, time, stress, and diet.

At no part am I saying, don’t set a big goal, those are great and we all have them. Just don’t forget to map out a realistic path to achieving that, with check-ins at regular intervals along the way to ensure you remain on track.

How can a DevOps Team use Kaizen?

An example of this, is one I actually witnessed at a previous employer. This company had 2-dozen separate business units (tons of merges and acquisitions) each with their own SaaS product, with entirely different legacy of development and technical debt.

In an effort to standardize processes, tech stacks, and stream line communication, all DevOps and SysAdmin teams became a company wide SRE organization. Suddenly we were in charge of release cadence, creating Service Level Indicators, and overall worshiping the Google SRE handbook, in addition to everything else we did previously.

Each group had to define Service Level Objectives that were better than our Service Level Agreements with our customers. We also had to call out areas that were in danger of keeping us from not meeting those goals, whether those were single points of failure in our tech stack or if they were toil issues, like endless support tickets that could and should be automated.

Once we had this all defined and agreed upon with key stakeholders, we used this live document to re-asses at monthly intervals how well we were doing.

The key point is that we didn’t just have utopian and mostly empty goals like 4 nines of uptime. We had to prove with real data that we were meeting our Service Level Objectives. If we started to dip below that, then we were within our rights to hit the breaks on all development. Circle the wagons, and swarm issues until they were fully resolved.

A few problems with the way we did this (telling you this, because these things are difficult to implement): 1) There wasn’t a clear understanding of what the Kaizen process was. 2) It took too long. We spent a good 2-3 months just documenting things, most of that was due to poor buy in from different departments, so it took weeks to get questions answered. By the time we had everything together, other stakeholders were tired of waiting and their attention fizzled out. 3) It can’t just be a DevOps/SRE thing, it has to include everyone; finance, development, project management, sales, etc

Kaizen and Agile

If everything I’ve described now sounds a lot like Agile, good, it should. But the thing is, Agile is mostly seen as just a way to manage software development. Kaizen is much bigger than that, focusing on the entire company.

The core focus of both Agile and Kaizen rest on continually having periods that one critically evaluates goals, performance, and the gap between them to make the company or product successful. In Agile this is expressed through sprint retrospectives, sprint planning, and having well known definitions of done.

If a lot of your time is stuck with handling issues and not developing new infrastructure or features, then maybe it is time to talk cross-departments. Get out of your silo and figure out a solution. Challenge yourself, your team, and your company. I’m positive the company will be better off if you solve these critical but time consuming bugs. Of course, that makes Kaizen all the harder to pull off, as you need buy in from more than just your team. The flip side of this, is that it gives your department a more holistic view of the problems on the other side. Maybe the ticket management workflow is too cumbersome for that other group, and so they keep personally contacting you to help with a problem. Maybe if you had a week and 3 other dedicated hands on keyboards, you could knock out several critical known problems that keep you awake at night.

Kaizen Events

A Kaizen event (also known as a burst or blitz), can be a great way to kickstart this process in your company. Here’s a few steps to do that:

1. Plan: Gather a lot of people (virtually now), from different cross-functional teams. Create a list of top 5-10 improvement ideas. Talk this through, make sure everyone involved sees the value of these improvements and has a part in achieving it.
2. Execute: During the event 80% of the action items should be achievable. The other 20% should be completable within 30 days of the event.
3. Check: Have a post event meeting where the state of improvements are discussed and documented - did they work, were they effective?
4. Act: Keep these newly adapted processes or fixes; then start a new cycle

Back at Toyota, they made these week long events - five days and one night. So they are intense focused action, all hands on deck. And because they are that way, they should only be conducted a few times a year to avoid over use and burn out.