A Tsunami Of Problems

The problems in Japan continue, as do the truly heroic efforts to contain the disaster. Still, how did this all happen? Now that we have some information, we can put together the pieces. The first piece is simply that all of Japan, not just the Fukushima Daiichi facility, lies in the Pacific Seismic Belt, or as it is often called… the Ring of Fire. This is the world’s most active earthquake and volcano zone. Not surprisingly,  Japan has some of the most advanced building techniques to deal with earthquakes.  The reactor met the most recent earthquake building codes. However, according to this week’s New York Times article, “Japanese Rules for Nuclear Plants Relied on Old Science”,  the real culprit wasn’t the earthquake, it was the Tsunami. Yes, the earthquake caused some damage but the fatal blow was the follow-up hit from the Tsunami. The plant is built on a cliff right on the edge of the sea, which is about 18 feet below the plant. Since Tsunami is a Japanese word, they must happen in Japan frequently enough for them to need a word to describe it! Until 2006, Japan had no specific Tsunami regulations, and the Fukushima Daiichi plant has had no Tsumani specific upgrades. Think about that. In 2004, we had the world’s most destructive Tsunami hit the pacific. In 1995, Japan had the disastrous Kobe earthquake. In between those two events… a whole pile of lesser earthquakes, Tsunami’s, rogue waves and other natural disasters.

On a really bad day we tend to all say, “We’re in a meltdown!” Luckily, however bad our problems get we don’t face consequences on the scale of a real meltdown. But should we have so many of these really bad days? A key takeaway from the Japanese example is that you shouldn’t just plan for a disaster, you need to plan for simultaneous disasters. They happen. We see smaller versions with some frequency. A winter storm AND a flu epidemic; a computer virus and a transit strike; a flood and a power problem. Lots and lots of permutations. One of the reasons that the Japanese under prepared for this disaster was because they chose to plan along deterministic rather than probabilistic lines. In other words, they planned based on  things that had happened, not on things that might happen. So, one-two punch disasters weren’t planned for. It is unrealistic for any of you to plan for every possible outage, and it is insanely unrealistic to assume that you’re going to get the funding to prepare for every possible outage. Still, we should give some thought to combination disasters. Take a look back on your service’s history. What are the sort of problems that have happened in the past? Are you able to deal with them? Are you able to two disasters happening at the same time? It’s an issue that’s worth thinking about.

In the Times’ article an engineer used a great term, “a cascade of stupid errors”, to describe how the Tsunami threat was underestimated. Don’t get caught up in your own cascade of errors by overlooking the data you already produce. There may be important indicators of the “little Tsunami’s” that are already impacting your operation. Increasingly, your “customers” live in a service world with little tolerance for outages. They pay for cell phone services that are supposed to work everywhere, flights around the world that are never supposed to be late, computers that are never supposed to fail. True, all of these things do fail, but the anger we all feel when a service fails seems to be growing every day. When personal services fail we often get some satisfaction from changing providers. Corporate services don’t work that way. Your customers don’t get to change to another service. Which is why your customers become absolutely livid when they don’t get what they want.  Go through your metrics, take a fresh look at your management reports and map out known all known serious outages. Plan for the things that haven’t happened yet, but probably will at some point. Worry about problems, but worry effectively. Convert your concerns into priorities and if it is a high enough priority, make your management aware of the possible consequences. Conversations about things that might go wrong can be difficult. Conversations about things that have gone wrong are even more difficult. And that’s my Niccolls worth for today!

This entry was posted in Backup Plans, Best Practices, Decision Making, Delivering Services and tagged , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.