Episode 14: Live Post-Mortem with David Zwieback, John Allspaw, and Mike Rembetsy

Date: 28 May 2012

Warning: This episode includes some profanity
Download Here

Panel

John Allspaw, twitter, blog
Mike Rembetsy, twitter
David Zwieback, twitter
Bryan Berry github, twitter, irc: bryanwb, blog: devopsanywhere
Nathen Harvey github, twitter, blog

Show Notes

After losing the original recording of Episode 15: How to Succeed in DevOps with Dave Zwieback, we invited John Allspaw and Mike Rembetsy onto the show to facilitate a blameless postmortem. John Allspaw recently wrote about Blameless PostMortems and a Just Culture over on Etsy’s Code as Craft blog.

During the episode we discussed the timeline of events that led to the failure. John applied a substitution test: given the same conditions would someone else have expected a different outcome. We came up with some remediation items aimed at preventing, detecting, and speeding recovery from a similar failure. We talked about how to deal with repeat offenses and why just telling people to “be more careful” doesn’t solve anything.

Want some tips for turning your postmortems from a whole lot of theatre into a whole lot of learning? Be sure to listen to this episode!

In the News

Productivity is the greatest scourge a workplace can face. You want to make sure your workers aren’t too productive, but how do you do that? Fear not! Jeremy Bingham just wrote up a blog post that will help! The blog post announces a new Chef Cookbook that will install and manage Dwarf Fortress on your workstation.
Matt Ray released Spiceweasel 1.1. This release adds support for extracting knife/json/yaml infrastructure & Ruby 1.8 support. I was able to attend Matt’s #ChefConf presentation on Spiceweasel and we’re starting to use it this week. Definitely worth checking out.
Over on CIO.com, they’ve posted an article suggesting that IT has figured out the development process but the “bookends” of Application Lifecycle Management (ALM) are where the most help is needed. Check the show notes for a link to the article: IT Should Forget Development, Focus on Requirements and Release Management
What do chimps and 90’s-era R&B star Montell Jordan have in common? They both like to tell the world “how they do it.” Over on theInfochimps blog, Nathaniel Eliot posted an article titled How We Do It. The article is a look at the hiring process, management style, career development, and more. A great read for any team working in this space.
John Willis has written a two-part series on Devops culture on the IT Revolution Press blog. Check the show notes for links to part one andpart two.
Are you looking for some reasons to implement Puppet? Over on the Tribily blog, you’ll find 8 reasons to implement Configuration Management using Puppet. Of course, these 8 reasons apply equally well to Chef and can serve as a nice way to help convince your team that automated configuration management with Chef or Puppet is the “right” way to go.
Zac Stevens is back with another post on his Cryptocracy blog this one will show you how ubuntu-vm-buildercan be used to go from zero to fully-installed Chef client in under 2 minutes.
There’s a new post on Evolving towards cookbook reusability in Chef. Find the article on the RealityForge blog and a link in the show notes.
The Godfather of DevOps, Patrick Debois is back with another thought-provoking and detailed blog post: Devops Areas - Codifying devops practices. This is a first stab at providing a structure to codify devops practices. The wording, descriptions are pretty much work in progress, but he’s looking for your feedback. Checkout the article and let Patrick know what you think.
Jesse Robbins was recently interviewed by O’Reilly Radar on the state of infrastructure automation.
Andrew Crump released version 1.3.0 of foodcritic. It features a number of bug fixes and new features such as FC026: Conditional execution block attribute contains only string rule added (related issue). Thanks to @mkocher for proposing this rule. Foodcritic now accepts multiple cookbook paths as arguments and supports linting of individual files only. Big thanks to @cgriego for these changes. These lay the groundwork for his new guard-foodcritic project.
Andrew Crump plans to extend foodcritic so that it can critique roles and environments. He would like to know what kind of rules you would like to see implemented. You can give him suggestions on this github issue for foodcritic.
The last item in our news for today is a link to the Food Fight github account and to a page where we’ve been collecting a list of slides, videos, and follow-up blog posts from #ChefConf. Checkout some of the great content from the conference. I you have something that should be added to the list, or know someone who does, pleas submit a pull request! https://github.com/foodfight/showz/blob/master/chefconf_slides.md

Cookbook Updates

14 new cookbooks were added to the community site since our last episode. That’s almost one new cookbook per day!

forkable_client v0.0.1 - chrisroberts

Like modifying code on the fly? Like reworking internals of running code to do interesting things? Like not losing a huge chunk of memory because some jerk cookbook decided to load a 100M json file just for the hell of it? Then this cookbook is for you!
This cookbook adds forking support to chef-client runs. It is based on the work within CHEF-3104. After this cookbook has loaded, client runs will fork themselves and then converge. What this provides is less worry about what cookbooks may be doing (and loading) that is going to bloat out the memory of the process. It’s just that simple.

log_rotations v0.0.1 - chrisroberts

Chris Roberts
This cookbook provides an extremely simple way to create log rotations via attributes. It depends on the logrotate cookbook.

simple_iptables v0.1.0 - dcrosta

Dan Crosta
Simple LWRP and recipe for managing iptables rules

disco v0.1.0 - dcrosta

Dan Crosta
Installs the Disco map-reduce framework using the default settings.

gitlab v0.2.1 - atomic-penguin

Eric G. Wolfe
This cookbook will deploy gitlab; a free project and repository management application. The github repo for this cookbook includes TravisCI build status.

readline v0.0.3 - atomic-penguin

Eric G. Wolfe
Installs platform specific readline binaries, and development libraries.

rsdns v0.0.3 - evan4498

Evan Ochs
Installs/Configures rsdns Rackspace Cloud DNS cli tool

raxmon-cli v0.0.3 - evan4498

Evan Ochs
Installs the python-pip package in Debian and RedHat based systems, and then uses pip to install the Rackspace Cloud Monitoring client, raxmon-cli, via pip