in which I continue to obsess about failure and organizational culture

Jerry Remy, beloved New England sportscaster and former Red Sox player, in his book Understanding Baseball, wrote something that has stuck with me hard: if your team never gets called out at home, your third base coach isn’t doing his job. That is, if he’s being so conservative about sending runners that he never sends the ones with a risk of failure, he’s also not sending a lot of guys with a risk of success, and you’re avoiding outs by passing up on a lot of runs you should have scored.

The same idea came up recently in a Slate interview with Peter Norvig, the director of research at Google:

[Having your company run by engineering rather than sales or marketing means] a very different attitude toward error. If you’re a politician, admitting you’re wrong is a weakness, but if you’re an engineer, you essentially want to be wrong half the time. If you do experiments and you’re always right, then you aren’t getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.

113/365: Flippin' coins by Pauli Antero, on Flickr
(Go read the whole thing; it’s great.)

He goes on to talk about how this pervades corporate culture and operations. If you assume there are going to be errors in your code (because there always are), you build a process that can route around that. If you assume your hardware will sometimes fail (because it always, eventually, does), you buy a huge pile of cheap servers instead of a few expensive bespoke machines, and design around them (and, coincidentally, save money). And you build an organization that can minimize the consequences of failure:

We do it by trying to fail faster and smaller. The average cycle for getting something done at Google is more like three months than three years. And the average team size is small, so if we have a new idea, we don’t have to go through the political lobbying of saying, “Can we have 50 people to work on this?” Instead, it’s more done bottom up: Two or three people get together and say, “Hey, I want to work on this.” They don’t need permission from the top level to get it started because it’s just a couple of people; it’s kind of off the books.

(I debated, up there, should I write, “minimize the consequences of failure”, or “encourage innovation”? I decided, really, they’re the same thing. The greater the downside if your innovations fail, the fewer people will be brave enough to try. The more resources (time, money, staff) have to be committed — put at risk — to try anything new, the fewer new things you’ll try.)

Danger of Death By Failing by AlmazUK, on Flickr
What not to do.

Norvig admits he’s lucky in that the stakes are lower with Google than they might be elsewhere; the “best” 10th result to a search query is highly subjective and it’s OK if you don’t return the same thing each time, in a way it’s not OK if you’re casual about how many zeros go on someone’s bank account. (Though if my bank wants to add a few zeros to my savings account, I’m cool with that. Just sayin’.)

Come to think of it, it’s striking how much subjectivity comes into play in that article. Something like server failure you can attack probabilistically: you may not know which servers will fail, but you can estimate how many and create a strategy that, with a quantifiable margin of confidence, can cope. But something like “will people think ads by their email are creepy” or “will anyone care about Android” are all focus groups and crystal balls and gut checks. Engineers are awesome at Bayesian probability; I doubt they’ve got any comparative advantage with crystal balls. So I’m wondering if there isn’t a meta-level of risk tolerance here: create a culture where people can innovate and fail about things in their comfort zone, and they’ll have the confidence to innovate (fail) (sometimes succeed) about things outside it. To step off into the unknown, where all the future happens.

You almost feel like you could fly without the plane... by addicted Eyes, on Flickr

an asymptotic approach to truth

Spent the weekend visiting the in-laws in Princeton, hence visiting my librarian crew down there; Monday lunch with Janie Hermann followed by Deep Thoughts with Peter Bromberg. Lovely! (As always: how lucky am I to have found this profession where people are so generous with their time, thoughts, and expertise?) And at one point Peter said, “Blog that!”, and so:

I spent much of library school in cultural whiplash between the library world, where I was being steadily acculturated, and the computer-geek world where my husband and so many of my friends work, and where, to an extent, my undergraduate background lies. And this is coming up again now that I’ve decided I really need to understand something about how ALA works.

Cthulhu Warning

Of course, the more I look into the committees and subcommittees and processes the more I get the sense that it is a Lovecraftian horror and if I ever understand it in full I will have lost my mind.

I have difficulty with bureaucracy. I see that ALA has a lot of it, and I don’t understand why — don’t know the historical antecedents that led to so many rules accumulated upon rules. (Please: do tell.) I sense there’s a story there, but I don’t know it, and even if I did I’d chafe against it.

I see, time and time again in the library world, the brittleness of perfection. If only we had the perfect rules — and then if we could implement them perfectly — and then everything would be great! And if something isn’t working, we need more perfect rules — we need thirteen years of committee process to write the perfect rules — and the cost of those years of inaction, the possibility that acting all along (however imperfectly) might bring more net benefit than waiting for the one true way, is politely not discussed.

And then I whiplash myself over to the computer science world and I see a merry agreement that things will be broken — that we will ship broken code, and we’ll know it’s broken, and we might not even know how, but that’s OK, because next month we’ll have banged out a new development cycle and shipped something else that — will be broken too, but will be better.

What the Real and Imaginary Parts of the Poles Do

It’s an asymptotic approach to the truth.

I was discussing this with my software-engineer husband after lunch and he said that, even in the wild-west world of software engineering, it was a tough paradigm shift, to go from implicitly to explicitly embracing this iterative mindset. But, we agreed, you have to have an iterative mindset if — following Lisa Carlucci Thomas‘s justly-retweeted Battledecks aphorism — failure is to be success in disguise.

If you aim do something once — perfectly — you either succeed or you fail, and failure isn’t an option, so we shy away from it — we shy away from even talking about it. I trace much of my networking success to having attended Andy Woodworth‘s Set Sail for Fail event at Midwinter, and I attended because that title was so striking — partly because it rhymed, but mostly because there was nothing else like it on the agenda.

Iteration liberates. Iteration frees us to walk toward the fail, to sit with it a while, to talk about it, and to take the time to understand how it’s success in disguise. Iteration means we have permission to fail because failures aren’t permanent, and because failures can be seen as mere perturbations of the system which tell us something more about how it’s shaped. Iteration means we always have the grace to fix things.

It means, as the world changes, we can change too, and our solution, so perfectly adapted for its little niche, need not die and fall apart when that niche changes.

I’m not going to be saying here that I wish everyone in the world would manage like a software engineer. (You engineers out there, I see you shuddering.) There’s a lot of deeply pathological issues in engineering management, and there’s a rough-and-tumble, high-energy aspect to the culture that is probably well-suited only to, well, software engineers. (And there are reasons we don’t generally let them near the customers.)

But I am saying that the perfect can be the enemy of a lot of things besides the good: things like learning, and adapting, and having the courage to look our mistakes in the eye. And I am saying that there’s a dovetail between the iterative mindset and something else Peter and I talked about — empowerment. You can’t empower your employees throughout the org chart unless you can be comfortable with everyone’s occasional failure, and you can’t do that unless you have a process wherein failure isn’t permanent, wherein there are second chances and we can boldly set sail for that fail because, when we get there, it won’t be a destination, but a conversation.

There’s some parallels crying out to be written here about Google Books metadata, and my experience as a teacher (and still-evolving understanding of what the culture of schools means), but this post is already long, and I’m eager to hear your thoughts on empowerment, iteration, cultures, fail.