an asymptotic approach to truth

Spent the weekend visiting the in-laws in Princeton, hence visiting my librarian crew down there; Monday lunch with Janie Hermann followed by Deep Thoughts with Peter Bromberg. Lovely! (As always: how lucky am I to have found this profession where people are so generous with their time, thoughts, and expertise?) And at one point Peter said, “Blog that!”, and so:

I spent much of library school in cultural whiplash between the library world, where I was being steadily acculturated, and the computer-geek world where my husband and so many of my friends work, and where, to an extent, my undergraduate background lies. And this is coming up again now that I’ve decided I really need to understand something about how ALA works.

Cthulhu Warning

Of course, the more I look into the committees and subcommittees and processes the more I get the sense that it is a Lovecraftian horror and if I ever understand it in full I will have lost my mind.

I have difficulty with bureaucracy. I see that ALA has a lot of it, and I don’t understand why — don’t know the historical antecedents that led to so many rules accumulated upon rules. (Please: do tell.) I sense there’s a story there, but I don’t know it, and even if I did I’d chafe against it.

I see, time and time again in the library world, the brittleness of perfection. If only we had the perfect rules — and then if we could implement them perfectly — and then everything would be great! And if something isn’t working, we need more perfect rules — we need thirteen years of committee process to write the perfect rules — and the cost of those years of inaction, the possibility that acting all along (however imperfectly) might bring more net benefit than waiting for the one true way, is politely not discussed.

And then I whiplash myself over to the computer science world and I see a merry agreement that things will be broken — that we will ship broken code, and we’ll know it’s broken, and we might not even know how, but that’s OK, because next month we’ll have banged out a new development cycle and shipped something else that — will be broken too, but will be better.

What the Real and Imaginary Parts of the Poles Do

It’s an asymptotic approach to the truth.

I was discussing this with my software-engineer husband after lunch and he said that, even in the wild-west world of software engineering, it was a tough paradigm shift, to go from implicitly to explicitly embracing this iterative mindset. But, we agreed, you have to have an iterative mindset if — following Lisa Carlucci Thomas‘s justly-retweeted Battledecks aphorism — failure is to be success in disguise.

If you aim do something once — perfectly — you either succeed or you fail, and failure isn’t an option, so we shy away from it — we shy away from even talking about it. I trace much of my networking success to having attended Andy Woodworth‘s Set Sail for Fail event at Midwinter, and I attended because that title was so striking — partly because it rhymed, but mostly because there was nothing else like it on the agenda.

Iteration liberates. Iteration frees us to walk toward the fail, to sit with it a while, to talk about it, and to take the time to understand how it’s success in disguise. Iteration means we have permission to fail because failures aren’t permanent, and because failures can be seen as mere perturbations of the system which tell us something more about how it’s shaped. Iteration means we always have the grace to fix things.

It means, as the world changes, we can change too, and our solution, so perfectly adapted for its little niche, need not die and fall apart when that niche changes.

I’m not going to be saying here that I wish everyone in the world would manage like a software engineer. (You engineers out there, I see you shuddering.) There’s a lot of deeply pathological issues in engineering management, and there’s a rough-and-tumble, high-energy aspect to the culture that is probably well-suited only to, well, software engineers. (And there are reasons we don’t generally let them near the customers.)

But I am saying that the perfect can be the enemy of a lot of things besides the good: things like learning, and adapting, and having the courage to look our mistakes in the eye. And I am saying that there’s a dovetail between the iterative mindset and something else Peter and I talked about — empowerment. You can’t empower your employees throughout the org chart unless you can be comfortable with everyone’s occasional failure, and you can’t do that unless you have a process wherein failure isn’t permanent, wherein there are second chances and we can boldly set sail for that fail because, when we get there, it won’t be a destination, but a conversation.

There’s some parallels crying out to be written here about Google Books metadata, and my experience as a teacher (and still-evolving understanding of what the culture of schools means), but this post is already long, and I’m eager to hear your thoughts on empowerment, iteration, cultures, fail.

4 thoughts on “an asymptotic approach to truth

    1. I suspect that Google is not trying to solve the general metadata problem, which is why their metadata is not very good.

      Google’s legal settlements oblige it to attempt to find copyright holders for the works it indexes. But this was impossible because Google didn’t even know, in more than a strictly probabilistic sense, what the works it indexes are (Are two scanned volumes the same book? A hard problem for a computer algorithm with currently existing metadata!). Google’s number one problem is actually a clustering problem. Given a set of, eg, a billion distinct metadata records, how can those records be arranged into clusters likely to represent the same book. Then they can try to refine the author and publisher metadata and work out the copyright state of each cluster.

      This is probably the reason that their big announcement was the number of books they know of. They neither know nor, at this stage, particularly need to know the actual identity and correct metadata of each of the books, just what metadata records are likely related to them.

      To solve this problem doesn’t require correct metadata. It only requires metadata with a statistically understandable distribution of errors.

      A fundamental rule of software development is to not prematurely optimize. You can expect that Google will not try to correct its metadata until that is the most valuable next use of its resources. Right now bad metadata is good enough. That will probably change as advertisers demand more correct filtering of what books their ads will appear near, and change further if readers start doing metadata sensitive searches in large numbers. Until that point the metadata is not the most urgent problem so long as it is good enough to yield decent clusters.


  1. My half baked reaction to this is to think of the difference between the data-culture I came from, where you know exactly the statistical model you want to use before you even see the data and it’s considered a moral failing to change course — versus the one I’m in now, where an iterative model-building process is considered sensible and desirable. I am too sleepy to continue baking this idea, however.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s