Programmers are the most optimistic bunch of people I have ever come across. Ask us how long something will take and chances are our estimates will be way off.

English: Programmer

English: Programmer (Photo credit: Wikipedia)

It’s not because we hate you. It’s not because we didn’t know any better, either. We’re just very optimistic about everything. The project, our skills, the tech, your specs, the unknown unknowns …

Especially the unknown unknowns are a big problem – it’s impossible to predict everything that might go wrong. The variables are just too many.

Right now it looks like we can do This ™ in an hour, but a weirdness or twenty and suddenly it’s taking a whole day.

As programmers we assume the least complications, even when multiplying by Pi.

But there’s another profession that gives a lot of predictions in the face of uncertainty – doctors. When a doctor gets their estimates wrong, people are often delighted. What?

Let me tell you a tale of two estimates – a conservative and a programmer estimate.

A tale of two estimates

Last Friday I messed up my shoulder, more specifically that tiny little joint between my clavicle and shoulder. I was longboarding, did something wrong and landed on tarmac. Hard.

In the morning I had to use my left hand to eat breakfast. Couldn’t even pick up a spoon.

At the ER the doctors told me that hey, the bones and joint seem fine (no dislocation etc.), but I must have messed up my tendons something bad. What I can expect is three weeks of pain and problems. “Just take some painkillers if it gets too bad”, they said.

Today, 6 days later, my shoulder is already at 90%. Clenching my teeth just a little, I could do 60 of my usual 150 pushups this morning. Even jumping rope wasn’t much of an issue. Yesterday I could even carry a four kilogram longboard in my right arm for extended periods of time.

In a day or two I will be fully healed, but essentially the doctor’s estimate was off by almost a factor of 3. And I couldn’t be happier, it’s brightened my whole day up!

A programmer’s estimate

I’m working on a web page segmentation algorithm. The idea is to take a website and decide which parts are headers, sidebars and so on. It’s an interesting problem since everyone else is only concerned with extracting the main juicy content.

The algorithm finally started working earlier this week, just a matter of merging the results back into the original HTML so they can be useful.

I’ll have it by Friday morning, I said.

I mean how hard can it be? I’m just using a language I hate (java), am not experienced with, a paradigm I dislike and try to avoid, and a library I only started poking at two weeks ago.

Surely I can do a task I don’t fully understand by Friday morning! It’s the estimate that popped into my mind when I tried to pull something out of my arse, so it must be correct. I’ve been a programmer for ages!

Nope, didn’t get it done.

Turns out I didn’t understand the object hierarchy of boilerpipe – took me many hours just to extend the relevant functions of a particular object.

Then I found out the HTML parser I’m using is linear, so there is no clear way of deciding which child nodes belong to what parent node.

In the end I spent all of my time just figuring out how to add the algorithm’s results to the HTML in the crudest way possible – add a number in front of every node in the DOM.

Well oops.

Programmers are optimists

Whenever you talk to a programmer, just remember, they are an optimistic bunch. Yes, even the most disheartened scruffy old programmer is insanely optimistic. We are the kind of person who:

  • Dives into a project with a spec that will change
  • Grabs a bunch of unknown tools, applies them to an unknown domain … and wins
  • Lies in a different world with every new project
  • Changes all their tools every couple of years

Anyone who isn’t an optimist just can’t survive as a programmer.

How many other professions can say that about themselves?

Enhanced by Zemanta

You should follow me on twitter, here.

  • Robert

    As for the shoulder, if I was you, I wouldn’t push it until it heals completely.

    As for the estimates… I always tend to take the “middle” path (not completely optimistic, not completely pessimistic, but somewhere in the middle). And then add 20% on that.

    But in general I do see myself as an optimist and a programmer (for a rather loose definition of programmer :P).

  • I remember this phrase «Programmers not only run on coffee and sugar and sushi and doritos, they run on happiness. They have a finite endowment of happiness and they spend it continuously, like drunken sailors.» (

  • Little theory what is behind this phenomena…

  • essiccf37

    You are being nice when you say opimist, sometimes I wonder if it’s not only foolishness ^^
    Great post, thanks.

  • If we weren’t optimists and had to swallow the daunting truth about how much we don’t know and how long and difficult our task(s) really are going to take, we’d quit… or go insane.

    I’ve found that as the years go by I become less and less optimistic and more and more cynical. It sucks. Pretty soon I’ll probably quit because I don’t want to be the “negative norman” in a company. All the more reason to quit and be insane working for myself…

  • Nick

    The doctor’s (over)estimate doesn’t cost anyone anything. A programmer’s estimation can affect cost/scope/planning for the rest of a company. That can be one thing which heavily influences programmers to give “pleasing” estimates.

  • Shreknel

    You must be the biggest optimist among programmers. Where we work our estimates are seen by management as ‘pessimistic’, but most of the time turn out about right.
    As for your doctor’s estimate; well, they’re working on even more incomplete information than you.

  • Sounds very much like a page out of Clean Code and it’s very true. Personally, I always estimate by the worst-likely-best case scenario formula and it’s remarkably accurate. Anyway: I enjoy your programming-philosophy-posts very much, so keep them coming.

  • Aaron Steed

    Wow, you must be my arch nemesis.

    My whole programming work-flow is based around the worst possible outcome.

    The result is that I have an uncanny knack to predict the future by way of all the possible things that could go wrong. This makes for excellent skills in complex logic, debugging and project management.

    Sure, I might be a miserable cunt, but I smash deadlines (by making it clear they need to be extended because of the disaster that’s about to happen).

  • Anon

    I’m always pessimistic that nothing I ever work on in software will ever be good enough, finished, or properly complete. It’s either confusing, half-way working, or in a state of ‘could be better.’

  • Mark Kethisouaran

    I agree that programmers are optimists, but you can’t really compare those two estimates. The right comparison (imo) would be if the doctor said that you’d be better in two days and it took more than a week. Under that scenario, I don’t think you would be happy.. Furthermore, another estimate that the doctor may go wrong is about someone’s life. They might say “He has at least 2-3 years” and suddenly after a couple of months…

    @04c8366319a7848c69e807f4eda3101c:disqus The doctor’s overestimate may cost a lot if someone is getting treated at the hospital!

  • That’s a really cool link! Thanks

  • As for the shoulder, I won’t know it’s healed if I don’t push it a tiny little bit.

    Unused joints also become “locked up” and painful. It sucks.

  • Pingback: In the News: 2012-05-26 | Klaus' Korner()

  • Pingback: A Clogwog in Oz()

  • Pingback: 程序员是乐天派 - 博客 - 伯乐在线()

  • Pingback: 程序员是乐天派 | IT & 数码()

  • Pingback: Reading Notes 2012-06-04 | Matricis()

  • caroldekkers

    Great post! I’d take a little bit of a different stance – programmers are often optimists because management insists on it! Even when project managers put forth a realistic estimate, management often cuts the budget (and/or schedule) in half insisting that the impossible be done, and programmers often are left to deliver (and do with overtime, yeoman efforts, creative and innovative solutions) to the “new and improved” 1/2 time schedule.

    When it succeeds (even if the quality sucks), management justifies cutting the project in half and will continue to do so. Programmers are often the unsung heroes of the development team!

    Looking forward to more posts.

    p.s., Wishing you a continued shoulder recover – as a young “pup” you likely heal faster than us older folks. Onwards and upwards!


  • Pingback: Understanding Programmers: a primer for Business People | ReStreaming()

  • Pingback: 程序员是乐天派 | OneCoder()