No, programming competitions don't produce bad engineers

A couple of times a year, someone finds this video by Peter Norvig and makes a blog post saying "Being good at programming competitions correlates negatively with being good on the job," and people jump to all kinds of conclusions about it.

What it's actually about

In the mid-'00s, a team at Google built a machine learning model to figure out what features of Google employees predicted good performance at Google. Each feature ended up with a positive or negative score, and the one for participation in programming competitions was negative.

What can we conclude from that? Not much.

Hire the highest jumpers

I just bought a minor league basketball team, so I need players. It's expensive to pay scouts; so I'll use scouts to hire only half my team. Jumping high is important, so I'll look at a list of the highest jumpers in the U.S. and hire them.

High jumpers suck

On my team, the people who can jump the highest are going to be the worst players. It isn't because good jumpers make bad basketball players: it's because I overemphasized the importance of being able to jump high.

Google overemphasized the importance of being good at programming contests when they were hiring in the early '00s. I can personally attest to that: I had no experience, a physics degree, and a good finish in Code Jam when I got hired as an intern after one phone interview.

All jumpers suck

When Google ran the study, the guy who won the TopCoder Open got the same label as a guy who lost a non-competitive ACM ICPC regional. The model didn't have a feature for doing well in programming contests... and some people put really lousy finishes on their resumes.

Machine learning sucks

No it doesn't, but the algorithm used for this study was bad at attribution. It could produce a score for a whole person, but because of correlations between features, you can't look at the score for a single feature and see what it "means."

I once worked on a machine learning system to predict the probability that a user would click on an ad. The features often had bizarre-looking scores. "The user and the advertiser were both in Germany": -10% chance of click. Why? Maybe "The user was in Germany and the ad had a German word in it" gave +20% chance of click. If you looked at the Germany/Germany feature in isolation, you'd come to a crazy conclusion.

The study was panned

After presenting their findings, the team was hammered by questions by senior engineers, and the project was shut down shortly thereafter. This perhaps speaks to what Googlers at the time thought about the quality of the study. (Note that this is hearsay, and based on old memories, so it's likely to be an exaggeration. Perhaps the study was brilliant and ahead of its time.)

High jumpers sucked a decade ago

Even if this study had a foolproof algorithm that did attribution correctly, studied the pool of programmers that you personally happen to care about, and correctly separated the best competitive programmers from everyone else... it was run over a decade ago, and the landscapes of software engineering and competitive programming have both changed a lot since then. There are a lot more software engineers, contest problems are more imaginative, and the competitors are better than we used to be when I was serious about it.

Sanity prevails

Given enough time, Reddit and Hacker News commenters get upvoted for posting reasonable things like "I've looked at these kind of machine learning results before and there are often seemingly surprising results, but you have to remember that it is in the context of a larger equation."[1] or "It's important to note that this was limited to people that were able to pass Google interviews and it's extremely unlikely that this holds for the general population of programmers."[2]

Who cares?

If you're an employer, you should look past the headline. No matter how carefully you phrase it, there's no compelling evidence that having a programming competition on your resume is a bad thing.

If you're thinking about becoming a participant in programming competitions, go for it! If you participate enough, you'll get a ton of experience doing a lot of simple things that you'll do all the time in your job, and doing more complex things that you might break out once in a while when you need them. It's a lot of fun, and you'll probably meet some pretty awesome people. I did.

-- Bartholomew Furrow, former engineering lead for Google Code Jam