The 10x Programmer and the End of the Myth of Research
In May of 2020, Bill Nichols got the world talking with news of a study that proved, says he, that the 10x programmer does not exist. One neat trick I clearly liked was putting his desired research result in the story title:
“The End to the Myth of Individual Programmer Productivity”.
Wow! It must be true! Did you see the title?!
Another smart move was not disclosing the researcher’s conflict of interest: they run programming classes. So the entire organization depend$ on its existence for software being something that can be taught, and on taught methods being more important than individual variation. Poke round, $1250 for one course that does not even include recognized academic credit.
But I am not here to talk about Mr. Nichols’s marketing. I am here to help folks think about research, with Mr. Nichols’s example as the cadaver for our post mortem. Let us research his research,
Good news, bad news
The good news? Quite a large sample of programmers!
Of the 3,800 students in our classes (from 2000 to 2006), …
Da-yamn! Thirty-eight hundred over six years! But wait…
…this study included only the 494 who used the C programming language and who also completed all 10 programming exercises.
Oh. In assessing ability distribution, they took the top 13% able performers as their sample. Hmmm. But there is more good news: a diverse sample!
Most students were professionals from industry with experience ranging from less than a year (half of the subjects) to 36 years, with an average of 3.7 years.
That seems quite representative! Now comes more bad news.
The sample population seems to be one that goes to a continuing education school to pay a ton of money to learn a new programming language. Full respect for the students’ ambition and diligence, but did we just filter out all the 10x programmers?
Hint: Yes.
C? In the 21st Century?
Next, they are using “C”. I had great fun programming hundreds of thousands of lines of “C”. Then I went looking for a better way and found Common Lisp. Quite late, I confess, 1992.
Are experienced 10x programmers paying to learn “C” in the 21st century? When we already had Common Lisp and Python? And when we could just buy K&R, one of the best tech books ever written, so great most of you knew what it was just from two initials? The birth of “Hello, world!”?!
Hint: No.
Bad news, bad news joke
So we have a disappointingly homogenous population of the 13% best students at a continuing education school taking a class to pad their resumes. Damn. We will need to run one Hell of a good study to identify 10x programmers. Let us look at what they did.
…we used data from the PSP class programming assignments for which students record effort, size, and defects for a series of programs…each student programmed solutions to the same 10-assignment specifications and collected the direct effort time…the instructions provided sufficient detail and examples for a non-specialist to implement the solutions. Each was the size of one or two small Agile user stories requiring two or more hours to implement. The student recorded time on the major activities needed to complete a solution that passed all required tests, including planning, design, coding, testing, and personal reviews.
The data was decided by the subject participants? The instructions were detailed and included examples? Exercises were well-specified and the size of small Jira tickets?
THEY SUPPLIED A TEST SUITE?!!!!!!?
Can we identify Michael Jordans with free-throw contests? Bobby Fischers with rock-paper-scissors tournaments? Do we look for Reinhold Messners on climbing walls?
Can we identify Michael Jordans with free throw contests?
Simply put, the variation in any skill can be measured only on tasks that can draw out variation.
- The goal and requirements are there but fuzzy.
- There is no spec.
- The structure to be built will have layers upon layers, and be so big that the construction itself must be automated.
- The result must be easily extendable as time goes by.
- When the requirements change half way through — no problem!
- It must be possible for other developers to complete the work if the designer steps in front of a bus that same half way through.
No, no one could never research a thing like that. One can only observe it in all the developers out there in tall buildings doing that now, recognized by management and their peers for their intensity, expertise, and productivity.
Punch line
The best part? Mr. Nichols did find a factor of two variation in productivity.
