Are AI Benchmarks the New LeetCode for Hiring Frontier Models?

For those that don’t know, LeetCode is a site used to prepare for computer science job interviews.

Every week brings a new LLM leaderboard. One model edges past another on a coding benchmark, the press release goes out, and the coverage treats it as a fact about which model is better at writing software.

The results are worthy of mention on the nightly news or the front page of the mainstream media, as they work into the story regarding the international race for AI dominance.

We’ve Been Here Before: The Vocabulary Part of the SAT Test

Consider the vocabulary section of the SAT. Why test vocabulary at all? What is the point of that?

The original point was that it indicated how much you had read, and at what level of difficulty. It was never something you were even supposed to be able to study for. You either had the words or you didn’t, and having them meant you had spent years reading things that contained them.

Over time the whole point was lost. You could learn the two thousand or so words on the test by rote and pass that section with flying colors having read nothing but Superman comics.

What the score measured was no longer what you had read. It was how much time you were willing to put into getting into a good college and playing their game.

Fast Forward to the Information Age and Enter LeetCode

As the information age grew, so did the desire for the hiring process to become more objective and consistent. The whole process was very short considering the commitment of years you would be making by hiring that person. Little by little, standardized programming tests became part of the entrance exam for getting a job.

It became a new version of SAT preparation. Theoretically you should not be able to study for the entrance exam to a high tech programming job. You should be a competent programmer for the task or not. But now you could study for them.

You could go to sites like LeetCode and learn all the problems you might encounter and the most popular questions. Getting a job became learning to study for the interview. People who wanted to change jobs would often spend months preparing. LeetCode was the programming part, but there were other sites for the other kinds of standardized questions you might get.

You could practice in depth for the entrance exam at particular companies. Meta, Google, AWS, Apple. You could know the whole drill. Having interviewed at those places myself, I can tell you that those sites give great preparation for that testing.

You could pass these entrance exams with flying colors and not even be able to solve these kinds of problems, because you could learn the answers to all the exact ones you were likely to see.

You just needed to remember what the program solution looked like. Even if you can’t remember it exactly, if the interviewer likes you they will walk you through what you can’t remember.

Despite the claims of objectivity, the interviewers will frequently put their thumb on the scale. I’m 70, and many interviewers’ jaws drop as soon as they see me. I know at that point I might as well stop the interview.

One time, the minute the interviewer saw me, before we had talked at all, he started trying to convince me that I did not want this job. I had just finished preparing all the LeetCode and was in perfect form. The one non-LeetCode problem was a complicated compiler problem that I happened to know because I had solved it once before in a real project, and I did it instantly. My reviewer told me that I took too long. That was not true, but it was defensible enough to put on the review and bar me.

So gaming these interviews involves some skills that have nothing to do with the job.

Back in the day, you wanted to talk to people about what they did and get a sense of how they think. That part is pretty absent nowadays from my experience. It is all about some kind of supposedly objective testing methodology.

This Already Happened to Compiler Benchmarks

The compiler world ran this experiment already. SPEC published its first suite in 1989. One of the programs was a dense matrix multiply, chosen because that sits at the core of numerical computing. Compiler writers noticed that restructuring the loops so each submatrix fit in cache transformed that one program and did almost nothing for anything else. It was retired in 1992.

The fixes that followed were not cleverer tests. They were rules. Declare whether you tuned per benchmark or ran one flag set across the suite. No compiler flags that reference benchmark names. Full disclosure of the configuration so someone else can reproduce your number. Retire any benchmark that has become a tuning target.

That took about twenty years and it is still gamed. It is just enormously better.

The Tell

I worked at a small compiler company that was very well known for top optimizing compilers. Intel in those days had a benchmarking compiler that existed mostly to sell its chips. Nobody really used it, but it put up numbers we had to beat.

One of my coworkers was a wizard at code generation optimization. There was a benchmark Intel was beating us on, and when he looked at the generated code it made no sense. The compiler was making choices it should not have been able to make from the code it was given.

Tennis fans will remember Boris Becker wondering how Andre Agassi seemed to read his mind when he served. Agassi was not reading his mind. Becker stuck his tongue out before he served, and the direction it pointed told you where the ball was going.

This particular benchmark was driving my friend crazy. Then he realized the compiler was using the names of the variables to guess things about the benchmark. He renamed them, changed no logic, and the improved performance numbers went away.

Tell Me About Some Problem at Work That You Solved Using Recursion

I worked in compilers a lot, which is an area where you are likely to need complex algorithms to solve the problems. Many of the LeetCode and other such tests involve solutions using recursion. Many compiler algorithms use some form of recursion, and reducing even higher level parts of compilers to some kind of recursive problem is key to getting a good solution.

My favorite interview question was: tell me something you solved at work using recursion.

I did not want some particular problem you could have learned in school or in a test preparation class. I wanted to know how you saw a real problem that, naturally in your mind, could be solved with recursion.

It is very rare that someone can think of an example.

I’m sure those same people could have answered LeetCode recursive problems all day without breaking a sweat.

It is like the SAT vocabulary test. I want to know if they can think recursively. Someone who can’t, but who can learn a hundred popular recursive programming solutions, is not useful for the kind of work I need them to do.

To me, if someone works on compilers and can’t think very clearly in recursive terms, the compiler is going to be a mess.

Enter the Frontier Model Benchmarks

I don’t claim to be an expert on the various benchmarks. What I do know is that anything solvable in less than a few months is not really what I need for real world programming projects. If a benchmark only shows you a thin slice of what the model can do, it is not going to tell you much.

As a personal example, I have a background project to make a full C++20 compiler using an LLM. There have been some partial solutions to making a C compiler, but nothing that could really pass a commercial test suite for C. And C++, despite the similar name, is about a hundred times more complex.

I use Anthropic for these more complex problems. When I started the project I began with another model that was trending in the top benchmark slot at that moment. It made a bad strategic choice early on that made the whole task much harder than it needed to be. Then it got stuck on a bug and hacked at it for days, eating a lot of budget. There were other issues I never dug into. I switched to Claude Code, which agreed with me about the high level mistake — we eventually reverted it — but Claude was still able to work inside the original framework.

The top trending model was not really able to solve the problems once they got more complex. It just churned.

All of these models have advanced and I am not here to advertise Claude. My team uses it exclusively now and we are happy with the results. Changing would be like firing your best employee when you have no complaints about him. If something becomes an issue I might look at the others. But I have logged more than a year and a half of solid time, and Claude has grown as our projects have grown.

It Is Not Just About Picking a Model Anymore

The same numbers are now driving national AI strategy and a news cycle about whether China is getting ahead of us.

On January 27, 2025, Nvidia lost roughly $589 billion of market value in a single day, the largest one-day loss in Wall Street history. The semiconductor index had its worst day since March 2020. The trigger was DeepSeek posting benchmark results comparable to the American models. Marc Andreessen called it a Sputnik moment.

When Congress took up a bill to bar Chinese AI systems from federal agencies, the stated basis was that Chinese models had reached near parity on the benchmarks.

Selecting Frontier Coding Models From Benchmark Leaderboards

If you use these to decide which models are better for coding, you are going to duplicate the long history I have outlined here. You will get models that are good at passing tests but may not be very good at all at programming on a real project.

I understand why the tests exist. If you are hiring fifty thousand programmers and need some consistency, a standardized test is cheap, uniform, and defensible. I have never run a company like that and I have no data to refute the case for it.

Of course there are more things on an interview than the programming tests, but they are a fundamental bar. The other parts are organized in their own way, so that you can learn how to pass them by studying and by networking with friends about the interview process. As with the SATs, they measure how much you were willing to go through to get the job, which is very different from how well you can perform it.

It also often shows that you did nothing at your current job for months ahead of the job change while you studied for your next interview.

Benchmark-driven model selection usually stops at the test. The leaderboard is the number and there is no second round.

A benchmark score is a fact about which model is better at the benchmark. Nothing follows from it about your codebase.

The only evaluation that matters is your own work on your own problems, which has the useful property that nobody selling to you can study for it.

Most importantly, success with LLMs is highly dependent on the skill of the people using them. Don’t confuse directing an LLM with programming yourself. They are different skills.

Make sure, both with the programmers and with the LLMs, that you know what you are hiring for.