AI can pass your coding test. Now what?

Not enough, unless you also know how they arrived at the answer.

We don't ban AI at Prorated. We let engineers use it, benchmark what AI can do alone, and look for the judgment that separates a strong engineer from someone accepting the first plausible answer.

A passing patch can still miss the problem

Take OPS-202, a ticket from our public code-comprehension challenge:

I was on a slow connection. I clicked Send once. The customer says they got my reply twice and the request now shows two replies. It does not happen every time. Please check rather than blaming my connection.

The obvious fix is to disable the Send button while a request is in flight. That looks sensible. It might even pass a quick manual test.

But the full requirement says the reply must still arrive if the first response is lost and the request is retried. Disabling a button doesn't make a retry safe.

The better investigation follows the request through persistence and email delivery. It checks what happens when a retry arrives, and verifies that the system produces one reply and one email, not two and not zero.

An engineer can use AI throughout that process. The signal isn't whether they typed the patch. It's whether they caught the incomplete fix, challenged it, and verified the actual requirement.

Five scores. Evidence behind each one.

Both AI-only runs and candidates are assessed on the same 0 to 4 rubric dimensions:

1. Delivered resultDid the work meet the requirements?
2. Diagnosis and validationWere conclusions grounded in the evidence inspected and checks performed?
3. Priorities and tradeoffsWere the scope and decisions sensible under the constraints?
4. AI supervisionWas the agentic work directed, questioned, and corrected when necessary?
5. Handoff and honestyDoes the final explanation match what was actually done?

A score alone is cheap. We want the receipts: the log someone read, the explanation they rejected, the test they ran, the change they made, and the limitation they acknowledged.

Model-assisted evaluation applies the rubric to each run, but it isn't infallible. A reviewer can inspect the underlying session and overrule a rating. If something wasn't observable, we mark it not observed rather than pretending a missing signal deserves a zero.

The AI benchmark is a reference, not a scientifically validated measure of human intelligence. Ten or fifteen runs are useful for seeing patterns, not for making grand statistical claims. Small candidate cohorts deserve the same restraint.

Hire for judgment, not résumé polish

Founders hiring their first engineers don't need another pile of polished submissions. They need a credible answer to a harder question: Who can take an ambiguous problem, use the tools available, and make a sound engineering decision?

That's what we're trying to surface. The benchmark tells us what AI can do. The work trace tells us what happened. Together, they give the hiring team specific evidence to discuss in an interview, not an automatic hire-or-reject verdict.