What does your lab measure when ChatGPT did the homework?
Honestly, not much. So the lab moves up the stack: ideate the solution, design the system, serve a real use case.
Write a Python function that reads a sales CSV and prints total revenue, units sold and the top 3 items.
AI can make mistakes. Check important info.
Lab 07 · what the desk receives · Fri 23:58
What the lab measured: it runs. Nothing about the student.
Where this is going
Software now ships with an agent in the loop, from the first draft to the last review.
That changed what a developer has to know, how companies hire, and what a degree has to prove. It changed the assignment too. Nothing here was retrofitted to that shift; it was built after it.
Not our numbers. Theirs, as reported.
Sundar Pichai said three quarters of all new code at Google is now AI-generated and then reviewed and approved by engineers, up from a quarter in late 2024.
Sundar Pichai, CEO
SemaforThe job moved up the stack. So does the grade.
The engineering stack, as hiring sees it
Problem framing and the use case
You grade thisSystem design and trade-offs
Verification and judgement
Implementation
Typing
- 01
Code became the cheap part
Agents write most first drafts now. Teams measure an engineer on what they asked for and what they accepted, not on lines typed.
- 02
The job moved up the stack
Specify the problem, decompose it, design the system, decide the trade-offs, review what comes back. Roles like forward-deployed engineer exist because of this.
- 03
Agents are in the loop, by default
Review, tests, deploys and docs run with an agent alongside. Working without one is now the unusual case, and interviews assume you can direct one.
- 04
Judgement is the scarce skill
Knowing when a confident answer is wrong separates the junior who ships from the one who takes production down. Nobody hires for typing speed any more.
The agent can write the handler. It cannot decide what the handler is for.
Only the student can decide it should be event-driven, why a malformed row must not stop the run, and what finance actually needs in the summary. That is what the checks are written against.
Ideation
From a use case to an approach
The student states the problem, who it is for, the constraints and the shape of the solution before any code exists. The checks read that approach, not just the output.
System design
Components, data flow, failure modes
Where the file lands, what triggers the work, what breaks, what the consumer needs back. The lab's checks are written against these decisions.
Practical use case
A real consumer, a real constraint
The finance team wants one JSON summary a day. The assignment is judged on whether it serves them, not on whether it compiles.
The brief does not change. The checks do.
An agent passes every check on the left in under a minute. Nothing on the right can be passed without understanding the system you asked for.
A check an agent passes alone
Parse the CSV and print the totals
Any agent does this in under a minute, correctly, first try. Passing it says nothing about the student.
Handle errors
Vague enough that a try/except wrapped around everything ticks the box.
Output the result
Print it, save it, either way the check is met and nothing was decided.
A check only a student can pass
Decide what triggers the run, and defend it
Event-driven or on a schedule? The student states the trade-off; the judge reads the reasoning and the binding in the code.
Name the row that must not stop the run, and prove it
A malformed row is named in the brief, and a test the judge can execute proves it is skipped rather than fatal.
Say what finance actually needs back
Which three numbers, in what shape, for whom. Judged on whether it serves the consumer, not on whether it compiles.