The phrase "AI skills" hides a lot of different things. Some of them matter for the job; some of them don't. Some of them surface naturally in an interview; some of them take careful design to see. If your team has decided to assess AI skills in a technical interview, the first question isn't how — it's which.
This is an attempt at a working taxonomy. Not exhaustive, just the four competencies most worth designing for.
1. Prompt precision
This is the skill most people mean when they say "prompting", and it's also the one most often mistaken for the whole category. Prompt precision is the ability to describe a problem to an AI in a way that produces useful output on the first or second attempt — context, constraint, specific question.
On the job, a candidate who can't do this wastes hours in conversational loops with the tool. Over a year, that's a real productivity gap.
In an interview, you surface this by reading the first two messages of the candidate's AI transcript and seeing whether they framed the problem or just pasted it. The answer is almost always obvious within thirty seconds.
2. Output evaluation
Prompt precision gets you a plausible-sounding answer. Output evaluation is the skill of deciding whether the plausible-sounding answer is actually correct for your situation. This is the competency that separates engineers who use AI well from engineers who get burned by it.
Output evaluation is hard to test directly, because it happens inside the candidate's head. What you can do is design a task where the AI's first suggestion is plausible but subtly wrong — misapplied to a constraint that was mentioned earlier, or based on an assumption that doesn't hold. The candidates with good output evaluation catch the mismatch and push back. The ones without it go along with the suggestion and produce a polished submission that's wrong.
A challenge statement with a specific constraint hidden in plain sight is usually enough to set this up. The shape of the challenge statement matters more than any individual test case.
3. Knowing when to stop the tool
This one is subtle and often overlooked. A competent AI user doesn't run everything through the model. They have a working sense of when they're at a point where the model will genuinely help, and when they're at a point where they need to think alone for five minutes.
Candidates who over-use AI show this by asking the model for its opinion on every decision, including decisions the model has no way to form a useful opinion on ("which of these two names is better for our internal service?"). Candidates who under-use it solve everything from scratch in a language they're only adequate in.
The signal is in the distribution of the conversation across the task. A transcript where the candidate goes quiet for twenty minutes in the middle and comes back with a reasoned position is usually better than one where every decision is mediated by the tool.
4. Staying the author
The last competency is the one that matters most on a team. A candidate who uses AI well is still the author of the final artefact. Their voice is in it, their reasoning is in it, and a colleague reviewing their PR can follow the shape of how they think.
A candidate who uses AI badly produces work that feels like the AI's work with their name on it. When pressed in an interview, they can't explain why something in the artefact is the way it is. They shipped the tool's output without internalising it.
This competency is testable in the live follow-up. Quote a specific line from their written answer back to them and ask them to defend it. If they wrote the line, they'll run with it. If the AI did, they'll stumble.
What not to bother testing
Two things worth skipping, because they don't predict on-the-job performance.
Familiarity with specific tools: the space moves so fast that the tool a candidate used last year isn't the tool they'll use next year. If they can work well with one modern AI assistant, they can work well with the next one within a week.
Prompt-engineering theory: the ability to recite patterns ("chain of thought", "few-shot", "role prompting") isn't the same as using AI well. The skill is applied; the theory is about the skill. Hire for the first, not the second.
The shape of the round
If you want a single round that surfaces all four competencies, the shape is fairly simple. A realistic problem, permission to use AI, an expectation that the candidate submits both the artefact and a short written reasoning about their choices, and — crucially — access to the transcript. Most of the assessment then happens at interview-prep time, not in the live round.
At CriticCode this is how the product works. The shape isn't specific to any tool, though. Any process that gives the candidate permission, collects the conversation, and asks for the reasoning will produce roughly the same signal.
The practical test
Before you design anything, answer this: which of the four competencies above are actually load-bearing for the role you're hiring? If the answer is "all of them", you probably haven't thought about it enough. Most roles put heavy weight on one or two and lean weight on the rest. Design the round around the heavy ones.