How to assess AI skills in an interview

How to run a work-sample task for AI skills

The quickest way to get past a yes is a work-sample task: a short, practical piece of work the candidate completes with an AI tool while you watch, or shortly before the interview. It shows you how they work with AI output, which no resume line can.

Meg Alexander, virtual event host at PowerToFly AI, sums up why this step matters:

"The credentials are how you know that someone can do the work. The interview is how you know what they are going to be like while doing the work."

Set up the task

Pick a task the person would actually do in their first month on the job, and go beyond asking AI to draft something. Strong AI users also analyze data, build reusable instructions that keep their work consistent, and automate repetitive steps. Use anonymized or made-up material, never real customer data.

Let them use the AI tool they know best, since what you want to see is their judgment. Tell candidates what the task involves before the interview, including the time limit. We recommend keeping the task under an hour and paying the candidate for their time, however long it takes. For bigger builds, like an automation, it's fine to ask for a plan and one working piece.

What to watch for

Ask the candidate to talk through their choices as they go, or to share their chats and setup afterward. If you're not sure what a strong result looks like, ask a teammate who does the job to watch along with you. Keep these four things in mind:

• How they break down the work. Strong candidates decide which steps AI should handle, which to automate, and which stay with a person.
• What they check. Watch whether they verify facts, numbers, and names before trusting the output.
• Where they push back. The best AI users will tell the tool it's wrong and redirect it.
• Whether the result can be reused. Look for work a teammate could run again next week and get the same quality.

Sample tasks for four roles

Here are four tasks you can adapt. Each one tests a different way people use AI on the job.

1. Ask a marketing manager to build a reusable setup. Share your brand guidelines and two product briefs. They create saved instructions in an AI tool, sometimes called a custom assistant or skill, that turn any brief into on-brand campaign copy, then test them on both briefs.
2. Have a customer success lead analyze data. Give them 50 anonymized support tickets and a short list of renewal dates. They use AI to group the tickets into themes, flag accounts that might not renew, and recommend where to focus this quarter. Ask how they confirmed the AI's groupings were right.
3. Give an ops lead a process to automate. Describe a manual workflow, like collecting vendor requests from email and logging them in a spreadsheet. They map out an AI-assisted version, build one step of it, and mark where a person should still step in, like a request with missing details.
4. Let a recruiter speed up their own week. Hand over anonymized notes from three interviews and your scorecard. They use AI to turn the notes into consistent candidate summaries and catch what the tool got wrong. Plant one tricky detail in the notes to make the test sharper.

Once you've seen how someone works, you'll have more concrete moments to ask about in the rest of the interview.

What interview questions show if a candidate can actually use AI?

The questions that work best ask for specific stories about AI in the candidate's own work, especially times it got something wrong. Someone who uses AI every day should be able to share detailed stories and examples. Let candidates know beforehand that mistakes are expected and they happen to the best of us.

Here are three questions to start with, and what to listen for in each.

"Tell me about the last time AI got something wrong in your work. How did you catch it?"

A strong answer describes the task, what went wrong, and how they spotted it. For example: "It invented a stat for a client report, so now I trace every number back to the source." A weak answer tends to be more general, like "AI isn't always accurate, so I double-check." If you hear the weak version, ask for a recent example.

"When did you decide not to use AI for something?"

Good AI users know the tool's limitations. Listen for a clear reason, like private customer data, a sensitive conversation, or a task where AI kept making the same mistake. For example: "I write the first draft of anything going to an upset customer myself, because the tone has to sound like a person." If the candidate can't think of an example, ask if there's any kind of task they wouldn't hand over to AI.

"Which parts of that did you do yourself, and which did AI do?"

Ask it after any project story, with real curiosity in your voice. Some candidates worry that admitting AI did part of the work will count against them, so make it clear you expect the answer to include both. Strong candidates draw a clear line between their work and the tool's, and they're proud of both parts.

You can learn more about hiring for judgment in our Who makes AI work? guide.

These same questions also make overstated experience easy to spot, which is what we'll look at next.

How do you tell if someone is exaggerating their AI experience?

You can usually tell someone is exaggerating their AI experience when their answers are overly vague, their tool list has no results attached, or they have no stories about mistakes. Ask for specifics and you should be able to gauge who has real experience and who doesn't.

Some exaggeration is to be expected, since candidates know employers want AI skills. In a separate Harris Poll survey for Express Employment Professionals, 22% of job seekers admitted to listing skills they don't have, and 86% of hiring managers said AI makes it too easy to embellish a resume.

Here are three warning signs, and a follow-up question to test each one.

1. Vague answers. "I use AI to be more efficient" could describe anyone. Ask, "What's one task you did with AI last week, step by step?"
2. Tool lists without outcomes. A candidate who names six AI tools but no results may have only tried them, which is a fine start but a long way from using them well. Ask, "What changed in your work after you started using that one?"
3. No mistake stories. AI tools make mistakes, so regular users tend to have stories about catching them. Ask, "What's a mistake AI made that nearly slipped past you?"

Nerves can make anyone give general answers, so give candidates a second chance with a more specific question. Real AI users usually light up once you ask about the details. Write down what you hear as you go, because the next step is turning it into a score.

How to score AI skills so they match your job description

Score candidates on the same AI skills your job description lists, using the same scale for everyone. That keeps your interview, your scorecard, and the job itself lined up.

Our guide on how to write a job description for a role that uses AI recommends listing three skills for most roles: checking AI output for errors, knowing when to skip AI, and explaining how they used it. If your job description lists different ones, swap them into the table below. Here's a simple way to score each skill from 1 to 3.

Score each skill separately, and keep a line of notes next to each score so you can reference it later.

Use the same questions, in the same order, with the same scorecard for every candidate. That's the core of a structured interview, as the U.S. Office of Personnel Management's structured interview guide lays out. And research shows this technique works. A 2022 meta-analysis in the Journal of Applied Psychology found that structured interviews ranked highest for predicting job performance among the selection methods studied. A shared scorecard is also fairer to candidates, since everyone gets the same questions and the same chance to show what they can do.

Apply the same thinking before the interview. If your resume screen filters for keywords, strong AI users can drop out before you ever meet them. Jennyn Herrera, head of talent acquisition at Biola University, has a simple rule:

"It's very important to have a human looking at everything. Don't rely on your tech tools to screen people out."

After you hire, check whether high AI skill scores matched strong performance on the job. Our guide on how to measure and improve quality of hire shows how to track that, so your scorecard gets better with every hire. With the scorecard ready, all that's left is the questions to go with it.

An AI skills interview question bank for your next interview

Copy these eight questions into your next interview plan, and ask every candidate the same ones so your scores stay comparable. For deeper explanations and more expert advice on interview questions, check out our Who makes AI work? guide.

1. "Tell me about the last time AI got something wrong in your work. How did you catch it?"
2. "When did you decide not to use AI for something?"
3. "Walk me through a project: the problem, why you chose your approach and what happened."
4. "Which parts of that did you do yourself, and which did AI do?"
5. "Tell me about an AI result you didn't trust. How did you check it?"
6. "How did you get up to speed on a tool you'd never used before?"
7. "If a teammate wanted to repeat what you did with AI on that project, what would you tell them?"
8. "What's one thing AI still does badly in your line of work?"

For questions that cover the rest of the role, see our list of the 20 best interview questions to ask candidates.

Frequently asked questions about assessing AI skills

How do you test for AI skills in an interview?

Give candidates a short, practical task to complete with an AI tool, and watch what they check, change, and push back on. We recommend keeping it under an hour and paying candidates for their time. Then ask follow-up questions about specific times AI got something wrong, and score every candidate on the same AI skills your job description lists, using one shared scale.

What interview questions show if a candidate can actually use AI?

Ask for specific stories about AI in their own work. Good options include "Tell me about the last time AI got something wrong. How did you catch it?" and "When did you decide not to use AI for something?" Candidates who use AI regularly answer with concrete details, like the task, what went wrong, and how they spotted it. More general answers can point to limited hands-on experience, so ask for a recent example.

How do you tell if someone is exaggerating their AI experience?

Watch for three warning signs: overly vague answers, long lists of AI tools with no results attached, or no stories about AI mistakes. Probe each one with a specific follow-up question, such as "What's one task you did with AI last week, step by step?" People with real AI experience can walk you through the details, including what went wrong and how they fixed it.

Hire AI-fluent experts who are ready to contribute

You now know how to spot real AI skills. PowerToFly helps you find more of the people who have them, with a community of 380K+ verified experts and 65K+ AI specialists across marketing, customer success, operations, and more.

Talk to PowerToFly about hiring AI-fluent experts who are ready to contribute from day one.