AI Sourcing Supervision: Writing the Brief and Auditing the Shortlist

AI sourcing supervision is the day-to-day craft of getting usable candidates out of a tool that searches faster than you can and understands the role less well than you do. The useful mental model is a capable junior teammate on their first week: fast, tireless, literal, and completely dependent on the brief you hand them. Supervise it like that and the output becomes workable. Treat it as a search box and you get volume with no judgement attached.

This article is about the practitioner's controls: the brief, the calibration loop, and the shortlist audit you run before any name reaches a hiring manager. It is a practice, not a job title.

Why the junior-teammate framing is the right one

A sourcing agent will do exactly what you asked, including the parts you did not mean. Ask for candidates with five years in logistics and it will return people whose profiles say logistics for five years, which is not the same population as people who can run a distribution site. A new team member makes the same mistake in their first fortnight, and you fix it the same way: show them two profiles you would shortlist, two you would not, and say why.

The difference is that a human junior generalises from the correction and an agent only generalises as far as your next brief carries it. That puts the learning in your hands rather than the tool's, which is why the brief and the feedback loop do more work here than the choice of platform.

It also sets the accountability correctly. The agent does not own a sourcing decision and cannot explain one. You do, and you will be asked to.

The sourcing brief that makes output usable

A usable brief is longer than a search string and shorter than a job description. Six parts, written once per role and refined as you go.

• The outcome - Include: what this person will have delivered by month six; Why: lets the agent weigh adjacent experience instead of matching titles
• Disqualifiers, in writing - Include: the two or three things that make a profile an automatic no; Why: the only instruction that reliably shrinks a bad list
• Acceptable adjacent backgrounds - Include: the industries and role titles that count as near enough; Why: stops the agent reading your spec as a title filter
• Two positive and two negative examples - Include: real profiles you have judged, with one line on each; Why: examples carry more signal than any amount of criteria prose
• Geography and work rights, stated plainly - Include: the actual commute or hybrid requirement, and visa constraints; Why: the most common source of false positives in Australian searches
• What you do not want it to infer - Include: fields it must not use as proxies, such as school, suburb or employer prestige; Why: a fairness control and an accuracy control at the same time

The last line is the one most recruiters leave out. An agent asked for "high-calibre candidates" will quietly build its own definition out of whatever correlates in its data, and brand-name employers and sandstone universities correlate with plenty of things that have nothing to do with the job. Naming the proxies you reject is cheaper than discovering them in an audit, and it is the same discipline behind our note on auditing recruitment AI for bias.

The calibration loop, run weekly

Calibration is where supervision actually happens. Four steps, about 20 minutes a week per active search.

Step one: rate a sample, not the whole list

Take ten profiles from the agent's output and mark each yes, no or maybe against the brief. Ten is enough to see a pattern and small enough that you will actually do it. Rating the full list is how calibration gets skipped in week two.

Step two: write the reason, not just the verdict

"No, warehouse experience but no team leadership" is a usable correction. "No" on its own teaches nothing and leaves you unable to reconstruct your own reasoning a fortnight later. The reasons are what you feed back into the brief.

Step three: change one thing

Adjust a single element of the brief per cycle, usually a disqualifier or an adjacent-background rule. Change three at once and you cannot tell which one moved the output.

Step four: keep the record

A short log of what you changed and what happened next is what turns this from fiddling into calibration. It is also the artefact that answers "why was this candidate surfaced and that one was not" months later, when nobody remembers.

Where the tool itself is the constraint rather than the brief, that shows up in the log within three cycles as output that does not move no matter what you change. Our comparison of free candidate sourcing tools is a reasonable place to start if you reach that point and want to test an alternative before committing budget.

The shortlist audit, before any name goes to a hiring manager

This is the checkpoint that makes the process defensible. Four checks, run on the list rather than on individual candidates.

• Fit against brief - Ask: for each name, which part of the brief does this person satisfy?; Red flag: a candidate you cannot map to a line of the brief
• False positives - Ask: how many names look right on paper and fail on a single disqualifier?; Red flag: more than two or three in ten, which means a disqualifier is not being read
• Missed segments - Ask: which population you would expect is absent from this list?; Red flag: no career changers, no part-time, no candidates from the adjacent industry you named
• Duplicate records - Ask: how many of these names are the same person twice?; Red flag: duplicates at all, because they inflate the list and distort your own sense of pipeline

Missed segments is the check almost nobody runs and the one that catches the expensive failures. A list of fifteen plausible names with no women in it, or no candidates over 50, or everyone from the same three employers, is telling you something about the brief rather than about the market. Looking for what is absent is harder than reviewing what is present, which is why it needs to be a written step.

Where the audit raises a question about a candidate's actual capability rather than their paper fit, that belongs in a structured evidence step rather than in more sourcing. A short AI assessment placed before the hiring manager's first conversation answers it better than another pass over profiles.

Human-in-the-loop is a record, not a reassurance

Saying a human reviewed the shortlist means nothing without evidence of what the human did. The record that holds up is dull and specific: the brief version in use, the calibration log, the audit you ran, and who signed off on the list. That is the material that answers a candidate complaint or an internal query six months later, and its structure sits inside our broader AI hiring compliance framework.

Keep the decision with the person. An agent can rank, cluster and summarise. It should not be the thing that removes a candidate from consideration without a named human having applied a stated criterion, because that is the point at which nobody can explain the outcome.

Frequently Asked Questions

How long before an agent's output is worth using?

Two to three calibration cycles for a role you have hired before, longer for something unusual. If the output has not improved after four cycles of single changes, the brief is probably not the problem and the tool may not fit the search.

Do I still need to know how to source manually?

Yes, and more than before. Judging whether a list is good requires knowing what the market actually holds, and that knowledge comes from having searched it yourself. The skill shifts from running the search to evaluating the output.

Who is accountable if the agent surfaces a discriminatory shortlist?

The employer, and in practice the recruiter who presented the list. A vendor contract does not transfer that, which is why the audit and the log matter more than the tool's own fairness claims.

Should candidates be told an AI tool was used to source them?

Our position is that transparency about the process costs nothing and protects you later. What your specific obligations are depends on your jurisdiction and the data involved, so take that question to your own privacy adviser rather than to a blog post.

Can I supervise more than one agent at a time?

In practice two or three active searches is the limit before the calibration loop gets skipped, and a skipped loop is worse than no agent because the output still looks authoritative. Supervision capacity is the real constraint, not licence count.

AI Sourcing Supervision Is a Weekly Practice

The temptation with a sourcing agent is to set it up well and then stop looking, because the output keeps arriving and looks reasonable. That is the failure mode. The brief, the weekly loop and the audit are the three places a recruiter adds judgement the tool does not have. Pick one active search and run the shortlist audit on it this week, starting with the missed-segments question.