Scorecards so interviewers rate the same thing

Two interviewers, same candidate, opposite conclusions, both sincere. It happens constantly and it is rarely about judgement. They assessed different things, because nobody said what to assess.

A scorecard is the cheapest fix available and most companies do not have one because writing it feels bureaucratic.

Generate it from the requirements, not from a template

A generic scorecard rates communication, technical skill and culture fit, which is another way of saying it rates nothing. What makes a scorecard useful is that it is about this role.

Using the Senior Designer requirements document, draft an interview
scorecard.

One row per hard requirement. For each:
  what we are assessing, in the words of the requirement
  a question that would actually reveal it
  what a weak, adequate and strong answer sounds like

Only include things this interview can actually assess. Do not add rows for
requirements a 45 minute conversation cannot test.

Do not include culture fit as a row.

Excluding "culture fit" is deliberate and worth defending. It is the row where unexamined preference gets a number and becomes evidence, and it is where two interviewers most reliably disagree without knowing why. If there is something specific you mean by it — works well without direction, argues with a client — write that as its own row with its own question, and it stops being a proxy.

The three-answer scale is the part that works

Describing what weak, adequate and strong sound like does more for consistency than any rating scale. It gives two interviewers the same yardstick without requiring them to have calibrated in advance.

It is also the hardest part to write, which is why generating a first draft is worth doing.

One scorecard per stage, not per role

Each stage should assess different things. If the portfolio review and the first conversation use the same scorecard, one of them is redundant.

Fill it during, not after

A scorecard completed from memory an hour later is a rationalisation. This is a process point rather than a product one, but it is where the value leaks.

What good looks like

Two interviewers rating the same candidate produce comparable notes, and where they disagree you can see which requirement they disagreed about.

The measure is your debriefs. If they are still about overall impressions, the scorecard is being filled in afterwards to justify a decision already made.