Interviews are judged on impressions nobody is required to evidence

Careers advice about interviews is overwhelmingly about impression management: rapport, energy, body language, the handshake, the story you tell about yourself. It is not bad advice, exactly. It is a rational response to a selection method that rewards those things, and the reason it feels unsatisfying is that the method is a poor one.

The method

The dominant hiring instrument is the unstructured interview: a conversation, different for each candidate, in which panellists ask what occurs to them and form an overall impression afterwards.

Its weakness is well established and rarely acted on. Structured interviewing — the same predetermined questions for every candidate, in the same order, scored against criteria fixed in advance — predicts subsequent job performance more reliably. The unstructured version substitutes the panel’s confidence in its own judgement for evidence of that judgement’s accuracy, and those two things are not correlated in the way the panel experiences them as being.

This has a specific consequence for anyone being interviewed. When a process is weakly predictive, outcomes carry a large random component. A rejection from a weakly predictive process is not a measurement of the candidate. It is closer to a coin weighted by things nobody wrote down.

The five reasons, honestly stated

Careers pieces usually list the candidate’s failings. A more accurate list distributes the causes:

1. The criteria were not the advertised criteria. Panels frequently converge on an attribute that never appeared in the person specification, and nobody records that this happened.
2. Somebody already had a preferred candidate. Not necessarily corruptly; often an internal applicant whose performance is already known, against whom every external is being implicitly compared.
3. The panel was assessing fit, and fit is unfalsifiable. It functions as a container for impressions that cannot be stated as criteria, which is also how it becomes a route for bias.
4. Order and contrast effects. Being interviewed after a strong candidate is materially worse than being interviewed after a weak one, and neither candidate can influence the schedule.
5. The answer genuinely was not good enough. This happens too, and it is the only item on the list you can act on, which is why advice concentrates on it.

Notice that four of the five are properties of the process rather than of the person, and that a candidate receives no information distinguishing which one applied.

The same weakness, from the other side

The strangest illustration of how unaccountable interview judgement is comes from the panel’s side of the table, not the candidate’s.

In January 2014 Thomas Docherty, a professor at the University of Warwick, was suspended. The allegations against him, as reported, included inappropriate sighing, making ironic comments, and projecting negative body language towards candidates during interviews for a position in his department. The conduct was characterised as showing disrespect to those candidates.

He was kept away from campus for almost nine months, during which he was prevented from attending a conference held there. The allegations were then dismissed and he was cleared, and the Times Higher Education reported in October 2014 that he was “deeply moved” by the support he had received. The Council for the Defence of British Universities covered the case under the title The man who sighed too much.

Read the two situations together. A candidate is assessed on impressions the panel is not required to evidence. A panellist was then assessed, at enormous cost to him, on impressions of his demeanour that also required no evidence beyond the impression itself. The same absence of a written standard produced both.

What follows for a candidate

Two things, and neither is inspiring.

Reduce variance instead of trying to win. If the process has a large random component, the way to improve your outcome is more attempts with adequate preparation, not fewer attempts with perfect preparation. Prepare specific evidence against the criteria in the advertisement, because those are the only criteria you have been shown.

Do not treat a rejection as data about yourself. It is data about one panel on one afternoon, filtered through a method that does not reliably distinguish the candidate who will do the job best. Reading it as a verdict is both painful and, statistically, unjustified.

The uncomfortable part of the Warwick case is what it shows about the institutional appetite for fixing any of this. A university that suspended a professor for nine months over his conduct in interviews was, in the same period, using interviews as its selection method.