Measuring Training Effectiveness Beyond Completion Rates

The quarterly learning review opens with a slide showing near-total course completion and a satisfaction score close to the top of the scale. It is structurally the same slide as last quarter. Nobody in the room disputes it, and nobody in the room can say what is different in the work because of any of it. Twenty minutes later the conversation moves on to next year’s budget, which will be defended with the same slide.

Completion measures attendance. Satisfaction measures the room, the delivery and, more than anyone likes to admit, the catering. Both are worth knowing. Neither is evidence that anybody can do something they could not do before.

Why the easy metrics persist

Because they are free. A learning platform emits them by default, they are unambiguous, and they are defensible in the narrow sense that nobody can prove them wrong. There is also a quieter reason: measuring properly creates the risk of discovering that an expensive programme does nothing. A function that has never been asked for harder evidence has little incentive to generate evidence that could be used against it.

That incentive problem is worth naming out loud at the start, because it determines whether the measurement programme you design will survive its first bad result. If the answer to a failed programme is blame, you will get good numbers and no learning.

Three measures worth the effort

Capability change, against the same instrument

Assess the same skills before and after, on the same scale, with the same kind of evidence. The immediate post-course measure is the least interesting one, because it largely captures short-term recall and the effect of having just been told the answers. The measure that matters is taken six to ten weeks later, once the material has either been used or forgotten.

Two failure modes to design against. Do not let the post-assessment be drawn from the material that was taught verbatim, or you are measuring memory of the slides. And use assessment that requires demonstration rather than self-rating, since confidence moves after training whether or not capability does. That distinction is the entire argument for assessing skills rather than surveying them.

Transfer into the work itself

Capability that never appears in the work is a hobby. Transfer is harder to measure but rarely impossible, because most work leaves artefacts: code review comments, incident write-ups, client proposals, call recordings, design documents, tickets. Pick one artefact type, define two or three observable things that should look different, and sample them before and a quarter after.

Manager observation is the other route, and it works if — and only if — it is a short checklist of specific behaviours at a fixed interval, rather than an open question at appraisal time. “Has Priya used the new escalation protocol on a live incident?” produces data. “Has Priya improved?” produces politeness.

The sponsor’s number, agreed in advance

Before the programme runs, ask whoever is paying what should be different afterwards, and write the answer down. Time for a new hire to reach independent delivery. Rework rate on a class of tasks. Internal fill rate for a role family that currently goes to external hiring. The measure does not have to be perfect. It has to be chosen before the result is known, because a metric selected afterwards will always be the one that looks best.

This also changes the conversation with the business. A programme built against a sponsor’s stated number is a joint commitment, which is a considerably stronger position than presenting attendance figures to someone who did not ask for them. It is also how an upskilling programme earns a second year of funding.

What you can honestly claim

You will rarely get a control group, and you should stop pretending otherwise. The practical substitutes are staggered rollout, where the group trained in March is compared with the group waiting until June, and cohort comparison across similar teams. Both are imperfect and both are far better than nothing.

When the confound is unavoidable, say so. A report that reads “capability moved, and we cannot separate that from the reorganization that happened in the same quarter” is more useful, and considerably more credible, than a clean attribution nobody in the room believes. Overclaiming once costs more trust than three honest null results.

Measure fewer programmes, properly

The instinct to instrument everything produces a dashboard that measures nothing, because the effort per programme drops below the level at which any of it is reliable. Pick the two or three programmes that cost the most or carry the most strategic weight. Measure those seriously, with pre and post assessment, a transfer measure and a sponsor number. Everything else gets completion data and a light satisfaction check, which is an honest statement of how much you care.

The point is to be able to stop things

A measurement programme that has never ended a course is not a measurement programme. When something does not work, there are three explanations worth separating, and only one of them is about the training. The content may be wrong. The population may be wrong — people sent who had no use for it. Or the environment may block transfer: the manager does not permit the new practice, the tooling does not exist, the incentive still rewards the old behaviour.

Telling those three apart is arguably the highest-value thing a learning function does, because the third one is not a training problem at all and no amount of redesign will fix it. Finding it is worth more than another course. Everything else — the measures, the instruments, the honest reporting, the way it connects to the gaps the organization actually has — is in service of being able to say which of the three you are looking at.