The Five Questions Academic Analytics Should Answer

A dean opens the analytics module before a semester review. There are twenty-eight charts. Attendance by department, by day of the week, by hour. A heat map of submission times. A pie chart of pass percentages by programme, with a filter for year of study. Every figure is accurate, the module cost a considerable amount, and nothing on that screen has ever caused a decision to be made differently.

The problem is not the data. It is that nobody wrote the questions down first. A dashboard is a set of answers, and a set of answers assembled without questions is a wall. Five questions justify the effort of building an analytics capability in an institution. Anything on the screen that does not serve one of them is decoration, and decoration is expensive because it competes for attention with the charts that matter.

One. Who needs help right now, and what kind

The output is a short, ranked list of named students, each with the reason attached and a recommended action. Not a score. A score on its own cannot be argued with by the tutor who knows the student, cannot be corrected when it is wrong, and cannot be explained to the student if they ask — which they are entitled to do.

The failure mode is a list nobody owns. Every flagged student needs a named person and a next action with a date, otherwise the list regenerates each week with the same names on it and the institution slowly learns to ignore it. Getting this right is mostly an operational design problem rather than a modelling one, which is the uncomfortable finding behind most early-warning programmes that quietly stop being used.

Two. Which parts of the curriculum break for everyone

Some units lose marks year after year, across sections, across instructors, across cohorts. That is not a teaching quality signal; it is a curriculum design signal, and it usually means the unit assumes something the students have not been taught, is scheduled at the wrong point in the sequence, or is assessed in a way that tests something other than what was taught.

This is the most actionable question on the list and the most frequently ignored, because acting on it means changing the syllabus, which is slow and political. It also pays back the furthest: fixing a unit fixes it for every cohort that follows, while helping an individual student helps one student.

The failure mode here is severe. The moment this view is used to rank faculty, the data stops being honest — assessment gets easier, marking gets generous, and the signal disappears within two semesters. Analysing the curriculum and evaluating the teacher have to be visibly separate systems, and the separation has to be stated rather than assumed.

Three. Are we where we said we would be

Syllabus coverage against plan, and assessment completed against plan. It is the least interesting question intellectually and often the most useful in practice, because it is the only one where the remedy is obvious and the timing still allows it. Two weeks behind in week six is recoverable with a schedule adjustment. The same two weeks discovered in week thirteen is a decision about what to cut.

The requirement is simply that the data arrives faster than the decision cycle it serves. A coverage report published at the end of term is a historical document. The same report available weekly is an instrument, and it is the single most reliable thing academic leadership can ask for from a system.

Four. Did what we did last time change anything

Institutions run interventions constantly: remedial sessions, mentoring, attendance drives, extra tutorials, revised assessment schedules. Very few record what was done, to whom, and when, in a form that allows anyone to re-measure afterwards. Without that record, the institution cannot tell an effective intervention from a popular one, and it will keep funding whichever is easier to organise.

The discipline is unglamorous: log the intervention, define what should move, re-measure the same thing at a set interval, and publish the result even when it is nothing. Expect a reasonable share to show no effect. That is not a failure of the programme, it is the reason for measuring, and an institution that cannot tolerate a null result will end up with analytics that only ever confirm decisions already taken.

Five. Are students becoming ready for what comes next

Pass rates describe performance inside the institution’s own system. They say relatively little about whether a graduating cohort can do the things the next stage requires, whether that is employment, a professional examination or postgraduate study. Answering this means holding a picture of what those destinations require and comparing it against what students can actually demonstrate, by cohort and by year, early enough that the final year is not the first time anyone looks.

This is the question that connects academic analytics to skills-gap analysis, and it is the one that tends to be politically hardest, because the honest answer in the first year is usually uncomfortable. It is also the only question on this list that speaks directly to what students and their families believe they are buying.

What a dashboard should not do

• Label a person. A flag is a prompt for a conversation, not a property of a student, and the language on the screen should make that obvious.
• Show a score without the evidence one click away. Anything that cannot be interrogated will eventually be either over-trusted or ignored, and both are bad.
• Report on individual staff in a view designed for curriculum analysis. Mixing those two purposes destroys the data.
• Refresh more often than anyone can act. A daily chart serving a monthly decision produces noise and the appearance of volatility where none exists.
• Grow without pruning. Every chart added without removing one dilutes the attention available for the rest.

Start with three

Most institutions should build the first, third and fourth questions before touching the others. They need the least modelling, they produce decisions immediately, and they establish the habit of connecting a number to an owner and an action. The fifth question is the most valuable in the long run and the one most likely to stall a programme if it is attempted first, because it requires vocabulary and evidence that most institutions have not yet built.

The test for any chart on the screen is the same throughout: name the decision it informs and the person who makes it. Charts that survive that question tend to be few, plain and heavily used. That is what an analytics layer should be aiming at — not comprehensiveness, which is easy, but the much harder property of being consulted before a decision rather than quoted after one.