The Test That Knows What You Know
In this article
A bathroom scale tells you that you weigh 82 kilograms. It cannot tell you whether the 82 is muscle, a winter of dinners, or water. A doctor looking at the same person asks a few questions, runs a test or two, and ends up knowing the thing the scale never will: what to do next.
Most exam practice is the bathroom scale. A candidate works through a set of questions and receives a percentage. The percentage is true and nearly useless: it arrives after the work is finished and folds several situations into one number. The last article here argued that the grind is a design decision. This one is about the design: a running underneath every session, and that uses it to choose what the candidate sees next.
Two ideas that belong together
The competency assessment is the doctor's half. Every examination authority publishes what a candidate must be able to do, split into topics with published weights. Each question here is tied to those topics, so each answer is evidence about one specific competency rather than a point added to a total. After a few dozen answers there is a working picture: strong here, shaky there, untouched over there.
Adaptive learning is the second half, and it only works if the picture exists. Compare a paper map with a . The map shows the whole country and leaves the driving to you. The satnav knows where you are, so it can say turn left. Take away the position and the satnav is a screen of road names. Adaptive learning without an assessment is that screen: movement with no idea of where the candidate stands.
Neither half is enough alone. An assessment with no adaptive learning is a report nobody acts on. Adaptive learning with no assessment is a shuffle. Together they make a loop: the assessment finds where the candidate stands, the next question moves that position, and the answer sharpens the picture.
What one answer can say
A candidate who answers quickly and correctly has the material. One who answers correctly after a long struggle has it, but pays in minutes, which on a timed examination is real money. One who answers correctly by guessing has neither the material nor any warning that it is missing. One who answers wrong at least knows it, and can act.
A percentage turns those four facts into one number. The guess is the dangerous one, because a guess that lands looks exactly like knowledge, like flipping a switch in the dark and happening to light the right room. So after each answer the candidate is asked whether it was a guess. Nothing in the score changes when the flag is set, which is the only reason the flag is worth collecting. A topic full of lucky guesses stops reading as settled.
The question that comes next
A fixed set of questions is an off-the-rack suit. It has to fit everyone who buys it, so it fits nobody exactly: too loose for the strongest candidate, too tight for the weakest.
The adaptive layer is the tailor, and what it makes is a : cut to one candidate's measurements, adjusted after every fitting. Each question carries a difficulty, and a floor for the chance of answering correctly without knowing the material. The pattern of a candidate's answers produces a current estimate of what that candidate knows, and the next question is drawn to be informative at that level. The framework is called , set out by Lord in 1980 and made practical on a computer by Bock and Mislevy and by Weiss in 1982.
The saving has been measured. A review of computerised testing in the health professions reports that fitting questions to the candidate cuts the number of questions and the testing time by half or more, with precision held steady. One caution belongs beside that. A study that shortened a physics inventory this way cut testing time to between 50 and 63 per cent of the original, and paid for it with 5 to 10 per cent of the measurement precision. A saving published with its cost is a saving a reader can check.
Answering is the study
Under all of this sits one choice: the candidate answers questions rather than rereading the chapter.
Re-reading is watching someone else lift weights. It feels like effort and builds little. Answering is lifting. Roediger and Karpicke measured how much of a passage students kept a week later: 61 per cent for students who had been tested on it, against 40 per cent for students who read it again. , the act of pulling the answer out, is what builds a lasting trace, and retrieval is what an answer is. Spacing helps further: a synthesis of 839 assessments across 317 experiments found that material spread over time is held longer than material crammed, so the scheduler here computes the gaps.
Teaching at the moment of the error
A wrong answer is the most useful event in a session and the easiest to waste. Here a wrong answer comes back with the principle the question turns on, the reason the right answer is right, and a short paragraph on why the option the candidate chose fails. Timing matters as much as content. A correction read while the question is on the screen lands on a candidate who remembers why the wrong option looked right. Read a week later, it lands on one who has forgotten. Hattie and Timperley's feedback research agrees.
The vocabulary works the same way. A licence is a language examination first. . Adjusted cost base. . A candidate meeting those terms for the first time pays twice: once for the meaning, again for the time spent leaving the question to look it up. So a new term opens in place, inside the question, and the candidate keeps the run. A definition that would give the answer away is withheld.
What you gain
You stop guessing about yourself. After every session you can see which topics are solid, which are shaky and which you have not touched, and the picture comes from your answers rather than your mood.
You stop spending evenings on what you already know. The strong topics go quiet on their own, and the practice thickens where you are thin.
You learn the language of the field where it is used, so the vocabulary stops being a second examination. A mistake gets fixed once, while the reasoning is still warm, instead of being met again on the day. And you walk into the examination knowing where you stand against its published weights, rather than hoping.
None of this removes the hours. It makes each one count for you in particular.
Why this is the right thing to do
Because the hours are not the problem. A professional examination asks for real work, and no method removes it. Anyone who promises a pass is promising something that belongs to somebody else's measurement. The examinations are hard, and they should be. What can be removed is the share of the hours that buys nothing: the chapter read twice, the questions too easy to teach anything, the percentage that arrives when nothing can be changed.
Because most candidates study around a job, often in a second language, for an examination that decides whether a career starts. For that person an hour is not an abstraction. A method that spends it where the candidate is weakest respects the candidate. A method that spends it on whatever came next in the book does not.
And because a claim about learning should be checkable. The engine is tested in simulation against synthetic candidates whose ability is known, with the bias and error printed beside every result. Question difficulties are earned from recorded answers rather than asserted, and a readiness figure is shown only once the questions behind it have earned them.
This is how we built the Compleat Exams platform. We did not start with a question bank and bolt a timer on. We started with the assessment, built the adaptive learning on top of it, and put the teaching inside the question. A platform built the other way round can only hand back a percentage. There is a better way, and this is it.
Every candidate still owes the hours. What the test knows decides what they buy.
Sources
- Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum.
- Bock, R. D., and Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431 to 444.
- Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473 to 492.
- The impacts of computer adaptive testing from a variety of perspectives. Journal of Educational Evaluation for Health Professions, 2017.
- Optimizing the length of computerized adaptive testing for the Force Concept Inventory. Physical Review Physics Education Research, 17, 010115 (2021).
- Roediger, H. L., and Karpicke, J. D. (2006). Test-enhanced learning. Psychological Science, 17(3), 249 to 255.
- Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall tasks. Psychological Bulletin, 132(3), 354 to 380.
- Hattie, J., and Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81 to 112.