I run a Claude-backed tutor for our data structures courses, and that tutor's main job is to not give students the answer. At some point I realised I had built a lot of machinery on an assumption I had never checked: that withholding the answer is good for learning. So I went and read everything I could find. That is the first half of this talk, and it is a literature review, not a results talk.
The second half is what happened when I tried to make a model actually do it, and measured whether it did. Three things I would keep: withholding protects but rarely accelerates, so budget about 0.2 SD rather than two sigma; the decision you care about belongs in ordinary code instead of a prompt, because prompt-only withholding leaks; and every rejection should state its reason, or a score never becomes a diagnosis. I built the system first and read the literature afterwards, which is the honest order rather than the tidy one.
What is in it
- What "Socratic" means. A countable behaviour, an active ingredient, and the effect size you should actually expect.
- How LLM tutors are built, and how they fail. Four method families, what the vendors ship, and failures at both ends of the assistance dilemma.
- Does it help students learn? Guardrails prevent harm. Gains come from structure. And students skip the help they most need.
- Computing education. The largest guardrailed deployments anywhere. All of them leak, and upper-level courses are barely studied.
- Building one that actually withholds. An eight-rung hint ladder, a trust boundary outside the model, and how you measure a refusal.
- The over-help ladder. What a capable model does when you tell it not to help, and keep checking.
- Live demo. I try to extract a solution from my own tutor. Second screen shows the machinery.
- What we still cannot say. The criticisms I cannot argue with, the open problems, and the study I do not know how to design alone.
The scope, stated plainly: the second half is an architecture and a calibration method, checked against safety gates on about two dozen scripted turns. No human subjects, no student data. It is not evidence that a single student learned a single thing.
Presenting it yourself
The slides are plain HTML with Reveal.js sitting next to them, so the folder works with no network. Arrow keys or space to advance, S for the speaker window (timer and next slide; the notes pane is empty in this copy), F for full screen, O for the slide overview, B to blank the screen, ? for help. The published copy carries no speaker notes. Feel free to borrow any of it.