Teacher models

How a teacher model trains a student

A teacher model gives a richer signal than a correct label. The 2015 paper on model distillation calls this signal “soft targets”.

1. Prompts or other inputs go to the teacher. These inputs can be unlabeled data.
2. The teacher gives its outputs: a probability for each answer, or full text.
3. The student trains to agree with these outputs.

4 types of teacher model

Type	What acts as teacher	Example
Larger model	A different, stronger model	Gemma 2 2B and 9B (Google, 2024)
Group of models	Models on disjoint data, with a vote	PATE (2016)
Average of weights	A moving average of the student weights	Mean Teacher (2017)
Same model, more context	The student, with the ground-truth solution	On-Policy Self-Distillation (2026)

3 risks of a teacher model

The teacher sets what the student learns. Thus the teacher is part of the supply chain of the student.

• Traits can go to the student in data that looks clean. In a 2025 study, students trained on number sequences from a misaligned teacher gave misaligned responses “almost 10% of the time”. Controls stayed below 1%. See misalignment.
• A special backdoor can go to the student. In a 2025 study, a trigger made for distillation reached 63.3% attack success for jailbreak in the best student pair of Table 2. The rate was 20.7% in the lowest pair. See backdoor attack.
• The teacher can belong to a different party. Anthropic, a vendor, reports more than 16 million exchanges with Claude through about 24,000 fraudulent accounts (February 2026). See model extraction.

The two studies are controlled tests, not results from production systems.

In practice: external teacher vs self-teacher

Three 2026 papers on post training train without a stronger external teacher. UniSD says that external teachers can be “costly, inaccessible, or undesirable”.

Question	External teacher	Self-teacher
What must you have?	A stronger model, or access to one	Only the model that you train
What can it add?	Skills that the student does not have	Mostly what the model and its data hold
What is the primary risk?	Bias or private content from the teacher	Mistakes that “may be reinforced across updates”
Who must you trust?	The owner of the teacher	The source of the data that the teacher role uses

The row “What can it add?” is our opinion from these papers. No source measures it.

What the numbers show

• Teacher result, 2015. On MNIST, one net with two hidden layers of 800 units trained in two ways. With no regularization, it made 146 test errors. With the soft targets of a large net, it made 74. This is one image task.
• Self-teacher cost, 2026. On-Policy Self-Distillation was 8 to 12 times more token-efficient than GRPO, a reinforcement learning method. This was one experiment on Qwen3-4B.
• Self-teacher limit, 2026. The same authors write that the method “can match GRPO performance but not significantly outperform it”. They also found that it under-performed at 1.7B scale.
• Backdoor conditions, 2025. Six older backdoor methods left students below 6% attack success. The paper summarizes the new trigger at about 60%, and its Table 2 shows up to 68.6%. The study assumes an attacker who anticipates the distillation dataset.

What a teacher model does not give

A teacher model does not make a student robust. Carlini and Wagner wrote in 2016 that defensive distillation “is not secure” against targeted misclassification attacks.

3 checks for a model that had a teacher

1. Get the name of the teacher model from the supplier.
2. Record this name in the model provenance data or the AI bill of materials.
3. Test the student for backdoors and misalignment. A clean test of the teacher data does not show a clean student.

“No external teacher” is a claim about one training step. Other models or data can have shaped the base model.

Where definitions disagree

Teacher. The 2015 paper does not use the word. It writes “cumbersome model” and “small model”. Subsequent papers use “teacher” for all 4 types in the table above. Only the first type is a different, stronger model.

External. Papers do not agree on what “no external teacher” excludes:

• On-Policy Self-Distillation (January 2026) gives “No External Teacher” as a property. Its teacher role receives the ground-truth solution.
• On-Policy Self-Training (October 2026) excludes more. It starts from “coarse, high-level topic descriptions”, “without access to any human-curated prompts or answers”.
• UniSD (May 2026) writes “stronger external teachers”. These words name only a stronger model.

Questions and answers

What is a teacher model in machine learning?

A teacher model is a trained model that gives the training signal for a student model. The student learns from the outputs of the teacher. These outputs can be a probability for each answer, or text. The usual goal is a smaller student that costs less to operate.

What does "no external teacher models" mean?

The phrase means that no different, stronger model supervises the training. The model learns from its own outputs, usually through self-distillation. Check the data too. In On-Policy Self-Distillation, the teacher role receives the ground-truth solution.

Can a backdoor go from a teacher model to a student?

Yes, if the attacker makes it for this path. In a 2025 paper, six older backdoor methods left students below 6% attack success. A trigger made for distillation reached about 60% in the summary of the paper, and 68.6% in one test of its Table 2. The attacker had to anticipate the distillation dataset.