Knowledge distillation and model extraction in AI

Knowledge distillation and model extraction in artificial intelligence can transfer selected behaviour or functionality from one model to another. They may reduce deployment costs and diffuse capabilities across firms or jurisdictions, but they are not the same technique and do not eliminate the need for compute, data, engineering or evaluation. The methods have no inherent state nexus and remain context unless a state directs, regulates or strategically exploits a specific campaign.

Distinct techniques

Knowledge distillation trains a student model to reproduce useful outputs or representations from a teacher, often to create a smaller or cheaper model. Synthetic-data training uses generated examples or labels, whether or not a student is smaller. Query-based model extraction uses repeated interactions to reproduce functionality, parameters or decision boundaries to a measured fidelity. Weight acquisition obtains the parameters themselves and is a different act again.

Technical access, contractual permission, intellectual-property rights and export authorisation are separate questions. Authorised compression, open research and licence-compliant distillation should not be conflated with adversarial extraction or acquisition of protected data. Accuracy, fidelity and safety properties also require separate evaluation.

Cost and capability claims

Distillation reallocates compute rather than replacing it. The teacher must exist, outputs must be produced or collected, and the student still requires training, inference, infrastructure and testing. The strategic effect depends on the capability retained, total resources used, time, detectability and ability to deploy.

The DeepSeek-V3 technical report estimated 2.788 million H800 GPU hours and approximately US$5.576 million for its final pre-training run under the report's stated price assumption. The estimate excluded prior research, ablation experiments, data work, infrastructure and other costs. It was not a total company cost or an estimate for developing DeepSeek-R1.

DeepSeek's R1 report describes smaller models distilled from reported teacher outputs. That supports a documented example of authorised capability transfer within the release. It does not prove that DeepSeek used any named rival model as the teacher for its frontier model. Such a claim would require evidence of the model, access path, outputs, method and legal basis.

Governance and statecraft relevance

Developers and cloud platforms can use identity controls, rate limits, monitoring, contractual terms and cybersecurity measures to manage output access. States may regulate chips, cloud services, model weights, technical data, software, end users or end uses under specific authorities. The controlled object and operative rule must be named rather than assuming that an export regime covers or excludes distillation as a category.

In May 2025 Commerce announced categorical non-enforcement of the Artificial Intelligence Diffusion Rule and initiated planned rescission, but it did not complete formal removal. As at 29 July 2026 the rule remained codified while not operating in practice under that enforcement policy. The legal split is treated at United States AI diffusion rule, non-enforcement and destination-specific controls (2025-present). It does not create a generally enforced rule for model outputs, and later semiconductor licensing changes likewise do not create one.

Anthropic's February 2026 account of alleged coordinated extraction campaigns is a private company's detection finding. Its account and interaction figures require attribution and independent corroboration before being treated as established fact. A commercial or unauthorised campaign also does not become statecraft without evidence of state direction, sponsorship or strategic use.

See also

Artificial-intelligence export controls · United States AI diffusion rule, non-enforcement and destination-specific controls (2025-present) · Model weights as controlled technology · Hyperscale cloud infrastructure · Technology diffusion

Sources

1. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, Distilling the Knowledge in a Neural Network (2015).
2. Cristian Buciluǎ, Rich Caruana and Alexandru Niculescu-Mizil, Model Compression, Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2006).
3. Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter and Thomas Ristenpart, "Stealing Machine Learning Models via Prediction APIs", 25th USENIX Security Symposium (2016).
4. Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin and Nicolas Papernot, "High Accuracy and High Fidelity Extraction of Neural Networks", 29th USENIX Security Symposium (2020).
5. DeepSeek-AI, DeepSeek-V3 Technical Report (2024, revised 2025).
6. DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in Large Language Models via Reinforcement Learning (2025).
7. DeepSeek, "DeepSeek-R1 Release", 20 January 2025.
8. Anthropic, "Detecting and Preventing Distillation Attacks", 23 February 2026.
9. United States Department of Commerce, Bureau of Industry and Security, "Department of Commerce Announces Rescission of Biden-Era Artificial Intelligence Diffusion Rule and Strengthens Chip-Related Export Controls", 13 May 2025.
10. United States Department of Commerce, Bureau of Industry and Security, Export Administration Regulations, Part 744: Control Policy, End-User and End-Use Based, current edition, checked 29 July 2026.
11. United States Department of Commerce, Bureau of Industry and Security, "Department of Commerce Revises License Review Policy for Semiconductors Exported to China", 13 January 2026.
12. United States Government Accountability Office, Applicability of the Congressional Review Act to the Rescission of the Artificial Intelligence Diffusion Rule, B-337935, 12 May 2026.