Knowledge Distillation (KD) is a technique for transferring knowledge in which a smaller student model is trained using the information generated by a larger or more capable teacher model. The main aim is to create models that need less memory and computational power while at the same time preserving as much of the teacher’s predictive power as possible. Starting from the early version of the teacher-student approach that relied on softened output distributions, KD has evolved into a wide range of methods that transfer responses, intermediate representations, relationships, attention information, or task-specific capabilities. More recent research has applied these ideas to Transformer encoders and Large Language Models (LLMs), since the sequential and generative nature of language modelling brings with it challenges which do not exist in the case of traditional classification. This paper looks at the basic principles of KD, its main methodological categories, some typical modern techniques, its use with Transformer and LLM models, and how it interacts with quantization. A comparison based on the existing literature is made in order to see how various approaches balance model size, computational efficiency, and the amount of capability retained. The review also points out some unresolved problems, such as a mismatch between the capacity of the teacher and student models, differences in architecture, knowledge loss, and the preservation of reasoning abilities in smaller language models.

Pranav Shaji, Ms. Riya K Prakash, Prof. Dr. L C Manikandan, "Knowledge Distillation: Evolution, Methodologies, and Emerging Applications in Large Language Models" International Journal of Scientific Research and Engineering Development, V9(5): Page(1727-1734) September - October 2026. ISSN: 2581-7175. www.ijsred.com. Published by Scientific and Academic Research Publishing.