Knowledge Distillation

AI model distillation, explained: the cheap way to inherit an expensive model’s brain

AI model distillation, also known as knowledge distillation, is a technique where a smaller "student" model learns from a larger "teacher" model. This process allows for the creation of cheaper, more efficient AI systems by inheriting the capabilities of expensive, pre-trained models. However, concerns exist regarding the potential loss of safety knowledge and the inheritance of biases during this distillation process.