Tao-Rui Guan

The law of AI knowledge distillation

Knowledge distillation—the practice of training a compact student model on the outputs of a larger teacher model—has emerged as the fastest‐growing technique for replicating and disseminating frontier artificial intelligence (AI) capabilities. Because distillers can achieve near‐parity with incumbents at a frac...