Paper review: Relational Knowledge Distillation(CVPR 2019)

MotivationRecent SOTA models require a high cost of computation and memory in inference. One promising direction for mitigating this computational burden is to transfer knowledge. Previous approches can be expressed as a from of training the student to mimic output activations of individual data examples represented by the teacher. So authors introduce relational knowledge distillation(RKD), tha..

2024.04.04