Semi-Supervised Contrastive Learning Based Knowledge Distillation

Semi-supervised knowledge distillation methods simultaneously distill labeled and unlabeled samples, effectively reducing reliance on labeled data while maintaining distillation performance. To address the problems where, in direct semi-supervised knowledge distillation, the teacher network is prone to generating erroneous knowledge for unlabeled samples, leading to cognitive bias in the student network, we propose Semi-supervised Contrastive Learning based Knowledge Distillation (SCLKD). This method exploits pseudo-label knowledge and class contrastive knowledge from unlabeled data. First, select high-confidence predictions as pseudo-labels from the teacher network for unlabeled data by threshold filtering, which are then used as real labels to guide the student training. Then, based on the pseudo-labels and soft predictions from the teacher model, we use contrastive learning methods with multi-positive samples and soft targets to further alleviate the confirmation bias. Experimental results on CIFAR-100, STL-10 and ImageNet datasets show that SCLKD has a performance improvement of up to 6.97%. In addition, SCLKD surpasses the results of full-supervised training methods using only 20% of the training labels, proving that it can reduce the model's dependence on labeled data while avoiding performance degradation caused by confirmation bias.