Gradient-balancing based transfer learning for class-imbalanced medical image classification

Abstract
In the process of utilizing Convolutional Neural Networks (CNNs) for the medical image classification task, a widely adopted solution to address suboptimal classification performance resulting from insufficient data in the target domain is transfer learning, particularly through fine-tuning. However, class imbalance can cause the gradients and parameter weights of fine-tuned models to disproportionately favor head samples, neglecting the optimal fine-tuning of tail classes, thereby reducing overall classification accuracy. Furthermore, when the volume of data in the target domain is insufficient, it leads to redundant parameters in deep models, resulting in overfitting and impacting the ultimate fine-tuning effect. To address these challenges, this paper proposes a Gradient-Balancing-based parameter Fine-tuning (GBF) for medical image classification. First, a sparse fine-tuning based on gradient attribution is used to identify parameters highly relevant to the target domain classification, which are then selectively fine-tuned. Next, during fine-tuning, the gradient components of the head samples are projected onto those of the tail samples to assist the update of the tail sample and optimize the update direction of the model. Finally, the loss components of the head and tail samples are recalculated and re-weighted to guide the update of model parameters. Experiments utilizing three mainstream CNN models and fifteen medical datasets demonstrate that the proposed GBF achieves higher BACC values across most target datasets, with improvements ranging from 1% to 6% over the baseline and 1% to 3% over the SOTA methods. Conclusions: The proposed method ensures optimal fine-tuning performance while addressing the challenge of class imbalance in medical image classification tasks.