An Empirical Evaluation of Fisher-Guided Adaptive Noise for Flat Minima in Few-Shot Class-Incremental Learning

Abstract

Flat minima search via noise injection during base training is a core component of F2M, a leading method for few-shot class-incremental learning (FSCIL). F2M injects isotropic Gaussian noise (constant σ = b for all parameters) into model parameters to encourage convergence to flat loss regions that are robust to incremental updates. I investigate whether replacing isotropic noise with Fisher Information-guided adaptive noise—scaling perturbation inversely with parameter importance—improves FSCIL performance. I term this extension A-F2M-G and evaluate it on CIFAR-100 with ResNet-18 across two optimizers and five noise configurations. My findings are negative but informative. With Adam, noise injection of any kind is redundant: all four configurations fall within 0.19 percentage points of each other. With SGD, noise injection provides a small, suggestive gain of +0.93 pp (p=0.0496), but adaptive noise offers no advantage over uniform perturbation (−0.09 pp, p=0.50). I trace this null result to a structural property of the Fisher Information distribution: a power-law skew spanning about 6 orders of magnitude (from approximately 10⁻⁹ at the median to 10⁻² at the maximum) causes min-max normalization to collapse adaptive noise to effectively isotropic noise. Additionally, in my experimental setup, SGD-trained models outperformed Adam-trained models by approximately 5 percentage points in aAcc.