Cross-Algorithm Steganalysis via Dual-Domain Feature Fusion: A Hybrid Deep Learning Approach for Payload Detection

Abstract

Stegware refers to malware payloads concealed within benign multimedia files that exploit weaknesses in traditional detection systems. This paper presents a hybrid deep learning framework called Hybrid StegNetA, designed for payload centric steganalysis with enhanced cross algorithm generalization. Five baseline architectures including CNN, RNN, ResNet, GAN, and Autoencoder were comparatively evaluated on stego images generated via LSB Matching, DCT/DWT, and Spread Spectrum techniques. The proposed model integrates residual learning and frequency domain encoding to isolate embedding noise from semantic content. Experimental results on a balanced CIFAR based dataset demonstrate improved stability, achieving consistent AUC ranging from 0.50 to 0.54 and minimal cross family variance of 0.0012, significantly outperforming ResNet which exhibited a cross family variance of 0.0028. The model maintains robust detection capability across four distinct steganographic families: LSB1, LSB3, Pixel Pair Matching, and Parity encoding. These results confirm that dual domain feature fusion enhances algorithm invariant payload detection capability and establishes a foundation for generalized stegware interception in dynamic threat environments.