Self-Supervised Contrastive Regularization for Robust Speech Enhancement
Keywords:
Speech Enhancement, Contrastive Learning, Self-Supervised Learning, Deep Neural Networks, Audio ProcessingAbstract
The problem of robust speech enhancement remains a critical bottleneck in the deployment of voice-controlled interfaces, telecommunication systems, and hearing aid devices, particularly in highly non-stationary and reverberant environments. While fully supervised deep learning models have demonstrated remarkable success in mitigating additive background noise and reverberation, they frequently suffer from severe performance degradation when exposed to unseen acoustic conditions. This generalization gap stems from the models overfitting to the specific noise distributions and acoustic characteristics present in the training corpus. To address this fundamental limitation, we propose a novel framework that integrates self-supervised contrastive learning as a regularization mechanism within a standard supervised speech enhancement architecture. By simultaneously optimizing a primary waveform reconstruction objective and an auxiliary contrastive loss, the proposed model learns to extract noise-invariant, high-level phonetic representations. The contrastive regularization forces the neural network to map augmented views of the same clean speech signal distorted by vastly different environmental noises to nearby points in a shared latent space, while pushing apart representations of different speech utterances. Comprehensive evaluations across multiple benchmark datasets demonstrate that the integration of self-supervised contrastive regularization substantially improves objective metrics such as perceptual evaluation of speech quality and short-time objective intelligibility, particularly under zero-shot testing conditions involving unseen noise types and severe signal-to-noise ratios. The findings emphasize the potential of hybrid learning paradigms to yield highly robust, generalizable audio processing systems without requiring exponentially larger annotated datasets.
References
1. Xu, X., Han, C., Zhang, Y., Tu, W., & Yang, Y. (2024, April). Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 10486-10490). IEEE.
2. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.
3. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.
4. Li, A., Sun, Z., Hao, F., Li, X., & Zheng, C. (2025). Neural vocoders as speech enhancers. arXiv preprint arXiv:2501.13465.
5. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of ICML.
6. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.
7. Xu, X., Tu, W., Yang, Y., Li, J., & Zhang, Y. (2025). Interactive Target Positive and Negative Features Modeling for Monaural Speech Enhancement. IEEE Transactions on Audio, Speech and Language Processing, 33, 4856-4869.
8. Shan, T., Cappelloni, M. S., & Maddox, R. K. (2024). Subcortical responses to music and speech are alike while cortical responses diverge. Scientific Reports, 14(1), 789.
9. Wang, J., Zhao, R., Wei, W., Wang, Y., Yu, M., Zhou, J., ... & Xu, L. (2026, March). Comorag: A cognitive-inspired memory-organized rag for stateful long narrative reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 39, pp. 33557-33565).
10. Zhang, K., Shuang, K., Yang, X., Yao, X., & Guo, J. (2023, July). What is overlap knowledge in event argument extraction? APE: A cross-datasets transfer learning model for EAE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 393-409).
11. Wang, W., Mi, L., Cen, S., Dai, H., Li, Y., Fu, X., & Liu, Y. (2025). Region-based Content Enhancement for {Efficient} Video Analytics at the Edge. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25) (pp. 613-633).
12. Wu, Y., He, Y., Liu, X., Wang, Y., & Dannenberg, R. B. (2023, June). Transplayer: Timbre style transfer with flexible timbre control. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
13. Shan, T., & Maddox, R. K. (2025). Comparing methods for deriving the auditory brainstem response to continuous speech in human listeners. Imaging Neuroscience, 3, IMAG-a.
14. Xu, Xinmeng, et al. "Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning." arXiv preprint arXiv:2605.01673 (2026).
15. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.
16. Li, A., Zheng, C., Zhang, L., & Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Applied Acoustics, 187, 108499.
17. Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., ... & Lu, Y. (2026, March). Sepprune: Structured pruning for efficient deep speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 38, pp. 31861-31869).
18. Vuruma, S. K. R., Wu, D., Gupta, S. S., Aust, L., Lookingbill, V., Henry, C., ... & Huang, M. (2024). Utilizing large language models to identify reddit users considering vaping cessation for digital interventions. arXiv preprint arXiv:2404.17607.
19. Shan, T., Wenner, C. E., Xu, C., Duan, Z., & Maddox, R. K. (2022). Speech-in-noise comprehension is improved when viewing a deep-neural-network-generated talking face. Trends in Hearing, 26, 23312165221136934.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Bert Dave, Rasmus Larsen, Anzu Matsuda

This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles are published under the Creative Commons Attribution 4.0 (CC BY 4.0) License. Users may share, copy, adapt, and distribute the work for any purpose, provided proper credit is given to the original author(s) and source. Authors retain copyright.
