Self-Supervised Contrastive Regularization for Robust Speech Enhancement

Authors

  • Bert Dave Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia
  • Rasmus Larsen Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia
  • Anzu Matsuda Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia

Keywords:

Speech Enhancement, Contrastive Learning, Self-Supervised Learning, Deep Neural Networks, Audio Processing

Abstract

The problem of robust speech enhancement remains a critical bottleneck in the deployment of voice-controlled interfaces, telecommunication systems, and hearing aid devices, particularly in highly non-stationary and reverberant environments. While fully supervised deep learning models have demonstrated remarkable success in mitigating additive background noise and reverberation, they frequently suffer from severe performance degradation when exposed to unseen acoustic conditions. This generalization gap stems from the models overfitting to the specific noise distributions and acoustic characteristics present in the training corpus. To address this fundamental limitation, we propose a novel framework that integrates self-supervised contrastive learning as a regularization mechanism within a standard supervised speech enhancement architecture. By simultaneously optimizing a primary waveform reconstruction objective and an auxiliary contrastive loss, the proposed model learns to extract noise-invariant, high-level phonetic representations. The contrastive regularization forces the neural network to map augmented views of the same clean speech signal distorted by vastly different environmental noises to nearby points in a shared latent space, while pushing apart representations of different speech utterances. Comprehensive evaluations across multiple benchmark datasets demonstrate that the integration of self-supervised contrastive regularization substantially improves objective metrics such as perceptual evaluation of speech quality and short-time objective intelligibility, particularly under zero-shot testing conditions involving unseen noise types and severe signal-to-noise ratios. The findings emphasize the potential of hybrid learning paradigms to yield highly robust, generalizable audio processing systems without requiring exponentially larger annotated datasets.

References

1. Xu, X., Han, C., Zhang, Y., Tu, W., & Yang, Y. (2024, April). Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 10486-10490). IEEE.

2. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.

3. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.

4. Li, A., Sun, Z., Hao, F., Li, X., & Zheng, C. (2025). Neural vocoders as speech enhancers. arXiv preprint arXiv:2501.13465.

5. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of ICML.

6. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.

7. Xu, X., Tu, W., Yang, Y., Li, J., & Zhang, Y. (2025). Interactive Target Positive and Negative Features Modeling for Monaural Speech Enhancement. IEEE Transactions on Audio, Speech and Language Processing, 33, 4856-4869.

8. Shan, T., Cappelloni, M. S., & Maddox, R. K. (2024). Subcortical responses to music and speech are alike while cortical responses diverge. Scientific Reports, 14(1), 789.

9. Wang, J., Zhao, R., Wei, W., Wang, Y., Yu, M., Zhou, J., ... & Xu, L. (2026, March). Comorag: A cognitive-inspired memory-organized rag for stateful long narrative reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 39, pp. 33557-33565).

10. Zhang, K., Shuang, K., Yang, X., Yao, X., & Guo, J. (2023, July). What is overlap knowledge in event argument extraction? APE: A cross-datasets transfer learning model for EAE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 393-409).

11. Wang, W., Mi, L., Cen, S., Dai, H., Li, Y., Fu, X., & Liu, Y. (2025). Region-based Content Enhancement for {Efficient} Video Analytics at the Edge. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25) (pp. 613-633).

12. Wu, Y., He, Y., Liu, X., Wang, Y., & Dannenberg, R. B. (2023, June). Transplayer: Timbre style transfer with flexible timbre control. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

13. Shan, T., & Maddox, R. K. (2025). Comparing methods for deriving the auditory brainstem response to continuous speech in human listeners. Imaging Neuroscience, 3, IMAG-a.

14. Xu, Xinmeng, et al. "Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning." arXiv preprint arXiv:2605.01673 (2026).

15. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.

16. Li, A., Zheng, C., Zhang, L., & Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Applied Acoustics, 187, 108499.

17. Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., ... & Lu, Y. (2026, March). Sepprune: Structured pruning for efficient deep speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 38, pp. 31861-31869).

18. Vuruma, S. K. R., Wu, D., Gupta, S. S., Aust, L., Lookingbill, V., Henry, C., ... & Huang, M. (2024). Utilizing large language models to identify reddit users considering vaping cessation for digital interventions. arXiv preprint arXiv:2404.17607.

19. Shan, T., Wenner, C. E., Xu, C., Duan, Z., & Maddox, R. K. (2022). Speech-in-noise comprehension is improved when viewing a deep-neural-network-generated talking face. Trends in Hearing, 26, 23312165221136934.

Downloads

Published

2026-05-01