Memory Compression for Small Language Models in Stateful Customer Service Workflows Under Strict Latency Constraints

Authors

  • Lydia Palmer Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia
  • Ivy Ellis Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia
  • Dylan Douglas Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia
  • Isaac Carter Faculty of Engineering and Information Technology, University of Melbourne, Melbourne, Australia

DOI:

https://doi.org/10.66421/int.j.datasci.intell.syst38

Keywords:

Small Language Models, Memory Compression, Stateful Workflows, Customer Service Agents, Natural Language Processing

Abstract

The deployment of artificial intelligence in customer service environments has increasingly relied on language models to automate interactions, resolve queries, and improve operational efficiency. While large language models exhibit remarkable capabilities in natural language understanding and generation, their deployment in stateful customer service workflows presents substantial challenges regarding computational overhead, latency, and memory utilization. Small language models offer a computationally viable alternative, particularly for edge deployment and privacy-preserving on-premise architectures, yet they inherently suffer from restricted context windows and limited capacity for retaining long-term conversational states. This paper investigates and proposes a comprehensive framework for memory compression tailored specifically for small language models operating within stateful customer service workflows. By leveraging semantic summarization, token-level pruning, and dynamic working memory allocation, the proposed methodology enables small language models to maintain contextual continuity across multi-turn interactions without exceeding memory constraints or degrading response accuracy. Through rigorous empirical analysis, this study evaluates the trade-offs between memory footprint reduction and context retention, demonstrating that optimized compression pipelines can significantly enhance the operational viability of small language models in complex, state-dependent tasks. The findings provide a scalable solution for enterprise applications requiring persistent conversational memory under strict computational limitations.

References

1. Zhang, K., Shuang, K., Yang, X., Yao, X., & Guo, J. (2023, July). What is overlap knowledge in event argument extraction? APE: A cross-datasets transfer learning model for EAE. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 393-409).

2. Yang, Y., Yang, X., Jiang, Y., Mu, N., Hu, H., Xie, R., ... & Xu, B. (2026). GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systems. arXiv preprint arXiv:2602.15776.

3. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.

4. Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization.

5. Deng, L., Deng, S., Chen, Y., Dai, Y., Zhong, Z., Li, L., ... & Huang, H. (2026). Tool-Augmented Agent for Closed-loop Optimization, Simulation, and Modeling Orchestration. arXiv preprint arXiv:2605.20190.

6. Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings of EMNLP.

7. Li, S. (2025). Momentum, volume and investor sentiment study for us technology sector stocks—A hidden markov model based principal component analysis. PloS one, 20(9), e0331658.

8. Chen, Y., Zhao, C., & Gao, Y. (2025). Blockchain applications in health insurance: a review of applications, challenges, and prospects. Frontiers in Blockchain, 8, 1699290.

9. Niu, Ke, et al. "Cme-cad: Heterogeneous collaborative multi-expert reinforcement learning for cad code generation." arXiv preprint arXiv:2512.23333 (2025).

10. Liu, T., Jia, Y., Zeng, S., & Yan, J. (2026). Social media use and attitudes toward premarital sex in China: a chain-mediating analysis. Sexuality Research and Social Policy, 1-17.

11. Liang, H., Huang, C., Xu, Y., Tang, C., Ye, W., Zhang, J., ... & Xu, L. (2024). Llava-slt: Visual language tuning for sign language translation. arXiv preprint arXiv:2412.16524.

12. Guo, J., Shuang, K., Li, J., & Wang, Z. (2021, August). Dual slot selector via local reliability verification for dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 139-151).

13. Zhang, Jiaquan, et al. "Lightweight llm agent memory with small language models." arXiv preprint arXiv:2604.07798 (2026).

14. Zhang, W., & Pei, S. (2026). Predictive prefetching for retrieval-augmented generation. Proceedings of Machine Learning Research. The Forty-third International Conference on Machine Learning, ICML 2026. https://par.nsf.gov/biblio/10686293

15. Deng, L., Guo, H., Bai, Y., Dai, Y., Huang, H., & Shi, Y. (2025). BrepLLM: Native Boundary Representation Understanding with Large Language Models. arXiv preprint arXiv:2512.16413.

16. Li, K., Shuang, K., & Guo, J. (2026). Bring order to the jumbled input: Layered attention for Large Language Model based Event Argument Extraction. Engineering Applications of Artificial Intelligence, 165, 113399.

17. Sang, Y. (2025, July). Towards explainable rag: Interpreting the influence of retrieved passages on generation. In 2025 4th International Conference on Robotics, Artificial Intelligence and Intelligent Control (RAIIC) (pp. 397-400). IEEE.

18. Huang, Z., Wang, J., Chen, L., Xiao, B., Cai, L., Zeng, Y., & Xu, J. (2025, October). MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 8797-8805).

19. Zhu, R., Jiang, Z., Wu, J., Ma, Z., Song, J., Bai, F., ... & He, C. (2025, April). Grait: Gradient-driven refusal-aware instruction tuning for effective hallucination mitigation. In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 4006-4021).

20. Zhu, R., Ma, Z., Wu, J., Gao, J., Wang, J., Lin, D., & He, C. (2025, April). Utilize the flow before stepping into the same river twice: Certainty represented knowledge flow for refusal-aware instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 24, pp. 26157-26165).

21. Wang, S., Zhang, X., Wang, P., & Li, X. (2026). Design of Magnetic Sensor Array-Based PUFs for IoT Security. IEEE Transactions on Instrumentation and Measurement, 75, 1-9.

22. Z. Hang, Y. Yashengjiang, J. Li, H. Dong, Y. Wei, Z. Hao, J. Ma, S. Bai, H. Kai, X. Yue, G. Si, D. Jiang, C. Yao, Z. Hu, J. Zhang, P. Liu, Y. Shen, X. Ren, L. Liu, Z. Xu, H. Li, Q. Yao, H. Dong, and H. Wang, “PDEAgent-Bench: A multi-metric, multi-library benchmark for PDE solver generation,” arXiv preprint arXiv:2605.09636, 2026, doi: 10.48550/arXiv.2605.09636.

23. Zhou, D. (2026). AI-Driven Hybrid SAST–DAST–SCA–IAST Framework for Risk-Based Vulnerability Prioritization in Microservice Architectures.

24. Guo, J., Yang, L., Chen, P., Xiao, Q., Wang, Y., Juan, X., ... & Wang, M. (2025). GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators. arXiv preprint arXiv:2512.19682.

25. Zhang, Jiaquan, et al. "Learning global hypothesis space for enhancing synergistic reasoning chain." arXiv preprint arXiv:2602.09794 (2026).

26. Wang, Juyuan, et al. "HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads." arXiv preprint arXiv:2604.17237 (2026).

27. Wang, Chenxing, et al. "When & How to Write for Personalized Demand-aware Query Rewriting in Video Search." arXiv preprint arXiv:2602.17667 (2025).

28. Zhou, J., Shuang, K., Wang, Q., Qian, B., & Guo, J. (2025). Bi-directional feature learning-based approach for zero-shot event argument extraction. Information Processing & Management, 62(5), 104199.

29. Mao, C., Shuang, K., Guo, J., Qian, B., Yang, Y., & Li, H. (2026). Cognition-aligned frequency filtering for sentence embeddings. Information Processing & Management, 63(2), 104415.

30. Liang, X., Tao, M., Xia, Y., Wang, J., Li, K., Wang, Y., ... & Wang, X. (2025). Sage: Self-evolving agents with reflective and memory-augmented abilities. Neurocomputing, 647, 130470.

31. Sang, Y. (2025, August). Robustness of fine-tuned llms under noisy retrieval inputs. In 2025 6th International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) (pp. 417-420). IEEE.

32. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.

33. Li, X., Wang, P., Li, G., Ni, L., & Zhang, Y. (2023). Design of interface circuits and lightweight PUF for TMR sensors. IEEE Sensors Journal, 23(11), 11754-11761.

34. Xiao, Q., & Zhou, K. (2026). MACD: Model-Aware Contrastive Decoding via Counterfactual Data. arXiv preprint arXiv:2602.01740.

35. Qu, D., & Ma, Y. (2025). Magnet-bn: markov-guided Bayesian neural networks for calibrated long-horizon sequence forecasting and community tracking. Mathematics, 13(17), 2740.

36. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 1–67.

37. Wang, J., Zhao, R., Wei, W., Wang, Y., Yu, M., Zhou, J., ... & Xu, L. (2026, March). Comorag: A cognitive-inspired memory-organized rag for stateful long narrative reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 39, pp. 33557-33565).

38. Sang, Y. (2025, October). AutoCrit: A Meta-Reasoning Framework for Self-Critique and Iterative Error Correction in LLM Chains-of-Thought. In 2025 6th International Conference on Machine Learning and Computer Application (ICMLCA) (pp. 1177-1180). IEEE.

39. Vinyals, O., Fortunato, M., & Jaitly, N. (2015). Pointer networks. In Advances in Neural Information Processing Systems.

40. Ma, Y., Qu, D., & Pyrozhenko, M. (2026). Bio-RegNet: A Meta-Homeostatic Bayesian Neural Network Framework Integrating Treg-Inspired Immunoregulation and Autophagic Optimization for Adaptive Community Detection and Stable Intelligence. Biomimetics, 11(1), 48.

41. Chen, L. (2026). Beyond external constraints: The missing dimension of ai governance. Available at SSRN 6449738.

42. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

43. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR.

44. Zhou, D. (2025, December). M-VP2: Microservice-Oriented Vulnerability Patch Planning-A Cost-Aware Approachusing Multi-Agent Reinforcement Learning. In 2025 5th International Conference on Computer, Internet of Things and Control Engineering (CITCE) (pp. 248-254). IEEE.

45. Zhu, Runchuan, et al. "AdaptFlow: Adaptive Workflow Optimization via Meta-Learning." arXiv preprint arXiv:2508.08053 (2025).

46. Ding, H., Fang, Y., Zhu, R., Jiang, X., Zhang, J., Xu, Y., ... & Wang, Y. (2024). 3ds: Decomposed difficulty data selection’s case study on llm medical domain adaptation.

Downloads

Published

2026-06-25