Lightweight Domain-Specific Language Models for Cloud Resource Scheduling

Authors

  • Liam Walker David R. Cheriton School of Computer Science, Faculty of Mathematics, University of Waterloo, Waterloo, ON, Canada
  • Aisha Rahman David R. Cheriton School of Computer Science, Faculty of Mathematics, University of Waterloo, Waterloo, ON, Canada
  • Victor Chen David R. Cheriton School of Computer Science, Faculty of Mathematics, University of Waterloo, Waterloo, ON, Canada

DOI:

https://doi.org/10.66421/int.j.datasci.intell.syst61

Keywords:

Cloud computing, resource scheduling, small language models, knowledge distillation, constrained decoding, integer programming

Abstract

Cloud schedulers place dependency-constrained tasks on heterogeneous machines while balancing completion time, utilisation and service-level objectives. Recent work shows that a large language model supplies useful scheduling intuition once its proposals are repaired by a mathematical solver, yet a foundation-model call costs hundreds of milliseconds and its price grows with the amount of cluster state that has to be described, which confines the reasoning component to a small fraction of decision epochs. This paper asks whether the reasoning component itself can be made small enough to run on every decision. We present SchedLM, a 0.36 M-parameter causal language model trained from scratch over a scheduling-specific token language. Three ingredients make a model of this size sufficient. A node-class abstraction keeps the prompt at 143 tokens whatever the cluster size. Distillation from a per-epoch integer-programming teacher supervises a defer-free action space, so that deferral is produced by the constraints rather than learned as an action. Feasibility-constrained decoding masks infeasible node classes at every decoding step, so the model emits only executable schedules. On trace-calibrated DAG workloads SchedLM retains 96.2–98.4% of the teacher's weighted job completion time and improves on the Kubernetes default scheduler by 7.1–8.5% in completion time, by 12.9–25.3% in SLA violations and by up to 14.9% in makespan, with zero infeasible assignments. Median decision latency grows 1.20

 

References

[1] Xing, S., Wang, Y., & Liu, W. (2025). Self-adapting CPU scheduling for mixed database workloads via hierarchical deep reinforcement learning. Symmetry, 17(7), 1109.

[2] Ji, E., Wang, Y., Xing, S., & Jin, J. (2025). Hierarchical reinforcement learning for energy-efficient API traffic optimization in large-scale advertising systems. IEEE Access.

[3] Chen, T., Shang, W., & Guo, Z. (2024). Xsla: An SLA-aware heterogeneous XPU scheduling framework for edge AI inference. World Journal of Information Technology, 2(2), 48–59.

[4] Zou, J., Qin, Y., & Rhee, M. (2024). Beyond model scale: A task-aware reliability benchmark for large language models in clinical decision support. Innovation and Technology Studies, 1(1), 39–47.

[5] Fang, Q. (2024). Bias-aware large language models for equitable educational feedback and assessment. Trends in Social Sciences and Humanities Research.

[6] V. K. Vavilapalli et al., “Apache Hadoop YARN: Yet another resource negotiator,” in Proc. ACM SoCC, 2013, pp. 1–16.

[7] H. Mao, M. Schwarzkopf, S. B. Venkatakrishnan, Z. Meng, and M. Alizadeh, “Learning scheduling algorithms for data processing clusters,” in Proc. ACM SIGCOMM, 2019, pp. 270–288.

[8] M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, “Omega: Flexible, scalable schedulers for large compute clusters,” in Proc. EuroSys, 2013, pp. 351–364.

[9] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica, “Dominant resource fairness: Fair allocation of multiple resource types,” in Proc. USENIX NSDI, 2011, pp. 323–336.

[10] R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella, “Multi-resource packing for cluster schedulers,” in Proc. ACM SIGCOMM, 2014, pp. 455–466.

[11] I. Gog, M. Schwarzkopf, A. Gleave, R. N. M. Watson, and S. Hand, “Firmament: Fast, centralized cluster scheduling at scale,” in Proc. USENIX OSDI, 2016, pp. 99–115.

[12] C. Delimitrou and C. Kozyrakis, “Quasar: Resource-efficient and QoS-aware cluster management,” in Proc. ACM ASPLOS, 2014, pp. 127–144.

[13] H. Topcuoglu, S. Hariri, and M.-Y. Wu, “Performance-effective and low-complexity task scheduling for heterogeneous computing,” IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, pp. 260–274, 2002.

[14] E. Cortez, A. Bonde, A. Muzio, M. Russinovich, M. Fontoura, and R. Bianchini, “Resource Central: Understanding and predicting workloads for improved resource management in large cloud platforms,” in Proc. ACM SOSP, 2017, pp. 153–167.

[15] R. Bhardwaj et al., “Cilantro: Performance-aware resource allocation for general objectives via online feedback,” in Proc. USENIX OSDI, 2023, pp. 623–643.

[16] Y. Bengio, A. Lodi, and A. Prouvost, “Machine learning for combinatorial optimization: A methodological tour d’horizon,” Eur. J. Oper. Res., vol. 290, no. 2, pp. 405–421, 2021.

[17] I. Trummer, “DB-BERT: A database tuning tool that ‘reads the manual’,” in Proc. ACM SIGMOD, 2022, pp. 190–203.

[18] S. Lai, Z. Xu, W. Zhang, H. Liu, and H. Xiong, “LLMLight: Large language models as traffic signal control agents,” 2023, arXiv:2312.16044.

[19] S. Kambhampati et al., “Position: LLMs can’t plan, but can help planning in LLM-Modulo frameworks,” in Proc. Int. Conf. Mach. Learn., 2024.

[20] K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati, “On the planning abilities of large language models: A critical investigation,” in Proc. Adv. Neural Inf. Process. Syst., 2023, pp. 75993–76005.

[21] G. Ding, S. Yang, H. Lin, Z. Chen, and J. S. Yang, “LLM-driven adaptive cloud resource scheduling: Bridging reasoning intelligence with optimization guarantees,” IEEE Open J. Comput. Soc., vol. 7, pp. 560–573, 2026.

[22] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015, arXiv:1503.02531.

[23] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,” 2019, arXiv:1910.01108.

[24] X. Jiao et al., “TinyBERT: Distilling BERT for natural language understanding,” in Findings of EMNLP, 2020, pp. 4163–4174.

[25] S. Gunasekar et al., “Textbooks are all you need,” 2023, arXiv:2306.11644.

[26] P. Zhang, G. Zeng, T. Wang, and W. Lu, “TinyLlama: An open-source small language model,” 2024, arXiv:2401.02385.

[27] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in Proc. Int. Conf. Learn. Represent., 2023.

[28] T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing incrementally for constrained auto-regressive decoding from language models,” in Proc. EMNLP, 2021, pp. 9895–9901.

[29] S. Geng, M. Josifoski, M. Peyrard, and R. West, “Grammar-constrained decoding for structured NLP tasks without finetuning,” in Proc. EMNLP, 2023, pp. 10932–10952.

[30] J. D. Ullman, “NP-complete scheduling problems,” J. Comput. Syst. Sci., vol. 10, no. 3, pp. 384–393, 1975.

[31] C. Reiss, J. Wilkes, and J. L. Hellerstein, “Google cluster-usage traces: Format + schema,” Google Inc., White Paper, 2011.

[32] J. Cao, A. Ganesh, J. Cai, R. Southwell, E. M. Perkoff, M. Regan, K. Kann, J. H. Martin, M. Palmer, and S. D'Mello, “A comparative analysis of automatic speech recognition errors in small group classroom discourse,” in Proc. 31st ACM Conf. User Modeling, Adaptation and Personalization (UMAP), 2023, pp. 250–261.

[33] Qin, Y., Zou, J., & Rhee, M. (2022). An AI-driven intelligent transaction execution framework for internet finance platforms: Cost-sensitive real-time integrity interdiction. Eurasia Journal of Science and Technology, 4(2), 36–44.

[34] Wang, Y., & Zhang, C. (2023). AERIS: Adaptive executor scoped resource management for privacy preserving data processing in Apache Spark. Eurasia Journal of Science and Technology, 5(2), 25-33.

[35] Zhao, W., & Wang, B. (2022). CAPA: A prediction‑driven autoscaling framework for SLO‑aware cloud‑native machine learning inference. Journal of Computer Science and Electrical Engineering, 4(1), 8-16.

[36] Guo, Z., Chen, T., & Shang, W. (2022). Class‑conditional intermediate‑domain adversarial adaptation for cross‑device acoustic scene classification. Journal of Computer Science and Electrical Engineering, 4(1), 26-35

[37] Shen, A., & Huang, S. (2022). A systems-aware framework for intelligent large-scale recommendation models. Journal of Computer Science and Electrical Engineering.

Downloads

Published

2026-09-17