Structure-Aware Reinforcement Learning for Energy-Constrained Background Update Scheduling

초록

Frequent background updates in mobile applications help maintain content freshness, but can lead to unnecessary energy consumption, especially when updates are discarded before user access. This paper studies the problem of scheduling background updates under an energy budget while accounting for stochastic user launch behavior. We formulate the problem as a constrained Markov decision process (CMDP) that minimizes user inconvenience subject to a long-term energy constraint. Analyzing the Lagrangian-relaxed MDP, we characterize the optimal policy as having a monotone threshold structure with respect to the update interval. Building on this structure, we propose SA-Q, a threshold-guided Q-learning algorithm that uses an online learned threshold to actively steer exploration toward informative regions and empirically reduce wasted sampling in constrained settings. This mechanism accelerates convergence and reduces training variance. Extensive simulations across diverse scenarios and real-world traces demonstrate that SA-Q consistently outperforms baseline methods, achieving up to 8.6% lower energy consumption while maintaining the same level of user experience. © 2026 IEEE.

키워드