A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 9 Issue 1
Jan.  2022

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 15.3, Top 1 (SCI Q1)
    CiteScore: 23.5, Top 2% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
Y. N. Wan, J. H. Qin, X. H. Yu, T. Yang, and Y. Kang, “Price-based residential demand response management in smart grids: A reinforcement learning-based approach,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 123–134, Jan. 2022. doi: 10.1109/JAS.2021.1004287
Citation: Y. N. Wan, J. H. Qin, X. H. Yu, T. Yang, and Y. Kang, “Price-based residential demand response management in smart grids: A reinforcement learning-based approach,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 123–134, Jan. 2022. doi: 10.1109/JAS.2021.1004287

Price-Based Residential Demand Response Management in Smart Grids: A Reinforcement Learning-Based Approach

doi: 10.1109/JAS.2021.1004287
Funds:  This work was supported in part by the National Natural Science Foundation of China (61922076, 61725304, 61873252, 61991403, 61991400) and in part by the Australian Research Council Discovery Program (DP200101199)
More Information
  • This paper studies price-based residential demand response management (PB-RDRM) in smart grids, in which non-dispatchable and dispatchable loads (including general loads and plug-in electric vehicles (PEVs)) are both involved. The PB-RDRM is composed of a bi-level optimization problem, in which the upper-level dynamic retail pricing problem aims to maximize the profit of a utility company (UC) by selecting optimal retail prices (RPs), while the lower-level demand response (DR) problem expects to minimize the comprehensive cost of loads by coordinating their energy consumption behavior. The challenges here are mainly two-fold: 1) the uncertainty of energy consumption and RPs; 2) the flexible PEVs’ temporally coupled constraints, which make it impossible to directly develop a model-based optimization algorithm to solve the PB-RDRM. To address these challenges, we first model the dynamic retail pricing problem as a Markovian decision process (MDP), and then employ a model-free reinforcement learning (RL) algorithm to learn the optimal dynamic RPs of UC according to the loads’ responses. Our proposed RL-based DR algorithm is benchmarked against two model-based optimization approaches (i.e., distributed dual decomposition-based (DDB) method and distributed primal-dual interior (PDI)-based method), which require exact load and electricity price models. The comparison results show that, compared with the benchmark solutions, our proposed algorithm can not only adaptively decide the RPs through on-line learning processes, but also achieve larger social welfare within an unknown electricity market environment.


  • loading
  • 1 Since the energy demand and consumption of loads is affected by many factors, the state transition is rather difficult to obtain. Therefore, we next employ a model-free Q-learning method to solve the dynamic retail pricing problem.
  • [1]
    X. Fang, S. Misra, G. Xue, and D. Yang, “Smart grid–The new and improved power grid: A survey,” IEEE Commun. Surveys Tuts., vol. 14, no. 4, pp. 944–980, Apr. 2012. doi: 10.1109/SURV.2011.101911.00087
    X. Yu and Y. Xue, “Smart grids: A cyber-physical systems perspective,” Proceedings of the IEEE, vol. 104, no. 5, pp. 1058–1070, May 2016. doi: 10.1109/JPROC.2015.2503119
    R. Deng, Z. Yang, M. Chow, and J. Chen, “A survey on demand response in smart grids: Mathematical models and approaches,” IEEE Trans. Ind. Informat., vol. 11, no. 3, pp. 570–582, Jun. 2015. doi: 10.1109/TII.2015.2414719
    L. Ding, L. Y. Wang, G. Y. Yin, W. X. Zheng, and Q. Han, “Distributed energy management for smart grids with an event-triggered communication scheme,” IEEE Trans. Control Syst. Technol., vol. 27, no. 5, pp. 1950–1961, Sept. 2019. doi: 10.1109/TCST.2018.2842208
    H. Hao, C. D. Corbin, K. Kalsi, and R. G. Pratt, “Transactive control of commercial buildings for demand response,” IEEE Trans. Power Syst., vol. 32, no. 1, pp. 774–783, Jan. 2017. doi: 10.1109/TPWRS.2016.2559485
    B. Shen, G. Ghatikar, Z. Lei, J. Li, G. Wikler, and P. Martin, “The role of regulatory reforms, market changes, and technology development to make demand response a viable resource in meeting energy challenges,” Appl. Energy, vol. 130, pp. 814–823, Oct. 2014. doi: 10.1016/j.apenergy.2013.12.069
    J. Qin, Y. Wan, X. Yu, F. Li, and C. Li, “Consensus-based distributed coordination between economic dispatch and demand response,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 3709–3719, Jul. 2019. doi: 10.1109/TSG.2018.2834368
    Y. Hung and G. Michailidis, “Modeling and optimization of time-of-use electricity pricing systems,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4116–4127, Jul. 2019. doi: 10.1109/TSG.2018.2850326
    R. Deng, Z. Yang, F. Hou, M. Y. Chow, and J. Chen, “Distributed realtime demand response in multiseller-multibuyer smart distribution grid,” IEEE Trans. Power Syst., vol. 30, no. 5, pp. 2364–2374, Sept. 2015. doi: 10.1109/TPWRS.2014.2359457
    Q. Dong, L. Yu, W. Song, J. Yang, Y. Wu, and J. Qi, “Fast distributed demand response algorithm in smart grid,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 2, pp. 280–296, Apr. 2017. doi: 10.1109/JAS.2017.7510529
    H. Hao, D. Wu, J. Lian, and T. Yang, “Optimal coordination of building loads and energy storage for power grid and end user services,” IEEE Trans. Smart Grid, vol. 9, no. 5, pp. 4335–4345, Sept. 2018. doi: 10.1109/TSG.2017.2655083
    R. Carli and M. Dotoli, “Decentralized control for residential energy management of a smart users’ microgrid with renewable energy exchange,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 3, pp. 641–656, May 2019. doi: 10.1109/JAS.2019.1911462
    L. Yu, Y. Sun, Z. B. Xu, et al., “Multi-agent deep reinforcement learning for HVAC control in commercial buildings,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 407–419, Jan. 2021. doi: 10.1109/TSG.2020.3011739
    D. Azuatalam, W. L. Lee, F. D. Nijs, and A. Liebman, “Reinforcement learning for whole-building HVAC control and demand response,” Energy and AI, vol. 2, p. 100020, 2020.
    K. Vanthournout, B. Dupont, W. Foubert, and S. Claessens, “An automated residential demand response pilot experiment, based on day-ahead dynamic pricing,” Appl. Energy, vol. 155, pp. 195–203, Oct. 2015. doi: 10.1016/j.apenergy.2015.05.100
    A. Mohsenian-Rad, V. W. S. Wong, J. Jatskevich, R. Schober, and A. Leon-Garcia, “Autonomous demand-side management based on game-theoretic energy consumption scheduling for the future smart grid,” IEEE Trans. Smart Grid, vol. 1, no. 3, pp. 320–331, Dec. 2010. doi: 10.1109/TSG.2010.2089069
    C. Li, C. Liu, K. Deng, X. Yu, and T. Huang, “Data-driven charging strategy of PEVs under transformer aging risk,” IEEE Trans. Control Syst. Technol., vol. 26, no. 4, pp. 1386–1399, Jul. 2018. doi: 10.1109/TCST.2017.2713321
    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 2011.
    S. Vandael, B. Claessens, D. Ernst, T. Holvoet, and G. Deconinck, “Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market,” IEEE Trans. Smart Grid, vol. 6, no. 4, pp. 1795–1805, Jul. 2015. doi: 10.1109/TSG.2015.2393059
    B. Kim, Y. Zhang, M. V. D. Schaar, and J. Lee, “Dynamic pricing and energy consumption scheduling with reinforcement learning,” IEEE Trans. on Smart Grid, vol. 7, no. 5, pp. 2187–2198, Sept. 2016. doi: 10.1109/TSG.2015.2495145
    E. Foruzan, L. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Trans. Power Syst., vol. 33, no. 5, pp. 5749–5758, Sept. 2018. doi: 10.1109/TPWRS.2018.2823641
    M. Imani and S. F. Ghoreishi, “Scalable inverse reinforcement learning through multifidelity Bayesian optimization,” IEEE Trans. Neural Netw. Learn. Syst., 2021. DOI: 10.1109/TNNLS.2021.3051012
    M. Sadeghi and M. Erol-Kantarci, “Power loss minimization in microgrids using bayesian reinforcement learning with coalition formation, in Proc. IEEE 30th Annu. Int. Symp. Personal, Indoor and Mobile Radio Communications, 2019, 1–6.
    T. Liu, B. Tian, Y. Ai, and F. Wang, “Parallel reinforcement learningbased energy efficiency improvement for a cyber-physical system,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 2, pp. 617–626, Mar. 2020. doi: 10.1109/JAS.2020.1003072
    M. Ahrarinouri, M. Rastegar, and A. R. Seifi, “Multiagent reinforcement learning for energy management in residential buildings,” IEEE Trans. Ind. Informat., vol. 17, no. 1, pp. 659–666, Jan. 2021. doi: 10.1109/TII.2020.2977104
    X. Zhang, D. Biagioni, M. Cai, P. Graf, and S. Rahman, “An edge-cloud integrated solution for buildings demand response using reinforcement learning,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 420–431, Jan. 2021. doi: 10.1109/TSG.2020.3014055
    Y. Zhao, Y. Cai, and Q. Song, “Energy control of plug-in hybrid electric vehicles using model predictive control with route preview,” IEEE/CAA J. Autom. Sinica, 2018. DOI: 10.1109/JAS.2017.7510889
    M. Jin, W. Feng, C. Marnay, and C. Spanos, “Microgrid to enable optimal distributed energy retail and end-user demand response,” Appl. Energy, vol. 210, pp. 1321–1335, Jan. 2018. doi: 10.1016/j.apenergy.2017.05.103
    R. Lu, S. H. Hong, and X. Zhang, “A dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach,” Appl. Energy, vol. 220, pp. 220–230, Jun. 2018. doi: 10.1016/j.apenergy.2018.03.072
    M. Yu, S. H. Hong, Y. Ding, and X. Ye, “An incentive-based demand response (DR) model considering composited DR resources,” IEEE Trans. Ind. Electron., vol. 66, no. 2, pp. 1488–1498, Feb. 2019. doi: 10.1109/TIE.2018.2826454
    Y. Wan, J. Qin, F. Li, X. Yu, and Y. Kang, “Game theoretic-based distributed charging strategy for PEVs in a smart charging station,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 538–547, Jan. 2021. doi: 10.1109/TSG.2020.3020466
    L. Yu, W. W. Xie, D. Xie, et al., “Deep reinforcement learning for smart home energy management,” IEEE Internet Things J., vol. 7, no. 4, pp. 2751–2762, Apr. 2020. doi: 10.1109/JIOT.2019.2957289
    H. Jahangir, S. S. Gougheri, B. Vatandoust, et al., “Plug-in electric vehicle behavior modeling in energy market: A novel deep learning-based approach with clustering technique,” IEEE Trans. Smart Grid, vol. 11, no. 6, pp. 4738–4748, Nov. 2020. doi: 10.1109/TSG.2020.2998072
    F. Li, J. Qin, and W. X. Zheng, “Distributed Q-learning-based online optimization algorithm for unit commitment and dispatch in smart grid,” IEEE Trans. Cybern., vol. 50, no. 9, pp. 4146–4156, Sept. 2020. doi: 10.1109/TCYB.2019.2921475
    Y. Du and F. Li, “Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 2, pp. 1066–1076, Mar. 2020. doi: 10.1109/TSG.2019.2930299
    “Sdge.com. Home-san diego gas & electric.” [Online]. Available: https://www.sdge.com
    “Real-time hourly prices,” Commonwealth Edison Company. [Online]. Available: https://rrtp.comed.com/live-prices/
    S. Koenig and R. G. Simmons, “The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms,” Machine Learning, vol. 22, no. 1, pp. 227C250, 1996.


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(14)  / Tables(3)

    Article Metrics

    Article views (888) PDF downloads(90) Cited by()


    • Study the price-based residential demand response management in smart grid considering PEV loads
    • Model the PB-RDRM from a social perspective, i.e., maximize the weighted sum of UC's profit and loads' cost
    • Propose a model-free reinforcement learning-based DR algorithm to address the uncertainties


    DownLoad:  Full-Size Img  PowerPoint