Real-time energy purchase optimization for a storage-integrated photovoltaic system by deep reinforcement learning
Authors:
- Waldemar Kolodziejczyk,
- Izabela Żółtowska,
- Paweł Cichosz
Abstract
The objective of this article is to minimize the cost of energy purchased on a real-time basis for a storage-integrated photovoltaic (PV) system installed in a microgrid. Under non-linear storage charging/dischargingcharacteristics, as well as uncertain solar energy generation, demands, and market prices, it is a complextask. It requires a proper level of tradeoff between storing too much and too little energy in the battery:future excess PV energy is lost in the former case, and demand is exposed to future high electricity pricesin the latter case. We propose a reinforcement learning approach to deal with a non-stationary environmentand non-linear storage characteristics. To make this approach applicable, a novel formulation of the decisionproblem is presented, which focuses on the optimization of grid energy purchases rather than on direct storagecontrol. This limits the complexity of the state and action space, making it possible to achieve satisfactorylearning speed and avoid stability issues. Then the Q-learning algorithm combined with a dense deep neuralnetwork for function representation is used to learn an optimal decision policy. The algorithm incorporatesenhancements that were found to improve learning speed and stability by prior work, such as experiencereplay, target network, and increasing discount factor. Extensive simulation results performed on real dataconfirm that our approach is effective and outperforms rule-based heuristics.
- Record ID
- WUT2f6d98592cdc45738565ff03ee54b9e3
- Author
- Journal series
- Control Engineering Practice, ISSN 0967-0661, e-ISSN 1873-6939
- Issue year
- 2021
- Vol
- 106
- Pages
- 1-12
- Publication size in sheets
- 5229.90
- Article number
- 104598
- Keywords in English
- Smart grid, Energy management, Storage control, Deep reinforcement learning, Q-learning, Neural networks
- ASJC Classification
- ; ; ;
- DOI
- DOI:10.1016/j.conengprac.2020.104598 Opening in a new tab
- URL
- https://www.sciencedirect.com/science/article/pii/S0967066120301763 Opening in a new tab
- Language
- (en) English
- Score (nominal)
- 100
- Score source
- journalList
- Score
- = 100.0, 14-05-2022, ArticleFromJournal
- Publication indicators
- = 6; = 1; = 10; : 2018 = 1.872; : 2020 (2 years) = 3.475 - 2020 (5 years) =3.415
- Citation count
- 9
- Uniform Resource Identifier
- https://repo.pw.edu.pl/info/article/WUT2f6d98592cdc45738565ff03ee54b9e3/
- URN
urn:pw-repo:WUT2f6d98592cdc45738565ff03ee54b9e3
* presented citation count is obtained through Internet information analysis and it is close to the number calculated by the Publish or PerishOpening in a new tab system.