Real-time energy purchase optimization for a storage-integrated photovoltaic system by deep reinforcement learning

Authors:

• Waldemar Kolodziejczyk,
• Izabela Żółtowska,
• Paweł Cichosz

Abstract

The objective of this article is to minimize the cost of energy purchased on a real-time basis for a storage-integrated photovoltaic (PV) system installed in a microgrid. Under non-linear storage charging/dischargingcharacteristics, as well as uncertain solar energy generation, demands, and market prices, it is a complextask. It requires a proper level of tradeoff between storing too much and too little energy in the battery:future excess PV energy is lost in the former case, and demand is exposed to future high electricity pricesin the latter case. We propose a reinforcement learning approach to deal with a non-stationary environmentand non-linear storage characteristics. To make this approach applicable, a novel formulation of the decisionproblem is presented, which focuses on the optimization of grid energy purchases rather than on direct storagecontrol. This limits the complexity of the state and action space, making it possible to achieve satisfactorylearning speed and avoid stability issues. Then the Q-learning algorithm combined with a dense deep neuralnetwork for function representation is used to learn an optimal decision policy. The algorithm incorporatesenhancements that were found to improve learning speed and stability by prior work, such as experiencereplay, target network, and increasing discount factor. Extensive simulation results performed on real dataconfirm that our approach is effective and outperforms rule-based heuristics.