Short-Term Forecasting of Electricity Imbalance Price and Day-Ahead Auction Price in the UK

Short-Term Forecasting of Electricity Imbalance Price and Day-Ahead Auction Price in the UK 

Authors: Lei Pu, Vanessa Adams 

As part of Utilidex’s R&D programme in collaboration with UCL School of Management we undertook research into understanding the predictability of short-term market prices. The energy market price discovery is designed to be driven by fundamental supply and demand factors which in theory, should be predictable given the quantity of transparent data available.


The world of energy is changing and in the absence of further regulatory intervention it is likely that the current volatility within day gas and power as well as day ahead will persist.  

Our concept of purchasing base-load and peak-load shapes particularly in the UK has never exactly corresponded to real demand profiles where peak-load is traded at a premium to baseload. Real energy demand is different from these approximate shapes.

See table below of forecasted demand vs actual 

Energy price volatility

Up until approximately 3 years ago the likelihood of the price spike was minimal so our standard peak and base products were effective hedging instruments.  

More recently due to variety of reasons including increasing renewable energy penetration, decentralisation of power plants, alongside changes in the how the market clearing price is calculated, generation intermittency has to be covered by a greater number of short-term peaking plant/batteries replacing many of the CCGT and coal fired steam turbines. The cost of such intervention creates greater volatility hence recent unprecedented price spikes where there has been scarcity of supply. 

This volatility if sustained for a long period of time are likely to impact energy suppliers and consumer’s ability to cover their hedge costs with standard products and, if they can’t hedge these costs with a certain degree of precision, imbalance costs are likely to be passed on to the end customer, especially if their customers are energy intensive users who wish to purchase energy with some certainty over the remaining year/ next year.  

The figure below depicts typical daily volatility of imbalance costs

Rather than exposing energy imbalances to the within day market, customers and energy suppliers have the option to use day ahead prices from N2EX to hedge short exposure and mitigate the risk of high price spikes. 

See table below of N2EX Day Ahead Auction Prices 

If spot prices or day ahead prices can be accurately predicted in advance, especially the likelihood of near-term price spikes, then suppliers and their end users can decide for themselves whether they buy energy, produce it themselves or just turn down their demand. 

Creating a predictive maths engine for prices 

There is a variety of data including gross demand, energy frequency data, individual generation unit dispatch, renewable generation forecasts, weather forecast or actuals and National Grid messaging/warnings. We explored each data set which affect electricity prices attempting to understand which features drive price more than others and when.  

More specifically we sought to discover which maths model can forecast the imbalance price and the day-ahead auction price on the next day more effectively using public data provided by National Grid. We reviewed 5 mathematical models.  

  • Multi- Agent 
  • Fundamental 
  • Reduced Form 
  • Statistical 
  • Computational Intelligence 

By reviewing the literature, the Statistical (LEAR) model and the Computational Intelligence (DNN) model were found to be more competitive against most of the models on the various energy market data. After a thorough comparison between the two state-of-the-art models and a comparison between the results obtained we concluded that both LEAR and DNN models can effectively forecast the day-ahead auction price with comparable performances.  

The graphs below show the real and predicted day-ahead auction prices


Though both models can effectively forecast the imbalance price up until July 2020, the performances are not favourable in recent months whilst the market seems to have fundamentally changed with the electricity price becoming highly volatile. This change in the market landscape is difficult to explain.  

The graphs below show the imbalance price

Additionally, it was found that combining the forecasts of several short and long calibration windows (Ensemble) cannot significantly improve the accuracy of the LEAR model, and the average forecast of multiple DNN models is not significantly better either. In terms of explanatory variables, it was found that the electricity generation by renewable sources are less important than other variables, especially demand, aggregated generation and load variables.  

Our Findings and Recommendations 

Based on the findings, recommendations are given as follows. First, the LEAR model with a calibration window of 52 weeks is preferable. Though the DNN model has a comparable performance, it is less practical considering its extensive training time. Second, the day-ahead auction price can be effectively forecasted, while the imbalance price forecast is not always easy to predict, especially in recent months due to the high volatility.   

Therefore, it is not recommended to make decisions according to the imbalance price forecast, but  the day-ahead auction price forecast is considered reliable. Further, if specific explanatory variables need to be focused on, the load, demand and aggregated electricity generation have higher priority compared to the renewable generation. 

Potential Applications 

Using the model, it is possible to hedge the day ahead market with reasonable accuracy as a method to lock in prices. But forecasting prices 2 hours in advance is much more difficult, especially over the last year. Therefore, as a hedging mechanism it easier to predict day ahead prices and consequently hedge and not be exposed to within day prices with little predictability. 

The model could be used for generators with energy despatch flexibility such as a fleet of batteries of CHP plants to bid into the markets. Typically, aggregators and energy generators can run similar calculations and reward generators with incentive payments or income based on a cost/ availability / efficiency hierarchy. The algorithm could also be used by end customers with flexible demand to moderate/ avoid costs, offering this as a service to using energy suppliers as a distribution channel to wholesale markets. Such application again might take the form of aggregation of demand flexibility across multiple sites based on changing manufacturing processes and run-times or utilising batteries.  

Contact us today to find out how we can help you.

Four reasons to switch across today


Easy and intuitive to use

Our modern and innovative technology solution has been built with energy professionals in mind and we’re constantly going out of way to make your job easier.


Make everyone part of the energy challenge.

The Energy.Hub is available across web, mobile, dashboards and wallboards, meaning you can get your entire energy estate behind you in your carbon and cost reductions.


Simple and affordable pricing with no hidden user costs

You purchase a single licence which allows you unlimited users, meaning all stakeholders can easily view the data that’s relevant to them without you having to pay more.


Technology that doesn’t stand still

Don’t settle for the outdated and frustrating systems of yesterday. The Utilidex Energy.Hub is a cloud-first built platform which means your system will get instantly updated with all the latest features – for free.

Applications to help you do more.

You can purchase any one of our software products direct as a customer, or through our partner consultancy firms (TPIs), or via energy suppliers. Our goal is to make it easy for you to work together, and stay in control of your energy estate.

Energy Analytics

Understand your energy data with our detailed energy insights and powerful reporting features.


Utility Bill Validation

Automate your bill validation with straight-through processing from EDI receipt to payments in minutes.


Trading and Energy Procurement

Provide full financial transparency to your energy procurement strategy, highlighting the risks and rewards in locking-in energy prices over time.


Our clients.