Changes to frequency keeping offer data will be deployed on 4 December 2018

  • 87 Views
  • Last post 15 hours ago
Phil Bishop posted this 4 weeks ago

This discussion describes some forthcoming changes to the Frequency Keeping (FK) offer data that we publish on EMI. The changes will most definitely impact those users of the FK offer data that have automated processes to download files from our Azure storage containers.

The XML and CSV files currently found in the FinalOffers and DispatchedOffers folders will cease to be updated from 4 Dec. This date will be confirmed nearer to 4 Dec.

The underlying Azure storage containers that will no longer be updated from 4 Dec are here and here.

The remainder of this discussion describes what will replace the aforementioned files. Please be aware that almost everything about the published FK offer data and the publishing regime will change on 4 Dec:

  • file structure, i.e. the number of columns and the column names
  • file names
  • file locations
  • file format, i.e. CSV versus XML
  • file contents, i.e. the actual data, in the case of final offers.

Frequency keeping offers (all FK offers)

The current files containing final offers will be replaced with a file that contains all offers, including modfied offers, i.e. the new files will have more records than the current files. The records that represent final offers will be denoted with an ‘IsLatest’ flag.

The FK offer files will be published daily in a CSV format and will be named YYYYMMDD_FKOffers.csv, i.e. the word ‘final’ is dropped from the file name, as all submitted offers will be included in the files.

The daily CSV files will be collated in yearly folders found in the file path ..\Wholesale\FrequencyKeeping\Offers\.

The published file for any given trading day will be published to the same file name for three days in a row, beginning the day after the trading day. For example, the file called 20181204_FKOffers.csv will be published on 5, 6 and 7 Dec 2018.

The current monthly files that replace the daily collections will be done away with, i.e. at the end of the calendar year, each yearly folder will contain 365 CSV files (366 in leap years).

The FK offer files will be found here on EMI datasets and here in the underlying Azure storage container.

The column names and the order of the columns in the daily FK offer CSV files are as follows:

  • Participant
  • OfferBlock
  • TradingDate
  • TradingPeriod
  • SubmissionDate
  • SubmissionTime
  • UTCSubmissionDate
  • UTCSubmissionTime
  • PointOfConnection
  • Unit
  • Block
  • IsLatest
  • Megawatt
  • MinimumMegawatt
  • MaximumMegawatt
  • Dollars

These column names should be self-explanatory to current users of these data but if you have any questions, please post to the bottom of this discussion. Within the next day or two, production files will begin being published at the locations noted above, ahead of the 4 Dec go-live date.

Frequency keeping dispatched offers

The FK dispatched offer files will be published daily in a CSV format and will be named YYYYMMDD_FKDispatchedOffers.csv. The daily CSV files will be collated in yearly folders found in the file path ..\Wholesale\FrequencyKeeping\DispatchedOffers\.

The FK dispatched offer files will be found here on EMI datasets and here in the underlying Azure storage container. Unless stated otherwise, all comments made above regarding the FK offer files will apply to the FK dispatched offer files.

The column names and the order of the columns in the daily FK dispatched offer CSV files are as follows:

  • Participant
  • TradingDate
  • TradingPeriod
  • SubmissionDate
  • SubmissionTime
  • UTCSubmissionDate
  • UTCSubmissionTime
  • PointOfConnection
  • Unit
  • Block
  • Island
  • Megawatt
  • Dollars

Constrained on/off costs 

The FK constrained on/off cost files will remain in their current location for now and will continue to be published monthly. However, at some point in the not too distant future we will announce some revisions regarding these files on this forum.

Accessing Azure storage accounts

Finally, see here for some general discussion on accessing data from Azure storage containers.

 

 

Order by: Standard | Newest | Votes
NigelCByron posted this 4 weeks ago

Thanks for the heads up - HOWEVER it is unrealistic for you to assume that we can resource a change to our automated systems with only 6 days notice, and the new files not being available yet.

We request an extension of the cessation date to at least end of December 2018.

We need a full definition of the file column data definitions to go with the new files when they start.

We are also totally unhappy about the same file being re-issued with the same name on threee successive days - this is a total departure from accepted norms and makes it impossible for us to determine if we have processed the particular file. (we reject the same file name if we have already loaded it).  We request that you put a 'published' datestamp in the file name eg 20181204_FKOffers_20181120.csv,  20181204_FKOffers_20181121.csv,  20181204_FKOffers_20181122.csv

Nigel Byron, Mercury

 

NigelCByron posted this 4 weeks ago

For clarity "The published file for any given trading day will be published to the same file name for three days in a row, beginning the day after the trading day".

Will the file be the SAME or UPDATED on each publish ?

If the SAME we can handle it.

If UPDATED, either you or us will have to pre/ap pend a time stamp for versioning

Nigel Byron, Mercury

Phil Bishop posted this 4 weeks ago

All signs are that we're all go for a 4 Dec cutover to the new files in the new location, as described above.

The FinalOffers and DispatchedOffers currently published as daily XML files will start being published as CSV files in the same location as the XML files starting today or on Monday 26 Nov. These new CSV files will have the same structure - column names, file names, contents, etc - as the daily XML files and their publication will continue each day until 18 December 2018.

As described above, the publication of the daily XML files will cease on 4 Dec 2018.

Cheers

Phil

 

Phil Bishop posted this 15 hours ago

Further to the previous post in this discussion, the publishing of FinalOffers and DispatchedOffers as CSV files using the old XML file structure will continue until 29 January 2019. As per above, this was previously scheduled to stop on 18 December 2018.

On 29 January 2019, the entire Ancillary services folder and all of its contents will be decommissioned.

Phil

 

Close