All users of the bids or offers data files that we publish on EMI Datasets should be aware that a replacement set of files has been published. The new files are available now at ..\Datasets\Wholesale\BidsAndOffers. Users should also be aware that Datasets on EMI is just a browser-based view of our FTP server, so this post applies also to bids and offers files obtained directly from the FTP server.
On 28 August 2017, the folder called ..\Datasets\Wholesale\Bids_and_offers and all of its contents will be deleted. Users that have scripts to automatically obtain and process these files will need to modify those scripts before 28 August 2017.
The Market Analytics team has been undertaking a revamp of our data warehouse. Among other changes, a common nomenclature and set of business rules is being applied across all datasets - this will be seen in the revised structure of the bids and offers files. The files to be decommissioned also contain some anomalies that the revamp corrects. For example, duplicate records, particularly those on the days that daylight savings ends, have been eliminated.
The remainder of this post presents the new column names as well as relates the new to the old.
Bids (new column names compared with the old - old in uppercase)
- PointOfConnection - PNODE_NAME
- Unit - PNODE_NAME
- OtherGridEntity - PNODE_NAME
- Trader - TRADER_NAME
- UTCSubmissionDate - SUBMISSION_DATE and SRC_COMIT_ID
- UTCSubmissionTime - SUBMISSION_DATE and SRC_COMIT_ID
- SubmissionOrder
- IsLatest - CURRENT_FLAG
- DispatchableYesNo
- TradingDate - TRADING_DATE
- TradingPeriod - TRADING_PERIOD
- Band - BAND
- Megawatt - MW
- DollarsPerMegawattHour - PRICE
BID_TYPE_DESCRIPTION and CONFORMING_BID_TYPE are columns from the old files that are not relevant and have no equivalent in the new files.
Offers (new column names compared with the old - old in uppercase)
- PointOfConnection - PNODE_NAME
- Unit - PNODE_NAME
- OtherGridEntity - PNODE_NAME
- Trader - TRADER
- ProductType - ORDER_TYPE and RESERVE_TYPE
- ProductClass - ORDER_TYPE and RESERVE_TYPE
- ProductSubClass - ORDER_TYPE and RESERVE_TYPE
- ProductCategory - ORDER_TYPE and RESERVE_TYPE
- UTCSubmissionDate - SUBMISSION_DATE
- UTCSubmissionTime - SUBMISSION_DATE
- SubmissionOrder
- IsLatest
- TradingDate - TRADING_DATE
- TradingPeriod - TRADING_PERIOD
- Band - BAND
- MaximumRampUpMegawattPerHour - MAX_RAMP_UP
- MaximumRampDownMegawattPerHour - MAX_RAMP_DOWN
- PartiallyLoadedSpinningReservePercent - PLSR_PERCENT
- MaximumOutputMegawatt - MAX_OUTPUT
- Megawatt - MW
- DollarsPerMegawattHour - PRICE
PNODE_TYPE is a column from the old files that is not relevant and has no equivalent in the new files.
Additional notes
The existing bids and offers data for the few months prior to 13 December 2012 have not been published in the new files as the data from this period is of poor quality and contains many anomalies that we are unable to resolve.
Submission dates have been provided in UTC format rather than some unspecified version of New Zealand time in order to conveniently deal with daylight saving.
SubmissionOrder provides an ordinal ranking of the submission order. For some purposes, this ordinal ranking is a more convenient means of determining order than parsing the submission time. IsLatest is a simple Y/N flag to indicate which submission is the final submission for any set of identifying keys.
The DispatchableYesNo column in the bids' files relates only to bids that are for demand that is to be dispatched by the system operator. In these cases, the trader may submit a bid and then quite late in the piece, certainly well after gate closure, decide that they no longer want the system operator to dispatch them. Rather than nullify all aspects of the previously submitted bid, the trader may simply change the dispatchable flag from yes to no. In other words, the status of this flag takes precedence over the price and quantity tranches that have previously been submitted.
The new files are arranged in subfolders by year, i.e. the monthly subfolders have been done away with.
In our revamped data warehouse, all data structures that denote a point on the grid or a network (e.g. field names such as pnode, node, GIP, GXP, POC, etc) are passed through a function and characterised as a:
- PointOfConnection (POC) - a varchar(7) of the form ABC1234, a
- Unit - a varchar of the form ABCx where x is generally a single-digit integer but very occasionally might be more than a single integer, and an
- OtherGridEntity (OGE) - a varchar of an arbitrary length but generally no more than about 10 characters.
The Other Grid Entity is something of a catch-all category to capture all point labels we encounter that don't fit the POC or Unit mould. Generally speaking, it is used to denote one end or the other, and maybe a flow direction, of the HVDC. There are no OGEs for either bids or offers.
Similarly, the myriad of product labels that we encounter in the data we receive has been classified in a four-level hierarchical product dimension as shown in the following table. Products are not relevant for bids. The product classification is mapped to the ORDER_TYPE and RESERVE_TYPE columns from the old offer tables in the table below.
