EMI datasets updated – NSP table, metering configurations, market structure and shape files

  • 445 Views
  • Last post 26 March 2018
Matthew Keir posted this 20 December 2017

We've made some changes to existing datasets on EMI. These changes are outlined below. The existing files and file structures will be maintained until after the data for January 2018 is published in mid-February. Users will have to migrate any automated processes to the new file names and locations before the February data is published in March. 

1. Network supply points table

Updated filename and location and the addition of region ID (one column) for network reporting regions to assist users when combining NSPs with other datasets that we publish. 

Obsolete file and location (to be disestablished mid-February 2018):

New file and location:

2. Distribution price category code added to metering configurations file

Some participants asked us to add distribution price category code to the metering configurations file that we publish. We agreed that this would help reduce barriers to entry for participants and third parties by helping parties setup billing processes or match consumers to retail plans.

Obsolete file and location (to be disestablished mid-February):

New file and location:

3. Rename of market structure files

This is simply a tweak to file names and location.

Obsolete file and location (to be disestablished mid-February 2018):

New file and location:

4. Network region shape files

Relocating these files to make them easier to find.

Obsolete file and location (to be disestablished mid-February 2018):

New file and location:

Order by: Standard | Newest | Votes
msouness posted this 18 January 2018

Hi Matthew,

Do these locations reflect the FTP file locations?

Matthew Keir posted this 18 January 2018

Yes, the FTP and datasets on EMI are the same underlying files and structure. I have only provided the old and new dataset URLs above. However, if you browse the FTP you'll see the locations are already there.

Alternatively, if you change the front part of the URLs above to the FTP equivalent, then the URLs above will work on the FTP server. For example: substitute "www.emi.ea.govt.nz/Wholesale/Datasets" for "ftp://emiftp.ea.govt.nz/Datasets/Wholesale/".

msouness posted this 18 January 2018

Filenames appear to have "_" in one, and no spaces in the other?

Matthew Keir posted this 18 January 2018

Hi Malcolm,

Sorry for the confusion. We are migrating to a camel case for file naming convention across our data warehouse including all files exported to datasets. This change will take some time and usually occur when we are doing something else with that piece of code. We'll advise of these changes through the forum and give a period of time for users to adjust their process as above.

Might you be looking at the old file still?

For example:

Obsolete file and locations (to be disestablished mid-February):

New file and locations:

Does that make sense?

Matthew Keir posted this 22 February 2018

Update...

The old files and locations have now been removed. However, some participants asked for more time to adjust processes around the NSP table. To accommodate these users we will keep the old location for the NSP table until mid-March.

The new NSP table location is:

Matthew Keir posted this 26 March 2018

The old NSP table files and location have now been removed.

Close