Daylight savings adjustment - 10/1997 to 9/2008

  • 57 Views
  • Last post 3 weeks ago
LJHS posted this 3 weeks ago

Hi,

I've been looking at the all of the Generation Metered Data available and finding some gaps on the daylight savings (DLS) days.

The description of the Generation Metered Data states the start date should have 46 trading periods and the end date should have 50 trading periods. This is apparent in the data for 2009 onwards. However, with data prior to 2009:

  • On the end dates (e.g. 15/03/1998), there are only 48 trading periods of data, and
  • On the start dates (e.g. 4/10/1998), there are 46 trading period as there should be but gaps at TP5 and TP6 corresponding to the hour that is skipped.

Is there an available explanation for this?

Sorry if this issue has been discussed previously!

Cheers

Order by: Standard | Newest | Votes
Phil Bishop posted this 3 weeks ago

I'll take a look into this over the next day or two and will update this response if I discover anything of note. In particular, I'll check to see if what has been published on EMI datasets is identical to what we have in our data warehouse. But I'm going to predict now that my update won't change anything.

The Market Analytics team at the Authority inherited the Authority's data warehouse in mid-2013. The decision was made at that time to get the business (i.e. the users of the data such as ourselves) to take over the governance and management of the data we collect, store, use and publish from the IT team. One of the first things we set about doing was establishing a specification for all of the data we received. The so-called metered data had no specification; that remains the situation today. That said, we know where it comes from and what it is supposed to represent.

However, what was actually received and processed, and how it was processed, back in the years you mention might be difficult for us to do anything about now. I'd be shocked if we could even locate the source files. Moreover, we've been planning for quite some time now to replace the wholesale volume data that we publish with a series that we derive from reconciliation data sources going back to about 1 Jan 2005. Unfortunately, we haven't got to that task yet but hope to knock it off in 2019. When we do this, it will replace the published metered data series, although obviously not for the years you mention.

BTW, I assume you've seen this discussion on daylight saving.

Cheers

Phil

LJHS posted this 3 weeks ago

Hi Phil,

Thank you very much!

Cheers

Luke

Phil Bishop posted this 3 weeks ago

Hi Luke

Just to be quite clear, you looking at files in https://www.emi.ea.govt.nz/Wholesale/Datasets/Generation/Generation_MD/, right?

The earliest we have data in the table from where this Generation_MD gets published from is 1 Jan 2003. So that means I'm going to have to do some digging to find where the Generation_MD files from Aug 1997 until Dec 2002 came from.

Cheers

Phil

 

Close