Understanding the CaseID field

  • 1K Views
  • Last post 03 April 2017
  • Discussion is solved
dwaterworth posted this 23 September 2016

Hi,

I'm trying to understand how the SPD caseID in the raw spdsolved files (http://www.emi.ea.govt.nz/Datasets/Wholesale/Final_pricing/Raw_case_files/2016) relates to the run types - i.e. interim / provisional / final.

i.e. file MSS_311112016081200937_0X.zip

contains file MSS_311112016081200937_0X.0_01-SEP-2016_00_00_0.SPDSOLVED

This file contains the data below. I don't see any way of working out what this is for other than it's interval 01-SEP-2016 00:00

The caseID appears to contain a date but it doesn't seem to relate to the interval or the solution generation date?

Regards

Dave

C,SPD,SOLUTION,SPD V39.2.0.001, Built at 2016-06-21 04:08:14, Solution generated at 2016/09/02,08:05:34

C,SPD solution to case 01Sep2016.1.nzx.311112016081200937 for uploading to market database

I,SOLUTION,CASE,1.0,CASEID,INTERVAL,TIMEPERIODLENGTH,SPDVERSION,NONPHYSICALLOSSSTATUS,SYSTEMLOSSES,ROUNDPOWERACTIVE,HVDCBIPOLEREMOVED

D,SOLUTION,CASE,1.0,311112016081200937,01-SEP-2016 00:00,30,V39.2.0.001,0,    92.469,0,0

 

 

Final pricing
Order by: Standard | Newest | Votes
Phil Bishop posted this 23 September 2016

Hi Dave

The short answer to your first question is that the case files are completely silent as to the status of final prices, i.e. the case file does not tell us if final prices are provisional, interim, or final. See also the second to last paragraph in our post on SPD case files.

Determination of the provisional-interim-final status is undertaken by the pricing manager in accordance with Part 13 of the Code. See in particular the provisions in Subpart 4. Moreover, this determination by the pricing manager is independent of the generation of an SPD case file. In other words, SPD is not used to produce interim or final prices per se. Rather, a final pricing case is generated and the pricing manager subsequently ascertains the status of those prices and publishes them.

Incidentally, we use the final pricing case files to produce GDX files for use by vSPD. We publish the GDX file as soon as we produce it, and then, subsequently, once we learn the status, we append the GDX file name with _P, _I, or _F to indicate the status. But of course, those GDX files contain no prices - they contain only the input data required to run the model and thereby generate prices.

I'm not sure I understand the last part of your question. Do you want to know the time a case file is generated?

Cheers

Phil

 

 

dwaterworth posted this 23 September 2016

Thanks Phil

The last part was more a statement - the caseid seems to contains a date 311112016081200937 but the interval is 01-SEP-2016 00:00 and the run time is 2016/09/02,08:05:34 so I wasn't sure how they're related but I see from the link that the date embedded in the caseid is utc which explains my confusion

From reading the link it appears there's nothing in the filename / caseid that allows any way of sorting either as the last 3 digits are random? I'd have to look at the runtime in the header of each file?

Regards

Dave

Phil Bishop posted this 23 September 2016

 Yep, those last three digits of the SPD case file name are random. And yes, look in the control file inside the zipped case file to see the generation time.

dwaterworth posted this 03 April 2017

 

 

Hi Phil,

Back on this, you said

"Incidentally, we use the final pricing case files to produce GDX files for use by vSPD. We publish the GDX file as soon as we produce it, and then, subsequently, once we learn the status, we append the GDX file name with _P, _I, or _F to indicate the status. But of course, those GDX files contain no prices - they contain only the input data required to run the model and thereby generate prices."

How do you learn the status of the case files? Do you publish the relationship anywhere - i.e. what casefile was used to create a corresponding GDX file?

Regards

Dave

 

Phil Bishop posted this 03 April 2017

We learn the status (provisional, interim or final) by having the pricing manager send us a file each afternoon that tells us the current status of all final pricing SPD case files they've sent us in the last 24 hours. This is a bit clunky as it requires us to then update records in our data warehouse and change GDX file names some time after those files were first created. We do not publish this status information other than indirectly, i.e. by appending the status flag to the GDX file name.

You can easily tell from what case file a GDX file was created because the case name is the first symbol loaded into the GDX file, i.e. you need to open the GDX file to ascertain the case name from whence it came. Of course the case name doesn't tell you the status.

Having said all of that, we are in the process of obtaining all prices directly from the WITS SFTP server rather than extracting them from SPD case files ourselves. Although WITS doesn't publish the case name associated with published prices, it does publish the run time of the SPD case. We can then match up the run times for the specific case types and assign a case name to every price record. Once we get done with all that, we can then push the price status through to the reports on EMI. All of this will occur almost immediately after the prices have been published on WITS (but that may still be some time - minutes or an hour or two rather than several hours - after we receive and process the SPD case file).

I hope this helps.

  • Liked by
  • dwaterworth
dwaterworth posted this 03 April 2017

Hi  Phil

Thanks, yes that helps. I can easily extract the status from the GDX filename and the CaseName set from inside the GDX file and create an cross reference to the SPD case files!

Regards

Dave

 

Categories
Find discussions by tag
See all
This week's high earners
Popular discussions

Our 1273 members have posted 1170 times in 327 discussions

Latest members:

94 users are currently online.

EA tel: + 64 4 460 8860 EMI email: emi@ea.govt.nz Legal information Glossary

© 2024 Copyright material on this site is protected by copyright owned by the Electricity Authority or its licensors.