ReadMe.text 21 October 2013. IMPORTANT NOTE : This is the final copy of the centralised dataset published by the Authority in this format. It contains data up to September 2013. See the Authority's website for access to more current data. The Authority’s Market Brief, published each Tuesday and available from the Authority’s website, will be used to make announcements in the coming months regarding arrangements for all data published by the Authority. Centralised Dataset September 2013 Electricity Authority Level 7, ASB Bank Tower 2 Hunter Street P.O. Box 10041 Wellington Email enquiries to info@ea.govt.nz Table of contents 1. Disclaimer and copyright statement 2. Installation instructions 3. Dataset contents a. Half hourly data b. Hydrology c. Network configuration data 4. More information 1. Disclaimer and copyright statement This Centralised Dataset (CDS) has been compiled by the Electricity Authority as part of its responsibilities under the Electricity Industry Participation Code 2010. Content has been compiled from raw data, which has been translated and published in a form suitable for use in industry planning processes. Although every care has been taken, some errors may have been introduced during translation of the raw data. The Authority, accordingly, accepts no liability for any loss that may result from reliance on CDS information. The Authority has used raw data provided to it by others. Although, under the Code, participants are required to use reasonable endeavours to provide accurate data, reliability is variable, and some of the historical data may be inaccurate to some extent. The Authority and participants who provided data accept no liability for any loss resulting from reliance on any CDS information. Some participants claim copyright over content they have provided and this is denoted by the copyright symbol (©). 2. Installation instructions The Centralised Dataset is provided in a compressed form for use in a Windows operating environment. The files have been compressed using an open source compression utility "7-Zip". A copy of the 7-Zip application has been included in the root directory. Run the '7z920.exe' file to install 7-ZIP. The file manager 7zFM.exe application can be run from the 7-Zip directory. By default the 7-Zip directory will have installed into your Program Files directory if you are running XP or Vista. Two compressed files have been provided: Source.7z contains the original 9000 source files used in compiling the Dataset; and CentralisedDataset.7z contains the various files required to access the contents of the Centralised Dataset. Installation of the Dataset files simply requires the unzipping of the CentralisedDataset.7z file into any directory on the users local or network drive. It is recommended that the .7z file be copied to a local hard drive before unzipping. Note: the unzipping process will take some time due to the size of the files. 3. Dataset contents The Dataset is broken into three main areas. a. Half hourly data b. Hydrology c. Network configuration data a. Half hourly data The data is held in the Gnash database contained in the HalfHourly directory. This area contains half hourly active and reactive meter data, final prices, bids and offers, and binding constraints data. Copies of the various the raw data files can be found in the 'Source' zip file. Binding constraints data is still provided in a separate database. To access this dataset use the command 'GRASP [filename_including_full_path]' in Gnash to switch the target database from the default database to the constraints database (the filename in this case is in the constraints folder called 'BigBag.Gnash', the path will varying depending on the directory Gnash has been installed into). Due to a number of process issues associated with the changeover in reconciliation manager provider in 2008, data for embedded generation plant smaller than 10MW in size is not available in the Dataset for any half-hourly periods after 1 April 2008. As a result, calculations that include the smaller embedded generators should be considered to be incomplete for periods after that date (for example, the gross demand calculations included in the TXSLosses_GrossDemand.txt file are invalid post April 2008). This version of Gnash includes a number of data series listing nominal generating capacity over time for the larger generating stations. The series start in 1888 when electricity was first produced for public supply in New Zealand. The capacity figures are from public sources and are only intended to give an approximate indication of capacity. The series containing capacity data start with the prefix 'Nominal.' -- MySQL-- A MySQL database containing load/generation, final prices, offers, and constraints data has been included in the CDS in the 'HalfHourly\MySQL' directory. This allows access using a SQL interface, which some may find more convenient. The MySQL database also includes: - summary tables (such as monthly average prices, and regional load totals). - mapping tables (which can be used to convert the nodal identifiers in the price and metering data tables to standard GXP labels, e.g. 'GEN_Hydro_Waikato_Waipapa_Price' -> 'WPA2201'). - tables which map generator metering data to generator name and information. - tables containing half-hourly HVDC transfer data and instantaneous reserve prices. The MySQL version of the Halfhourly Database may differ from the original Gnash data in some respects; for example, some missing or misleading data have been removed from the MySQL version. Where the two systems differ, the original Gnash system should be considered to be authoritative. -- Cleared instantaneous reserves quantities -- Half hourly cleared instantaneous reserves quantities, aggregated at an island level, have been included in the Dataset. The data has been extracted from EMS's EM6 data service, and in this version of the Dataset is provided in a raw .csv format. See the ClearedReserveQuantities sub-directory in the HalfHourly directory. -- Reserve offers -- Reserves offers have also been included. These are also in raw .csv format in this Dataset version. See the ReservesOffers sub-directory in the HalfHourly directory. b. Hydrology This area includes hydrological flow data, lake level history data, and lake storage history data. -- Flows -- This section consists of the "Generator SPECTRA Update - Issue 8" (Spectra Report 2010 (Final Draft).pdf) prepared by Opus International Consulting, which contains hydrological flow data by catchment to the end of June 2008. This Spectra Update is in the same format as the previous ones that were published in the Centralised Dataset. It includes two more years of flow data, various corrections and improvements to the data, and a report describing the information provided. The report contains maps and diagrams of hydro schemes, which were not included in previous versions. Error reports and corrections are invited - please provide feedback to jack.mcconchie@opus.co.nz, copied to matthew.keir@ea.govt.nz. The Authority thanks Genesis Energy, Meridian Energy, Mighty River Power, Contact Energy, Trustpower, Todd Energy, the King Country Power Company, and Pioneer Generation for supporting the project and providing data. The Authority also thanks the National Institute of Water and Atmospheric Research (NIWA) for supplying data from the Water Resources Archive (funded by PGSF). Also included in this area is a note 'Inflow description.doc' on the use of these inflow sequences in SDDP modelling, a summarised monthly flow dataset, denominated in GWh rather than cumecs for users' convenience, and flow files in SDDP format. -- Lake levels -- This section consists of the 'Lake Level History' prepared by Opus International Consulting, which contains historical lake level data to the end of December 2008. The report provides a brief history of the fluctuations in the levels of the major lakes used for hydropower generation in New Zealand. The lakes covered are Ohau, Tekapo, Pukaki, Te Anau, Manapouri, Hawea, Wanaka, Wakatipu, Taupo and Waikaremoana. The report describes the recording equipment used to measure levels and provides some guidance on how to interpret the data. The report is accompanied by a dataset which includes average daily lake levels for the entire length of record available. These data have been obtained from the Power Archive, which is maintained by OPUS International under contract to the various power generation companies. The Authority thanks Genesis Energy, Mighty River Power, Meridian Energy, and Contact Energy for supporting the project and providing data. -- Lake storage -- This section consists of the 'Lake Generation Potential History' prepared by Opus International Consulting, which contains historical lake storage (GWh) data to the end of June 2010. The report provides a brief history of the fluctuations in the storage of the major lakes used for hydropower generation in New Zealand. The lakes covered are Ohau, Tekapo, Pukaki, Te Anau, Manapouri, Hawea, Wanaka, Wakatipu, Taupo and Waikaremoana. The report is a sequel to the 'Lake Level History' - it was felt that converting the lake levels (masl) to hydro generation potential (GWh) would add value for some users. The report is accompanied by a dataset of lake storages (GWh) over Jan 1990 - Jun 2010. These are based on lake level data from the Power Archive, which is maintained by OPUS International under contract to the various power generation companies. A second dataset provides lake storage in volumetric terms (million cubic metres). The Authority thanks Genesis Energy, Mighty River Power, Meridian Energy, and Contact Energy for supporting the project and providing data. We also acknowledge the use of data from the National Hydrometric Archive administered by NIWA. c. Network configuration data This area contains PSS/E and DigSilent system load flow model files, lines diagrams, bus design fault levels, asset outage information, circuit breaker ratings,generation station data and transmission line locations. The line diagrams have been provided by Transpower in a PDF format. Please check the ReadMe.txt file in the directory containing the line diagrams if you intend to use them. The circuit breaker rating information, asset outage information, and generation station data has been provided in Excel spreadsheet format. A new directory 'NetworkConfiguration\Demand' has been added as of April 2010. This directory contains a datasheet providing information about GXPs/GIPs where there is demand, including: - creation date (for relatively new nodes) - location - lines company - ICP counts by retailer - offtake statistics - plots of demand at the node - local embedded generators - local direct-connect customers - some comments on major loads, and - a breakdown between CBD, residential, commercial/industrial and agricultural load. 4. More information --Tutorial-- A tutorial has been provided to give a basic understanding of the Gnash application and its commands. The tutorial 'CDSDataExtractionTutorial.pdf' can be found in the 'HalfHourly' directory. --Quick reference-- A quick reference to commands 'Quick reference.pdf' can be found in the 'HalfHourly' directory. --Comments— A number of corrections have been made to historical data provided by industry participants. The data contained in this version of the Dataset should be considered as being 'correct' if there are conflicts between data from this Dataset and a prior version.