"Binding Constraints" These rather sporadic data first manifested in a file of fixed columns, with peculiar texts to be recognised: "Grp Constrnt" might appear. The file finished with data for 15/8/2004, then a later data file arrived, as a spreadsheet, starting at the same point but continuing beyond 15/8/2004. On saving it to text format (.csv style), the previous "Grp Constrnt" now appeared as "Branch Group Constraint", and the other state, previously unmentioned, now appeared with the text "Branch Constraint". First format: Trading Date Trading Periods Issue 1/07/1998 36-37 COB_UTK1.1 on its 24MW limit 6/07/1998 17-24 COB_UTK1.1 on its 26MW limit 10/07/1998 38 TKU_WKM1.1 & 2.1 on their 200MW limits 17/07/1998 47-48 BEN_T2.L2 & T5.L5 on their 200MW limits 18/07/1998 47-48 BEN_T2.L2 & T5.L5 on their 200MW limits Second format: "Trading Date","Trading Periods","Type","Description" 01/07/1998,"36-37","Branch Constraint","COB_UTK1.1 on its 24MW limit" 06/07/1998,"17-24","Branch Constraint","COB_UTK1.1 on its 26MW limit" 10/07/1998,"38","Branch Constraint","TKU_WKM1.1 & 2.1 on their 200MW limits" 17/07/1998,"47-48","Branch Constraint","BEN_T2.L2 & T5.L5 on their 200MW limits" 18/07/1998,"47-48","Branch Constraint","BEN_T2.L2 & T5.L5 on their 200MW limits" 18/07/1998,"36-39","Branch Constraint","TKU_WKM1.1 & 2.1 on their 200MW limits" Since the second lot of data encompassed the first (and apparently with the same values, except for the text changes), reading the first format is abandoned in favour of the second. Notice that the second field is a list of half-hour ranges, so the use of quoted strings is for once useful. The first format lacked such quotes and the first-order commas between the fields. On feeding the file to Gnash, bizarre run names result, and for the moment, they are left as they stand. However, the supplied texts are declared not to be a list of names but a specification of names. With handwaving, it was noted that the sequence "TKU_WKM1.1 & 2.1" does not mean the name "TKU_WKM1.1 & 2.1" that incorporates spaces and an ampersand. Instead, it is to be parsed as a specification of two names, "TKU_WKM1.1" and "TKU_WKM2.1" without the tedious repetition of the initial part, both of which are to receive the specified value for the date and list of half-hour numbers previously specified. These two names have much better manners. There being two names not one is further confirmed by grammar and the usage of "their" rather than "its". Accordingly, the scan of the name specification proceeds by first accepting nameish text, such as "TKU_WKM1.1". Subsequently (passing any spaces), if a comma or an & are encountered then a new name is sought, and if the length of the new name is less than the established name, the trailing text of the established name is replaced by the new name (now thought of as a suffix replacement) to produce the second name. Thus, "TKU_WKM2.1" Otherwise the new name becomes the second name. And then (passing any spaces) a third name is sought should a comma or & be encountered afresh. Should the third field contain "Branch Group Constraint", the names are prefixed by "Group.". Whee! Actual data files proved to contain many grammatical errors, some of which might be data errors instead of data-entry errors but I haven't queried them. See file Gnash.Trail.txt for details of the reading. So the file with the second lot of data is given the name bc1998m07d01-2005m04d30.csv This file turned out to have three occurrences of "AVI WTK.1" but seventeen of "AVI_WTK.1", so this looks like a mistype. Likewise "BEN T2.L2","BEN HAY2.1","INV ROX1.1","INV ROX2.1","SFD TMN.1","ROX T10.M10","LIV WTK.1","CYD TWZ2.2 Conversely, "BEN_ T2.L2" Inversely, omitted space in 14/12/2001,"15-17, 21-25, 27-36","Branch Constraint","ARI_HAM1.1 & 2.1on 26MW limit" 16/12/2001,"18, 20-21, 23-25, 27-28, 40, 42","Branch Constraint","ARI_HAM1.1 & 2.1on 26MW limit" 17/12/2001,"25, 27-41","Branch Constraint","TKU_WKM1.1 & 2.1on their 158MW limits" 18/01/2002,20,"Branch Constraint","ARI_HAM1.1 & 2.1on their 26MW limits" Forgetting to use _ 02/12/2003,"9-12","Branch Group Constraint","MHO T3 reverse flow on 10 MW limit" And despite the usage of &, ten occasions when "and" was diligently written out instead. And on further scrutiny, mention of hh145 (for 9/6/2001) was noticed by Gnash, that looks as if it should be a hh45. This demonstrates that these data have been hand-entered, and, the file has never been checked by having a computer programme read it with suspicion. Additional files have arrived, with the longest names yet. "m" means that "macros" were supplied, and "l" that there were "links" to other files, which were of course inaccessible. Supplied Reformatted. FinalPricing_BindingConstraints_20050501_20050531.xls bc2005m05d01.csv FinalPricing_BindingConstraints_20050601_20050630.xls bc2005m06d01.csv l FinalPricing_BindingConstraints_20051201_20051231.xls bc2005m12d01.csv Then silence settled in on this front: nothing arrived for months, nor did anyone seem to want these data. But in fact there had been occasional arrivals, that were being stashed quietly. Then one day a file search via Agent Ransack found a whole bunch of files, as follows: "Sheet" name. ml FinalPricing-BindingConstraints-20061001-20061031.xls bc2006m10d01.csv "cum_type4_constraints" The original file name uses hyphens, not underlines, so first in order... FinalPricing_BindingConstraints_20050701_20050731.xls bc2005m07d01.csv "Current Format" A "New Format" sheet, even more bloated and not used again, removed. m FinalPricing_BindingConstraints_20050801_20050831.xls bc2005m08d01.csv "Sheet1" FinalPricing_BindingConstraints_20050901_20050930.xls bc2005m09d01.csv "Sheet1" Sheets "Sheet2" and "Sheet3" removed, being empty. ml FinalPricing_BindingConstraints_20051001_20051031.xls bc2005m10d01.csv "cum_type4_constraints" And file Binding-Constraints-sum-Jan-Apr-2006.zip, which contained FinalPricing_BindingConstraints_20060101_20060131.xls bc2006m01d01.csv "cum_type4_constraints" FinalPricing_BindingConstraints_20060201_20060228.xls bc2006m02d01.csv "cum_type4_constraints" FinalPricing_BindingConstraints_20060301_20060331.xls bc2006m03d00.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20060401_20060430.xls bc2006m04d01.csv "cum_type4_constraints" Three "HAM_HLY_1_110kV_ SPLIT_S_O_2" changed to "HAM_HLY_1_110kV_SPLIT_S_O_2" And Binding-Constraint-Sum-May-Sep-2006.zip, the singular form, yet containing more than one item, ml FinalPricing_BindingConstraints_20060501_20060531.xls bc2006m05d01.csv "cum_type4_constraints" m FinalPricing_BindingConstraints_20060601_20060630.xls bc2006m06d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20060701_20060731.xls bc2006m07d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20060801_20060831.xls bc2006m08d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20060901_20060930.xls bc2006m09d01.csv "cum_type4_constraints" Mz. Amy takes over from Mz. Nicole and now some files start appearing as per the declared protocol: FinalPricing_BindingConstraints_20061001_20061031.xls bc2006m10d01.csv "cum_type4_constraints" m FinalPricing_BindingConstraints_20061101_20061130.xls bc2006m11d01.csv "cum_type4_constraints" FinalPricing_BindingConstraints_20070101_20070131.xls bc2007m01d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20061201_20061231.xls bc2006m12d01.csv "cum_type4_constraints" And after a few queries, m FinalPricing_BindingConstraints_20070301_20070331.xls bc2007m03d01.csv "cum_type4_constraints" Monthly arrivals resume... m FinalPricing-BindingConstraints-20070201-20070228.xls bc2007m02d01.csv "cum_type4_constraints" Mislaid, then found via Jenny complaining to Brian, who had a sporadic e-mail collection, forwarded at times from Bruce. ml FinalPricing_BindingConstraints_20051101_20051130.xls bc2005m11d01.csv "cum_type4_constraints" Encouraged by this, a prowl through Brian's acumulated emails from Bruce found... ml FinalPricing_BindingConstraints_20070401_20070430.xls bc2007m04d01.csv "cum_type4_constraints" Back to monthly arrivals... m FinalPricing_BindingConstraints_20070501_20070531.xls bc2007m05d01.csv "cum_type4_constraints" m FinalPricing_BindingConstraints_20070601_20070630.xls bc2007m06d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20070701_20070731.xls bc2007m07d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20070801_20070831.xls bc2007m08d01.csv "cum_type4_constraints" ml FinalPricing_BindingConstraints_20070901_20070930.xls bc2007m09d01.csv "cum_type4_constraints" FinalPricing_BindingConstraints_20071001_20071031.xls bc2007m10d01.csv "Sheet1" m FinalPricing_BindingConstraints_20071101_20071130.xls bc2007m11d01.csv "cum_type4_constraints" Changed two "ISL_LIV_1 _O_1" to "ISL_LIV_1_O_1". l FinalPricing_BindingConstraints_20071201_20071231.xls bc2007m12d01.csv "Electricity Commission _2_" A new layout, requiring more hand editing: remove redundants statements before the heading, remove apostrophes from the start of each date field so that a date is recognised and then converted to dd/mm/yyyy form. A query led to the provision of file Binding Constraint Detail.xls which reverted to the prior usage, though with startlingly large type. Writing this as a .csv file gave the same result as my edited version, except that it failed to have "limit" plural where agreement was needed as per proper grammar. m Binding Constraint Detail(2).xls bc2008m01d01.csv "Electricity Commission" - the (2) because the name of the supplied file is not distinct from the previous month's name. After changing the format of the date column, all seemed well. A second "sheet" bore "Macro1" of unclear purpose. Binding Constraint Detail FEB08.xls bc2008m02d01.csv "Electricity Commission" Binding Constraint Mar08.xls bc2008m03d01.csv "Electricity Commission" m Binding Constraint Detail.xls bc2008m04d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. m Binding Constraint Detail_fixed.xls bc2008m05d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding-Constraint-Detail-fixed.xls deviant format. bc2008m06d01.csv "Transpower" With a second "sheet" called Macro1, of unclear purpose. Reformatted, with guesswork. The original is now in BadData/bc2008m06d01.xls Binding-Constraint-Detail-0708.xls bc2008m07d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Aug08.xls bc2008m08d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. m Binding Constraint Data.xls bc2008m09d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. m Binding Constraint Detail_fixed.xls bc2008m10d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2008m11d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. m Binding Constraint Detail_EC_Dec08.xls bc2008m12d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_Jan09.xls bc2009m01d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2009m02d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_March09.xls bc2009m03d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2009m04d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2009m05d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2009m06d01.csv "Electricity Commission", very large text, no second "sheet". Binding Constraint Detail_fixed.xls bc2009m07d01.csv "Electricity Commission" With a second "sheet" called Macro1, of unclear purpose. Binding Constraint Detail_fixed.xls bc2009m08d01.csv "Sheet 1". The first column is given the name "Day", not "Trading Date" as required, and has the wrong date order. Binding Constraint Detail_fixed.xls bc2009m09d01.csv "Sep09". The first column is given the name "Day", not "Trading Date" as required, and changed the span form to "(10-48) rather than "10-48" as established. Binding Constraint Detail_EC_Oct09.xls bc2009m10d01.csv "Sheet 1" Supplied in column order 3,4,2,1. Binding Constraint November09.xls bc2009m11d01.csv "Electricity Commission" Supplied in the right order, but with extra large text, spurious heading and footing lines, and not the recognised column headings. EC_DEC.XLS bc2009m12d01.csv "Sheet 1" Supplied in column order 2,4,3,1. For this format, the agreed and recognised form for the column header is "Trading Date", not "Date". binding_constraint_201002.xls bc2010m01d01.csv "Sheet 1" Data for January, but supplied in February so obviously, month two. Column order 2,4,3,1 again, with column headings "C_3", "C_1", "C_2", and "E263793" with dates in month/day/year order and with a spurious time of 00:00:00. binding_constraint_201002.xls bc2010m02d01.csv "Sheet 1" Data for February, but supplied in March so obviously, month two again. Column order 2,4,3,1 again, with column headings "TRADING_PERIODS", "DESCRIPTION", "TYPE", "ACTUAL_DATE" with dates again in month/day/year order and with a spurious time of 00:00:00. binding_constraint_201003.xls bc2010m03d01.csv "Sheet 1" Column order 2,4,3,1 again, with column headings "TRADING_PERIODS", "DESCRIPTION", "TYPE", "ACTUAL_DATE" with dates in day/month/year order this time. Sheet2 and Sheet3 were blank. Data hiccoughs (see Gnash.Trail.txt) unresponded to by the data supplier. binding_constraint_201004.csv bc2010m04d01.csv At last a text file, but, in column order 4,1,3,2 this time, with column headings TRADING_PERIODS,DESCRIPTION,TYPE,ACTUAL_DATE bc201005.csv bc2010m05d01.csv Again a text file, but another column order 1,2,4,3, with column headings Trading_date,TRADING_PERIODS,DESCRIPTION,TYPE and many fields no longer within quotes. binding_constraints_201006.csv bc2010m06d01.csv Again a text file, and the same column order 1,2,4,3, with column headings supplied as ACTUAL_DATE,TRADING_PERIODS,DESCRIPTION,TYPE bc2010m07d01.csv bc2010m07d01.csv After a great deal of repetition, the agreed format remanifests. attachment.csv bc2010m08d01.csv And continues into another month! Though with a rather different file name. attachment.csv bc2010m09d01.csv attachment.csv bc2010m10d01.csv Supplied in order 4,1,3,2 and with the wrong format dates and the wrong column names... Thanks to an initiative from Brian Kirtlan, these data were to be supplied in a new protocol while the existing protocol would remain. This failed, and reminders have got nowhere. And then suddenly, on Tuesday the third of May, a surge from the disrupted protocol thanks to an ad-hoc conversion from the new protocol... bc_20101101_20110228.csv bc2010m11d01_2011m02d28.csv A text file, but supplied in order 4 1 3 2 and the wrong column names as well. The usual. And additional files arrive, continuing the demonstration of the inability to stick with a file name scheme or data layout and now introducing even more bizarre names. With no explanation. BindingConstraints20110403.csv bc2011m03d01.csv Wrong header, order 1 2 4 3 bc_201104.csv bc2011m04d01.csv Wrong header, order 4 1 3 2 BindingConstraints_20110510_094935.csv bc2011m04again.csv Wrong header, order 1 2 4 3 BindingConstraints_20110607_095135.csv bc2011m05d01.csv Wrong header, order 1 2 4 3 BindingConstraints_20110711.csv bc2011m06d01.csv Wrong header, order 1 2 4 3 BindingConstraints_20110907.csv bc2011m08d01.csv Wrong header, order 1 2 4 3 BindingConstraints20111007.csv bc2011m09d01.csv Wrong header, order 1 2 4 3 BindingConstraints20111111.csv bc2011m10d01.csv Wrong header, order 1 2 4 3 BindingConstraints20120115.csv bc2011m12d01.csv Wrong header, order 1 2 4 3