Initial investigation into the London blackout revel the cause.

http://news.bbc.co.uk/1/hi/england/london/3096098.stm

Wrong equipment caused blackout

An incorrect piece of equipment caused the power cut which plunged much of
London and the South East into darkness, according to a preliminary report
by the National Grid.
The report says it was the largest power cut from the National Grid for 10
years and the company expressed "its deep regret".
About 1,600 trains were stopped and 60% of the London Underground network
was closed, stranding 250,000 during the evening rush hour.
Electricity was restored in 41 minutes and the outage affected 410,000 homes
and businesses, according to the report.
The company says a failed transformer at the Hurst substation near Bexley in
Kent on 28 August, triggered the outage.
Maintenance work
National Grid switched the transformer out of service at 1820 BST and power
was re-routed through other circuits so it could still flow.
But within seven seconds another fault occurred which stopped the flow of
electricity on a 275,000-volt underground cable between the New Cross and
Wimbledon substations.
There are usually two circuits operating between the substations but one was
not functioning because of maintenance work.
According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with the
supply, because of the action taken to compensate for the defective
transformer.
The report adds that this occurred because "an incorrect protection relay
was installed when old equipment was replaced in 2001".
National Grid has so far checked 9,000 out of 45,000 other similar pieces of
equipment across the network and found them to be working as normal.
Blackout trends
The remaining equipment will be checked within four weeks.
But it added: "The incident on the areas of south London was exacerbated by
the loss of supply to the [London] underground and railway services."
It comes as the Department of Trade and Industry and energy regulator Ofgem
said they will appoint specialist consultants to carry out initial
fact-finding investigation into the blackout.
The inquiry will also cover blackouts in the West Midlands last Friday and
power failures in the last five years to see if any trends can be
identified.

In message , Cast_Iron
writes
According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with the
supply, because of the action taken to compensate for the defective
transformer.

Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.
--
Roland Perry

On Wed, 10 Sep 2003 14:03:12 +0100, Roland Perry wrote:

In message , Cast_Iron
writes
According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with the
supply, because of the action taken to compensate for the defective
transformer.

Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error.

"According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with
the supply, because of the action taken to compensate for the defective
transformer. The report adds that this occurred because "an incorrect
protection relay was installed when old equipment was replaced in 2001".

The second 'fault' seems to have been a protection malfunction.

The key phrase is perfectly clear and obvious, at least to anyone with a
knowledge of the industry - "an incorrect protection relay was installed
when old equipment was replaced in 2001".

On Wed, 10 Sep 2003 15:11:48 +0100, Wanderer
wrote:
On Wed, 10 Sep 2003 14:03:12 +0100, Roland Perry wrote:
In message , Cast_Iron
writes
According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with the
supply, because of the action taken to compensate for the defective
transformer.

Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error.
...
The key phrase is perfectly clear and obvious, at least to anyone with a
knowledge of the industry - "an incorrect protection relay was installed
when old equipment was replaced in 2001".

But triggered by the first fault, no? If the first fault hadn't
occured, neither would the second one (at that particular time).

I guess it all depeneds on how you want to define the chain of
cause-and-effect...

Sam
--
Sam Holloway, Cambridge
www.samholloway.co.uk

On Wed, 10 Sep 2003 15:17:25 +0100, Sam Holloway wrote:

On Wed, 10 Sep 2003 15:11:48 +0100, Wanderer wrote:
On Wed, 10 Sep 2003 14:03:12 +0100, Roland Perry wrote:
In message , Cast_Iron
writes
According to the report, the second fault occurred because automatic
protection equipment shut off the power thinking there was a fault with the
supply, because of the action taken to compensate for the defective
transformer.


Finally! Confirmation that the two faults were indeed related. As some of
us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error. ... The key phrase is perfectly clear and
obvious, at least to anyone with a knowledge of the industry - "an
incorrect protection relay was installed when old equipment was replaced in
2001".


But triggered by the first fault, no? If the first fault hadn't occured,
neither would the second one (at that particular time).


I guess it all depeneds on how you want to define the chain of
cause-and-effect...

Or whether a member of the public would consider a protection
malfunction to be a genuine fault, in quite the same way that they would
seeing a bloody great hole blown in a cable or a transformer on fire.

Certainly for the purposes of the National Fault reporting scheme it
would be classed as a fault, but the implications in this particular
case are quite far-reaching. If, in the final analysis it can be proven
beyond all reasonable doubt that the outage happened because of
negligence, which seems highly likely, then National Grid could be
liable for substantial damages. Might be time to dispose of the
shares...

In message , Wanderer
writes
Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error.

The *reason* for the second outage might have been human error [1], but
the second outage was most definitely *caused* (ie triggered) by the
first. It wasn't an "unrelated" incident.

[1] Although you could argue that it was really caused by failure to
implement a procedure that would have noticed and rectified that human
error.
--
Roland Perry

On Wed, 10 Sep 2003 15:21:57 +0100, Roland Perry wrote:

In message , Wanderer
writes
Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error.

The *reason* for the second outage might have been human error [1], but
the second outage was most definitely *caused* (ie triggered) by the
first. It wasn't an "unrelated" incident.

I'll concede that point, but the discussions that were going on at the
time were much more of the nature that one 'bang' caused another 'bang'
because the circuits were overloaded. It seems *that* wasn't the case.

[1] Although you could argue that it was really caused by failure to
implement a procedure that would have noticed and rectified that human
error.

Yes. Knowing the very strict and thorough commisioning and testing
procedures that once were enforced, one is left wondering whether
corners have been cut since privatisation.

In message , Wanderer
writes
The *reason* for the second outage might have been human error [1], but
the second outage was most definitely *caused* (ie triggered) by the
first. It wasn't an "unrelated" incident.

I'll concede that point, but the discussions that were going on at the
time were much more of the nature that one 'bang' caused another 'bang'
because the circuits were overloaded. It seems *that* wasn't the case.

No, the debate was about whether the second 'bang' was related to the
first in *any* way shape or form. Lots of people insisted it wasn't.

One possible failure mode would be that the first 'bang' caused the
second circuit to overload, and cut out.

If the actual failure mode is the first 'bang' causing the second
circuit to *think* it's overloaded, and cut out, then the difference is
entirely academic to those people sat in the ensuing darkness.
--
Roland Perry

Wanderer wrote in message . ..
On Wed, 10 Sep 2003 15:21:57 +0100, Roland Perry wrote:

In message , Wanderer
writes
Finally! Confirmation that the two faults were indeed related. As some
of us insisted (against solid opposition) from the start.

Not quite true. You conveniently snipped the *real* reason, which was
almost certainly human error.

The *reason* for the second outage might have been human error [1], but
the second outage was most definitely *caused* (ie triggered) by the
first. It wasn't an "unrelated" incident.

On any interconnected system, the National Grid being a prime example,
events are by definition related. This incident was caused by a
classic very simple installation 'cock-up' made 2 years ago and not
discovered or exposed since; system design, configuration and incident
response were correct and supplies would have been securely maintained
otherwise.

The second outage (Wimbledon - New Cross 275kV circuit No.2) which
caused the loss of supply was a 'phantom' overload trip, there was no
equipment fault and total load was 33% of line rating. Overload
detection relays, connected via 1,200:1 transformer, are rated 1Amp or
5Amp with a wide ranging multiplier, 5A was specified but 1A installed
and through all the checks nobody spotted the error. This meant that
tripping was initiated (time delay involved, irrelevant here) at 23%
of rating and 20% of correct overload setting, so in the past 2 years
this line can't have loaded above that level even at peak winter
demand indicating the generous level of redunancy provided.

It's accepted that when the system is being reconfigured, as it was
after the first, potentially explosive, fault warning; there can be 5
to 10 mins 'switching time' when supply is via a single circuit. The
risk of a second fault on that circuit during this time is estimated
at 1:40,000 though that doesn't apply here with the system wrongly
set-up. The engineers planning the switching could not have known the
line would trip.

[1] Although you could argue that it was really caused by failure to
implement a procedure that would have noticed and rectified that human
error.

Yes. Knowing the very strict and thorough commisioning and testing
procedures that once were enforced, one is left wondering whether
corners have been cut since privatisation.

They were carried out but I guess those engineers assumed they had
been handed over correctly installed equipment and never thought to
check the relay rating, pity they aren't brightly colour coded.
Simulating real life faults is simply not possible or wise.

Reading the full report my impression is of a well run and financed
operation not obviously damaged by privatisation with London area
investment running at £50m pa. Ironically the duplicate circuit,
Wimbledon - New Cross No.1, is out from 1/7 to 28/9 for upgrade and
refurb.

Lessons have been, painfully, learnt and the relay error is not likely
to happen again. All 45,000 are being checked (none found after
9,000). I would say the system is now secure, building extra lines
would be very expensive. The LT loss of supply is rather laid on EDF,
the London distribution operator, who chose to operate their Wimbledon
132kV Substation in a way that resulted in LT and others losing supply
when the Grid was able to maintain the full load without interuption,
and should have reconnected LT in a couple of mins at most. I get the
impression there was only one engineer in their Control who didn't
know what hit him...

Source:
http://image.guardian.co.uk/sys-file...on28082003.pdf

Roland Perry said:

The *reason* for the second outage might have been human error
[1], but the second outage was most definitely *caused* (ie
triggered) by the first. It wasn't an "unrelated" incident.

[1] Although you could argue that it was really caused by failure
to implement a procedure that would have noticed and rectified
that human error.

So much for redundancy as a method of protection. Doesn't anybody
test these things?

--
=============================================

Chris Game chrisgame@!yahoo!dotcodotuk
=============================================