BYZANTINE FAILURE MODES

Air Transat A330-200 Fuel Loss

 (24 Aug 01)

Some of the content of this risks forum article (below) by Prof Dr Peter Ladkin may cast some light - see the highlit sections.

....particularly if the C of G cruise-control fuel cross-transfer adjustments cut in after a precautionary shut-down of an engine.

Extracted From a prior incident (A340) -see below

"Action to correct CG control errors
[the center of gravity is adjusted in cruise by moving fuel around, to give
efficient cruise performance] is contained in a software (only) upgrade to
standard 6.1. [...]

and:

"They `dialed in' the ILS using a "back-up method",
and while doing so received an ECAM warning of low fuel state and
instructions to open crossfeeds (airplanes like this have many tanks and
fuel is pumped around between them). The warning reoccurred and readings
indicated they had some 2 tonnes (2000kgs) of fuel less than expected."

Stir in one fuel management system computer connector with a non-standard pin-out, and hey presto you have an uncommanded fuel-dump as soon as your Centre of Gravity cruise-control cuts in.

So perhaps it was dumb software -  that was dumb simply because it was NOT intolerant of a hardware fault producing a totally unacceptable outcome (i.e. pouring all the onboard fuel load over the side). Which pin-out? Probably the one that's designed to stop dumb pilots dumping all their fuel.

It's not as if non-standard pin-outs hasn't been seen in an Airbus before.....

In Particular:  see at  X   below

Airbus software

Subject: RISKS DIGEST 16.92



RISKS-LIST: RISKS-FORUM Digest  Thursday 16 March 1995  Volume 16 : Issue 92

Date: 15 Mar 1995 15:56:26 +0000

From: Les Hatton <les_hatton@prqa.co.uk>

Subject: A340 shenanigans



The BBC news at 08.30 reported a slight problem which occurred on the

morning of 15 Mar 1995 with the ultra high-tech, packed full of software

and therefore utterly wonderful Airbus A340.



Apparently on the final part of its approach to Gatwick, both the pilots

screens went blank, to be replaced by a polite little message saying "Please

wait ...".  Somewhat unnerved, the pilots requested that the plane turn

left, but it turned right instead.  They then tried to get it to adopt a 3

degree approach to the runway, but it chose a 9 degree plummet instead.  At

this point, from the report, they appeared to gain manual control and landed

safely.  It is not clear who will pick up the dry-cleaning bill.



Vis a vis this sort of thing, I was at a talk recently, given by the CAA (UK

Civil Aviation Authority), at which it was stated that in the past

generation of civil aircraft, most of the software problems were reported in

the Flight Management System.  Not surprisingly, this was the most complex

part of the aircraft software system.  Not any more it isn't.  During the

talk, it was also admitted that the newer generation of aircraft such as the

A340, other software systems including active systems were "at least as

complicated".  So what next ?

 I suppose it follows on nicely from the story in the October 1994 Risks

whereby a Japanese Air Force T-4 jet trainer ejected one of its pilots.

Perhaps it didn't like him.  :-)



Les Hatton, Ph.D. C.Eng, Director of Research & Engineering, Programming

Research Ltd, England   les_hatton@prqa.co.uk    +44 (0) 1 932 888 080

End of RISKS-FORUM Digest 16.92 

Date: Wed, 22 Mar 95 14:56:20 PST

From: RISKS Forum <risks@csl.sri.com>

Subject: RISKS DIGEST 16.96



RISKS-LIST: RISKS-FORUM Digest  Weds 22 March 1995  Volume 16 : Issue 96

Date: Wed, 22 Mar 1995 19:50:56 +0100

From: ladkin@techfak.uni-bielefeld.de

Subject: Re: A340 incident at Heathrow (Hatton, RISKS-16.92)



Les Hatton reports in RISKS-16.92 on an incident involving an Airbus A340 on

a trip from Japan to London Heathrow. The aircraft is one of the

A320/330/340 (also A319/321) family of Airbuses, some of whose primary

flight control systems are computer-controlled (that is, the pilots

control-stick movements are input to a computer that guides the control

systems).  The A340 is a very long-haul aircraft, capable of flying the very

longest routes without refueling (it holds the world record for length of

route flown without refuelling, for normally-equipped civil transport

aircraft).  The incident is of greatest significance for RISKS readers

because it is the first time that an accident report on an A320/330/340

series aircraft specifically cites software and hardware reliability as the

main problem.



The incident concerned a Virgin A340 at Heathrow on 19 September 1994 (cf.

the incorrect info reported by Hatton). A short article by Christian Wolmar

appeared in The Independent newspaper, one of Britain's major dailies, on

Wednesday March 15, 1995. After talking to Christian, I obtained a copy of

AAIB (Britain's Air Accident Investigation Board) Bulletin No. 3/95, the

report on the incident.



I'll spoil the tale for everyone by giving the punchlines first: the

description of the problem areas identified during the incident, and the

report's conclusion, the `Safety Recommendation 95-1'.  [during quotes, my

editorial comments and elisions are contained within square parentheses such

as these. PBL.]



[begin quote]



Problem Areas



The AAIB identified and investigated the following problem areas: RTF [radio

communication] phraseology; ATC [air traffic control] vectors and ILS

[instrument landing system] performance; fuel quantity indications; double

Flight Management Guidance System (FMGS) failure and aircraft type

certification.



[end quote]



The radio phraseology can be ignored by RISKS readers.  ATC vector problems

had to do with capturing the `glideslope', the radio beam angled up into the

sky from the end of the runway, down which an aircraft flies in order to

land, under instrument conditions.  The aircraft at one point encountered a

`false glideslope' at about 9\deg at 5 miles from touchdown and 4,800ft

altitude caused by a `shallow sidelobe' of the ILS. Such problems are known

(glideslope is assured for between 1.35\deg and 5.25\deg in the UK) and the

airplane wouldn't have got there had it not been vectored there by ATC - but

one hastens to add that normally this is not a problem. Just in this

case....see below. All the other problems concern the on-board computers,

and the AAIB has written to the JAA (Joint Aviation Authority, which does

for Europe what the FAA does for the USA) to determine if the JAA was "aware

of some of the more significant shortcomings of the A340's fuel and flight

management systems before the type certificate was granted".



[begin quote]



Safety Recommendation.



It is recommended that the reliability of the Airbus A340 FMGS and the fuel

management system should be reviewed to ensure that modified software and

hardware required to achieve a significant improvement in reliability should

be introduced as quickly as possible and the subsequent system performance

closely monitored.



[end quote]



Here is why they made this recommendation:



[begin quote]



Autopilot and Flight Director heading performance



The reason for the wrong response of the autopilot and one flight director

to the left turn demand was a software error [...] [This error] was known to

Airbus Industrie and corrective measures for this and several other software

deficiencies were contained in [...] standard L-5 that has been issued and

incorporated in most A340s on the UK register.



Fuel Quantity indications



In July 1994 Airbus Industrie issued an Operations Engineering Bulletin on

the subject of fuel quantity indication. [The bulletin gives detailed

descriptions of anomalies, when they occur, and how to take them into

account.]  Action pending by Airbus Industrie to correct fuel quantity

errors involves the installation of five additional fuel probes in each

inner tank and software standard 6.0.X Action to correct CG control errors

[the center of gravity is adjusted in cruise by moving fuel around, to give

efficient cruise performance] is contained in a software (only) upgrade to

standard 6.1. [...]



FMGS double failures 



After landing the aircraft's Central Maintenance System had logged a fault

in No 2 FMGEC. This was removed and sent to France for data extraction and

fault analysis. No fault was found within the hardware and a comparable

software fault could not be reproduced on the test bench.  Nevertheless, the

BITE data dump showed that at 1435 hours the No 2 FMGEC had detected a CLASS

1 HARD failure within itself and a simultaneous fault within FMGEC No 1. The

investigation was complicated by the involvement of several sub-contractors

in the manufacture of the FMGEC and its database.



[...]



Airbus Industrie were aware of the double FMGS failure mode that had first

emerged on the A320 series aircraft. On A320/330/340 aircraft, each FMGEC is

linked to its own set of peripherals and inertial reference system. Both

FMGECs achieve their own computations and exchange data through a cross talk

bus. One FMGEC is declared as master and the other as slave; the master

FMGEC is related to the engaged autopilot. Some data in the slave FMGEC is

synchronised to the master but all data inserted on any MCDU is transferred

to both FMGECs and to all peripherals.



According to Airbus Industrie, there are several ways in which the exchange

of data and/or a problem in one computer can affect the other computer.

Often the computers reset themselves after a few seconds but occasionally a

fault results in repetitive resets or attempts to resynchronise. The fifth

reset relatches the computer, which will not recover without a power

interrupt. Reset breakers for manual power interrupts are on the flight deck

overhead panel. Dual resets occur when both FMGECs encounter failures at the

same time.  They generally occur after a pilot entry that involves use of

the navigation database or to an event synchronised between both flight

management systems. Latched double failures usually occur if pilots

successively perform three inputs that cause a reset, or if an `impossible'

computation of predictions occurs.



Airbus Industrie have succeeded in radically reducing the frequency of

double FMGS failures on the A320 series aircraft; they are also addressing

the problem on the A330 and A340 series. [...]



[end quote]



I point out that so-called Byzantine failure modes and algorithms for

avoiding them in distributed systems were first identified and studied in

the 70s by my former SRI colleagues Lamport, Shostak and Pease under the

auspices of the SIFT project, and since then by many, many others. As for

other such topics in computer science, this was regarded as `theory' for a

few years. I could hazard a guess that many aerospace engineers still have

not heard about this area. The account above of the problems with the

A320/330/340 master/slave FMGECs may give RISKS readers reason to inform

themselves about the `theory' of how all this anomalous and possibly

dangerous behavior can be avoided in the first place. Many papers in the

January 1994 issue of the Proceedings of the IEEE speak about the current

state of the art.



Finally, the story.



On the ground in Tokyo before departure, one FCMC indicated numerous faults.

It is accepted procedure to depart with only one FCMC operative - they did

so, and followed the appropriate procedures for calculating fuel with only

one FCMC. Early in cruise, the map symbology on the commander's EFIS

(Electronic Flight Instrument System) disappeared and his MDCU

(Multifunction Control and Display Unit) ceased calculating.  They slaved

both off the copilot's DMC (Display Management Computer).  The EFIS is the

pretty screen in front of the pilot that tells him/her which way is up, which

way is forward, and which way, as well as how fast (in three dimensions),

and where he/she is.  The MDCU displays flight plan info, and a bunch of

other things. About an hour later, they found that the commander's EFIS had

restored. Logical indications from the No.2 FCMC were also restored later by

`resetting the computer'.



Now come a few things which one should really think about hard.



Getting close to home (Heathrow), the copilot tuned in the Lambourne VOR (a

radio beacon) manually, to ensure that the EFIS displays were still

accurate. They were cleared to fly direct to the beacon, but a few miles

east, "the commander's EFIS map display symbology froze and lost all

computed data [..]. His MDCU displayed the message `PLEASE WAIT' together

with a page normally seen only when loading in data before flight. He was

unable to obtain any other display. At [roughly the same time], the

[copilot's] EFIS and MCDU exhibited identical behavior."



Notice that not all flight control info was lost from the EFIS - they could

still fly the airplane. X They `dialed in' the ILS using a "back-up method",

and while doing so received an ECAM warning of low fuel state and

instructions to open crossfeeds (airplanes like this have many tanks and

fuel is pumped around between them). The warning reoccurred and readings

indicated they had some 2 tonnes (2000kgs) of fuel less than expected.  They

discussed traffic density with ATC, and eventually declared an emergency in

order to get priority for landing.



They had the autopilot automatically capture the ILS, which is when they hit

the sidelobe. `The glidepath bar moved rapidly down the ILS display before

moving rapidly up once again; the autopilot's attempt to follow the

glidepath resulting in unusually high pitch rates and so the autopilot was

disconnected.' The commander informed the tower they were having problems

with the glideslope and requested an SRA (Surveillance Radar Approach). In

an SRA, the controller talks the airplane down localiser and glideslope

continuously, by giving an uninterrupted stream of position information

relative to the glideslope/localiser pair. It's very impressive.



Aircraft are on `final approach' when they're lined up with the runway

centerline and heading down the glideslope to land. (This should not be

confused with when the flight attendant says `final approach' to the

passengers, which is usually when the aircraft is even before initial

approach phase.) The approach was for Runway 09 Right at LHR (the `09' means

that it's pointing roughly 90\deg to North). The crew were on the SRA, being

vectored (given magnetic headings to fly) to intercept the final approach

course. They were flying a heading of 180\deg and were commanded to change

to 130\deg.  When they turned the heading selector knob on the autopilot,

both commander's and copilot's heading `bugs' moved correctly (that's an

indication on the directional gyroscope of which heading you want the

autopilot to fly to and hold - I had a lower-tech version on my Piper

Archer), but the flight director bars went in opposite directions and the

airplane followed the false movement of the copilot's bar, and turned right

instead of left. At this stage, the copilot disconnected autopilot and

flight directors and flew the plane `manually' (see first paragraph for why

this is not quite an accurate expression for these aircraft).



They landed without further incident; after taxiing in and shutting down,

the fuel indications recovered; and thankfully everyone lived happily ever

after.



Peter Ladkin

ftp.csl.sri.com/pub/{reports|pvs}

http://www.csl.sri.com/cgi-bin/WebObjects/CSL.woa/wa/Publications 

http://commons.somewhere.com/rre/1995/Airbus.software.html

Airliner Lands Without Power in Azores, 11 Hurt

By Ian Simpson

LISBON (Reuters) - A Canadian Air Transat Airbus with suspected fuel trouble made an emergency landing, gliding in without engine power, on Portugal's Azores islands on Friday, slightly injuring 11 passengers, authorities and company officials said.

Transat Flight 236 carrying 291 passengers and 13 crew from Toronto to Lisbon overnight was prepared to put down in the Atlantic Ocean but managed to land at 5:46 a.m. at the Lajes airport on the Azores' Terceira Island, Paulo Lagarto, a spokesman for Portugal's civil aviation agency, told Reuters.

The Azores are a group of nine Portuguese islands about 900 miles west of the mainland.

A spokesman for the island's hospital said nine people were treated for minor injuries. In addition, one Portuguese woman was admitted to hospital for treatment of a fractured kneecap and another for a cracked vertebrae, but neither injury was consider serious.

Some of the airliner's tires ruptured during the landing and its undercarriage was damaged.

The Airbus A330-200, which is powered by two Rolls-Royce engines, ``told the control tower about 20 minutes before landing that it probably would have to put down in the sea since it was losing fuel,'' he said.

Jose Angeja, the airport director, said authorities suspected the aircraft had trouble with its fuel system.

``There are eyewitnesses and even passengers that say that when it landed, it had its engines off,'' Angeja said.

Portuguese authorities are investigating the incident.

Montreal-based Air Transat, Canada's biggest charter airline, confirmed the jet landed without power.

``I cannot confirm whether it was a fuel or motor problem or something else, but at the moment of landing the engines were out,'' Michel Lemay, a company spokesman, told Reuters.

``I don't for how many minutes that was the case, but in effect the aircraft glided the last moments of the flight.''

LOSS OF FUEL, ENGINE POWER Passenger Maria de Fatima told Portuguese television that passengers realized about an hour before landing that the plane was in trouble. ``We didn't know what was going to happen,'' she said, weeping. ``The captain was very brave.''

Jason Srancoz, a Canadian passenger from Toronto, called the pilot a hero for landing the plane safely.

``It was very scary. We were prepared to do a water crash,'' he told the Canadian Broadcasting Corp.

Worried officials with Canada's Transport Ministry in Ottawa did not ground Air Transat, but limited the airline's operating authority on its three Airbus 330s, requiring the company to fly the jets closer to airports on long-haul routes.

Instead of flying in a more direct line to Europe, which meant Air Transat's Airbus 330s could be up to two hours away from the nearest airport, the jets must now follow a more northerly route near Greenland or Iceland to ensure they are no more than one hour from an airport, Canadian officials said.

``We're very, very concerned about this,'' Art LaFlamme, director general of civil aviation at Transport Canada, told Reuters.

``To my knowledge this is the first instance of this occurring in Canada or even worldwide,'' he added.

LaFlamme was referring to indications that the airliner continued to lose fuel during the flight despite design specifications for the Airbus which allow the flight crew to shut down a troubled engine and reroute or conserve fuel while using the remaining functional engine.

``The system is designed so that they can fly on one engine...so the continued loss of fuel is probably the most perplexing situation here that has to be explained,'' he said.

Transport Canada also plans to audit Air Transat's flight and maintenance operations to ensure they meet regulatory requirements.

Air Transat's Lemay said the airliner had been in service since 1999.

Another Air Transat jet was due to arrive in the Azores late on Friday at the airport at Ponta Delgada. The passengers from Flight 236 were being transported by boat to Ponta Delgada where they would board the plane for a short flight to Lisbon.

Air Transat is part of travel services company Transat A.T. Inc. Its shares were down 30 Canadian cents at C$10.65 in Toronto on Friday.

    MONTREAL, Aug. 25 /CNW/ - Air Transat confirms that passengers aboard

flight TS 236 bound for Lisbon, which made an emergency landing in Terceira in

the Azores at 6:46 a.m. GMT on August 24, arrived in Lisbon at 4:17 a.m. local

time (11:17 p.m. EST, August 24), on August 25, aboard a Lockheed L1011

airplane dispatched to the site by the Company. Air Transat personnel and

professionals, who were deployed to provide on-site psychological support,

accompanied the passengers. Other resources were made available in Lisbon to

further assist passengers.

    Given that the Terceira airport is closed, passengers were transported to

the Ponta Delgada airport, where the airplane awaited them. Seven passengers,

two of whom suffered fractures at the time of the evacuation, chose to remain

in Terceira and be transported to Lisbon once the runway reopens. In total,

flight TS 236 transported 291 passengers and 13 crew members. The majority of

passengers were Canadian of Portuguese descent.

    "We have experienced a very serious incident, one whose fortunate outcome

was largely due to the competence and professionalism of the pilots and crew.

Despite the stress and anxiety, aircraft evacuation was completed within 90

seconds, an undertaking that requires clear direction, exceptional execution

and speed, especially an aircraft of this size (362 seats). Many passengers

have acknowledged the crew and Air Transat wishes to join them in this

recognition. We apologize for any inconvenience caused by this incident,"

declared Denis Jacob, Air Transat President and Chief Executive Officer.

    The Company has already advised the passengers of the TS 236 flight in

question that their airfare will be fully refunded.



    THE TS 236 FLIGHT INCIDENT AND INVESTIGATION

    Flight TS 236 left Toronto on time at 8:10 p.m. EST on August 23. While

the Airbus A330-200, operating since 1999, was flying at a cruising altitude

of 39,000 feet and was around 30 minutes from the Azores, a technical problem

caused a significant loss of fuel. The captain then took appropriate measures.

He decided to direct the aircraft towards Terceira, the closest airport, and

had passengers prepare for a possible ditching - a procedure required for all

emergencies over water. Both aircraft engines ceased functioning several

minutes prior to landing. Eight of the ten landing gear wheels burst on

touchdown. No fire or smoke was reported in the cabin. Emergency evacuation

procedures were rapidly and systematically implemented. The hypothesis of

incorrect fuelling in Toronto has been definitely ruled out.

    Forty-nine-year-old Captain Robert Piché, who has thirty years of airline

experience and has been employed by Air Transat for close to five years, was

the pilot in command. Twenty-eight-year-old First Officer Dick Dejager has

been employed by Air Transat for three years. The crew is currently in

Terceira and will soon return.

    "It seems clear that our pilots did an outstanding job. Because of their

professionalism, flying skills and training, our pilots handled a most

difficult situation. We commend their achievement, as well as that of the

entire crew. Evacuating such an aircraft in 90 seconds required quick and

decisive action on the part of cabin personnel, with the goal being a rapid

evacuation of the aircraft. We salute the entire crew," added Mr. Jacob.

    The Portuguese authorities will carry out an investigation on the cause

and circumstances surrounding the incident. The Transportation Safety Board of

Canada and Transport Canada will participate in the investigation. Air Transat

is fully collaborating, as are Airbus and Rolls Royce. Air Transat will also

carry out its own internal investigation, which is already underway.



    MEASURES TAKEN BY TRANSPORT CANADA

    Air Transat will fully abide by all measures undertaken and announced

yesterday (August 24) by Transport Canada. Following the incident, Transport

Canada announced an ETOP (Extended Range Twin-Engine Operation) audit of Air

Transat's three Airbus-A330 aircraft. Until further notice, these three

aircraft will have to conform to the basic standard for two-engine aircraft.



    THE MEASURES ANNOUNCED BY TRANSPORT CANADA HAVE A MINOR IMPACT ON THE DAY

TO DAY OPERATIONS OF AIR TRANSAT'S 3 AIRBUS A330S AND NO IMPACT ON ITS OTHER

21 AIRCRAFT, NOR DOES IT IMPACT ITS LICENSE, NOR ITS TRANSATLANTIC OPERATIONS.



    OTHER FLIGHTS

    Air Transat advises its customers that all flights are operating as

scheduled. As always, it is recommended that passengers confirm their

departure by contacting the Company 48 hours in advance at 1-877-TRANSAT.



    SAFETY AT AIR TRANSAT

    Air Transat, a Canadian carrier, currently operates 24 aircraft, three of

which are Airbus A330s (two Airbus A330-200s and one Airbus A330-300), as well

as 4 Airbus A310-300s, 6 Lockheed L1011-500s, 7 Lockheed 1011-100s and 4

Boeing 757-200s. Air Transat travels to 90 destinations in 25 countries. The

Company transported 3.5 million passengers last year and 23 million passengers

since its start in 1987. In 14 years, the Company has not had any accidents

causing injury and the incident involving flight TS 236 was its third

emergency evacuation since the company's inception. The two-year-old Airbus

A330, which was involved in the emergency landing in Terceira, has never been

implicated in any other incident. The Company's safety record compares

favourably to all other Canadian airline carriers. Also, the Company's

aircraft maintenance and training programmes, including training in emergency

situations, strictly conform to Transport Canada standards.



    THE DAMAGED A330 AIRBUS

    Specialized Air Transat personnel, as well as representatives of the

authorities, Airbus and Rolls Royce, are on site in Terceira. Measures have

already been taken, with the full cooperation of Portuguese military

authorities who operate the base in Terceira, to move the aircraft and re-open

the runway, which is a priority now that our passengers have reached their

destination. The extent of landing gear repairs required for the aircraft to

return to Air Transat's Montreal base, as well as the delays required for the

authorities to conduct the on-site investigation of the aircraft, are not yet

known.





-30-

For further information: Seychelle Harding, Communications Specialist, 

Air Transat, (450) 476-1011, ext. 3069

Following some comments made at an AT Press Conference, it would appear that Portuguese mechanics are saying that there appears to have been a leak from the LP fuel system.

This would call into question the reason why or how fuel could still be leaked over the side in toto. For that to happen, the cross-ship valve would need to have been open or opened. (either by the system or by the crew). This could have been due to an illogical non-sequitur when switching from one emergency procedure in the QRH to another, i.e. a step that required the cross-ship valve to be opened in order to restart the "failed" engine.

This Pprune posting would seem to lead in the correct direction:

R. Cramden Just another number Member # 21591

posted 29 August 2001 03:31                Not trying to speculate but I believe there is a situation on 777 where you can get into trouble by blindly following checklists. A massive fuel leak will lead to low fuel on one side (obviously). After the fuel imbalance checklist, later will come the low fuel check which asks for either cross-feed valve to be opened. This then drains the other tank. Some of my details may not be absolutely precise, but have heard of this scenario in the sim. Posts: 3 | From: Dubai | Registered: Oct 2000  |  IP: Logged

http://www.pprune.org/cgibin/ultimatebb.cgi?ubb=get_topic&f=1&t=015188&p=3

TRICKY SYSTEMS need trickier crews

    Double Flame-out Checklist         Fuel Line had leaked following engine change

    Forced Landing Checklist            Air Transat Press Conference

to Hot off the PRESS