Children of the Magenta (Automation Paradox, pt. 1)

RM: This is 99% Invisible Roman Mars.

RM: On the evening of May 31st, 2009, 216 passengers 3 pilots, and 9 flight attendants boarded an Air France flight in Rio de Janeiro Brazil. The flight was headed across the Atlantic Ocean, to Paris.

KM: A warning, this story might not be a good one to listen to if you’re sitting on a plane right now.

RM: That’s producer Katie mingle letting you know that the closest exit could be behind you.

KM: The plane that took off that evening from Rio with an Airbus 330. The takeoff was unremarkable. The plane reached cruising altitude of 35,000 feet. The passengers read and watched movies and slept as they flew over the floor of the Atlantic Ocean passing the equator…

WL: There was a lot of thunderstorm activity, quite typical nothing really unusual about that.

KM: That’s William Langewiesche, he wrote an article for Vanity Fair about this flight and has written extensively about aviation. He also used to be a pilot.

WL: So I spent, many many thousands of hours in cockpits.

KM: Which means he’s particularly well suited to tell the story of Air France flight 447.

WL: It was a long flight. Um, so they had two co-pilots known as first officers and one captain.

RM: They would fly the plane in shifts, two of them flying while one slept.

WL: We know, that the flight proceeded normally for several hours.

KM: And then…

WL: Then with no communication to the ground or air traffic control, it suddenly disappeared.

RM: Some pieces of the plane and several bodies were found days later floating in the Atlantic. But it would be two more years before most of the wreckage was found deep in the ocean. All 228 people on board had died.

KM: When the wreckage was found they also recovered the cockpit voice recorder and the flight data recorders.

WL: And they were in good shape.

RM: The recordings were able to tell a story about how flight 447 ended up in the bottom of the Atlantic Ocean.

KM: And the story they told was about what happened when the automated system flying the plane suddenly shut off and the pilots were left surprised, confused, and ultimately unable to fly their own plane.

RM: Automation in flight has been around for a long time. The first so-called autopilot was invented by the Sperry Corporation in 1912 and it allowed the plane to fly straight and level without the pilot’s intervention.

KM: In the 1950s the autopilots got better.

WL: So that you could program an autopilot to follow a route, and not just to keep the wings level. This was making the pilot’s job a lot easier.

RM: But there was still a lot to think about in the cockpit of an airplane.

WL: Controlling the electric systems, controlling the hydraulic systems.

KM: And because of all this, most planes had three people in the cockpit. Two pilots, and an engineer.

WL: A flight engineer sat behind the pilots and they manipulated and managed these increasingly complex systems.

RM: But by the 1970s, all of those complex systems were also automated and flight engineers lost their necessity, and their jobs.

KM: By this time the jet engine had replaced the piston engine, making planes much more reliable. And studies were showing that most accidents were being caused not by mechanical error but by human error.

RM: And a French company called Airbus thought that they could make planes safer by letting automation carry an even bigger part of the burden of flying.

WL: Airbus was a leader. Airbus has always been quite radical in its design philosophy.

KM: Led by a guy named Bernard Ziegler, Airbus set out to design what they hoped would be the safest plane yet. A plane that even the worst pilots could fly with ease.

WL: Bernard Ziegler, he famously said that he was building an airplane that has concierge could fly.

RM: Ziegler’s plane not only had auto-pilot, it also had what’s called a fly-by-wire system.

KM: So, this can get a bit technical, but basically an autopilot just does what a pilot tells it to do. Fly-by-wire is a computer-based control system that can interpret what the pilot wants to do and then do it smoothly and safely. So if the pilot pulls back on his or her stick, the fly-by-wire system will say “Oh, I see that you want to pitch up and here, I’ll do it for you at just the right angle and rate.”

RM: Importantly, the fly-by-wire system will also protect you from getting into an aerodynamic stall. So in a car, a stall happens when the engine stops turning. But in a plane, stalling is different.

WL: Basically getting so slow that the wings no longer function correctly, and the airplane you know, loses altitude. Let’s put it that way.

KM: A stall in a plane can happen when the nose of the plane is pitched up at too steep an angle. A steep angle can cause the plane to lose lift and start to descend

RM: Stalling in a plane is not good. Stall for long enough and you will crash. But fly by wire automation makes it impossible to do as long as it’s on.

KM: Unlike autopilot, the fly-by-wire system cannot be turned on and off by the pilot.

WL: You can’t turn this automation off.

KM: But, here’s the thing…

WL: It can turn itself off.

RM: And that’s exactly what it did on May 31st, 2009 as 5447 flew through the night sky, over the Atlantic Ocean.

WL: At some point, having flown normally for a few hours, the captain had gone back to sleep.

KM: This meant the two co-pilots were in control of the plane.

WL: They encountered some weather. It wasn’t rough, they never hit significant turbulence.

KM: So far nothing out of the ordinary. And then…

WL: Suddenly they lost airspeed indications.

RM: A pressure probe on the outside of the plane had iced over. The automation could no longer tell how fast the plane was going.

KM: At this point autopilot disconnected.

WL: At the same time the fly-by-wire system degraded by one step.

RM: So the autopilot turned off and the fly-by-wire system shifted into a different mode. A mode that did not protect against aerodynamic stall.

KM: None of this was ideal, but it also it also wasn’t a reason to panic, according to William. The plane was still basically flying straight and steady.

WL: The airplane was not upset. Had they done nothing, they would have done exactly what they needed to do. Nothing.

KM: But that’s not what happened. The co-pilot in the right seat put his hand on his control stick a little joystick like thing to his right.

WL: And he pulled it back.

RM: Not just a little, he pulled it three-quarters of the way back.

WL: Why he did that is that a major question.

KM: Perhaps he was startled, or reacting to some turbulence. it’s important to note that he did not tell the other pilot in the cockpit, the guy in the left seat, what he had just done. If he had, maybe his co-pilot would have known the dangerous situation they were about to be in.

RM: In an Airbus, the two pilots have separate control sticks that move independently of each other. In other commercial airplanes, the controls would move in unison, so that you would actually feel the moves your copilot was making on their controls. This design variation may also have also have contributed to what was about to become a very bad situation. In any case, as the pilot in the right seat pulled back on his stick…

WL: The airplane pulled, put his nose up and it started to soar upward.

RM: When stall protections are in place, the plane won’t just warn you not to pitch it at such extreme angle, it physically will not let you do it. But remember the plane has shifted into a mode in which it wasn’t offering that underlying protection.

KM: As the angle got steeper and steeper, the air couldn’t flow smoothly across the wings and the plane began to stall.

RM: The plane was angled up trying to gain altitude, but it actually began to lose altitude.

KM: A warning began to sound. The plane couldn’t prevent a stall but it could still warn that a stall was happening.

WL: It said, “Stall, Stall, Stall”

KM: It would have sounded like this: “Stall Stall [ring], Stall Stall [ring] Stall Stall [ring]”

KM: And you’d think that if you heard this alarm someone would say “Hey, I think we might be stalling.” But this never happened.

RM: At one point, one of the copilots did say, “Get your nose down.”

WL: But he didn’t say “we’ve stalled.” and the man the right seat, who’s flying the airplane, the junior copilot, he would put the nose down a couple times then bring it right back up again.

RM: The co-pilots began to ring frantically for the captain to come into the cockpit. At one point where they say “[Bleep!] Where is he?”

WL: The airplane was shaking all kinds of alarms are going off…

KM: One minute and thirty-eight seconds after the episode started, the captain came into the cockpit and asked, “What’s going on here?”
RM: The co-pilots did not say, “we are in a stall.” instead one of them said “we’ve completely lost control of the airplane and we don’t understand anything. We’ve tried everything.”

KM: The captain had walked into a difficult scene to understand. Everyone was confused. Were they descending or climbing? At times they couldn’t seem to figure it out. Finally, they seem to realize they’re in a fast descent. Losing thousands of feet per minute. But why? They couldn’t figure out why.

WL: The captain never assessed correctly what was going on. He was obviously trying to, he wasn’t there to drink coffee or suck his thumb, but he was never able to figure out what was going on even though it should have been amply obvious what was going on.

KM: But for whatever reason, it wasn’t obvious to them what was going on. They didn’t know they were in a stall. If they’d realized it, Williams says the fix would have been clear. It’s basic, it’s pilot school 101.

WL: The recovery would have required them to put the nose down, power doesn’t matter, just get the nose down get a below the horizon regain flying speed and then pull out of the ensuing dive.

RM: The plane continued falling down at a jaw-dropping pace, losing several thousands of feet per minute. But since they started at 37,000 feet for a while there was still time for recovery.

WL: So for quite a long time, a normal crew would have been able to recover and at the very end, at around 16,000 feet the best crew in the world only probably would have been able to recover, and below that altitude, no crew would have been able to enable to recover because they simply didn’t have the space beneath them to execute the dive before hitting the water. But in any case, they never tried to do it. And so they rode this airplane down, expressing confusion the whole time and finally expressing certainty that they were going to die.

KM: Four minutes and twenty seconds after the incident started

WL: They pancaked into the water at a very high descent rate and of course it killed everyone instantly.

KM: Six years after the crash of flight 447 what’s clear is that the pilots didn’t understand what was happening to them. The big question is how? How could they have a computer yelling “Stall!” at them and not understand they were in a stall?

RM: There were various things that contributed to the crash of 447. But automation, which has overwhelmingly made airline travel safer also played a role in this accident.

WL: However much automation has helped the airline passenger by increasing safety, it has had some negative consequences. In this case, it’s quite clear that these pilots had had experience stripped away from them, for years.

RM: William actually did the math on this. The captain of the Air France flight had logged 346 hours of flying over the past 6 months. But of that time there were only about 4 hours in which he was actually in control of an airplane. Takeoffs and landings. The rest of the time, autopilot was flying the plane. That’s 4 hours in 6 months. The co-pilot’s would have had even less time at the controls.

KM: This is not to say pilots do nothing when autopilot is on. They still have an important role to play.

WL: What the good pilots do in flight, they watch the systems that handle navigation and communication, they occasionally program things in flight like path changes, they deal with anticipation of weather, they think about fuel management.

RM: Increasingly Williams says, pilots have become automation managers with fewer and fewer chances to actually fly the plane. He believes this lack of experience at the controls left the pilots of 447 unprepared to take over when the automation turned off.

WL: The pilots were hideously incompetent. Um, that’s, I think no one would disagree with that.

NS: So, I have certainly heard that position from other people before. I think we all have to be very careful none of us was there when this actually happened.

KM: That’s Nadine Sarter, a systems engineer at the University of Michigan who’s studied a lot of aviation accidents related to automation.

NS: My approach to things with different I guess. These pilots were in that airplane with the passengers and I’m sure they have every motivation to keep that flight safe and yet they didn’t. So so then I try to understand well, why did they not? Did they not have the right information? Did they not have the proper training?

RM: When one of the copilots hauled back on his stick he pitched the plane into an angle that eventually caused the stall. But it’s possible he didn’t understand that he was now flying in a different mode. One that would not regulate and smooth out his movements.

KM: This confusion about how the fly-by-wire system responds in different modes is referred to aptly as “Mode Confusion” and it’s come up in other accidents. The plane switches into a different mode and suddenly…

NS: Pilots find it difficult to understand what’s going on at that point. They do not exactly know whether the system is still taking care of the airplane or whether they are in charge now, they don’t necessarily understand whether all of that has disconnected or just parts of it have disconnected.

WL: A lot of what is happening is hidden from view from the pilots. It’s buried. When the airplane starts doing something that is unexpected and the pilot says, “Hey what’s up, now what’s it doing?” That’s a very very standard comment in cockpits today. “What’s it up to now?”

RM: The pilots of Air France flight 447 never quite say, “What’s it doing now?” but they say similar things.

KM: William wasn’t the only person to point out that “What’s it doing now?” is a commonly heard question in the cockpit. It seems to be almost a running joke in the industry. A joke that everyone agrees is a serious problem.

WV: What’s the most often asked ask question in our cockpit? “What is it doing now?” (laughter)

KM: That’s American Airlines captain Warren Vanderburgh addressing a group a group of pilots in 1997. People have been talking about this issue way before the crash of flight 447. It’s just that no one’s really figured out how to solve it yet.

WV: But you see, we have become what I call, Children of the Magenta.

RM: The Children of the Magenta are too dependent on the guiding magenta-colored lines on their screens.

WV: You know we think we have to have those magenta lines on the map, and that magenta bar that’s steering us toward that line.

RM: The Children of the Magenta are too dependent on automation in general.

WV: And if this is you I’m talkin’ about, do not be defensive. In the industry, we created you like this.

KM: Vanderburgh recommends in certain situations, to turn off the autopilot and fly the plane yourself.

WV: When you deem the situation is right to practice your skills, turn off your autopilot, and your autothrottles. Fly your planes. Maintain your skills.

KM: Nadine Sarter agrees that this could help. She also recommends using simulators to let pilots explore different kinds of emergency scenarios.

RM: And then, of course, there are people who think the problem can be solved with more and better automation. But this is a paradox of automation.

WL: Automation accommodates incompetence. In fact, it’s designed to do that.

RM: The Airbus was designed after all for Bernard Ziegler’s concierge to be able to fly.

WL: We appear to be locked into a cycle in which automation begets the erosion of skills or the lack of skills in the first place. And this then begets more automation.

KM: Nadine Sartor thinks it’s a mistake to throw more automation at the problem.

NS: The position I would adopt is one that has been termed human-centered automation, where we see what you need to do instead is make the automation that you have, make it smarter, but importantly make it what we call a team player. And part of that is to provide better feedback, part of it is to have the automation communicate more effectively with the pilot.

KM: No one I talk to is advocating for going back to the bad old days before automation.

NS: So automation in general, the fly-by-wire system all of those certainly do increase safety I would argue.

KM: And even though a lot of the accidents we see these days have to do with the interaction between pilots and automation, William says you have to remember..

WL: One of the reasons for that is that because of automation there hardly any other accidents.

KM: And as far as how we should feel about Air France flight 447 and other accidents involving automation, should they be seen as an acceptable loss in an industry that’s become mostly safer through automation?

WL: I think as a society, the answer to that is yes, um, you know for the end of it individuals is no such thing as an acceptable loss, alright. I mean this is a terrible tragedy, terrible. And for the pilots who died as well.

RM: Airbus planes are not any less safe or any more safe than any other plane on the market planes have accidents. But it should be said that they’re pretty safe too. The accident rate for air travel is very, very low. About 2.8 accidents for every 1 million departures and there are plenty of pilots out there that know how to take control in an emergency. So breathe deep everyone, just relax. It’s gonna be okay.

KM: It’s true there are lots of good pilots out there but in a far-off future we might be flying in pilotless planes.

WL: We’re a long way from that but the technology is definitely coming, and we will, I mean, this will happen it just makes a lot of sense. I don’t think anyone is sort of saying “Great, that’s going to be a wonderful thing.” Um, but I think reasonable people are also not tearing your hair out about it.

KM: Because by the time it happens the automation will be so good and so reliable that humans with all of our human emotions and human fallibility, the Children of the Magenta will really just be in the way.

99% invisible was produced this week by Katie Mingle with Sam Greenspan, Avery Trufleman, and me Roman Mars. We relied heavily on the reporting and expertise of William Langewiesche for this piece, so special thanks to him he went above and beyond for us. And if you want an even more in-depth a minute-by-minute analysis of the crash of flight 447, check out his piece in Vanity Fair it’s called, The Human Factor.

And on the next episode of 99% Invisible, we’re going to take this idea of the automation paradox and use it as a lens to view the next great design challenge in automation when we all potentially become the Children of the Magenta in our self-driving cars. It’s our first 2 part series, so stay tuned.

Now last week I mentioned that we have a new addition to the Radiotopia collective and I’m pleased to announce that Nate DiMeo’s The Memory Palace is doing the gang. When I was first thinking about starting 99% invisible Nate was one of the first people I called because I really loved what he was doing with his show featuring these poignant historical moments and I wanted to do something in the same spirit with stories about design. And so, more than any other program I can say if you love 99% Invisible you will love, love The Memory Palace. Nate & I are friends and he once recommended history book to me and on page 325 I noticed a couple sentences, just a couple of sentences that were a tiny, but really striking digression from the main narrative. And I’d like to think that I would have noticed the charm of its historical footnote on my own but the real reason it stood out to me it was because I knew it sparked an episode of The Memory Palace.

ND: Most Memory palace pieces start on page 325 of some book.

RM: That voice you just heard is Nate DiMeo of the memory palace has a special way of zeroing in on these moments in history that light up your imagination and hit you in the gut or make you smile.

ND: I’d noticed that there were these historical things that would jump out at you from long Ken Burns documentaries or that one moment in that historic home tour when that object on the mantelpiece like actually jumps out and says “wow, that’s amazing! Look at that beautiful thing.” and so often those like, I noticed that the things that move me where these focused things. They were recognizing the single object they were the incredible twist in someone’s story and I wanted to find out on some level if there was a way to cut to the chase, if there was a way to focus so tightly, it was almost just an aesthetic, artistic challenge.

Like, is there a way that I can replicate this feeling of being moved or having my head blown off, or having my heartbroken. This thing that I have once felt. Is there a way that I can get people to feel that same thing. Like, can I retell the story in such a way that they don’t have to take the whole historic home tour. Or they don’t have to sit through all 9 1/2 hours of this documentary series. Is there a way to kind of crystallize it and turn this sweeping historical story into a pop song.

RM: But The Memory Palace is never just a simple historical anecdote there’s always something more going on.

ND: Like there is a subject and then there is meaning. Like there are themes which are different than plot. On some level I’m not merely trying to tell a historical story, I’m trying to conjure it. Trying to have it sort of like, live in the listener’s mind for a bit and then melt away.

RM: I really like his show so much that I had a hard time selecting which episode to present so I asked Nate which episode really stood out to him that did all the things he was hoping to achieve in Memory Palace and he picked this episode, enjoy.

ND: This is the memory palace I’m Nate DiMeo. The world loved the World’s Fair. The Exposition Universelle in Paris in 1889 just killed. The exhibits in the grounds were unparalleled and impeccable. And at the center of it all was this audacious steel structure that managed to be imposing, and elegant in the tallest thing on Earth and unmistakably French all at once. Paris had the Eiffel Tower, and the men planning the next fair the Columbian Exposition in Chicago just a couple of years away needed something that good. Something Eiffel Tower good. And that wasn’t easy to find. The proposals ran from the ridiculous, like a structure that would soar more than 1000 feet above the Land of Lincoln made entirely out of stacked logs and topped with a replica of Abe’s boyhood home, to the extra ridiculous. Something so tall that visitors would take an elevator to the top of a slide that they’d ride down until it dropped them off in New York or San Francisco. The fair’s organizers were panicking. They demanded that America’s designers and engineers step up. And a man named George Ferris stepped forward. When his ferris wheel was completed that summer it rose 264 feet above the ground, which was a lot shorter than the Eiffel Tower, but whatever, that thing didn’t even move. The idea that something so massive but that looks so fragile like a bicycle wheel whose spokes look too thin to keep a bike up, was thrilling, and pretty terrifying. Despite what the engineers said, despite the math and laws of physics, despite the many assurances of the fair’s organizers, there were people who were sure that this ludicrous machine was going to be a disaster; that people going to die. Not only was there no way that that flimsy thing could stand up to a prairie wind or a gale off the lake. Even if it did, the prospect of tumbling through the air in a cage 20 stories above Chicago was full-on craziness. Who in their right mind was going to want to ride that thing? Hundreds of thousands of people did. Despite the fact that during its first test run, hundreds of bolts and loose parts rained down on spectators below. Despite real stories of panicked riders trying to escape through the windows when you realize exactly how high 200 feet was. Despite apocryphal stories of suicides, and severed limbs. This Ferris wheel, this thing that is basically a kiddie ride today was a bigger thrill ride than any quintuple-loop-open-car-reverse-twist-rocket-coaster they might have at Six Flags.

The papers, even some in France, said it was the marvel of the age. Better than the Eiffel Tower. That almost mundane sensation we have now of looking down from above, of moving through space. Up and out and down and back around. No one had ever felt those things before. And of course, now we can’t really feel them again. We’ve gone around too many times, we’ve looped too many loops. But back in 1893 you could pay your $0.50 and climb into a car right after sunset, during the golden hour and experience something entirely new. You could rise up above the World’s Fair where down below Americans were eating hamburgers for the first time. Where Buffalo Bill, and Frederick Douglass, and Mark Twain were sightseeing. Where entire villages from Egypt and Algeria had been brought and reconstructed. Where people heard ragtime for the first time. Saw hula dancers, and belly dancers. You can come back to Earth and walk out, and be among the first people in human history walk around at night, with lights on. But fairs end and they shut down. They pack up and leave, and the Ferris wheel did too. They moved it up to a different park in the north side of the city. And after a while, the novelty was gone, and the Ferris wheel became just a Ferris wheel, and they tore it down. The salvage company bought it for eight grand, blew it up with dynamite and sold the pieces for scrap.

RM: That’s the memory palace from Nate DiMeo. Now from Radiotopia. After a couple years of not put not putting the show out very regularly, he was running TV shows and books and stuff, I’m pleased to say he is releasing weekly episodes through the summer. So get on board. You’re going to want to set aside 5 minutes or so to listen to it the moment each new episode comes out. That’s what I’m going to be doing.