The ELIZA Effect

Roman Mars:
This is 99% Invisible. I’m Roman Mars.

Roman Mars:
When Joseph Weizenbaum was still alive, he liked to tell this story about a computer program he’d created back in the 1960s as a professor at MIT. It was a simple chatbot named ELIZA, that could interact with users in a typed conversation. Weizenbaum asked his secretary to try it out.

Joseph Weizenbaum:
I asked her to my office and sat her down at the keyboard and then she began to type, and of course I looked over our shoulder to make sure that everything was operating properly.

Roman Mars:
Weizenbaum watched as the secretary typed a message to ELIZA, and then as the program responded to her.

Joseph Weizenbaum:
After two or three interchanges with the machine, she turned to me and she said, “Would you mind leaving the room please?”

Roman Mars:
The secretary was so entranced by this chatbot that she didn’t want to stop exchanging messages.

Voiceover:
“Weizenbaum’s secretary fell under the spell of the machine.” (sound of typing on keyboard)

Roman Mars:
Weizenbaum saw this happening over and over again. People would reveal very intimate details about their lives to the program. It was like they’d just been waiting for someone, for something, to ask.

Delaney Hall:
ELIZA was a simple computer program.

Roman Mars:
That’s producer Delaney Hall.

Delaney Hall:
The way it worked was it would recognize a keyword in the user’s statement and then reflect it back in the form of a simple phrase or question. When ELIZA got confused or couldn’t parse a statement, it would fall back on set phrases like, “Please go on,” or, “Tell me more.”

Roman Mars:
Weizenbaum had programmed ELIZA to interact in the style of a psychotherapist and it was pretty convincing. It gave the illusion of empathy even though it was just simple code. Here’s an exchange, dramatized in an old documentary segment, that begins with the human.

Woman: Men are all alike.

Roman Mars:
And then the computer program answers.

ELIZA: In what way?
Woman: They’re always bugging us about something or other.
ELIZA: Can you think of a specific example?
Woman: Well, my boyfriend made me come here.
ELIZA: Your boyfriend made you come here?
Woman: He says, I’m depressed much of the time.
ELIZA: I’m sorry to hear that you’re depressed.
Woman: It’s true, I am unhappy.
ELIZA: Do you think coming here will help you not to be unhappy?

Delaney Hall:
ELIZA was one of the very first computer programs that could convincingly simulate human conversation, and Weizenbaum got weirded out by the way that people reacted to it. He didn’t expect them to be so captivated. He feared that they were being fooled, that they didn’t really understand it was just a bunch of circuits on the other end. And he wondered if it was so easy to mimic a sense of human understanding, then what did that say about human understanding?

Roman Mars:
Weizenbaum started raising these big difficult questions at a time when the field of artificial intelligence was still relatively new and mostly filled with optimism. Many researchers dreamed of creating a world where humans and technology merged in new ways. They wanted to create computers that could talk with us and respond to our needs and desires.

Delaney Hall:
But Weizenbaum would take a different path. He would begin to speak out against what he saw as the eroding boundary between humans and machines, and he’d eventually break from the artificial intelligencia, becoming one of the first and loudest critics of the technology he helped to build.

Roman Mars:
People have long been fascinated with mechanical devices that imitate humans. Ancient Egyptians built statues of divinities from wood and stone and consulted them for advice. Early Buddhist scholars described ‘precious metal-people’ that would recite sacred texts, and weep.

Pamela McCorduck:
The Greeks had a wonderful attitude toward robots and machines. They welcomed them.

Delaney Hall:
This is Pamela McCorduck, and she wrote ‘Machines Who Think’, one of the first modern histories of artificial intelligence. And she says that in Greek mythology, the God Hephaestus, who was the God of blacksmithing, loved to make robots.

Pamela McCorduck:
He created all kinds of robots, one to help him out in the forge and all kinds of party help. Things would roll into the parties of the gods with trays and whatnot, and there’s a wonderful description in Homer, “They are like lovely young woman and there is intelligence in their hearts.”

Delaney Hall:
Flash forward nearly 2,500 years to the 1940’s.

Voiceover:
“In the mid 1940’s of the 20th century, a different kind of tool was invented. A tool for extending certain of the powers of man’s mind. This tool is the electronic computer.”

Delaney Hall:
As modern computers became more powerful and widely available, people begin to believe that this fantasy of intelligent machines that could think and communicate might actually be within grasp. The idea was most famously articulated by the British mathematician Alan Turing.

Brian Christian:
And he wrote this very famous paper in 1950 called ‘Computing Machinery and Intelligence’, which begins with this famous line, “Can machines think?”

Delaney Hall:
This is Brian Christian. He wrote ‘The Most Human Human: What Talking with Computers Teaches Us about What it Means to be Alive’.

Brian Christian:
Turing is already sort of seeing ahead into the 21st century and imagining, what if someday we do manage to build a machine that can actually “think”? How would we know that we had built such a machine?

Roman Mars:
This is a hard task, because it’s difficult to even define what it means to think, let alone prove that a computer is doing it. But Turing approached the question like the engineer he was, proposing a simple test. He said that if a computer could regularly fool people into thinking it was in fact a human, it would pass. Turing predicted that this would eventually happen so consistently-

Brian Christian:
That as a result we would just as a society, he says, come to this point where we would speak of machines as being intelligent without expecting to be contradicted.

Roman Mars:
The ‘Turing test’ brought up really profound questions. Ones we’ve been thinking about since the days of Greek robots and precious metal-people.

Brian Christian:
What does it mean to have a mind? How do you kind of bridge that gap from one mind to another? And is language kind of the best or only means that we have for doing that? How does empathy work? What are the limits of empathy? They’re not fundamentally technological questions, they’re fundamental human questions.

Delaney Hall:
And they’re questions that would come to occupy Joseph Weizenbaum, the creator of ELIZA. Before turning against artificial intelligence, Weizenbaum helped develop it. He was part of an early generation of AI researchers. Back in the 50’s they started exploring how to give computers new, human like powers.

Roman Mars:
They programmed computers to reason, plan and perceive. They created programs that could play checkers, solve word problems and prove logical theorems. The press at the time described their work as astonishing. Herb Simon, one of the most prominent AI researchers at the time predicted that within 20 years, by the 1980’s, machines would be capable of doing any work a person could do.

Delaney Hall:
Just like today, these advances made people pretty anxious. For millennia, humans had defined themselves as rational creatures. It was our ability to reason and use logic and play chess, that made us special.

Brian Christian:
It is exactly, those things that the computer came in and within a handful of years, was better than any human could ever hope to be. And so it really pulled that rug out from under us.

Delaney Hall:
But despite all the big leaps forward, there was one realm in particular where computers remained quite dumb. They struggled to master human language, which is known in AI as natural language.

Melanie Mitchell:
Natural language processing or understanding if you will, is probably the hardest problem for AI.

Delaney Hall:
Melanie Mitchell is a computer scientist and AI researcher.

Melanie Mitchell:
When we communicate in language, we’re using all of our thought processes, we’re using our vast knowledge of the way the world works, about other people and their goals and motivations, we’re disambiguating words. It’s really almost equivalent to cognition itself.

Delaney Hall:
So as an example of this, Mitchell says, think about this sentence.

Melanie Mitchell:
‘A steel ball fell on a glass table and it shattered.’

Delaney Hall:
If you ask-

Melanie Mitchell:
What does it refer to?

Delaney Hall:
Humans will understand immediately.

Melanie Mitchell:
Of course we know that if a steel ball falls on a glass table, the glass table shatters. But that’s because we know about what glass is, what steel is, we know about things falling on other things and we know a lot about the way the world works.

Delaney Hall:
But computers don’t know that stuff. Unless they’re programmed to, they don’t actually know about glass and steel and gravity. And actually programming all that information into a computer is really labor intensive and difficult, because that knowledge is mostly subconscious for us. It’s so intuitive as to be almost invisible.

Melanie Mitchell:
And this is something that comes up again and again in natural language processing systems, is that they don’t have the kind of knowledge about the world that we humans have and so they make mistakes.

Roman Mars:
This clumsiness with human language meant that early chatbots built in the 1950’s and 60’s were tightly constrained. They could converse about some very specific topic like, say baseball, so you could ask about Joe DiMaggio’s batting average in 1949, and the computer would tell you.

Delaney Hall:
And by limiting the world of possible questions and answers, researchers could build machines that passed as intelligent, but talking with them was like having a conversation with Wikipedia, not a real person.

Naomi Weizenbaum:
I was very aware of the notion that he was doing something with computers that gave these computers a power to understand that was leaps and bounds above anything that had come before.

Delaney Hall:
Naomi Weizenbaum is Joseph Weizenbaum’s youngest daughter. She didn’t know a lot about the specifics of his work when she was growing up, but she remembers his mustache, she says he always had one, and that he was serious and studious, and he spent a lot of time in his office at home.

Naomi Weizenbaum:
My mother, she’d call me downstairs and say, “Okay, time to go walk the Dad,” and I had to go down and fetch Dad out of his study and ask him to come for a walk with me.

Delaney Hall:
She also remembers her dad as someone who couldn’t help but see the world from a critical perspective.

Naomi Weizenbaum:
I think my father was very deeply an outsider right from the very beginning, and that when he went to MIT, it was no exception.

Roman Mars:
Shortly after he arrived at MIT in the 1960’s, Weizenbaum had an insight. He realized he could create a clever workaround for the natural language problem. He could build a chatbot that didn’t really need to know anything about anything. It wouldn’t spit out facts, it would just reflect back at you like a mirror.

Joseph Weizenbaum:
“You know the cocktail party conversation for example…”

Delaney Hall:
This is Joseph Weizenbaum in an archival interview.

Joseph Weizenbaum:
“Where someone says something to you that you really don’t fully understand, but you are in fact able to give a response which appears appropriate, and in fact the conversation can continue for quite a long time. And we do that all the time, we do that all the time.”

Roman Mars:
But Weizenbaum still had to create a framework in which that conversation would happen, a style of interaction the program could follow. Weizenbaum had long been interested in psychology and he recognized that the speech patterns of a therapist might be easy to automate.

Delaney Hall:
Because when you think of a therapist, certain phrases come to mind like, “And how does that make you feel?” And so, Weizenbaum decided to create a bot that like a therapist, would listen, pick out keywords and then mirror statements back to the user.

Naomi Weizenbaum:
How Dad explained it was that it was kind of, “simple dimple,” psychology, more or less very similar to a parrot. That you pick out one element of what the person said, then you ask them to expand on that.

Delaney Hall:
Joseph Weizenbaum was very clear. He didn’t intend for ELIZA to be an actual therapeutic tool. It wasn’t even programmed with information about human thought or emotion. But nevertheless, ELIZA seemed to understand.

Roman Mars:
And as Weizenbaum watched people like his secretary converse with the program, he quickly realized they weren’t just having simple dimple exchanges, they were having meaningful conversations with the machine.

Brian Christian:
People began having these really long kind of soul-bearing conversations with this program, and reporting a meaningful therapeutic experience. And this for him was horrific.

Delaney Hall:
Brian Christian, again.

Brian Christian:
He talks about, “I would have assumed that some sort of meaningful experience like this would require someone to be present on the other side of the conversation and to actually invest something.” So the fact that the merest mechanical parody of this process can still apparently produce something meaningful for people, he found really unsettling.

Roman Mars:
But while Weizenbaum was horrified by his own creation, others saw ELIZA as an important breakthrough, and a real opportunity.

Delaney Hall:
Before coming to MIT, Weizenbaum had spent time at Stanford where he became friends with a psychiatrist named Dr. Kenneth Colby. Colby had worked at a large underfunded state mental hospital where patients maybe saw a therapist once a month.

Pamela McCorduck:
And that was why when he encountered even this rather crude caricature as Weizenbaum put it, of the therapeutic process, he thought, “Boy, I can put some therapeutic knowledge into this and make it happen.”

Roman Mars:
Colby started promoting the idea that ELIZA might actually be therapeutically useful and the medical community started to pay attention. They thought maybe this program, or other programs like it, could help expand access to mental health care. And maybe in some ways it could even be better than a human therapist. It would be cheaper and people might actually speak more freely with a robot.

Pamela McCorduck:
“Yeah, I can tell this bunch of tubes anything I want, I can tell the truth.” We feel free to open ourselves to what we think is a nonjudgmental machine.

Kenneth Colby:
Many therapists claim there has to be a live, warm human body present for therapeutic change to take place. Again, that’s an opinion, that’s not a fact.

Delaney Hall:
Here’s Colby in a news interview years later.

Kenneth Colby:
If you say to the program, “I’m feeling quite depressed,” I would sympathize with you, and so will the machine.

Roman Mars:
So the idea of the robot therapist started to take off.

Brian Christian:
You know, in the late 60’s, early 70’s, you have these peer reviewed psychology journal articles saying, “This represents a potential breakthrough for therapy.”

Delaney Hall:
The famed scientist Carl Sagan wrote about the idea in 1975, imagining a network of psychotherapeutic computer terminals, kind of like phone booths.

Brian Christian:
You walk to a city corner, put in a nickel and you talk to this therapeutic robot for a few minutes, to kind of take your mind off of things. So it was very warmly received by the medical community at large.

Roman Mars:
And while the idea of therapy terminals on every corner never materialized, people who worked in mental health would continue to experiment with how to use computers in their work. Colby, for instance, went on to create another chatbot called PARRY, which simulated the conversational style of a person with paranoid schizophrenia. He later developed a program called Overcoming Depression.

Brian Christian:
Again for Weizenbaum, he felt like he had sort of let this genie out of the bottle and there was kind of no going back.

Joseph Weizenbaum:
“The real break, or the real controversy finally arose over the claim that ELIZA was of therapeutic significance. My God, could people really believe that? What are we doing?”

Delaney Hall:
Weizenbaum objected to the idea that something as subtle, intimate and human as therapy, could be reduced to code. He worried that if machines snuck into the therapist’s office, then where else might they end up? He began to argue that fields requiring human compassion and understanding just shouldn’t be automated.

Roman Mars:
Weizenbaum also worried about the same future that Alan Turing had described. One where chatbots regularly fooled people into thinking they were interacting with a human. Weizenbaum would write of ELIZA, “What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful, delusional thinking in quite normal people.”

Joseph Weizenbaum:
“The whole thing is a con job, the whole thing is. It’s very much like fortune telling. It’s an illusion-creating machine and all that sort of thing.”

Delaney Hall:
Weizenbaum quickly went from someone working in the heart of the AI community at MIT, to someone preaching against it. And his friend Kenneth Colby became one of his first targets.

Pamela McCorduck:
He began attacking Dr. Colby in public, in professional meetings, and he would say, “This is a wicked thing to do, it should not be done, and he should be punished for doing it.”

Roman Mars:
Over time, Weizenbaum escalated his arguments. His family had escaped Nazi Germany in 1936, and sometimes when critiquing AI, Weizenbaum would invoke the Holocaust. He worried powerful technologies could be abused by governments and corporations. Here’s one of his other daughters, Miriam Weizenbaum.

Miriam Weizenbaum:
These happen to be incredibly powerful tools, and the really important question is for whom are they being used? What’s the purpose of them? Who benefits? Who is subject to them? Who controls them? Who doesn’t?

Pamela McCorduck:
At that point, he began extending that and saying all artificial intelligence is questionable and probably wicked. It shouldn’t be done. And oh dear, he was really swimming upstream on that one.

Voiceover:
“The computers are coming.”

Roman Mars:
Over the next few decades, AI allowed for automation, not just in therapy but in fields like medicine, law and finance.

Voiceover:
“Computers fly our planes. Increasingly, they do our banking. The computer is king in business, finance and information. And increasingly, say experts, computers will get smarter and smarter. Companions, advisors, oracles to all of us.”

Delaney Hall:
Eventually, researchers began trying to create computers that were flexible enough to learn human language on their own.

Voiceover:
“To take those first faltering steps as babies do toward understanding language, to break down and diagram sentences, to remember new meanings, verb tenses, and to ask questions to make itself smarter. The computer wants to know, is the past tense of make, maked? No, says Martin, made.”

Delaney Hall:
These programs could take a long time to figure things out.

Voiceover:
“It has been a tedious three year project. The computer now knows about 300 words. Still, Paul Martin is convinced computers can learn.”

Roman Mars:
Starting in the 1980’s and 90’s, there were new breakthroughs in natural language processing. Scientists began relying on statistical methods.

Melanie Mitchell:
Taking documents or recordings of language and performing statistics on things like, how often does one word come after another? Or how often does this sort of triplet of words occur?

Delaney Hall:
And then in the 2000’s and 2010’s, researchers began using what are called ‘deep neural networks’. They’re trained using the huge amounts of data that only became possible with the rise of the internet.

Melanie Mitchell:
So now we can take every single webpage online and process the language in it, and have the neural network analyze the language, and they work so much better than anything that came before.

Delaney Hall:
And these techniques Mitchell is describing, they’ve been applied to chatbots specifically.

Melanie Mitchell:
They might be trained on actual conversations, recorded conversations. Sometimes people put their chatbot online and then other people interact with them, and the systems learn by just interacting with people online.

Roman Mars:
As a result, contemporary chatbots and their talking cousins like Siri and Alexa have gotten a lot better. There are still limitations, of course. They still can’t really understand the world in the same way we do, but they have become a lot more reliable, personable and convincing.

Delaney Hall:
And humans remain as open and sometimes gullible as ever when it comes to interacting with chatbots. There’s even a term that’s emerged in computer science – ‘the ELIZA effect’ – that refers to our tendency to anthropomorphize computers and to believe that programs understand, even when they really don’t.

Roman Mars:
There are recent examples that make it feel as though we are living firmly in the world that Alan Turing predicted and that Joseph Weizenbaum feared. Machines fool humans all the time now.

Delaney Hall:
In 2018 for example, Google revealed something called Duplex, a conversational AI that can make phone reservations, and it is uncanny.

Salon Receptionist: Hello, how can I help you?
Duplex: Hi, I’m calling to book a woman’s haircut for a client. I’m looking for something on May 3rd.
Salon Receptionist: Sure, what time are you looking for around?
Duplex: At 12:00 PM.

Melanie Mitchell:
They programmed in human-like stuttering, and saying, “um” and “uh” and made it very convincing.

Duplex: Do you have anything between 10:00 AM and 12:00 PM?
Salon Receptionist: Depending on what service she would like. What service is she looking for?
Duplex: Just a woman’s haircut for now.
Salon Receptionist: Okay.

Melanie Mitchell:
And it sounded very human. People were really alarmed, they wanted to know that they were speaking to a computer.

Duplex: The first name is Lisa.
Salon Receptionist: Okay, perfect. So I will see Lisa at 10 o’clock on May 3rd.

Roman Mars:
This issue of transparency has become central to the ethical design of these kinds of systems, especially in sensitive realms like therapy. Because even if ELIZA was the first chatbot to be used in the field of mental health, it wasn’t the last.

Alison Darcy:
Transparency is the basis of trust and you must have trust with a service like this.

Delaney Hall:
This is Alison Darcy.

Alison Darcy:
My name is Alison Darcy, I’m the CEO and founder of Woebot Labs.

Delaney Hall:
Darcy is a psychologist and she’s long been interested in how to use digital tools to expand access to mental healthcare.

Alison Darcy:
In the United States, a third of the population are living in areas that are designated no access areas, so they effectively don’t have a mental health professional anywhere near them.

Delaney Hall:
And even if you live in a major city with lots of therapists, it’s often not covered by insurance and can be super expensive.

Roman Mars:
So a few years back, she and her team began thinking about how to build a digital tool that would make mental healthcare radically accessible. They experimented with video games before landing on the idea of Woebot, a chatbot guide who could take users through exercises based on cognitive behavioral therapy, which helps people interrupt and reframe negative thought patterns.

Delaney Hall:
Woebot is not even trying to pass the Turing test. It’s very transparently a robot. It’s represented by a little robot avatar and part of its personality is that it’s curious about human feelings, like a student learning a foreign language.

Alison Darcy:
I think he’s somewhere between Kermit the Frog, circa 1980, and Spock. Spock who’s always trying to, you know, struggle with his human emotions.

Delaney Hall:
As Darcy’s team built a prototype and started watching people interact with Woebot, right away they could tell something interesting was happening. They noticed that if Woebot didn’t check in for a day or two, people would reach out.

Alison Darcy:
People were saying things like, “Woebot, where were you? I need you to check in on me every day.”

Delaney Hall:
Just like with ELIZA, people were forming emotional connections with the program. They liked interacting with Woebot and relied on it.

Alison Darcy:
People when we spoke to them, would say, “Look, I know this is just a chatbot. I know this isn’t anything, but at the same time it feels like this fun little dude checking in on me,” and so people were willing to just go with the suspended reality. They were effectively stepping in to this kind of fictional world and it made it playful and fun.

Delaney Hall:
I’ve spent some time with Woebot and I feel the same way. It’s like texting back and forth with a relentlessly positive and sweet little cartoon character. It offers high fives and sends you GIFs. It feels less like a human therapist leading me to deep insights and more like a tiny personal cheerleader I can carry around in my pocket.

Roman Mars:
A tiny personal cheerleader doesn’t sound very dystopian.

Delaney Hall:
No, it doesn’t, but I mean of course Woebot raises tough questions about tech more broadly. If chat bots are built with nefarious intentions, they can do stuff like detect and manipulate human feelings. They can fool humans into thinking they’re people.

Alison Darcy:
I have anxiety about this kind of tech misapplied like anything else. Like a surgeon’s scalpel, this can be a tool that helps people survive and it can also be a weapon.

Delaney Hall:
And with Woebot in particular, there are other issues too around privacy and safety and even thornier stuff like, is this tool a digital bandaid that somehow enables the status quo? Meaning the situation we’re in now where the vast majority of people don’t have access to high quality comprehensive mental health care. Darcy is clear that she does not see Woebot as a replacement for talk therapy, and she says her team is always thinking and talking about those ethical questions.

Alison Darcy:
It’s something we always keep an eye on, but I think the potential for good just so much outweighs the risk right now.

Delaney Hall:
Darcy thought about this a lot when Woebot first launched. They had been working in relative obscurity and then suddenly their numbers began to climb. Quickly within the first five days they had 50,000 users. Woebot was exchanging millions of messages with people each week.

Alison Darcy:
And I remember going home after our first day and sitting down at my kitchen table and having the realization that Woebot on his first day of launch, had had more conversations with people than a therapist could have in a lifetime. You know, it just really hit home.

Roman Mars:
Joseph Weizenbaum eventually retired from MIT, but he continued speaking out against the dangers of AI until he died in 2008 at the age of 85. For a while he became a go to critic if a TV program needed someone to cast doubt on some new technology. For example, here he is in a segment about computers in schools, criticizing the tendency to reduce every human problem to a technological one.

Joseph Weizenbaum:
“It’s a very dangerous attitude. It can’t help but destroy our culture if it ever becomes totally victorious.”

Delaney Hall:
And while he was an important humanist thinker, a guy way ahead of his time in many ways, some people felt like he went too far. Pamela McCorduck knew Weizenbaum over several decades. She’s the one who conducted the interview you heard with him earlier in the story. And she says he burned a lot of bridges in the AI community and became almost a caricature of himself towards the end. She describes being at a conference in Austria where Weizenbaum was also in attendance. At one point he got up and delivered what had become his usual tirade.

Pamela McCorduck:
“We must not use computers, the German army uses computers and look what’s going to happen.” Well, I happened to be sitting next to a young German artist and he looked at me and he said very quietly, “The German army uses knives and forks, let us not use knives and forks,” well, I had to put my hand over my mouth to keep from laughing.

Delaney Hall:
And Weizenbaum also may have missed something that Darcy was just talking about with Woebot. The idea that humans engage in a kind of play when we interact with chatbots. We’re not necessarily being fooled, we’re just fascinated to see ourselves reflected back in these intelligent machines. That was my experience chatting with Woebot and was probably true for ELIZA too.

Miriam Weizenbaum:
Also, I was going to say about the ELIZA and the story about his secretary and stuff…

Delaney Hall:
Here’s Miriam Weizenbaum again. And I want to be clear, she had many lovely things to say about her dad in our conversation. That he was brilliant and insightful and funnier than people realized, but she also had this to say about the secretary story.

Miriam Weizenbaum:
I always heard that just really kind of soaked in arrogance. Like, how could she be so stupid to think that this is actually a meaningful communication? Here you have somebody who so needs a landing pad for her feelings that she’s willing to embrace it, right? And he totally misses that, he totally misses the human need and instead speaks of just evidence of this person’s stupidity rather than evidence of their humanity.

Delaney Hall:
I think we’re all a little like Dr. Weizenbaum’s secretary now. We’re living in this highly automated world. We have these little computers in our pockets that we can talk to, and we’re under no illusion that they really understand us, but they still captivate us. And we still rely on them to tell us the weather, and how to get to where we want to go, and maybe even on occasion to ask us if we’re doing okay.

Roman Mars:
Coming up after the break…. A recent development in natural language processing that would make Joseph Weizenbaum VERY nervous.

[BREAK]

Roman Mars:
So I’m back in the studio with Delaney Hall, and you’ve got more to tell us about some interesting stuff that’s happened with natural language processing in just the past year.

Delaney Hall:
Yeah, a lot of interesting stuff has happened just recently. So as I was interviewing people for this story, you know, I was talking with computer scientists and historians and we were talking about Weizenbaum and his legacy and these fears he had about chatbots fooling people. And totally unprompted, a couple of people brought up this one recent development in natural language processing.

Brian Christian:
It’s something that’s called GPT-2.

Melanie Mitchell:
So GPT-2 stands for generalized pre-trained transformer. I can never remember what it stands for.

Delaney Hall:
So, Melanie Mitchell, one of the computer scientists I spoke with is close. It stands for Generative Pretrained Transformer 2, and it’s this relatively new neural network that generates incredibly convincing texts.

Melanie Mitchell:
Somebody described it as a mouth without a brain, which is a good statement.

Delaney Hall:
And the way that GPT-2 works is that you can input a word or a sentence or a paragraph, and the program then generates text based on that input. It basically conjures up a whole coherent piece of writing based on just a few words that you give it.

Brian Christian:
It is this giant neural network that is trained on a huge corpus of text to predict the next word in a sentence or a paragraph given the preceding text.

Delaney Hall:
Here’s Brian Christian who we also heard from in this story.

Brian Christian:
You get what appear to be plausible if very slightly weird screenplays and, you know, news bulletins and chat message histories and all sorts of things.

Roman Mars:
Okay. So I’m having a hard time picturing that. So like, what are some examples of the type of texts that generates? Like why and how is it so convincing?

Delaney Hall:
So the example that kind of broke the internet when all of this first came out was this fake news story about unicorns in the Andes.

Roman Mars:
I must have missed that one. Well, that’s notable.

Delaney Hall:
Yeah, I know. So this will sort of help you understand a little bit better, kind of how it works, I think. So the prompt that GPT-2 was given, and so this was a couple of sentences that were written by a human and given to the machine. And I guess I should be clear, this was written in the style of a news story, that was intentional. So the prompt was “In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes mountains. Even more surprising to the researchers was the fact that the unicorn spoke perfect English.” So a human wrote those sentences and then fed them to the machine.

Roman Mars:
And so what did the machine spit back out?

Delaney Hall:
So totally independently based on those sentences. This is what GPT-2 to generated, and it actually goes on for quite a while, but I’ll redo the first few lines just because they’re amazing. It goes: ‘The scientists named the population after their distinctive horn, Ovid’s unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Perez and evolutionary biologists from the University of La Paz and several other companions were exploring the Andes mountains when they found a small valley with no humans or animals. Perez noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of rock and silver snow.’ And so it goes on for a few more paragraphs, but Roman, a machine! A machine wrote that, a machine!

Roman Mars:
And added all the detail about Dr. Perez and the University of La Paz and everything. That’s just from the machine.

Delaney Hall:
Yes! I know. I know. And it’s like, there’s a couple things that really stand out to me when I’m reading it. And the first of course is that, you know, the sentences are coherent. Like if you told me a human wrote them. I would not doubt you either.

Roman Mars: Neither would I. That’s incredible.

Delaney Hall:
And then it has those details like the names and the location of the university, which align upright with the prompts. And then the other thing it does is that it does this pretty convincing job of mimicking the format of a news article. So it starts with like lead sentences, they give you the most crucial information, and only then does it begin to tell the story of how the discovery unfolded.

Roman Mars:
Wow. It’s so cool. I almost can’t even make sense of it – how real that feels. I mean, this is pretty silly with unicorns in the Andes, but I mean, I imagine, you know, this is making people nervous that a computer can generate this convincing a text that as if it came from a human.

Delaney Hall:
Yeah, absolutely. So the company that created the program called OpenAI released this paper and it had various examples of text generated by this neural NEC net, and it caused a huge stir online, partly because it did such a good job. And partly because it made people immediately nervous, like the idea that this text is so convincing gets back to that idea that we talked about in the episode, which is like, will people know this was written by a machine? And because it generated a very convincing and totally fake news article, people are worrying that the tool could supercharge the spread of misinformation on the internet.

Roman Mars:
Right. I mean, so what do you do? Like once you make a thing like that, how do you make sure it doesn’t do that? How do you make sure it isn’t used for nefarious ends?

Delaney Hall:
So the way that OpenAI approached it was when they first released news about it, they said “We’ve made this thing. Here’s some examples of the text it generates. It is extremely convincing, and because of that, we’re not going to actually release the code. It’s too dangerous.”

Roman Mars:
Whoa. Yeah. The only thing I see a problem with that is like the Streisand effect. Like if you say you can’t use it and don’t recreate it, it just means that more people are going to try to create and use that sort of thing. It just calls more attention to it.

Delaney Hall:
So yeah, it did. It did call a lot of attention to it. Here is Melanie Mitchell again.

Melanie Mitchell:
A bunch of people just like jumped on them. They were saying this is just a publicity stunt, and there were, of course, the popular press doomsday headlines about, you know, OpenAI creates an AI that’s so powerful. It’s too dangerous to release.

Delaney Hall:
So basically OpenAI got a lot of pushback, but the whole situation also inspired this interesting conversation in the AI community about these, you know, really tough questions that they’re facing now, now that they can generate what are known as deep fakes, you know, not just in text, but in audio and video too. These are really convincing computer-generated pieces of media. So here’s Melanie.

Melanie Mitchell:
How should people think about publishing results in AI? Is it better to publish these things and get them out there so that people can do research on how to protect or defend against them? Or should they not be published at all? So there is this big conversation about that.

Roman Mars:
I guess I wonder, like, you know, this obviously has a problem with fooling people, but you know, couldn’t a human just write an entire fake news article or a bunch of them and post online? I mean, how does GPT-2 make things really all that different.

Delaney Hall:
That was something I asked Melanie and she said that the issue is really scale. Like, sure she could sit down and write a story about unicorns herself and post it online and tweet it out and try to make it go viral. But it would take time. It would take effort and instead of GPT-2 could generate literally millions of these stories very quickly.

Melanie Mitchell:
And here you can just do it automatically and generate as many of these as you want. You could put them online and somebody is going to believe them. It just makes it easy to create these things and publish them and it’s hard to fight back, you know?

Delaney Hall:
And the tough thing is that, you know, we’re already living in an online information environment where it can be hard to sort the real from the fake.

Brian Christian:
Here’s an account that says they’re the Tennessee Republican Party.. Are they? Here’s this thing that says it’s the Trump Fan Club of Missoula. Is it? You know, like, here’s this person who claims they’re a retired Navy Seal. Are they? We’re increasingly moving into this world in which it’s just really hard to make sense out of the discourse on the web without being extremely scrupulous and vigilant.

Roman Mars:
It seems like this generation, from now on, we’re going to have to have major media literacy training. Just to be able to spot the fakes from the real thing is going to take a whole new amount of effort and part of our brain.

Delaney Hall:
Yeah, it is. And to bring it back to Joseph Weizenbaum, when he first started thinking about this stuff, computers were still really rudimentary. They had about as much processing power as we have in like a parking meter today, which is to say they couldn’t do much. But today it’s this totally different story.

Brian Christian:
How do we survive like this coming reality in which you can produce for all intents and purposes, Turing test-passing content that espouses whatever viewpoint you happen to want infinitely at scale? Increasingly these extensively philosophical questions of the Turing test become not merely one of the hassles of everyday life, but one of, I think, the more pressing questions affecting us at kind of this moment in history.

Roman Mars:
Wow. So it seems like Weizenbaum may have been ahead of his time, but dead on.

Delaney Hall:
It’s interesting cause I think he sounded to many people in his time, alarmist. But increasingly the questions that he was raising seem very, very relevant.

Brian Christian:
I find myself wishing that we had both Turing and Weizenbaum by our side and in the coming decade because I think we need all the perspective we can get.

Roman Mars:
99% invisible was produced this week by our senior editor, Delaney Hall. Mix and tech production by Sharif Youssef. Music by Sean Real. Katie Mingle is our senior producer. Kurt Kohlstedt is the digital director. The rest of the team is Emmett FitzGerald, Joe Rosenberg, Chris Berube Vivian Le, Avery Trufelman, Sofia Klatzker, and me, Roman Mars.

Special thanks this week to Katie Mingle who first came up with this idea to do a story about ELIZA. Thanks also to Dr. Alena Buyx, who has studied the ethics of AI in mental health settings. You can find a link to the paper she co-authored on our website. Thanks also to the Carnegie Mellon University Archives for letting us use the recording of Joseph Weizenbaum from the Pamela McCorduck Collection. And finally, thanks to Naomi and Miriam Weizenbaum, who generously shared their memories of their father.

You can find the show and join discussions about the show on Facebook. You can tweet at me at Roman Mars and the show at 99PIorg. We’re on Instagram and Reddit too, but our true home on the web is 99pi.org.

Melanie Mitchell:
I remember one time when I was probably about 10 asking my dad if I could work on the computer and I asked if I could be alone. Not that I knew anything at that time about the famous story of his secretary wanting to be left alone to work with ELIZA. I didn’t know anything about that, but I somehow knew that this computer was going to have powerful insights into people and who I was and something like that. And unbeknownst to me, he had turned the computer on so that it operated basically as a typewriter, but I didn’t know that. And so I sat down at the typewriter and I typed in, what is my sister Miriam doing right now? And I waited for a really long time and the computer didn’t answer me. And then I said, what is Miriam wearing right now? Cause I knew the answer to that. So I thought, well then maybe the computer would too, and nothing happened. I waited for a really long time and finally, I got up in disgust and thought this stupid computer doesn’t know sh*t.