Bot tricks people into thinking it is human – Artifical Intelligence at last? Nope, I’m skeptical

I’ve tripped over an interesting little snippet in New Scientist where there is a claim that some software called Cleverbot has passed one of the key tests of artificial intelligence: the Turing test. Now before we proceed, lets clarify what this test actually is. It was introduced by Alan Turing (hence the name) in his 1950 paper Computing Machinery and Intelligence, which opens with the words: “I propose to consider the question, ‘Can machines think?‘” In essence, the test is to have folks engage in a natural language text only conversation (via an instant messaging interface and work out if they are conversing with a machine or a human. If folks cannot reliably tell the machine from the human, the machine is said to have passed the test.

The claim is that cleverbot has passed that test, but you need not take the claim at face value, you can go play with cleverbot yourself. I did, and so I’m skeptical about this claim.

Here is an extract of what New Scientist actually reports:

The Cleverbot test took place at the Techniche festival in Guwahati, India. Thirty volunteers conducted a typed 4-minute conversation with an unknown entity. Half of the volunteers spoke to humans while the rest chatted with Cleverbot. All the conversations were displayed on large screens for an audience to see.

Both the participants and the audience then rated the humanness of all the responses, with Cleverbot voted 59.3 per cent human, while the humans themselves were rated just 63.3 per cent human. A total of 1334 votes were cast – many more than in any previous Turing test, says Cleverbot’s developer and AI specialist Rollo Carpenter.

“The world had better decide rather than me – it’s either passed or it’s come very close in this particular test,” says Carpenter.

You know it is just an illusion, right? All you have here is a smart rules engine that uses a database of previous conversations to reply with an apparently appropriate response, there is no consciousness and there never will be. Nor is this really new either, I’ve played with a simpler version called ELIZA in the 1970’s. Yes, I’m that old, but this is even older, Eliza actually dates back to 1966. That original pattern has simply been copied over and over and has been improving over the years, not primeraly due to software innovations, but instead by leveraging technology advances.

OK, I need to be fair, if you go and check out the on-line version of Cleverbot you will not be impressed at all, but it is just a simpler variation of what was deployed for the actual test. “In what way?”, you might ask.  Well, the online version does about three searches then delivers an answer. For the test it (yes, it is an ‘it’) was beefed up to do 42 searches, so has a far better chance of a superior reply. That does not make it smart, just more efficient at fooling you into thinking it is real. The NS article nails the problem, and also speculates about where all this might go …

The problem is the Turing test itself. A result of 50 per cent or more is often considered to be a pass, but there is clearly still a gap between Cleverbot and humans.

Passing the test does not prove intelligence, says Carpenter, but merely shows that the machine can imitate intelligence. That doesn’t mean attempting to pass the test is pointless, however.

Cleverbot learns from conversations and the same technique could be used to create a more general artificial intelligence. “We humans learn from enormously richer sensory data than that,” says Carpenter. “Imagine a robot with learning techniques that has the capability to do the same.”

Of course you can also point out that the Turing test is not just measuring the strength of the proposed AI, it is also measuring how easily those taking part can be fooled, and as we are all aware (I hope), some folks are indeed very prone to being conned, so perhaps the Turing test is also a Gullibility test.

Bot to Bot – “Do you Believe in God?”

Now here is a bit of fun, what happens when you wire up Cleverbot to itself and set it chatting? It starts out being polite, but would you believe that  it soon starts talking about God and gets a tad aggressive about it?

 

A Real Conversation – Yes you can be fooled:

Lets finish off with the following folkloric legend of AI from 1960’s (yes that far back), to illustrate that being fooled is nothing new. The anectode comes from Daniel Bobrow, who wrote STUDENT, one of the earliest natural language processing programs that could read, understand, and solve high-school level algebra problems. Here is how Bobrow tells the story:

One Saturday morning about 9 a.m., a Vice President of Bolt, Beranek, and Newman in charge of selling our Telcomp commercial service arrived at our PDP-1 computer room hoping to be able to use the computer for a demonstration. He found a note on the PDP-1 teletype that read: “TYPE ALL MESSAGES INTO THIS TELETYPE AND END THEM WITH A PERIOD.” This note was left there by one of the programming bums who had just succedeed in making run a LISP version of the DOCTOR program (based on ELIZA as developed by Weizenbaum). However, the VP thought that this note indicated that this teletype was linked with the one in my home, since I often use the computer all night. Therefore, he started typing to “me,” but in fact was typing to this version of the DOCTOR program. The following conversation ensued:

vp: Do you think that I can use the computer this morning?

eliza: Why do you ask?

vp: I might be able to make some additional Telcom sales.

eliza: Why aren’t you so sure?

vp: My perspective customers have never seen a demonstration of the system.

eliza: Does this mean very much to you?

vp: Of course it does.

eliza: You seem very positive about that.

vp: This is ridiculous.

eliza: What do you mean, this is ridiculous?

vp: Is there any reason why I can’t use the computer now?

eliza: Tell me more about your perspective customers.

vp: Please dial me up on 491-1850

Note that after the last remark the VP did not obey instructions, and left out the period. Therefore, of course, the computer didn’t answer him. This so infuriated the VP, who thought I was playing games with him, that he called me up, woke me from a deep sleep, and said:

vp: Why are you being so snotty to me?

bobrow: What do you mean I am being snotty to you?

The VP angrily read me the dialog that “we” had been having, and couldn’t get any response but laughter from me. It took a while to convince him it really was a computer

The final observation is this. Bots as such do not tell us very much about AI, but instead reveal a great deal about how we can be so easily be fooled into attributing intelligence and consciousness where none exist at all.

Links

• New Scientist article about the Cleverbot claim
• Link to cleverbot itself – Interesting, but clearly not human
• Wikipedia article on the Turing Test
• ELIZA