Loebner Prize

The contest will take place on Wednesday 19th October 2011 in the Harrison Building.

Programme for the day is as follows:

• 9.00am - Welcome from Mark Overton, Deputy Vice Chancellor (External Affairs) - Room 101

• 9.30am-12noon - Main Loebner Prize Contest (Rooms 101-103)

• 12.20pm - Main contest prize giving (Rooms 101-103)

• 12.30pm - Junior Loebner Prize Contest begins (Rooms 101-103)

• 12.30pm onwards - Buffet lunch and Tours available (Room 103)

• 2pm - 4.00pm Presentations from Judges (Room 170)

• 4.00pm - Coffee (Room 103)

• 4.15pm - 5.00pm - Panel discussion (Room 170)

All timings are approximate and subject to change.

Location

Harrison Building, North Park Road, Exeter, Devon, UK (Google Map).

Contest Organisation

The contest is being organised by Dr Ed Keedwell with kind assistance from Dr Zena Wood, Prof. Richard Everson and Dr Hugh Loebner.

Entries will be selected in a process similar to that undertaken in previous years in that a series of questions will be posed to each entry and the responses recorded.  The responses will then be voted on by an audience of participants at the University of Exeter, with the 4 top-ranked selected for entry to the main contest. 

Update:  Entries will be tested on a Core i7 PC with 6GB RAM running Windows 7.  Please note that the machines used for the contest itself (if not provided by entrants) may be of lower specification but will use the same OS.

The selection process for entries will operate as follows:

1. Each entry will be tested against a number of questions using the Loebner Protocol by an automated system that ensures each entry is tested in the same fashion. The questions themselves will be similar in nature to those used last year which are shown below.
2. The results for each question will be collated onto a presentation slide showing the question and answers from all entries presented in a random order.
3. An audience consisting of staff and students at the University of Exeter will be asked to vote for the most human-like of the responses for each question using audience participation technology.
4. The top-4 rated entries will be selected for the Loebner Prize in October based on the summation of votes for all entries across all questions. In the unlikely event of a tie for the last place, the organiser will have the final decision.

Last Year's Questions:

• My name is Bill. What is your name?
• Which is larger, a pineapple or a pine tree?
• What is a calendar?
• What is my name?
• John is taller than Mary and Mary is taller than Sue. Who is shorter, John or Sue?
• What day of the week is it?
• What will be tomorrow's date?
• My friend Bob likes to play tennis. What game does Bob like to play?
• Are you a human or a computer?
• Do you have any brothers or sisters?
• What number comes after twelve?
• The ball was hit by Bill. What did Bill hit?
• Who hit the ball?
• What time do you go to be at night?
• What is your favorite food?
• What is the name of my friend who likes to play tennis?
• What would I do with a knife?
• How many letters are in the alphabet?
• Have you read a good book lately?
• What time is it?
• What's your favorite color?
• Do you prefer cats or dogs?

The selection event took placeat the College of Engineering, Mathematics and Physical Sciences at the University of Exeter on the 24th June. 

13 entries were received and all were received before the postal deadline of the 6th June 2011. Each entry was presented with 20 questions (where possible with the Loebner Test Program) as shown at the end of this page.

Questions and responses were recorded for each entry in a file, each of which is available to view on the 2011 Loebner Prize website. Where no response to a particular question was forthcoming, entries were allowed 3 minutes before moving on to the next question. The response files were then used to inform the selection process as follows.

Pre-Selection

Due to technical limitations of the voting technology, a maximum of 10 entries could progress to the audience voting stage. To remove the 3 worst performing entries, each entry was scored by the organising team using the same scoring system used in 2010 (1 point for a correct and human-like answer, 0.5 for a partially correct and human-like answer, 0 points otherwise). The worst performing entries received scores of 0, 3.5 and two entries were tied on 5.5. The responses from the two tied entries were very similar and could not be separated and so both were excluded from the final selection phase, leaving 9 entries to progress to the final stage.

Selection

12 staff and students at the University of Exeter took part in the selection process. The organisers involved in pre-selection did not take part in this second stage. The audience was shown a slide containing each question and the 9 responses from the entries in a numbered list presented in a single random order (e.g. entry 1 remained at position 1 throughout). They were then asked to determine which 4 answers were most human-like and to enter the number of the best entries into their audience participation handsets. This was repeated for each of the 20 questions and the results collated. Entry names were not revealed until the voting process was completed.

Results

So congratulations to Rosette, Zoe, ChipVivant and Tutor and commiserations to the rest.

Questions

• My name is Ed. What is your name?
• Which is larger, an ant or an anteater?
• What month of the year is it?
• What is my name?
• Dave is older than Steve but Steve is older than Jane. Who is youngest, Steve or Jane?
• What day will it be tomorrow?
• What's your favorite food?
• My friend Chris likes to play football. What sports do you like to play?
• Are you a human or a computer?
• What letter comes after T?
• What is the name of my friend who likes to play football?
• What is your name?
• What would I do with a screwdriver?
• How many letters are in the word 'banana'?
• Have you watched a good film lately?
• What year will it be next year?
• What's your favorite fruit?
• Do you prefer white or black coffee?
• How old are you?
• The football was kicked by Fred. Who kicked the football?

Thankyou all for your hard work in producing the entries this year and well done to the finalists.

Transcripts

Below are the transcripts of the question & answer sessions (all PDFs).

The 2011 Loebner Prize was held at the University of Exeter on the 19th October.

Judging Panel

Confederates

Mark, Steve & Dawn (University of Exeter), Samantha (University of Plymouth)

Finalists

Results

The contest was comprised of four rounds.  In each round, the judge was paired with a new AI entry and human confederate ensuring that each judge interacts with every AI and human during the course of the contest.  At the end of each round the judges must decide which entity is the computer.  At the end of the contest, the judges ranked the AI entries and humans.  The results are as follows:

Main Contest

None of the AI systems fooled the judges, therefore the Turing Test has not been passed and the Silver Medal was not awarded.  The bronze medal and cash prizes were therefore awarded based on the ranks awarded by the judges which were as follows:

Congratulations to Bruce Wilcox's Rosette as the 2011 Loebner Prize winner!

Junior Prize

None of the AI systems fooled the junior judges which were conducted over shorter, 10 minute rounds.  However the ranks were significantly different. 

Opening

Contest organiser Dr Ed Keedwell, Deputy Vice Chancellor Prof. Mark Overton and Dr Hugh Loebner spoke at the beginning of the day.  Dr Keedwell welcomed spectators, confederates and judges alike and Prof. Overton formally welcomed the audience and described the University of Exeter's continuing success in research and the links between AI  research at Exeter to the contest.  Dr Loebner explained the rules of the contest to participants and spectators.

Main Contest

The main contest was run in rooms 101-103 in the Harrison Building.  One conversation was projected onto a screen in the confederates room and two 40" LCD screens in the Judges room also relayed what was happening to the audience seated behind.

Junior Loebner Prize

Following the main contest, the Junior Loebner Prize took place with four junior judges from Torquay Boys Grammar School.  The junior judges were only permitted 10 minutes with each AI, but still managed to correctly identify the machine each time, so well done to them.

Talks

After lunch, Dr Loebner and judges Prof. Noel Sharkey and Dr Antony Galton presented their thoughts on their research, experience of the Loebner Prize and thoughts on the Turing Test as a whole.

Panel Discussion

The day concluded with a panel discussion including Jonny O'Callaghan, Dr Galton, Prof. Sharkey,  Dr Loebner and Dr Keedwell.  Questions were posed by the audience and vibrant discussions were had as to the suitability of the Turing Test as a test of intelligence, the question of whether biological systems could be considered artificial in the AI sense, and the question of whether systems that perform brute-force search can be considered to be truly understanding the problem they are solving.  The judges were then presented with a small gift and the day was concluded.

Media interest

The Loebner Prize always receives media interest from around the world, this year was no different. We had reporters and film crews at the event as well as Paul Marks, senior technology correspondent from New Scientist and Noel Sharkey, Professor of artificial intelligence and robots (University of Sheffield) who writes for the Guardian they took part as a confederate and a judge respectively.

• Professor Noel Sharkey wrote an article in the Guardian entitled "Would artificial intelligence outsmart me? I needn't have worried"
• Paul Marks wrote in New Scientist "Chatbots fail to convince judges that they're human"
• Bruce Wilcox winner from this year's competition took part in an interview for the AiDreams website
• Forums including AI Nexus and Chatbots.org have been discussing the competition and future developments
• iProgrammer website have also reported on the day, displaying the winning transcript from Rosette.

Further media items will follow as well as independent filmmaker Miranda Yousef's film about the day.

The 2012 Loebner Prize took place in Bletchley Park on Tuesday 15th May 2012 to mark Alan Turing's centenary celebrations. The contest is being run by David Levy who won the prize in 1997 and 2009. Further information about the prize can be found on the Loebner website.

A team from the University of Exeter's Computer Science department (Dr Ed Keedwell, Max Dupenois and Kent McClymont) will be running a webcast of the Loebner Prize and so, for the first time, interested people from around the world will be able to follow the conversations the judges have as they happen. The URL for the webcast will appear above shortly.