Selection Results

The selection event took place at the College of Engineering, Mathematics and Physical Sciences at the University of Exeter on the 24th June. 

13 entries were received and all were received before the postal deadline of the 6th June 2011. Each entry was presented with 20 questions (where possible with the Loebner Test Program) as shown at the end of this page.

Questions and responses were recorded for each entry in a file, each of which is available to view on the 2011 Loebner Prize website. Where no response to a particular question was forthcoming, entries were allowed 3 minutes before moving on to the next question. The response files were then used to inform the selection process as follows.

Pre-Selection

Due to technical limitations of the voting technology, a maximum of 10 entries could progress to the audience voting stage. To remove the 3 worst performing entries, each entry was scored by the organising team using the same scoring system used in 2010 (1 point for a correct and human-like answer, 0.5 for a partially correct and human-like answer, 0 points otherwise). The worst performing entries received scores of 0, 3.5 and two entries were tied on 5.5. The responses from the two tied entries were very similar and could not be separated and so both were excluded from the final selection phase, leaving 9 entries to progress to the final stage.

Selection

12 staff and students at the University of Exeter took part in the selection process. The organisers involved in pre-selection did not take part in this second stage. The audience was shown a slide containing each question and the 9 responses from the entries in a numbered list presented in a single random order (e.g. entry 1 remained at position 1 throughout). They were then asked to determine which 4 answers were most human-like and to enter the number of the best entries into their audience participation handsets. This was repeated for each of the 20 questions and the results collated. Entry names were not revealed until the voting process was completed.

Results

Rosette 18.01%
Zoe 15.52%
ChipVivant 12.98%
Tutor 12.30%
Mitsuku 11.84%
Cleverbot 10.57%
Alice 6.96%
Eugene_Goostman 6.95%
UltraHal 4.88%
8pla 5.5
Trane 5.5
Adam-L 3.5
SEARS 0

So congratulations to Rosette, Zoe, ChipVivant and Tutor and commiserations to the rest.

Questions

  • My name is Ed. What is your name?
  • Which is larger, an ant or an anteater?
  • What month of the year is it?
  • What is my name?
  • Dave is older than Steve but Steve is older than Jane. Who is youngest, Steve or Jane?
  • What day will it be tomorrow?
  • What's your favorite food?
  • My friend Chris likes to play football. What sports do you like to play?
  • Are you a human or a computer?
  • What letter comes after T?
  • What is the name of my friend who likes to play football?
  • What is your name?
  • What would I do with a screwdriver?
  • How many letters are in the word 'banana'?
  • Have you watched a good film lately?
  • What year will it be next year?
  • What's your favorite fruit?
  • Do you prefer white or black coffee?
  • How old are you?
  • The football was kicked by Fred. Who kicked the football?

Thankyou all for your hard work in producing the entries this year and well done to the finalists.

Transcripts

Below are the transcripts of the question & answer sessions (all PDFs).

 8pla Tom Joyce
 Adam-L  Daniel Burke
 ALICE  Richard Wallace
 ChipVivant  Mohan Embar
 Cleverbot  Rollo Carpenter
 Eugene_Goostman  Vladimir Veselov, Eugene Demchenko, Sergey Ulasen
 Mitsuku  Steve Worswick
 Rosette  Bruce Wilcox
 SEARS  Martin Lefley
 Trane  Robert Mitchell
 Tutor  Ron Lee
 Ultrahal  Robert Medeksza
 Zoe  Adeena Mignona
Google+