The College is excited to host the 2011 Loebner Prize on 19 October. The top four AI programs will compete to win a share of a prize fund worth $5750, with additional prizes available this year for the Junior Loebner Prize.
Hear from the event organiser discussing Will we ever create thinking machines?
2011 Loebner Prize
The Loebner Prize, named after its founder and philanthropist Hugh Loebner, is an annual world-wide contest to test the state-of-the-art in artificial intelligence (AI).
The contest is based on the Turing Test conceived by the British polymath Alan Turing in 1950 which tests whether machines can be considered as artificially intelligent. In this test a panel of human judges must ask questions of a human and a computer program and determine which is which. The task for the computer program is to answer questions in as a human would and thus fool the judges into mistakenly judging them as human rather than machine. Turing stated that machines that could achieve this task could be said to be 'thinking'.
The Loebner Prize is an annual contest based on the Turing Test and has run continuously since 1991. In the 2011 contest $5750 prize money and the Bronze medal will be awarded as follows:
1st place $4000 and the Annual Bronze Medal to the submitters of the most human-like machine as determined by the panel of judges.
2nd place $1000
3rd place $500
4th place $250
At risk will be the $25,000 Silver Medal Prize.
A Gold Medal (pictured) and $100,000 prize is awarded to a machine that generates responses that are indistinguishable from a human.
Junior Loebner Prize
For the first time in its history the Loebner Prize will also include a separate prize known as the Junior Loebner Prize in which the judging panel will consist of students between 12 and 14 years old. This prize arose after a paper and discussion at AISB 2010 where contest organiser, Dr Keedwell proposed a Turing Test based on child development. It should be fascinating to see if the students discriminate between the machine and human differently to the adult judges. An additional small prize fund is available for the machines in this contest. To choose the judges, the College is running a competition involving local schools as part of our outreach programme.
Visual Turing test
Dr Antony Galton and Michael Barclay from the University of Exeter have devised a test that asks machines to mimic our visual recognition abilities. New Scientist magazine have reported (Take the visual Turing Test) on the test and have a copy of the test online for you to try out. Take the test to see what you deem as the most "human" answer.
The contest will take place on Wednesday 19th October 2011 in the Harrison Building.
Programme for the day is as follows:
- 9.00am - Welcome from Mark Overton, Deputy Vice Chancellor (External Affairs) - Room 101
- 9.30am-12noon - Main Loebner Prize Contest (Rooms 101-103)
- 12.20pm - Main contest prize giving (Rooms 101-103)
- 12.30pm - Junior Loebner Prize Contest begins (Rooms 101-103)
- 12.30pm onwards - Buffet lunch and Tours available (Room 103)
- 2pm - 4.00pm Presentations from Judges (Room 170)
- 4.00pm - Coffee (Room 103)
- 4.15pm - 5.00pm - Panel discussion (Room 170)
All timings are approximate and subject to change.
Harrison Building, North Park Road, Exeter, Devon, UK (Google Map).
The contest is being organised by Dr Ed Keedwell with kind assistance from Dr Zena Wood, Prof. Richard Everson and Dr Hugh Loebner.
Entries will be selected in a process similar to that undertaken in previous years in that a series of questions will be posed to each entry and the responses recorded. The responses will then be voted on by an audience of participants at the University of Exeter, with the 4 top-ranked selected for entry to the main contest.
Update: Entries will be tested on a Core i7 PC with 6GB RAM running Windows 7. Please note that the machines used for the contest itself (if not provided by entrants) may be of lower specification but will use the same OS.
The selection process for entries will operate as follows:
- Each entry will be tested against a number of questions using the Loebner Protocol by an automated system that ensures each entry is tested in the same fashion. The questions themselves will be similar in nature to those used last year which are shown below.
- The results for each question will be collated onto a presentation slide showing the question and answers from all entries presented in a random order.
- An audience consisting of staff and students at the University of Exeter will be asked to vote for the most human-like of the responses for each question using audience participation technology.
- The top-4 rated entries will be selected for the Loebner Prize in October based on the summation of votes for all entries across all questions. In the unlikely event of a tie for the last place, the organiser will have the final decision.
Last Year's Questions:
- My name is Bill. What is your name?
- Which is larger, a pineapple or a pine tree?
- What is a calendar?
- What is my name?
- John is taller than Mary and Mary is taller than Sue. Who is shorter, John or Sue?
- What day of the week is it?
- What will be tomorrow's date?
- My friend Bob likes to play tennis. What game does Bob like to play?
- Are you a human or a computer?
- Do you have any brothers or sisters?
- What number comes after twelve?
- The ball was hit by Bill. What did Bill hit?
- Who hit the ball?
- What time do you go to be at night?
- What is your favorite food?
- What is the name of my friend who likes to play tennis?
- What would I do with a knife?
- How many letters are in the alphabet?
- Have you read a good book lately?
- What time is it?
- What's your favorite color?
- Do you prefer cats or dogs?
The selection event took placeat the College of Engineering, Mathematics and Physical Sciences at the University of Exeter on the 24th June.
13 entries were received and all were received before the postal deadline of the 6th June 2011. Each entry was presented with 20 questions (where possible with the Loebner Test Program) as shown at the end of this page.
Questions and responses were recorded for each entry in a file, each of which is available to view on the 2011 Loebner Prize website. Where no response to a particular question was forthcoming, entries were allowed 3 minutes before moving on to the next question. The response files were then used to inform the selection process as follows.
Due to technical limitations of the voting technology, a maximum of 10 entries could progress to the audience voting stage. To remove the 3 worst performing entries, each entry was scored by the organising team using the same scoring system used in 2010 (1 point for a correct and human-like answer, 0.5 for a partially correct and human-like answer, 0 points otherwise). The worst performing entries received scores of 0, 3.5 and two entries were tied on 5.5. The responses from the two tied entries were very similar and could not be separated and so both were excluded from the final selection phase, leaving 9 entries to progress to the final stage.
12 staff and students at the University of Exeter took part in the selection process. The organisers involved in pre-selection did not take part in this second stage. The audience was shown a slide containing each question and the 9 responses from the entries in a numbered list presented in a single random order (e.g. entry 1 remained at position 1 throughout). They were then asked to determine which 4 answers were most human-like and to enter the number of the best entries into their audience participation handsets. This was repeated for each of the 20 questions and the results collated. Entry names were not revealed until the voting process was completed.
So congratulations to Rosette, Zoe, ChipVivant and Tutor and commiserations to the rest.
- My name is Ed. What is your name?
- Which is larger, an ant or an anteater?
- What month of the year is it?
- What is my name?
- Dave is older than Steve but Steve is older than Jane. Who is youngest, Steve or Jane?
- What day will it be tomorrow?
- What's your favorite food?
- My friend Chris likes to play football. What sports do you like to play?
- Are you a human or a computer?
- What letter comes after T?
- What is the name of my friend who likes to play football?
- What is your name?
- What would I do with a screwdriver?
- How many letters are in the word 'banana'?
- Have you watched a good film lately?
- What year will it be next year?
- What's your favorite fruit?
- Do you prefer white or black coffee?
- How old are you?
- The football was kicked by Fred. Who kicked the football?
Thankyou all for your hard work in producing the entries this year and well done to the finalists.
Below are the transcripts of the question & answer sessions (all PDFs).
|Eugene_Goostman||Vladimir Veselov, Eugene Demchenko, Sergey Ulasen|
The 2011 Loebner Prize was held at the University of Exeter on the 19th October.
|Professor Noel Sharkey||Professor of Artificial Intelligence & Robotics, University of Sheffield|
|Dr Antony Galton||Reader in Knowledge Representation, University of Exeter|
|Paul Marks||Chief Technology Correspondent, New Scientist|
|Jonny O'Callaghan||Staff Writer, How it Works Magazine|
Mark, Steve & Dawn (University of Exeter), Samantha (University of Plymouth)
The contest was comprised of four rounds. In each round, the judge was paired with a new AI entry and human confederate ensuring that each judge interacts with every AI and human during the course of the contest. At the end of each round the judges must decide which entity is the computer. At the end of the contest, the judges ranked the AI entries and humans. The results are as follows:
None of the AI systems fooled the judges, therefore the Turing Test has not been passed and the Silver Medal was not awarded. The bronze medal and cash prizes were therefore awarded based on the ranks awarded by the judges which were as follows:
|Entry||Mean Rank |
(lower is better)
|Rosette||1.5||Bronze Medal & $4000|
Congratulations to Bruce Wilcox's Rosette as the 2011 Loebner Prize winner!
None of the AI systems fooled the junior judges which were conducted over shorter, 10 minute rounds. However the ranks were significantly different.
|Entry||Mean Rank |
(lower is better)
Contest organiser Dr Ed Keedwell, Deputy Vice Chancellor Prof. Mark Overton and Dr Hugh Loebner spoke at the beginning of the day. Dr Keedwell welcomed spectators, confederates and judges alike and Prof. Overton formally welcomed the audience and described the University of Exeter's continuing success in research and the links between AI research at Exeter to the contest. Dr Loebner explained the rules of the contest to participants and spectators.
The main contest was run in rooms 101-103 in the Harrison Building. One conversation was projected onto a screen in the confederates room and two 40" LCD screens in the Judges room also relayed what was happening to the audience seated behind.
Junior Loebner Prize
Following the main contest, the Junior Loebner Prize took place with four junior judges from Torquay Boys Grammar School. The junior judges were only permitted 10 minutes with each AI, but still managed to correctly identify the machine each time, so well done to them.
After lunch, Dr Loebner and judges Prof. Noel Sharkey and Dr Antony Galton presented their thoughts on their research, experience of the Loebner Prize and thoughts on the Turing Test as a whole.
The day concluded with a panel discussion including Jonny O'Callaghan, Dr Galton, Prof. Sharkey, Dr Loebner and Dr Keedwell. Questions were posed by the audience and vibrant discussions were had as to the suitability of the Turing Test as a test of intelligence, the question of whether biological systems could be considered artificial in the AI sense, and the question of whether systems that perform brute-force search can be considered to be truly understanding the problem they are solving. The judges were then presented with a small gift and the day was concluded.
The Loebner Prize always receives media interest from around the world, this year was no different. We had reporters and film crews at the event as well as Paul Marks, senior technology correspondent from New Scientist and Noel Sharkey, Professor of artificial intelligence and robots (University of Sheffield) who writes for the Guardian they took part as a confederate and a judge respectively.
- Professor Noel Sharkey wrote an article in the Guardian entitled "Would artificial intelligence outsmart me? I needn't have worried"
- Paul Marks wrote in New Scientist "Chatbots fail to convince judges that they're human"
- Bruce Wilcox winner from this year's competition took part in an interview for the AiDreams website
- Forums including AI Nexus and Chatbots.org have been discussing the competition and future developments
- iProgrammer website have also reported on the day, displaying the winning transcript from Rosette.
Further media items will follow as well as independent filmmaker Miranda Yousef's film about the day.
The 2012 Loebner Prize took place in Bletchley Park on Tuesday 15th May 2012 to mark Alan Turing's centenary celebrations. The contest is being run by David Levy who won the prize in 1997 and 2009. Further information about the prize can be found on the Loebner website.
A team from the University of Exeter's Computer Science department (Dr Ed Keedwell, Max Dupenois and Kent McClymont) will be running a webcast of the Loebner Prize and so, for the first time, interested people from around the world will be able to follow the conversations the judges have as they happen. The URL for the webcast will appear above shortly.