A metal ruler

Funded by EPSRC logo (EP/G03995X/1).

In conjunction with ENLG 2009.

Following the NSF/SIGGEN Workshop on Shared Tasks and Comparative Evaluation in NLG in April 2007, and the INLG'06 Special Session on Sharing Data and Comparative Evaluation , there has recently been a lot of activity on shared tasks in Natural Language Generation (NLG). The Pilot Attribute Selection for Generating Referring Expressions (ASGRE) Challenge took place between May and September 2007; the Referring Expression Generation (REG) Challenge 2008 has just been completed; the GIVE Challenge is currently underway; and other challenges are being prepared.

To provide a common forum for these activities, we are organising Generation Challenges 2009, an umbrella event designed to bring together a variety of shared-task evaluation efforts that involve the generation of natural language. Evaluation results and participating systems will be presented at the Generation Challenges 2009 Special Session at the 12th European Workshop on Natural Language Generation (ENLG'09).

Shared Tasks

There will be three shared tasks at Generation Challenges 2009 which will be organised independently (for details, please refer to the Call for Participation for each task):

  1. The GIVE Challenge (Koller, Byron, Moore, Oberlander & Striegnitz): Generation of natural-language instructions to aid human task-solving in a virtual environment. This task has started, for more information go to: http://homepages.inf.ed.ac.uk/v1akolle/proj/give
  2. GREC Task (Belz, Gatt, Kow & Viethen): Generation of references to named entities in discourse; exact task definition to be announced. For more information go to: http://www.nltg.brighton.ac.uk/research/genchal09/grec
  3. TUNA Progress Test (Gatt, Belz & Kow): An opportunity to improve on REG'08 Task 3, i.e. mapping from TUNA domain representations to referring expressions. For more information go to: http://www.nltg.brighton.ac.uk/research/genchal09/tuna

Special Track - Proposals for Future Tasks

We invite submissions of papers describing ideas for future shared tasks in the general area of language generation. Proposed tasks can be in the area of core NLG, or straddle more than one research area in which language is generated, e.g. core NLG and MT, or core NLG and text summarisation. Submissions should describe possible future tasks in detail, including information regarding organisers, task description, motivating theoretical interest and/or application context, size and state of completion of data to be used, and evaluation plans.

For submission details, see the Call for Papers.

Open Track

We invite submissions of papers reporting any language generation method and results obtained using the training and development data from any of the shared task tracks. The task definition should differ from that in the corresponding shared task track, and submissions will be reviewed according to standard workshop reviewing criteria.

For submission details, see the Call for Papers.

Evaluation Methods Track

We invite submissions of papers describing any evaluation method that can be used to evaluate systems submitted to any of the shared tasks. Papers should report results for applying the evaluation method to one or more systems implementing a solution to one of the shared tasks (although different data may be used). Submissions will be reviewed according to standard workshop reviewing criteria.

For submission details, see the Call for Papers.

Dates

30 November 2008 Submission of task proposal papers
31 December 2008 Notification of acceptance
12 January 2009 Submission of papers in Open and Evaluation Methods tracks
early/mid February 2009 Expected deadline for camera-ready papers
30-31 March 2009 ENLG'09 (provisional)

Review Committee

  • Robert Dale, Macquarie University, Australia
  • Kevin Knight, Information Sciences Institute, University of Southern California, USA
  • Amanda Stent, Stony Brook University, USA
  • Chris Mellish, Aberdeen University, UK
  • Mirella Lapata, ICCS, University of Edinburgh, UK

Organisation

  • Anja Belz, NLTG, University of Brighton, UK
  • Albert Gatt, Computing Science, University of Aberdeen, UK
  • Eric Kow, NLTG, University of Brighton, UK

GenChal'09 homepage: http://www.nltg.brighton.ac.uk/research/genchal09
GenChal'09 email:

Last modified: 2009-01-19 12:13

News

2 December 2008
EPSRC to fund Generation Challenges 2009
9 September 2008
Second Call for Papers
5 September 2008
GREC First Call for Participation
15 August 2008
Tuna First Call for Participation
11 July 2008
Call for Papers
01 April 2008
Announcement and Call for Tasks