*********************************************************************** News: After careful consideration and consultation with participants we have decided to move the GREC'09 results meeting to the ACL-IJCNLP'09 Workshop on Language Generation and Summarisation in Singapore on 6 August 2009. This enables us to extend the system development period for the GREC'09 Shared Tasks until 1 June 2009. Again in consultation with the 8 registered participating teams, we have decided to re-open the GREC Shared Tasks to new participants. Anyone wishing to take part can register via the GREC'09 homepage (url see below). The results reports by the organisers and system reports by the participants will be published in the proceedings of the ACL-IJCNLP'09 Workshop on Language Generation and Summarisation. *********************************************************************** *NEW* CALL FOR PARTICIPATION - Submission deadline: 1 June 2009 GENERATION OF REFERENCES IN CONTEXT (GREC) TASKS 2009 ----------------------------------------------------- Part of Generation Challenges 2009. Generation Challenges 2009 is being organised to provide a common forum for a number of different NLG Shared Tasks (see http://www.nltg.brighton.ac.uk/research/genchal09/). As part of Generation Challenges 2009, we are organising two GREC Shared Task Competitions. The first is the GREC-MSR (Main Subject References) Task which uses the GREC-2.0 Corpus of 2,000 Wikipedia introduction sections in which references to the main subject of the Wikipedia article have been annotated, and the task is to develop a system that can select (from a given list) an MSR that is appropriate in the context. The second is the GREC-NEG (Named Entity Generation) Task which uses the new GREC-People Corpus of 1,000 Wikipedia introduction sections about people in which single and plural references to all people mentioned in the text have been annotated. The task in GREC-NEG is to select appropriate referential expressions for all mentions (singular and plural) of people. Submissions to both tasks will be evaluated using a range of intrinsic and extrinsic measures, some assessed automatically, some manually. Submitted systems and evaluation results will be presented in a special session at the ACL-IJCNLP'09 Workshop on Language Generation and Summarisation on 6 August 2009 in Singapore, and accompanying reports will be published in the workshop proceedings. 1. Background -------------- There has been increasing interest recently among text summarisation researchers in postprocessing techniques to improve the referential clarity and coherence of extractive summaries, and among language generation researchers in generating referential expressions in context. The GREC tasks are aimed at researchers in both of these groups, and the objective is the development of methods for generating chains of referential expressions for discourse entities in the context of a written discourse, as is useful for postprocessing extractive summaries and repeatedly edited texts (such as Wikipedia articles). 2. Data -------- The GREC data resources consist of introduction sections collected from Wikipedia articles in which several broad categories of overt reference to named entities have been annotated. The annotations include features encoding basic syntactic and semantic information. The GREC-2.0 corpus consists of 2,000 texts in five different domains (cities, countries, rivers, people and mountains). In this corpus, only references to the single entity that is the main subject of a Wikipedia article (e.g. "Michael Faraday") have been annotated. The new GREC-People corpus consists of 1,000 texts in just one domain, people. Here, all references to all people mentioned in a text have been annotated. GREC-People therefore includes explicit coreference annotation for one or more coreference chains (whereas in GREC-2.0 texts there is always just one annotated coreference chain). For GREC-2.0 and GREC-People we have test sets of 200 and 100 texts, respectively, where referential expressions have been selected by participants in an elicitation experiment. In these test sets, there are three versions for each corpus text, in each of which the referential expressions have been manually selected by a single participant in the experiment. 3. The GREC'09 Tasks -------------------- The GREC-MSR Task has the same task definition as the GREC shared task at REG'08. Participating systems need to select the referential expression (RE) from a given set of alternatives that is most appropriate in the given context, which may involve e.g. ensuring that pronouns can be resolved. Systems will be evaluated both against the REs in the corpus and against human-selected topline solutions for this task. Results and descriptions of participating systems from the REG'08 run of this task can be found here: http://www.aclweb.org/anthology-new/W/W08/#1100 The new GREC-NEG Task is an extension of GREC-MSR in that it requires participating systems to select appropriate referential expressions for all discourse entities of the same type (people in this round) as the main subject of the article. 4. Evaluation ------------- For both tasks, the data has been randomly divided into training, development and test data. Participants will compute evaluation scores on the development set (using code provided by the organisers), and the organisers will perform evaluations on the test data set. We will use a range of different evaluation methods, including intrinsic and extrinsic, automatically assessed and human-evaluated. The intrinsic methods will include string-accuracy, feature-accuracy and string-similarity measures, as well as human-produced quality assessments. The extrinsic methods will include a reading/comprehension experiment and measuring coreference resolver success (for details about the previous edition, see http://www.aclweb.org/anthology/W/W08/W08-1127.pdf). Full details of the evaluation methods for GREC'09 are contained in the Participants' Pack that is distributed to registered participants. 6. Participation ---------------- Registration is now open at the GREC'09 homepage (http://www.nltg.brighton.ac.uk/research/genchal09/grec). Once registered, participants in the GREC-MSR and GREC-NEG Tasks will receive the complete training and development set, evaluation software and detailed documentation (collectively known as the Participants' Pack) for their task(s). 7. Proceedings and Presentations -------------------------------- The GREC'09 meeting will be held as a special session at the ACL-IJCNLP'09 Workshop on Language Generation and Summarisation on 6 August 2009 in Singapore. The session will include overviews of the GREC'09 Tasks and presentations of the evaluation results. The participating systems will additionally be presented as papers in the workshop proceedings, and as short presentations during the GREC'09 special session. Participants are encouraged, but not required, to attend the results session. GREC'09 papers will not undergo a selection procedure with multiple reviews, but the organisers reserve the right to reject material which is not appropriate given the participation guidelines. 8. Important Dates ------------------ May 15-Jun 01 GREC'09 submission of test data outputs: 1. submit system report; 2. download test data; 3. submit outputs within 48h. Jun 01, 2009 Final deadline for submission of GREC'09 test data outputs Jun 01-30, 09 GREC'09 Evaluation period Aug 06, 2009 GREC'09 meeting at ACL-IJCNLP'09 Workshop on Language Generation and Summarisation 9. Organisation --------------- Anja Belz, NLTG, University of Brighton, UK Eric Kow, NLTG, University of Brighton, UK Jette Viethen, Macquarie University, Australia Albert Gatt, Computing Science, University of Aberdeen, UK GREC'09 homepage: http://www.nltg.brighton.ac.uk/research/genchal09/grec Generation Challenges homepage: http://www.nltg.brighton.ac.uk/research/genchal09 Generation Challenges email: nlg-stec@itri.brighton.ac.uk