Enter Your Score
Does this paper fit in SIGMORPHON 2012?
(The focus of SIGMORPHON 2012 is on applying computational
techniques to problems in phonetics, phonology, and morphology; see the
Call for Papers for more details.)
5 = Appropriate for SIGMORPHON 2012. (most submissions)
4 = Computational linguistics or NLP, though it's not typical SIGMORPHON material.
3 = Possibly relevant to the audience, though it's not quite computational linguistics or NLP.
2 = Only marginally relevant.
1 = Inappropriate.
For the reasonably well-prepared reader, is it clear what was done
and why? Is the paper well-written and well-structured? Does the
English or the mathematics need cleaning up? Would the explanation
benefit from more examples or pictures?
Is there sufficient detail for an expert to validate the work, i.e., by replicating experiments or filling in theoretical steps?
(Take into account whether any obscurity or minor English errors could be fixed with relatively
little effort, or whether the paper requires more work than is
likely to be carried out in the 2.5 weeks available.)
5 = Admirably clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing.
|Originality / Innovativeness (1-5)|
How original is the approach? Does this paper break new ground in
topic, methodology, or content? How exciting and innovative is the
research it describes?
(Note that a paper could score high for originality even if the
results did not show a convincing benefit.)
5 = Surprising: Noteworthy new problem, technique, methodology, or insight.
4 = Creative: Relatively few people in our community would have put these ideas together.
3 = Somewhat conventional: A number of people could have come up with this if they thought about it for a while.
2 = Rather boring: Obvious, or a minor improvement on familiar techniques.
1 = Significant portions have actually been done before or done better.
|Soundness / Correctness (1-5)|
First, is the technical approach sound and well-chosen? Second,
can one trust the claims of the paper -- are they supported by
proper experiments, proofs, or other argumentation?
5 = The approach is very apt, and the claims are convincingly
4 = Generally solid work, though I have a few suggestions
about how to strengthen the technical approach or evaluation.
3 = Fairly reasonable work. The approach is not bad, and at least the
main claims are probably correct, but I am not entirely ready to accept them
(based on the material in the paper).
2 = Troublesome. There are some ideas worth salvaging here,
but the work should really have been done or evaluated differently, or
1 = Fatally flawed.
|Impact of Ideas and Results (1-5)|
How significant is the work described? If the ideas are novel, will they also
be useful or inspirational? If the results are sound, are
they also important?
5 = Will affect the field by altering other people's choice of research topics or basic approach.
4 = Some of the ideas or results will substantially help other people's ongoing research.
3 = Interesting but not too influential. The work will be cited, but mainly
for comparison or as a source of minor contributions.
2 = Marginally interesting. May or may not be cited.
1 = Will have no impact on the field.
|Meaningful Comparison (1-5)|
Does the author make clear where the
problems and methods sit with respect to existing literature?
Are the references adequate? Are
any experimental results meaningfully compared with
the best prior approaches?
5 = Precise and complete comparison with related work. Good job
given the space constraints.
4 = Mostly solid bibliography and comparison, but I have
3 = Bibliography and comparison are somewhat helpful, but it could
be hard for a reader to determine exactly how this work relates
to previous work.
2 = Only partial awareness and understanding of related work, or a flawed empirical comparison.
1 = Little awareness of related work, or lacks necessary empirical comparison.
|Impact of Resources (1-5)|
In addition to its direct intellectual contributions, does the
paper promise to release any new resources, such as an
implementation, a toolkit, or new data?
If so, is it clear what will be released and when? If so, will
these resources be valuable to others in the form in which they are
released? Do they fill an unmet need? Are they at least sufficient
to replicate or better understand the research in the paper?
(This question encourages authors to help the field advance, by releasing their systems, data, or tools.)
5 = Enabling: The newly released resources should affect other people's
choice of research or development projects to undertake.
4 = Useful: I would recommend the new resources to other researchers or developers for their ongoing work.
3 = Potentially useful: Someone might find the new resources useful for their work.
2 = Documentary: The new resources are useful to study or replicate
the reported research, although for other purposes they may have
limited interest or limited usability. (this is a positive rating)
1 = No usable resources released. (most submissions)
are many good submissions competing for slots at SIGMORPHON 2008; how
important is it to feature this one? Will people learn a
lot by reading this paper or seeing it presented?
In deciding on your ultimate recommendation, please think over all
your scores above. But remember that no paper is perfect, and
remember that we want a conference full of interesting, diverse, and
timely work. If a paper has some weaknesses, but you really got a
lot out of it, feel free to fight for it. If a paper is solid but you
could live without it, let us know that you're ambivalent. Remember
also that the author has a couple of weeks to address reviewer
comments before the camera-ready deadline.
Should the paper be accepted or rejected?
5 = Exciting: I'd fight to get it accepted
4 = Worthy: I would like to see it accepted
3 = Borderline: I'm ambivalent about this one
2 = Mediocre: I'd rather not see it in the conference
1 = Poor: I'd fight to have it rejected
|Reviewer Confidence (1-5)|
5 = Positive that my evaluation is correct. I read the paper
very carefully and am very familiar with related work.
4 = Quite sure. I tried to check the important points carefully,
and checked for uncited prior work. It's unlikely, though
conceivable, that I missed something that should affect my ratings.
3 = Pretty sure, but there's a chance I missed something.
Although I have a good feel for this area in general, I did not
carefully check the paper's details, e.g., math, experimental design,
2 = Willing to defend evaluation, but it is fairly likely that I
missed some details, didn't understand some central points, or can't
be sure about the novelty of the work.
1 = Not my area, or paper is very hard to understand. My
evaluation is just an educated guess.