Dreamsongs Wiki

ReviewCriteria

• Technical contribution—how substantial is the contribution
• Novelty—how novel or innovative are the ideas
• Substantiation—how well substantiated is the contribution (e.g., proof supplied, implementation completed and tested, experiments performed according to accepted methods)
• Presentation—how clearly written and presented is the material
• Argument—how compelling or well-made are the arguments in the paper
• Art/Craft—how well does the paper demonstrate, describe, or promote excellence of artistry or craft in architecture, design, implementation, methodology, or documentation

A technical contribution is a technique or a technology—something concrete that can be applied either to create better software or to foster better technical and scientific understanding. Further, the description of the technical contribution must include a description of the benefit—the benefit is that which gets better based on the contribution.

Novelty is what is new about the contribution. The idea itself, its use in a particular setting, or its use in conjunction with other technologies or techniques can be new. Novelty is measured in terms of published literature as well as known applications, especially those whose source code is publicly available. Moreover, the contribution should not be obvious to a typical practitioner or researcher in the field.

Substantiation is why the contribution or its claimed benefit should be considered valid. The substantiation should not itself be novel.

Presentation is the sum total of all the methods and techniques used to convey the substance of the paper to its audience. This includes the quality of the writing, clarity of explanation, and effectiveness of the narrative and rhetorical structure. Some reviewers are willing to work hard to understand a paper; others aren’t.

A successful submission is not only technically interesting, but well written. Writing is important to many program committee members. Thucydides’ quote from Pericles’ last speech applies to OOPSLA submissions:

A man who has the knowledge but lacks the power to express it clearly is no better off than if he never had any ideas at all. Book

Argument comprises the rhetorical and logical structure of the paper. Argument is how the presentation of the paper convinces a reader that the thrust of the paper is believable in the absence of hard substantiation.

Art and craft touches on what makes either what the paper presents or the paper itself uplifting, elegant, or beautiful. A beautiful and effective design for an application can demonstrate good art/craft, as can a well-crafted, well-argued, and well-written presentation.