CICLing verifiability, reproducibility, and open source policy

Instructions for software reviewers

Each paper will be assigned to two kinds of reviewers: scientific reviewers and software reviewers. The task of the traditional scientific reviewers is to review the paper (they are likely not to see the accompanying software) and judge the proposed method for novelty and significance. In contrast, the task of the software reviewers is to judge whether the provided software generates the results reported in the paper and not to judge the method itself.

They would probably skip the state-of-the-art part of the paper and address directly the main algorithm and the experimental methodology and results, trying to locate the description of the input data and the obtained output.

Finally, they should evaluate the following aspects:

   Clearness of presentation. Is it easy to understand from the paper what were the input, the processing steps, and the output?

   Correctness. Does the software completely and accurately support all results provided in the paper?

   Clearness of instructions. Are the installation instructions easy to understand and follow? Are the results easily interpretable? Is it easy to see how the generated results support the claims presented in the paper?

   Simplicity and readability. Is it easy for a reader (willing to do so) to follow the logic of the program? Is it easy to locate the main algorithm and to separate it from interface technicalities? Are the comments in the source code helpful? Does the program contain minimum details needed to prove the correctness of the results and nothing else? It is undesired for the program to include fancy graphical interface or other complicated technical details that obscure its main purpose: to show exactly how the algorithm is implemented and to prove that it does produce the results reported in the paper. Ideally, the input and output should be formatted in such a way that most of the program code would implement the algorithm itself and would not struggle with input-output, user interface, or other things not related to the algorithm.

   Self-containedness. Does the provided material reasonably contain all the data needed to run the software? Will it be easy to run this software in twenty years, when current versions of compilers or operating systems will not be available? It is better if all the necessary files are included; however, it is acceptable if the installation instructions require downloading some commonly used and easily accessible software or data from stable sources that are likely to be easily available in the future; examples may include Perl and WordNet. In the latter case the reviewer is encouraged to include this software in the archive, for convenience of future users. Relying only on free software is a plus, but it is acceptable that the software require items, such as compilers, commercially available from major and stable vendors and widely used by the community. Otherwise, any non-standard item required to run the software should negatively affect the score.

   In summary, are the claims of the paper verifiable using this software? Are the results reproducible? Does the paper, together with the software, contribute to human knowledge, i.e., provide the reader with a new methodology, algorithm, or finding that they could use in their own work, build on it, or compare their own results with it? -- as opposed to merely communicate that the authors have some methodology, without properly transmitting it to the reader.