Reproducibility and information preservation
Measuring reproducibility in computer systems research
The reproducibility, or lack thereof, of scientific research is a hot topic recently. Among real scientists, reproducibility of experiments has always been an important goal, although as experimental materials and volumes of data they can generate have grown rapidly, it is becoming more and more difficult to achieve this goal. In principle, the "Methods and Materials" section of a scientific journal article should provide the information needed (at least, for another qualified scientist in the same area) to reproduce the results. In practice, this is not always possible, although one must at least pay lip service to this in order to get a paper accepted. In some domains, however, it is estimated that up to half or more of reported results are irreproducible; however, once a result is published (especially by a high-profile journal) it can become difficult for others to publish results that contradict apparently-established facts.
Read more »