Reproducibility is routine in programming. Science should learn from it.

Edward Miller
2 min readDec 9, 2020

--

There is a major reproducibility crisis across the sciences. Despite multiple centuries of experience, we have yet to come up with a solution.

Every study is detailed in the form of unstructured natural language, from the experiment design to the hypothesis to the results.

We can do better.

Programmers have a unique appreciation for formal specification, automated error checking, standards, and structured data. These are the basic building blocks of our profession.

In recent years, in particular, there has been a strong push towards Continuous Integration (CI), Continuous Delivery (CD), Infrastructure as Code (IaC), Software Configuration Management (SCM), and so on using structured syntax like dockerfiles, vagrantfiles, Chef recipes, and so forth.

Using these tools it is possible to deploy reproducible binaries from source code in a completely automatic way. If you are methodical enough, you can create Reproducible Builds that are identical down to the bit regardless of how many times you build it, or who executes the build, and what type of computer they use.

This is important because it is all too easy for someone to make an error when transforming source code into a binary.

In a very similar way, it is all too easy for someone to deviate from the experiment setup procedures because of unclear documentation. It may simply be impossible to reproduce at all, if something was missed.

It would be entirely possible to create a Domain-Specific Language based on a simple structured syntax such as XML, and use this to define experiment design. Such a syntax could have rules that would tell you if there are certain classes of obvious errors or omissions, using an automated check, either when it is uploaded to arXiv or before then when it is still only on your computer.

It could throw either warnings or errors if you fail to document any tool or input, its manufacturer, purity, container, storage temperature, production date, and any other relevant information.

Additionally, this kind of structured syntax if used for recording hypotheses and presentation of data could make it much easier to automatically parse, analyze, and display the aggregated results of many experiments, and assist in the creation of automated meta-analyses using various forms of artificial intelligence.

--

--

No responses yet