Source code serves a dual purpose. It communicates program instructions to machines and programmer intent to people. Unfortunately, it is common for a person to come to a different conclusion about the behavior of a piece of code than a machine does when executing the program. In other words, a programmer’s interpretation of a piece of code’s behavior differs from that of the machine. While a difference of interpretation can naturally happen in some situations (such as those involving randomness, poorly understood APIs, or undefined behavior), it can also occur in small, self-contained lines of code. These design patterns, which are easy to misinterpret, naturally lead to bugs in code.

We are studying these atoms of confusion using empirical, objective techniques. We have and will continue to run multiple human subject studies. We design our studies around gathering measures of performance and correctness over subjective data.

All of our experiments are performed specifically to make the data available to other researchers online. Many projects make their data available by request, but we have decided to make all of our data (once it’s been scrubbed of personally-identifying information) available online by default. It is our hope that this free exchange of data will make it easier for other researchers to participate alongside us. Additionally we hope that this may encourage others doing separate work to also share their data with the general public.