The streetlight effect in software regulation

Happy New Year, everyone! The start of this new year also brings my second term in Italy, after a very productive (but exhausting!) first term. In the last few months, I learned much and meet various colleagues and professors, and joined some interesting projects. However, I have not blogged as much as I planned, and that is something I intend to change, as blog posts seem a good medium for texts that are too big for a reasonable Twitter thread and too informal (or still undeveloped) for a paper.

To start this year, I would like to follow up on a post I committed in October. There, I presented an argument against the claim that tech law scholars do not need to learn to program, mainly because the skills gained from programming courses are not necessarily the ones required for critical engagement with software systems.

Research on computer technologies, IMHO, should not rely (just) on researchers with double degrees, but rather on multidisciplinary research teams which allow for a division of labour and the frank exchange of ideas¹. Nevertheless, I would be remiss if I did not point out that this approach to research introduces its own threats to the validity of a research project.

For a legal scholar based in a university, the natural place to seek a computing collaborator is the university itself. This is a wise choice, as computer science scholars (as well as those from other computing-heavy areas such as statistics) have access to a wealth of specialised knowledge and experience that can provide the desired insights on computer systems. In fact, there are various examples of this kind of scholarship, such as Brkan and Bonnet (2020)’s analysis of black boxes, whichs present insights on the technological and legal feasibility of approaches to system transparency.

Computing researchers have much to contribute to studies of tech law. However, if one wants to understand how computer systems operate in practice, it is important to keep in mind that software development in academia is different from software development in the industry. As professor Matt Might puts it, academic code frequently is meant as a “proof of concept”, that is, as a demonstration of the feasibility of a given method. This means that the novelty of the presented algorithms or software techniques, in detriment of other values, such as ease of maintenance or code legibility.

Software developed for end consumers or corporations follows a different set of priorities.² At the opposite end of the spectrum from academic code, enterprise software is…boring, with interfaces and functionalities that seem to be years behind the state of the art. This follows from the fact that enterprises require certain properties from their software, such as security and the possibility of long-term maintenance, which are rarely expected on academic proofs of concept.

Since enterprise software and academic software are driven by different goals and constraints, their development also follows different practices. For example, academic software seldom implements unit tests — tests, if present, are usually aimed at showing functionalities —, while industry professionals tend to use software testing for validating software at various levels and for various desiderata, such as security and reliability. In particular, academic software development places a high premium at the development of algorithms for tasks, with less attention to the system engineering required to apply the algorithms to specific problems.³ A legal analysis that follows the same lines might thus end up overlooking the relevance of factors beyond the algorithm, such as the hardware and software structure necessary for running the implemented algorithm.

This does not mean that the knowledge of computer scientists (and other academic software people) is not relevant for understanding non-academic software. After all, non-academic software development relies on the same fundamental concepts developed and used by academic researchers, even if the differences in the implementation of those concepts have substantial practical effects. However, one must be careful to avoid the streetlight effect:

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, “this is where the light is”.

Overlooking the differences between academic and non-academic practices in software development can lead to a misleading image of software systems. An inadequate description of how software is developed and used may, in turn, lead to inadequate legal analysis, which may fail to diagnose relevant issues — such as those involved creating a deployed software system from an algorithm — or miss some potentially useful possibilities for legal intervention, such as requirements for black box accountability.

Avoiding the streetlight effect does not require that one avoids collaboration with academic computer scientists. That would be throwing the baby out with the bathwater, as those researchers are not only familiar with the requirements of scholarship and have the conceptual tools needed for understanding the technical side of software systems. Instead, one should address this problem when outlining the object of research: is it a system that is applied outside of academia? If so, how is it developed and deployed by organisations such as enterprises or government agencies? By looking at a broader range of actors and practices, and not just those closest to a researcher’s personal experiences and acquaintances, it will be possible to obtain a richer image of how computer systems are constructed, situated, and used, which in turn enables a more fruitful analysis of the legal problems related to those systems.