Measuring and Assessing Open Source Project Impact and Community Health. A CZI EOSS Community Call Resource

This resource was first developed by attendees at the October, 2024 Chan Zuckerberg Initiative community call for the Essential Open Source Software program.

Projects define and measure impact in a number of ways. In scientific OSS, impact is often defined as the intensity of usage in a given discipline: How much of the field and its work relies on scientists using the software? How critical is the software to advancing science in the domain? What other OSS projects depend on the software? What kinds of breakthroughs does the software enable that other options do not?

Projects tend to measure impact in a number of ways, most of which do not adequately answer the questions proposed above. The following activities instead act as rough proxies for assessing impact, often because they involve metrics that are easy to extract from open source tools:

Recording package download counts Tracking citations in the scientific literature Monitoring downstream uses of the software (e.g., dependencies) Counting GitHub stars Quantifying engagement at in-person and online events Assessing interest from institutions, science and computing facilities, and industry partners in supporting software on their systems Measuring changes in the number of contributors or contributions Calculating past funding support and projecting future support needs The above metrics can be useful as a starting point in assessing interest in and usage of software. But as pointed out in a 2015 blog post by Daniel S. Katz, the most difficult metrics to track often offer the most value when compared to easier-to-track metrics. Download counts, for example, are easy to record, but those counts tell us little about how the software is being used to advance science or any other goal. Tracking citations (and chains of citations) in scientific papers can be labor-intensive (particularly when examining what the authors used the software for) yet the activity paints a much clearer picture of actual usage and impact.

In the ideal scenario, then, projects would be able to track usage at a more detailed level and address critical questions, such as those in the table below. For each question, we have proposed possible metrics (or information-gathering activities) that might help provide a detailed view of impact. More work is certainly needed to identify other existing ways to answer these questions and to build new approaches to impact assessment.

The full document with resources can be access on the community call webpage.