I have a complicated relationship with bibliometrics (i.e. “research impact metrics”), as does the field of academic librarianship and academia itself, if we’re honest. On one hand, there is something undeniably fascinating about how statistical formulas can help us understand “scholarship as conversation” at a macro-level. As a field, bibliometrics has been heavily influenced by information scientists. Eugene Garfield, often cited as a founder of modern bibliometrics and cocreator of Journal Citation Reports, held a degree in Library Science from Columbia University, for example, and created the infamous journal impact factor not to assess the quality of a researcher’s CV, but to help librarians make decisions about which journal subscriptions to purchase. This relationship with librarianship makes me feel some level of responsibility to/for the field of bibliometrics even if I’m not a bibliometric scholar per se. Librarians’ collection development responsibilities have traditionally positioned us as arbiters and assessors of information quality, for better or worse, which tracks with the project of research assessment.
On the other hand, several high-profile bibliometric formulas have been coopted by corporate entities to serve for-profit purposes and have been misapplied in the academe as markers of excellence for individual researchers. In the 2010s, research assessment experts denounced the use of Clarivate’s journal impact factor and overreliance on quantitative citation-based evaluation of research “impact” through statements like the San Francisco Declaration on Research Assessment (now operating as DORA) and the Leiden Manifesto for research metrics. I share concerns cited in those statements about whether folks who use these numbers in retention, promotion, and tenure (RPT) processes can accurately explain the insight each formula is supposed to convey. Journal impact factor and Elsevier’s “CiteScore” are not generic proxies for journal quality. They aggregate citation rates over a (relatively short) period. In a vacuum, that seems like potentially useful information to consider as one part of a comprehensive assessment of a journal’s and/or a researcher’s portfolio. As a key determiner of whether a journal or its authors offer any value to the scholarly conversation, it becomes dangerous.
It is also well established that these popular metrics don’t map well across academic disciplines because different disciplines have significantly different publishing rates and citation practices, whereas metrics that normalize citation attention between disciplines like field-weighted citation impact are not commonly used in RPT procedures (from my understanding). Consequently, the perceived value of scholarship in the arts, humanities, and social sciences is diminished when their citation numbers are compared to STEM fields. In the fields that bibliometrics favor, their outsized influence in RPT processes forces faculty to “study for the test,” thereby limiting the list of suitable publication venues to a select, well-established few (often owned by one of the big five publishers). From a simple assessment standpoint, Goodhart’s law reminds us “when a measure becomes a target, it ceases to be a good measure.”
Although Goodhart’s law is certainly applicable to the current use of bibliometrics in academia, I think the problem requires us to zoom out a bit. The core issue leading to the misapplication of bibliometrics is academia’s attempt to assess a concept as sweeping and nebulous as “impact” using a few numbers that address one specific type of impact—contribution to the scholarly conversation. Furthering the progress of knowledge is a valid and core reason why many scholars do research, but it certainly isn’t the only reason why people do research, and it’s not the only value that research provides to the world, so it shouldn’t be the only aspect of research impact we evaluate and thereby ascribe value to.
Of course, making any kind of structural change in academia is easier said than done for many reasons, including the entrenchment of bibliometric-based evaluation in the publishing industry and in academic tenure processes. But, there are folks exploring alternatives: in 2023, the University of Illinois Urbana-Champaign introduced a Public Engagement Research Option (PERO) for Promotion and Tenure, designed for faculty who “address a societal problem to contribute to the public good; produce traditional and non-traditional outputs; collaborate with communities or organizations; [and] engage in a mutually beneficial exchange of knowledge and resources with community partners.” The HumetricsHSS Humane Metrics Initiative provides workshops and toolkit resources for academic departments to consider how they can embed their stated values in their evaluation of scholarly life, reminding us, “if we don’t measure what we value, we will only value what we can measure.” Just this month, DORA published their Practical Guide to Implementing Responsible Research Assessment at Research Performing Organizations.
I heard about that DORA resource while attending a new (to me) conference in April. The Center for Advancing Research Impact in Society hosts an annual summit around “broader impacts”—the National Science Foundation’s terminology for research’s “potential to benefit society and contribute to the achievement of specific, desired societal outcomes.” I’d been debating which research impact professional development opportunities I should explore, given that “research impact” is included in my job title. This one intrigued me because of its necessarily broad conception of impact, and I came away energized by the creative research development work happening around the world. Research development offices are logically very interested in understanding and tracking the impact of research conducted at their institutions, and they clearly spend a lot of time thinking about how to do research assessment well, and how to encourage researchers to embed the public good in their plans for research outputs. Two favorite sessions included the Association of Public & Land-grant Universities’ walk-through of “Modernizing Scholarship for the Public Good: An Action Framework for Public Research Universities,” and the Research Development team at University of Texas at Austin’s presentation on their Research-to-Policy workshops that teach faculty how to make their work accessible to legislators.
Although librarians are welcomed at the ARIS Summit, I didn’t meet any other library folks wandering between conference rooms. I mention this not to admonish anyone or toot my own horn, but rather to raise the question: why shouldn’t we participate at these types of conferences, contributing to and learning from conversations about research impact/assessment? I would argue the prevailing (mis)conception of research impact is an information literacy issue. As a librarian, I feel a responsibility to remind scholars of the ways “authority is constructed and contextual” and how our interpretations of information (like bibliometrics) influence the way those specific pieces of information come to have value. These issues are especially relevant to those of us in the scholarly communication subfield, as we play a particular role in helping scholars decide where to communicate and share their work. I now see the “open knowledge” and “research impact” components of my job title as much more intertwined that I originally thought, but it required me to conceptualize a holistic vision of research impact that incorporates the societal contribution we believe many types of scholarship can make. Academic libraries are increasingly involved at every stage of the knowledge lifecycle, and as such, I think research assessment/evaluation is an area where librarians could reconceptualize our approach, training, responsibilities, and campus partnerships.