Tag: research assessment

OpenAlex and values-aligned tools

It is within our sphere of influence to learn about and use values-aligned infrastructure and leverage our positions as information professionals to teach and encourage others to use values-aligned infrastructure as well.

Two different kinds of bridges crossing a river. Photo by Katelyn G on Unsplash

Academia needs to untangle from corporate entities our ability to understand and evaluate scholarly activity. Companies like Elsevier and Clarivate have effectively embedded evaluation methods into academia that rely on their proprietary tools (Scopus and SciVal; Web of Science and Journal Citation Reports and InCites, respectively). Conveniently, these products also cast judgement on the “legitimacy” of scholarly publication venues and whether they are valuable as lines in a tenure portfolio and/or in library subscription decisions, which raises questions about conflict of interest. One would assume journals owned by Elsevier are rarely excluded from the Scopus corpus, for example.

The need to sever indexing and evaluation of scholarly activity from corporate control dovetails with the need to regain a degree of control over the entire scholarly publishing enterprise. One path might look like a significant shift toward venues that are owned-and-operated by academic institutions or scholarly societies or academic alliances as opposed to for-profit publishers, but a basic barrier to this shift is the inconsistent and incomplete indexing of such venues in scientific knowledge graphs like Scopus and Web of Science. In their recent article on the subject, Nazarovets et al. (2026) conclude with this suggestion: “…the uneven visibility of UJs [university journals] highlights a structural blind spot in global research assessment: unless more inclusive infrastructures like DOAJ and OpenAlex are more widely recognized and further improved in terms of comprehensiveness in coverage and metadata reliability, large parts of scholarly production will remain invisible even if a move is made away from using WoS [Web of Science] and Scopus.” This is just one possible benefit of investing in open infrastructure like OpenAlex (as recommended by UNESCO). Open infrastructure can be defined as free, foundational tools and resources upon which open science/scholarship can be conducted, shared, and explored.

For those unfamiliar with OpenAlex, it is essentially a large database of scholarly works and related metadata. OpenAlex focuses on comprehensive inclusion of scholarly works as opposed to “curation” of legitimate sources as practiced by proprietary services. It has a web interface and can be queried using an API, but at its core, it is a piece of infrastructure dedicated to the public domain via CC0 and actively maintained by a 501(c)3 nonprofit. Of course, relying on publisher-reported citation data may be a dubious proposition to begin with, but in the meantime, as we work to recognize and reward other types of research impact evidence, let’s at least try to use comprehensive citation data.

Is OpenAlex as good as its competitors? In my experience, it has usually been as good or better for most purposes, like generating a list of an authors’ publications inclusive of preprints, or charting an institution’s open access publishing trends over time. Most folks who have taken a deep dive seem to agree. Culbert et al. (2024) analyzed OpenAlex reference coverage in comparison to Web of Science and Scopus and found it to be “comparable” even before OpenAlex’s recent improvements, which included an addition of over 50 million new works. In a Katina review, Ho (2025) called OpenAlex “…a promising and reliable alternative to traditional subscription-based citation databases for researchers, university administrators, research institutions, and government funding bodies that are interested in research activities and potential collaborations.” Other scholarly discovery tools like Overton are using OpenAlex for foundational data.

Folks will need to try it for themselves to see if the usability holds up to their standards, but for those on the fence, consider how simply using such tools might be a tiny contribution to a different future for research infrastructure. Coordinators involved in the Barcelona Declaration on Open Research Information had this to say about the declaration’s commitment to supporting infrastructures for open research information:

“What we think is important for organizations is to take seriously their responsibility for supporting these infrastructures—which can only continue to exist, and develop further, if they are both used and supported. That includes financial support but could also involve participating in governance and in-kind efforts, e.g., contributing data and helping improve data quality. Regarding financial contributions, here again Sorbonne University is a great example—when they unsubscribed from Web of Science, they redirected part of that budget to supporting OpenAlex. In general, we advocate making investments in open infrastructure an integral part of institutional budgets.”

For many of us in academic libraries, we don’t have significant resources to invest or divest, although voting with our money is still worth considering if we have authorization to do so. However, what is certainly within our sphere of influence is to learn about and use values-aligned infrastructure and leverage our positions as information professionals to teach and encourage others to use values-aligned infrastructure as well. At my institution, we’ve begun with small projects like integrating OpenAlex into our local scholarly profile platform and working to add our institutional repository as a source of OpenAlex data. Whether it be OpenAlex or another values-aligned tool, we can vote with our usage as a small daily act of support and/or resistance. By teaching about these tools, separate small acts become collective power.

Reconsidering research impact

I have a complicated relationship with bibliometrics (i.e. “research impact metrics”).

The silhouette of a person looking at a huge swirling modern art piece on a digital screen.
Photo by Gunnar Ridderström on Unsplash

I have a complicated relationship with bibliometrics (i.e. “research impact metrics”), as does the field of academic librarianship and academia itself, if we’re honest. On one hand, there is something undeniably fascinating about how statistical formulas can help us understand “scholarship as conversation” at a macro-level. As a field, bibliometrics has been heavily influenced by information scientists. Eugene Garfield, often cited as a founder of modern bibliometrics and cocreator of Journal Citation Reports, held a degree in Library Science from Columbia University, for example, and created the infamous journal impact factor not to assess the quality of a researcher’s CV, but to help librarians make decisions about which journal subscriptions to purchase. This relationship with librarianship makes me feel some level of responsibility to/for the field of bibliometrics even if I’m not a bibliometric scholar per se. Librarians’ collection development responsibilities have traditionally positioned us as arbiters and assessors of information quality, for better or worse, which tracks with the project of research assessment.

On the other hand, several high-profile bibliometric formulas have been coopted by corporate entities to serve for-profit purposes and have been misapplied in the academe as markers of excellence for individual researchers. In the 2010s, research assessment experts denounced the use of Clarivate’s journal impact factor and overreliance on quantitative citation-based evaluation of research “impact” through statements like the San Francisco Declaration on Research Assessment (now operating as DORA) and the Leiden Manifesto for research metrics. I share concerns cited in those statements about whether folks who use these numbers in retention, promotion, and tenure (RPT) processes can accurately explain the insight each formula is supposed to convey. Journal impact factor and Elsevier’s “CiteScore” are not generic proxies for journal quality. They aggregate citation rates over a (relatively short) period. In a vacuum, that seems like potentially useful information to consider as one part of a comprehensive assessment of a journal’s and/or a researcher’s portfolio. As a key determiner of whether a journal or its authors offer any value to the scholarly conversation, it becomes dangerous.

It is also well established that these popular metrics don’t map well across academic disciplines because different disciplines have significantly different publishing rates and citation practices, whereas metrics that normalize citation attention between disciplines like field-weighted citation impact are not commonly used in RPT procedures (from my understanding). Consequently, the perceived value of scholarship in the arts, humanities, and social sciences is diminished when their citation numbers are compared to STEM fields. In the fields that bibliometrics favor, their outsized influence in RPT processes forces faculty to “study for the test,” thereby limiting the list of suitable publication venues to a select, well-established few (often owned by one of the big five publishers). From a simple assessment standpoint, Goodhart’s law reminds us “when a measure becomes a target, it ceases to be a good measure.”

Although Goodhart’s law is certainly applicable to the current use of bibliometrics in academia, I think the problem requires us to zoom out a bit. The core issue leading to the misapplication of bibliometrics is academia’s attempt to assess a concept as sweeping and nebulous as “impact” using a few numbers that address one specific type of impact—contribution to the scholarly conversation.  Furthering the progress of knowledge is a valid and core reason why many scholars do research, but it certainly isn’t the only reason why people do research, and it’s not the only value that research provides to the world, so it shouldn’t be the only aspect of research impact we evaluate and thereby ascribe value to.

Of course, making any kind of structural change in academia is easier said than done for many reasons, including the entrenchment of bibliometric-based evaluation in the publishing industry and in academic tenure processes. But, there are folks exploring alternatives: in 2023, the University of Illinois Urbana-Champaign introduced a Public Engagement Research Option (PERO) for Promotion and Tenure, designed for faculty who “address a societal problem to contribute to the public good; produce traditional and non-traditional outputs; collaborate with communities or organizations; [and] engage in a mutually beneficial exchange of knowledge and resources with community partners.” The HumetricsHSS Humane Metrics Initiative provides workshops and toolkit resources for academic departments to consider how they can embed their stated values in their evaluation of scholarly life, reminding us, “if we don’t measure what we value, we will only value what we can measure.” Just this month, DORA published their Practical Guide to Implementing Responsible Research Assessment at Research Performing Organizations.

I heard about that DORA resource while attending a new (to me) conference in April. The Center for Advancing Research Impact in Society hosts an annual summit around “broader impacts”—the National Science Foundation’s terminology for research’s “potential to benefit society and contribute to the achievement of specific, desired societal outcomes.” I’d been debating which research impact professional development opportunities I should explore, given that “research impact” is included in my job title. This one intrigued me because of its necessarily broad conception of impact, and I came away energized by the creative research development work happening around the world. Research development offices are logically very interested in understanding and tracking the impact of research conducted at their institutions, and they clearly spend a lot of time thinking about how to do research assessment well, and how to encourage researchers to embed the public good in their plans for research outputs. Two favorite sessions included the Association of Public & Land-grant Universities’ walk-through of “Modernizing Scholarship for the Public Good: An Action Framework for Public Research Universities,” and the Research Development team at University of Texas at Austin’s presentation on their Research-to-Policy workshops that teach faculty how to make their work accessible to legislators.

Although librarians are welcomed at the ARIS Summit, I didn’t meet any other library folks wandering between conference rooms. I mention this not to admonish anyone or toot my own horn, but rather to raise the question: why shouldn’t we participate at these types of conferences, contributing to and learning from conversations about research impact/assessment? I would argue the prevailing (mis)conception of research impact is an information literacy issue. As a librarian, I feel a responsibility to remind scholars of the ways “authority is constructed and contextual” and how our interpretations of information (like bibliometrics) influence the way those specific pieces of information come to have value. These issues are especially relevant to those of us in the scholarly communication subfield, as we play a particular role in helping scholars decide where to communicate and share their work. I now see the “open knowledge” and “research impact” components of my job title as much more intertwined that I originally thought, but it required me to conceptualize a holistic vision of research impact that incorporates the societal contribution we believe many types of scholarship can make. Academic libraries are increasingly involved at every stage of the knowledge lifecycle, and as such, I think research assessment/evaluation is an area where librarians could reconceptualize our approach, training, responsibilities, and campus partnerships.