Category Archives: In The Disciplines

Evaluating Research By the Numbers

This month’s post in our series of guest academic librarian bloggers is by Bonnie Swoger, Science and Technology Librarian at the State University of New York (SUNY) Geneseo. She blogs at The Undergraduate Science Librarian.

Last week I taught an information literacy class to a group of senior Chemistry students. We didn’t talk about databases or indexes, we talked about numbers. We talked about impact factors and h-indexes and alternative metrics, and the students loved it. Librarians have used these metrics for years in collection development, and have looked them up to help faculty with tenure and promotion packets. But many librarians don’t know where the numbers come from, or what some of the criticisms are.

The students in this class needed to select a research topic, and the professor was tired of reading about obscure and “uninteresting” topics. He wanted his students to be able to find out what’s “hot” right now in chemical research.

At this level, the students are just starting to develop a sense about the nature of chemical research. It is hard for them to look at a journal article and know if that item is “hot” (or not). Librarians are often in the same boat. But there are some strategies for helping non-specialists do this. One is to look at science news sites such as C&E News, and the news wings of Science and Nature.

Another strategy is to make use of the metrics used to quantitatively assess journals, authors and articles.

We started the class by talking about the Journal Impact Factor (JIF) developed by Eugene Garfield and Irving Sher almost 50 years ago (see this article for the history of the JIF). It is a simple calculation:

JIF = Number of Citations/Number of articles

I had asked the students to read a brief commentary prior to class discussing the use (and abuse) of this metric, and in class we discussed some of criticisms of the number:

  • The numerator and denominator count different things (commentary articles are included in the numerator but not the denominator, so a journal can get an extra boost if commentary-type articles are cited)
  • The publication of review articles can quickly increase the impact factor because they are more likely to be cited.

These students were particularly interested in how the JIF could be manipulated and intrigued to learn about the story of how a single article increased the impact factor of Acta Crystallographia – Section A from 2 to 50 in a single year.

Importantly, we talked about how the impact factor was never meant to assess individual articles or authors.

So we explored alternatives.

The h-index was first suggested by physicist Jorge Hirsch, and and is now sometimes used to assess the influence of particular authors.

It works like this: Let’s say that professor Jane Smith has published 5 articles. Each article has been cited a different number of times:

Article Citations
Article 1 9
Article 2 10
Article 3 4
Article 4 2
Article 5 1

The h-index is the number that fills in the phrase “x number of articles have been cited x number of times.” In this case, we can easily say that 3 of Jane’s papers have been cited at least 3 times, so she has an h-index of 3. The major citation indexes (Scopus, Web of Knowledge) can calculate this number easily.

Like all other measures, h-index isn’t perfect. It never decreases, even as a researcher’s influence in their field decreases. It favors fields that tend to have larger numbers of authors on each paper (like high energy physics), and it can easily be manipulated by citing your own papers (or those of your friends and relatives). It does provide a way to try to sort out those authors who just write a lot from those authors who write a lot of good stuff.

We then turned to a brief discussion about some of the alternative metrics now being proposed by various journals and publishers. Some of the simplest measures in this category are the number of on-site views of an article and the number of times a PDF has been downloaded. Other tools include article ratings, comments, and how many times an article has been bookmarked. I think these developments are exciting, and it will be interesting to see how scholars react as more publishers offer these services.

Of course, none of these numbers are useful without context. Is an impact factor of 12 in organic chemistry considered good or bad? What about an h-index of 7 for a cancer researcher? And when an article is downloaded 457 times, what does that actually mean?

At the end of the class, I gave students an article citation and asked to students to determine if the research topic (and the article) was “hot” or not. They were asked to find some of the relevant metrics, and asked to provide a bit of background to give some context to their numbers. They had fun exploring the numbers, and I think they felt more confident in their ability to determine how important or buzz-worthy their prospective research topics might be as a result of our in-class discussion.

The numbers without context aren’t very helpful. But if you can find the numbers, and gain a sense of context, they can help non-specialists gain a sense of perspective about particular journals, authors and articles.

Faculty Blog Round Up: Teaching with Technology

Editor’s Note: A few weeks ago we put out a call for someone to be our new faculty blog correspondent. With this post I’d like to introduce Laura Wimberley, the librarian we’ve selected to keep us up-to-date on what’s happening in the faculty blogosphere. Laura works at the Medical Center Library at the University of California San Diego. In addition to her MLIS – which she just completed – she also has an MA and PhD in Political Science. Her research interests include information policy, scholarly communication, and collection development. In addition to her posts here, you can read her at Libri & Libertas. We look forward to Laura’s future posts.

Much of what’s going on with faculty is very similar to what’s going on with librarians: Conferences are great, highly specialized, but exhausting! Or: Why, oh why, do students not cite sources after we work so hard with them? These experiences, we know.

What we don’t usually observe is the teaching, and this is one of the parts we need to stay in tune with. Here I’ve highlighted three posts with really innovative technology teaching techniques – ideas that you might not have thought about how to support from the library. Or maybe you’re dying to include blogging, Wikipedia, and gaming, and you didn’t know how to find faculty who are doing it, too. Either way, here’s a sample.

Acephalous is the blog of Scott Eric Kaufman, who teaches English at the University of California Irvine; he also contributes to the faculty group blogs The Valve (mostly literature) and Edge of the American West (mostly history).

SEK is blogging with his students in his undergraduate writing course the Rhetoric of Heroism. Because the course relies so heavily on detailed analysis of film and other visual iconography, a blog with embedded images seems like a wonderful way to communicate the material. I expect they’re watching and discussing the films together in class, but images are usually not the kind of thing students are accustomed to taking notes on (especially in the dark).

Jeremy Boggs, who blogs at ClioWeb, is a graduate student in American history at George Mason University. He’s also creative lead at the Center for History and New Media, so it’s not too surprising that he’s willing to take on the bete noire – Wikipedia. In his undergraduate American History Survey course, he assigns students to not just use, but create, Wikipedia articles, including citating sources, monitoring for follow-up collaboration, and writing a reflective essay. One of his students wrote the article that developed into the entry for Living Newspapers.

Another history professor, Rob MacDougall of the University of Western Ontario, blogs at Old is the New New (with a charming original steampunk blog theme). Rob uses the game Civilization to frame the course Science, Technology, and Global History. He asks his students to write an essay that reconceptualizes technology not as a serial, linear progress of development – as the game depicts it – but in some other way. How could we play a game that thinks of history as more contingent or branching or cyclic?

In this assignment, the game is laying bare a lot of social assumptions we carry around without realizing, and making them something students can analyze. If you ever need to justify a games collection in your library, this kind of work is a stellar example of such a collection could do.

Why are You a Librarian?

No, that isn’t meant to be said in the voice of a slightly-tipsy relative at a family gathering. You? A librarian? Why on earth . . .

It’s an invitation to a meme started over at Free Exchange on Campus, where I occasionally blog. It was inspired by Dr. Crazy’s wonderful post, “Why I Teach Literature.” And now a number of academic bloggers have weighed in. What they have to say is an excellent way to learn more about faculty perspectives and the passion that drives them into the classroom.

I was asked to join in just as I was mulling over Steven’s post on faculty status, so in part my response to “why I am a librarian” is an extension of my responses to that post and a reflection on the ACRL/AAUP statement on librarians and faculty status. In my post I tagged a handful of librarians – you’re it! – but then felt a bit silly because in those blogs, the answer is pretty obvious, even if it the question isn’t posed that way.

But I hope some librarians will be moved to pick up the meme – here in comments, or on your own blogs. I sometimes think some of our colleagues in the academy respect what we do … they just aren’t exactly sure what it is or why it matters.

You mean I can’t throw these out?

James Cortada, a historian of computing who works for IBM, has a nice screed (Save the Books!) over at the American Historical Association that heaps a bit of anger on us lil’ old academic librarians.

Fresh from reading Nicholson Baker and full of Google digitization anxiety, Cortada charges that a new spectre is haunting libraries: heartless librarians ruthlessly discarding old PC-DOS manuals. (Wah! I had to scrounge second hand bookstores to write my 3 volume history of computing! Bad librarians! Them not book people!) Apparently no one told Cortada that when librarians discard books it’s called deselection, and we have rigorous protocols in place for that kind of thing.

Kidding aside, I agree with much of what Cortada has written and don’t think librarians and historians are as far apart on the issue as he claims. The future of print collections in light of the Google digitization project is a serious issue that is seeing ongoing discussion by librarians. In New Jersey, academic librarians gathered at Fairleigh Dickinson University for a one day symposium on the Future of Print in the Academic Library that included suggestions for collaborative solutions. Obviously, all libraries can’t and shouldn’t be holding on to everything, therefore choices must be made as to who saves what.

IUPUI Library Dean David Lewis demonstrates a nuanced understanding of the issues in his recent “Strategy for Academic Libraries in the First Quarter of the 21st Century.” In his section on “retire legacy print collections” Lewis talks about regional collection management and the use of OCLC’s WorldCat as a tool for this purpose. He writes perceptively:

Whether it will be possible to build a national consensus and to implement a concerted program of action or whether a laissez-faire approach will be adequate is unclear. Until one approach or the other is proven to work, individual libraries will either have to delay decisions or make them on faith. Neither choice will be attractive to tradition-minded librarians who do not wish to antagonize faculty who value proximity to “their” books.

That sounds familiar, as I’ve been purposely procrastinating on a project of sending more of our history collection to remote storage for a while now. I’ve knocked off some low hanging fruit, like multi-volume outdated reference books in foreign languages, but more difficult decisions loom. Historians do tend to feel that the library should buy everything and hold on to it forever. Cortada’s piece can be a jumping off point for communication between librarians and historians. If librarians can understand more about the importance of holding on to ephemera (and non ephemera) for future historical writing, historians can understand more about the realities and economics of space planning. Beginning the conversation early is better than doing the evil mad laugh while running from the dumpster.

HRN Joins SSRN

First – if you support the NIH plan to make tax-funded research publicly available, take a minute to call your senators. Right now. There are some amendments to be voted on today that could gut the NIH proposal. Tell them to vote no on Senator James Inhofe’s amendments #3416 and #3417 to the 2008 Labor-HHS-Education bill.

Okay – are you done? Good.

Now, here’s some other news about access to research. The Chronicle reports that humanists will have a place to share their work in progress just as scholars in the social sciences have done for over ten years. This site for sharing documents is something between informal blogging and formal publication – more like a conference presentation without the hotel bill or airfare to pay for. The social sciences have had such an Internet forum since the SSRN was founded in 1994. There, some 131,000 papers have had over 4 million downloads in the past year. These are clustered into “networks” – rather like conferences – where people working in related areas can share their work. If you look toward the bottom of the page, classics, US and British literature, and philosophy have new networks. It looks as if these will be spun off into a new HRN – Humanities Research Network.

I’m not that familiar with how SSRN works and would be interested in hearing from anyone who has an insider’s view. All I know is that I’ve downloaded a lot of good articles from there, so I’m happy to see it expand into new fields.