Tag Archives: research

Evaluating Research By the Numbers

This month’s post in our series of guest academic librarian bloggers is by Bonnie Swoger, Science and Technology Librarian at the State University of New York (SUNY) Geneseo. She blogs at The Undergraduate Science Librarian.

Last week I taught an information literacy class to a group of senior Chemistry students. We didn’t talk about databases or indexes, we talked about numbers. We talked about impact factors and h-indexes and alternative metrics, and the students loved it. Librarians have used these metrics for years in collection development, and have looked them up to help faculty with tenure and promotion packets. But many librarians don’t know where the numbers come from, or what some of the criticisms are.

The students in this class needed to select a research topic, and the professor was tired of reading about obscure and “uninteresting” topics. He wanted his students to be able to find out what’s “hot” right now in chemical research.

At this level, the students are just starting to develop a sense about the nature of chemical research. It is hard for them to look at a journal article and know if that item is “hot” (or not). Librarians are often in the same boat. But there are some strategies for helping non-specialists do this. One is to look at science news sites such as C&E News, and the news wings of Science and Nature.

Another strategy is to make use of the metrics used to quantitatively assess journals, authors and articles.

We started the class by talking about the Journal Impact Factor (JIF) developed by Eugene Garfield and Irving Sher almost 50 years ago (see this article for the history of the JIF). It is a simple calculation:

JIF = Number of Citations/Number of articles

I had asked the students to read a brief commentary prior to class discussing the use (and abuse) of this metric, and in class we discussed some of criticisms of the number:

  • The numerator and denominator count different things (commentary articles are included in the numerator but not the denominator, so a journal can get an extra boost if commentary-type articles are cited)
  • The publication of review articles can quickly increase the impact factor because they are more likely to be cited.

These students were particularly interested in how the JIF could be manipulated and intrigued to learn about the story of how a single article increased the impact factor of Acta Crystallographia – Section A from 2 to 50 in a single year.

Importantly, we talked about how the impact factor was never meant to assess individual articles or authors.

So we explored alternatives.

The h-index was first suggested by physicist Jorge Hirsch, and and is now sometimes used to assess the influence of particular authors.

It works like this: Let’s say that professor Jane Smith has published 5 articles. Each article has been cited a different number of times:

Article Citations
Article 1 9
Article 2 10
Article 3 4
Article 4 2
Article 5 1

The h-index is the number that fills in the phrase “x number of articles have been cited x number of times.” In this case, we can easily say that 3 of Jane’s papers have been cited at least 3 times, so she has an h-index of 3. The major citation indexes (Scopus, Web of Knowledge) can calculate this number easily.

Like all other measures, h-index isn’t perfect. It never decreases, even as a researcher’s influence in their field decreases. It favors fields that tend to have larger numbers of authors on each paper (like high energy physics), and it can easily be manipulated by citing your own papers (or those of your friends and relatives). It does provide a way to try to sort out those authors who just write a lot from those authors who write a lot of good stuff.

We then turned to a brief discussion about some of the alternative metrics now being proposed by various journals and publishers. Some of the simplest measures in this category are the number of on-site views of an article and the number of times a PDF has been downloaded. Other tools include article ratings, comments, and how many times an article has been bookmarked. I think these developments are exciting, and it will be interesting to see how scholars react as more publishers offer these services.

Of course, none of these numbers are useful without context. Is an impact factor of 12 in organic chemistry considered good or bad? What about an h-index of 7 for a cancer researcher? And when an article is downloaded 457 times, what does that actually mean?

At the end of the class, I gave students an article citation and asked to students to determine if the research topic (and the article) was “hot” or not. They were asked to find some of the relevant metrics, and asked to provide a bit of background to give some context to their numbers. They had fun exploring the numbers, and I think they felt more confident in their ability to determine how important or buzz-worthy their prospective research topics might be as a result of our in-class discussion.

The numbers without context aren’t very helpful. But if you can find the numbers, and gain a sense of context, they can help non-specialists gain a sense of perspective about particular journals, authors and articles.

Countdown to the Conference

I’ve found myself with less time than usual for blogging lately as I’ve been busy working on the poster I’m presenting with colleagues at the upcoming ACRL National Conference. In the handful of years since I’ve been a librarian I’ve been to many smaller conferences and symposia in and around New York City (where I live), but this will be my first time attending the national conference, and as the date draws closer I find that I’m really looking forward to it.

In my past life as an archaeologist I went to lots of scholarly conferences, though I imagine that National will be somewhat different. While I enjoyed hearing about the latest research in my field back then, it always seemed odd to me that the convention was for presenters to stand at a podium and read straight through their scholarly papers. Of course some people are better at public speaking than others, and archaeologists tend to illustrate their talks with lots of site photos, charts, and graphs. But I find the very formal presentation style to be a bit monotonous, and I vastly prefer the more interactive and conversational style that most librarians seem to use at conferences.

Another big difference from my prior experiences is that the ACRL Conference has several keynote speakers, which is not the usual fare at other scholarly conferences I’ve been to. I find this a bit confusing: though I know that keynotes are a standard feature of both ALA conferences, it’s not what I expected to travel to an academic librarianship conference and hear speakers who are not involved in academic librarianship. I have to admit that I’m less interested in the keynote speakers than in other parts of the conference, though I’ll be curious to hear how they relate to academic libraries in their presentations.

I’m lucky to have many events at which I can connect with colleagues from my university and across NYC, but as a still-somewhat-new librarian I haven’t had many opportunities to mingle with librarians from across the country. I’m most looking forward to the two things I remember fondly from the anthropology conferences I used to frequent (and I suspect this is true for many of us attending National):

1) the opportunity to share and discuss my and my colleagues’ work with others in our field, and

2) the opportunity to learn about research and practice in academic libraries from the other conference presenters and attendees

Conferences are a concentrated experience with no distractions — all academic librarianship all the time! — which I always find refreshing and invigorating (if sometimes exhausting). But I’ve got my reusable coffee cup, so I’m ready to go.

If you’re going to National, what are you most looking forward to?

After The Values Study

ACRL has received a considerable amount of positive feedback about the Values of Academic Libraries Study. Perhaps you’ve had an opportunity to catch one of the presentations about the study that Megan Oakleaf, author of the study, or ACRL President Lisa Hinchliffe, have conducted at a number of different conferences.

At the Midwinter conference, during a meeting of ACRL’s Leadership Council (the Board, section chairs, and other miscellaneous representatives), a question was raised about what we do next with the Values study, or rather what comes after the study. If anyone at the meeting had a good idea, he or she chose not to share it because there were no responses to the question – and perhaps folks just had not yet had much time to give thought to that particular question. The study provides abundant information, from a mix of qualitative and quantitative studies, to help academic librarians provide evidence of the ways in which our libraries make valuable contributions to student and faculty success, and help to improve higher education. But the report itself is not a research study that provides concrete documentation of the value of academic libraries. What it does well is provide ammunition for library leaders who will want to argue for the value of academic libraries, and use it to make a case for institutional support. So the question about what comes next – what more can be done to create a strong connection between academic librarians and the value they provide – is a good one. I suspect ACRL is already cooking up some plans for next steps to extend the “value of academic libraries” initiative, but I’m not sure what they are.

As I’ve been thinking about this “what comes next” question, two possibilities have come to mind. I continue to believe that some of the most essential areas in which we can demonstrate the value of our work are student retention, persistence to graduation and student success beyond graduation. How do we connect our contributions to these higher education performance issues? I wanted to share some thoughts about this, and would like to hear what you think might make a good follow-up to the values study. One inspiration for a next step is the recently released book Academically Adrift that has created quite a stir in higher education circles with its finding that for many of our students there is little learning in their four years of college. The findings are based on data collected from a sample of 2,000 students from 24 four-year colleges. The students took standardized learning assessment tests three times during their college years.

That approach could offer some possibilities for a next step. With enough grant money a sample of students could be tracked in order to assess changes in their research skills. As seniors would they still be starting their research at Google? If asked, to what extent would they point to the librarians at their institution as playing a role in their academic success? Did the librarians have any impact on their ability to stay enrolled? The authors of Academically Adrift are already moving on to the “next step” in their research on student learning, and they’ll be looking more closely at alumni and what happens after college. Targeting alumni might even work better as a way to document the value of the academic library. If asked, what would alumni have to say about their library experience? I could see that as a more qualitative study, interviewing alumni to get more in depth information about their library experience, what value it provided and whether it was making a difference for them in their careers (assuming they’ve started careers).

A few colleagues and I previously did some quasi-experimental research on the use of LibGuides and whether, by examining the annotated bibliographies produced by the students in control and experimental groups, we could ascertain if the LibGuides made a difference in the use of library resources. While it was difficult to determine if higher quality work could be attributed to having access to the LibGuide, one thing we did notice is that there were clear outliers within the study groups. Some students performed far better, and perhaps that’s not unusual in any academic setting. Looking specifically at library research skills though, especially evaluation of content, what leads some students to excel? Another possible follow-up to Values Study could track the outliers into their post-graduate years to determine whether or not they still use their learned library skills in the workplace – and can any post-graduate success with work that involves research and/or writing be attributed to library research skills education. If we could link library research skill building with positive post-graduate or career performance that could definitely speak volumes about the value of academic librarians. There’s no question that these types of research projects are involved, somewhat complicated and almost a full-time job in themselves. That’s where ACRL’s connection with LIS educators to conduct the research makes good sense.

I’m not sure what will come after the Values Study. Given its success and value as a starting point, there is strong support in the library community for further research into the value of academic librarians and their libraries. In this post I focused on student retention and persistence to graduation. The Values Study also points to the academic librarian’s contribution to faculty research and productivity, as well as institutional prestige. There are important areas too for “next steps” research. ACRL is open to ideas for what comes next. Let ACRL know what you think would be a good next step. A great idea for what comes after the Values Study could come from anywhere in our profession.

Turning the Research Lens on Ourselves

I’m working on a research project again this year exploring the scholarly habits of undergraduate students at my university. One of the methods we’re using to collect data is a mapping diary. We ask students to record all of their movements through the course of one typical school day–time, location and activity–and draw a map to accompany their time logs. Last year’s responses from students at my own campus were fascinating, and I’m looking forward to interviewing this semester’s students when they finish their logs.

Many of last year’s participants told me that they really enjoyed the opportunity to reflect on what they do and where they go all day. Now that the semester is firmly underway and things are busy as usual, I wonder whether it might be a good idea to do some research on myself. I’ve often wanted to join the Library Day in the Life project in the past, but it always seems to be scheduled for days that I’m either out on vacation or before the semester has begun (that is, not really a typical day for me). Maybe it’s time for me to pick a day (or week, or month) to record my activities?

It’s all too easy to fall into the trap of thinking that there’s not enough time for everything I want to do. Of course that’s true on one level, because no one can do everything, but I also think that we may be less busy than we realize. A post on Prof Hacker over the summer popped into my mind when I was considering this, a review of a book called 168 Hours: You Have More Time Than You Think (168 is the number of hours in a week). The review isn’t completely positive, but does highlight the use of time logging to inject a dose of reality into how we perceive that we spend our time.

Judging from my interviews with students last year, this kind of reflection can help with both time management and task prioritization. Though it sounds like more work to add a time log to my to-do list here in the thick of the semester, I think it’s worth a try. And maybe the next time the Library Day in the Life date rolls around I’ll be ready to participate, too.

Practice, Practice, Practice

The semester is drawing to a close at my college and students in the information literacy course that I’m teaching are deep into their work on their final projects. I’m taking a breath before the grading begins and already starting to reflect on the semester: what worked well, what didn’t, what I’ll tweak over the summer and what I can use again in the fall.

One thing has been apparent since my students turned in their annotated bibliographies last month. To put it bluntly: their sources are awesome. Each of them has found solid information on their research topics from a wide variety of sources including scholarly books and articles, conference proceedings, academic websites, specialized reference materials, newspapers, magazines, blogs, and other internet sources. I can honestly say that it was a delightful experience to read their bibliographies.

The students chose topics of interest to them which definitely seems to have helped them embrace the research process. But I think that the main reason they were able to find such excellent sources is time. We had time over the course of the semester to explore where information comes from; how and by whom it’s produced and distributed; how to search for, find, and evaluate it. We also spent time discussing when to use different kinds of information, for example, when it’s appropriate to use a journalistic source and when it’s better to find something scholarly. Like the old joke about Carnegie Hall, this semester my students had time to practice.

I don’t know that I’ve emerged on the other side of this assignment believing that credit-bearing courses are the one and only best way to teach information literacy, but my experiences this semester have certainly been eye-opening. It’s not that taking one course magically creates information literate students — as with English Composition courses and writing, this is just the beginning. But I do feel that the students have built a solid foundation that will serve them well as their information competencies continue to develop over the rest of their time in college and, I hope, throughout their lives.

Realistically, it would be difficult at my college to require an information literacy course of all students; there just aren’t enough available credits in most degree programs. So another thing I’ll be thinking on over the summer is how to port some of the successful strategies I used during the course over to the one-shot sessions that still represent most of the library and information literacy instruction we provide. And I’m hopeful that strategies from both kinds of instruction can continue to evolve and inform each other.