Category Archives: Research Issues

Use this category for posts related to research projects, research conducted by librarians, the need for certain types of research, research in higher education

Evaluating Research By the Numbers

This month’s post in our series of guest academic librarian bloggers is by Bonnie Swoger, Science and Technology Librarian at the State University of New York (SUNY) Geneseo. She blogs at The Undergraduate Science Librarian.

Last week I taught an information literacy class to a group of senior Chemistry students. We didn’t talk about databases or indexes, we talked about numbers. We talked about impact factors and h-indexes and alternative metrics, and the students loved it. Librarians have used these metrics for years in collection development, and have looked them up to help faculty with tenure and promotion packets. But many librarians don’t know where the numbers come from, or what some of the criticisms are.

The students in this class needed to select a research topic, and the professor was tired of reading about obscure and “uninteresting” topics. He wanted his students to be able to find out what’s “hot” right now in chemical research.

At this level, the students are just starting to develop a sense about the nature of chemical research. It is hard for them to look at a journal article and know if that item is “hot” (or not). Librarians are often in the same boat. But there are some strategies for helping non-specialists do this. One is to look at science news sites such as C&E News, and the news wings of Science and Nature.

Another strategy is to make use of the metrics used to quantitatively assess journals, authors and articles.

We started the class by talking about the Journal Impact Factor (JIF) developed by Eugene Garfield and Irving Sher almost 50 years ago (see this article for the history of the JIF). It is a simple calculation:

JIF = Number of Citations/Number of articles

I had asked the students to read a brief commentary prior to class discussing the use (and abuse) of this metric, and in class we discussed some of criticisms of the number:

  • The numerator and denominator count different things (commentary articles are included in the numerator but not the denominator, so a journal can get an extra boost if commentary-type articles are cited)
  • The publication of review articles can quickly increase the impact factor because they are more likely to be cited.

These students were particularly interested in how the JIF could be manipulated and intrigued to learn about the story of how a single article increased the impact factor of Acta Crystallographia – Section A from 2 to 50 in a single year.

Importantly, we talked about how the impact factor was never meant to assess individual articles or authors.

So we explored alternatives.

The h-index was first suggested by physicist Jorge Hirsch, and and is now sometimes used to assess the influence of particular authors.

It works like this: Let’s say that professor Jane Smith has published 5 articles. Each article has been cited a different number of times:

Article Citations
Article 1 9
Article 2 10
Article 3 4
Article 4 2
Article 5 1

The h-index is the number that fills in the phrase “x number of articles have been cited x number of times.” In this case, we can easily say that 3 of Jane’s papers have been cited at least 3 times, so she has an h-index of 3. The major citation indexes (Scopus, Web of Knowledge) can calculate this number easily.

Like all other measures, h-index isn’t perfect. It never decreases, even as a researcher’s influence in their field decreases. It favors fields that tend to have larger numbers of authors on each paper (like high energy physics), and it can easily be manipulated by citing your own papers (or those of your friends and relatives). It does provide a way to try to sort out those authors who just write a lot from those authors who write a lot of good stuff.

We then turned to a brief discussion about some of the alternative metrics now being proposed by various journals and publishers. Some of the simplest measures in this category are the number of on-site views of an article and the number of times a PDF has been downloaded. Other tools include article ratings, comments, and how many times an article has been bookmarked. I think these developments are exciting, and it will be interesting to see how scholars react as more publishers offer these services.

Of course, none of these numbers are useful without context. Is an impact factor of 12 in organic chemistry considered good or bad? What about an h-index of 7 for a cancer researcher? And when an article is downloaded 457 times, what does that actually mean?

At the end of the class, I gave students an article citation and asked to students to determine if the research topic (and the article) was “hot” or not. They were asked to find some of the relevant metrics, and asked to provide a bit of background to give some context to their numbers. They had fun exploring the numbers, and I think they felt more confident in their ability to determine how important or buzz-worthy their prospective research topics might be as a result of our in-class discussion.

The numbers without context aren’t very helpful. But if you can find the numbers, and gain a sense of context, they can help non-specialists gain a sense of perspective about particular journals, authors and articles.

Research Librarianship in Crisis: Mediate When, Where, and How?

This month’s post in our series of guest academic librarian bloggers is by Bohyun Kim, Digital Access Librarian, Florida International University Medical Library. She blogs at Library Hat.

The talk about the crisis of librarianship is nothing new. Most recently, back in May, Seth Godin, a marketing guru, has written on his blog a post about the future of libraries. Many librarians criticized that Godin failed to fully understand the value of librarians and libraries.  But his point that libraries and librarians may no longer be needed was not entirely without merit (See my post “Beyond the Middlemen and the Warehouse Business”). Whether we librarians like it or not, more and more library users are obtaining information without our help.

One may think academic research libraries are an exception from this. Unfortunately, the same trend prevails even at research libraries. In his guest editorial for the Journal of Academic Librarianship, “The Crisis in Research Librarianship (pre-print version)”, Rick Anderson makes the case that patrons are finding information effectively without librarians’ help, citing the drastic decline of reference transactions in Association of Research Libraries (ARL).  According the ARL statistics, the number of reference transactions went down by more than 50-60 % since 1995.

This is particularly worrisome considering that at research libraries, we tend to place reference and instruction services at the center of the library operation and services. These services delivered by physical or online contact are still deemed to be one of the most prominent and important parts of the academic library operation. But the actual user behavior shows that they can and do get their research done without much help from librarians.  To make matters worse, existing library functions and structures that we consider to be central appear to play only a marginal role in the real lives of academic library users.  Anderson states: “Virtually none of them begins a research project at the library’s website; the average student at a major research university has fewer than four interactions with a reference librarian in a year (and even fewer of those are substantive reference interviews); printed books circulate at lower and lower rates every year.”

We have heard this before. So why are we still going in the same direction as we were a decade ago? Could this be perhaps because we haven’t figured out yet what other than reference and instruction to place in the heart of the library services?

For almost three years, my library has been offering workshops for library users. Workshops are a precious opportunity for academic librarians to engage in instruction, the most highly regarded activity at an academic library. But our workshop attendance has been constantly low. Interestingly, however, those who attended always rated the workshops highly. So the low attendance wasn’t the result of the workshops being bad or not useful. Library users simply preferred to spend their time and attention on something other than library workshops.  I remember two things that brought out palpable appreciation from users during those workshops: how to get the full-text of an article immediately and how to use the library’s LibX toolbar to make that process even faster and shorter.

What users seemed to want to know most was how to get the tasks for their research done fast, and they preferred to do so by themselves. They appreciated any tools that help them to achieve this if the tools were easy to use.  But they were not interested in being mediated by a librarian.

What does this mean?  It means that those library services and programs that aim at increasing contact between librarians and patrons are likely to fail and to be received poorly by users. Not necessarily because those offerings are bad but because users prefer not to be mediated by librarians in locating and using information and resources.

This is a serious dilemma. Librarians exist to serve as a mediator between users and resources. We try to guide them to the best resources and help them to make the best use of those resources.  But the users consider our mediation as a speed bump rather than as value-added service. So where do research libraries and librarians go from here?

I think that librarians will still be needed for research in the digital era. However, the point at which librarians’ mediation is sought for and appreciated may vastly differ from that in the past when information was scarce and hard to obtain.  Users will no longer need nor desire human mediation in basic and simple tasks such as locating and accessing information. Most of them already have no patience to sit through a bibliographic instruction class and/or to read through a subject guide.

But users may appreciate and even seek for mediation in more complicated tasks such as creating a relevant and manageable data set for their research.  Users may welcome any tool that libraries offer that makes the process of research from the beginning to the final product easier and faster. They will want better user interfaces for library systems. They will appreciate better bridges that will connect them with non-library systems to make library resources more easily discoverable and retrievable.  They will want libraries to be an invisible interface that removes any barrier between them and information.  This type of mediation is new to librarians and libraries.  Is it possible that in the future the libraries and librarians’ work is deemed successful exactly in inverse proportion to how visible and noticeable their mediation is?

In his guest editorial, Anderson presents several scenarios of research libraries “going out of business.” Libraries being absorbed into an IT group; Libraries losing computer labs, thereby losing a source of transaction with users as laptops and handheld devices become widely adopted; Libraries budget taken away for better investments; Libraries’ roles and functions being eroded slowly by other units; Information resources that libraries provide being purchased directly by users.

So if a library comes to lose its facilities such as a computer lab, a reading room, carrels, and group study rooms, would there still remain the need for librarians? If a library ends up removing its reference desk, workshops, and other instruction classes, what would librarians be left to do?  If we consider the library space that can be offered and managed by any other unit on campus as the essential part of library services and operation, the answer to these questions would be negative.  As long as we consider reference and instruction – the direct contact with users to mediate between them and resources – as the primary purpose of a library, the answer to these questions would be negative.

Libraries may never lose their facilities, and the need for users to have a direct contact with librarians may never completely go away. But these questions are still worth for us to ponder if we do not want to build a library’s main mission upon something on which the library’s patrons do not place much value. The prospect for the future libraries and librarians may not necessarily be dreary. But we need to rethink where the heart of research librarianship should lie.

After The Values Study

ACRL has received a considerable amount of positive feedback about the Values of Academic Libraries Study. Perhaps you’ve had an opportunity to catch one of the presentations about the study that Megan Oakleaf, author of the study, or ACRL President Lisa Hinchliffe, have conducted at a number of different conferences.

At the Midwinter conference, during a meeting of ACRL’s Leadership Council (the Board, section chairs, and other miscellaneous representatives), a question was raised about what we do next with the Values study, or rather what comes after the study. If anyone at the meeting had a good idea, he or she chose not to share it because there were no responses to the question – and perhaps folks just had not yet had much time to give thought to that particular question. The study provides abundant information, from a mix of qualitative and quantitative studies, to help academic librarians provide evidence of the ways in which our libraries make valuable contributions to student and faculty success, and help to improve higher education. But the report itself is not a research study that provides concrete documentation of the value of academic libraries. What it does well is provide ammunition for library leaders who will want to argue for the value of academic libraries, and use it to make a case for institutional support. So the question about what comes next – what more can be done to create a strong connection between academic librarians and the value they provide – is a good one. I suspect ACRL is already cooking up some plans for next steps to extend the “value of academic libraries” initiative, but I’m not sure what they are.

As I’ve been thinking about this “what comes next” question, two possibilities have come to mind. I continue to believe that some of the most essential areas in which we can demonstrate the value of our work are student retention, persistence to graduation and student success beyond graduation. How do we connect our contributions to these higher education performance issues? I wanted to share some thoughts about this, and would like to hear what you think might make a good follow-up to the values study. One inspiration for a next step is the recently released book Academically Adrift that has created quite a stir in higher education circles with its finding that for many of our students there is little learning in their four years of college. The findings are based on data collected from a sample of 2,000 students from 24 four-year colleges. The students took standardized learning assessment tests three times during their college years.

That approach could offer some possibilities for a next step. With enough grant money a sample of students could be tracked in order to assess changes in their research skills. As seniors would they still be starting their research at Google? If asked, to what extent would they point to the librarians at their institution as playing a role in their academic success? Did the librarians have any impact on their ability to stay enrolled? The authors of Academically Adrift are already moving on to the “next step” in their research on student learning, and they’ll be looking more closely at alumni and what happens after college. Targeting alumni might even work better as a way to document the value of the academic library. If asked, what would alumni have to say about their library experience? I could see that as a more qualitative study, interviewing alumni to get more in depth information about their library experience, what value it provided and whether it was making a difference for them in their careers (assuming they’ve started careers).

A few colleagues and I previously did some quasi-experimental research on the use of LibGuides and whether, by examining the annotated bibliographies produced by the students in control and experimental groups, we could ascertain if the LibGuides made a difference in the use of library resources. While it was difficult to determine if higher quality work could be attributed to having access to the LibGuide, one thing we did notice is that there were clear outliers within the study groups. Some students performed far better, and perhaps that’s not unusual in any academic setting. Looking specifically at library research skills though, especially evaluation of content, what leads some students to excel? Another possible follow-up to Values Study could track the outliers into their post-graduate years to determine whether or not they still use their learned library skills in the workplace – and can any post-graduate success with work that involves research and/or writing be attributed to library research skills education. If we could link library research skill building with positive post-graduate or career performance that could definitely speak volumes about the value of academic librarians. There’s no question that these types of research projects are involved, somewhat complicated and almost a full-time job in themselves. That’s where ACRL’s connection with LIS educators to conduct the research makes good sense.

I’m not sure what will come after the Values Study. Given its success and value as a starting point, there is strong support in the library community for further research into the value of academic librarians and their libraries. In this post I focused on student retention and persistence to graduation. The Values Study also points to the academic librarian’s contribution to faculty research and productivity, as well as institutional prestige. There are important areas too for “next steps” research. ACRL is open to ideas for what comes next. Let ACRL know what you think would be a good next step. A great idea for what comes after the Values Study could come from anywhere in our profession.

Building Smart Collections for Today’s Users

This month’s post in our series of guest academic librarian bloggers is from Anna Creech, Electronic Resources Librarian at the University of Richmond, Virginia. She also blogs at Eclectic Librarian.

Some days I look at my projects list and tasks and wonder how in the world I ended up here. They often appear to be more like what one might expect to be doing in an office of institutional research rather than in a library.

I am an electronic resources librarian, which I have found to be a title used for everything from online reference instruction to cataloging to acquisitions. In my case, I do little instruction or cataloging, and spend most of my time analyzing the digital resources we have acquired.

Increasingly, as libraries are forced to cut their resources even more severely, and in some cases, justify their existence, we have had to use more metrics to determine the value of our resources, whether they are personnel or materials. While this has been a tradition in libraries for as long as I’ve known them, it’s not what most of us thought we would be doing when we entered the profession. But, we can’t keep our heads in the sand any longer.

Just as we have many people who are passionate about the preservation of materials, we need to have as many if not more people in libraries who are passionate about the stewardship of the resources we purchase. We can no longer afford to purchase material that sits on a shelf and may never be touched. We need to be smarter about the things we acquire and a big part of that is looking at trends in the past to predict the future.

When I analyze usage data, I am looking for the anomalies that indicate a problem with a resource, such as sudden drops in use, declining patterns, etc. I talk to the public service librarians about resources that seem to be declining in use to make sure they are still relevant to our programs and researchers. We consider accessibility issues and course offering patterns before ultimately deciding whether or not to renew the resource or continue to collect in that area.

I hope that someday, we will be able to shift the 80/20 rule towards 100% circulation so that more of the resources in undergraduate libraries are used and not just sitting on the shelf waiting for someday to arrive. Alternative purchasing models like patron-driven acquisitions and collaborative collection development agreements indicate a trend towards making more purchasing decisions based on what users want now, and less towards purchasing things they might want later.

I know that some librarians are concerned that just-in-time collections will have significant gaps that may not be filled later on, but I don’t think we can afford to continue to maintain large just-in-case collections of materials. Academic libraries need to transition from being warehouses of books to being collaborative and individual learning spaces where research and innovation happen, and in part that means using ILL, document delivery, and online content to supplement materials that are not on the shelf.

If a publication is significant enough to be of value to a researcher someday, then it’s likely that a library somewhere has purchased a copy. Besides, we live in the future now. There’s no reason why a book needs to be out of print when it could be sold or otherwise made available in electronic formats. The argument of “we must purchase everything now or it may not be available later” is becoming less and less relevant.

I also hope that someday, libraries will have business intelligence tools to help them assess the return on investment for their collections. We do the best we can with the tools we have, but I think we could better make use of staff time if we didn’t spend so much of it getting our mish-mash of systems to spit out comparable data. This is why I believe we should be actively supporting standards initiative like COUNTER (Counting Online Usage of Networked Electronic Resource), SUSHI (Standardized Usage Statistics Harvesting Initiative), and CORE (Cost of Resource Exchange). They’re just the tip of the iceberg, but it’s a start.

We librarians are an intelligent and resourceful bunch. With the right set of tools, I believe we could come close to creating “perfect” collections to meet the needs of our users. With the right set of tools, we can be better stewards of the financial resources provided by our institutions. It’s time to work smarter, not harder.

Underground Resource Sharing

One outcome of the Netflix discussion that took place in the library community is that there seems to be general agreement that adhering to licensing agreements is the right thing for academic librarians to do for a number of good reasons. Not only is it a good way to avoid a potential lawsuit from Netflix or a movie studio, but it sets the right example for students and faculty. How can we expect them to abide by fair use guidelines and licensing agreements if the campus librarians are openly flouting them. We need to take the moral high ground, even if Netflix represents a reasonably good solution to the DVD distribution challenge.

So I find it interesting that this blogger is complaining about not having access to JSTOR as an alumnus of some college or university. Dr. Koshary writes:

I didn’t think this would happen, now that I’m out of grad school, but I’m feeling a fresh surge of hatred for Dear Old University. I tried to log in to JSTOR to look up an article, and found that I no longer have access to JSTOR through my DOU affiliation.

I’m pretty sure Dr. Koshary knows that JSTOR is a restricted database, and that most libraries are prohibited from allowing alumni to gain access (unless they make some sort of arrangement which likely isn’t cheap – and Dr. Khosary suspects his alma mater has such an agreement). At the end of the rant against his alma mater he asks:

I don’t suppose any of my readers has a better/cheaper idea for me to regain access to JSTOR?

Turns out they do, and most of those offering advice don’t seem too concerned about taking the moral high ground – or even abiding by their university or library’s guidelines for sharing accounts:

Do what everybody I know who’s been in your position has done: get a friend who has access to a research library and its databases to share their log-in and password with you. I know I’ve helped a few people out in this way, and I’ve done it with a spring in my step and a song in my heart. Sure, it’s technically “wrong” but I’d argue that it’s more wrong to charge underemployed people money for access to scholarly resources.

I just ran into this, where my new school has some journal accesses but not many, and I crowdsourced it on facebook — some current Gradschooland students offered me their proxy server login, and another was already in the library and emailed me the pdf.

Everyone does it. Hell, I’ll give you MY login if you want

Virtually everyone I know who’s not employed by a top-tier R1 has a bootlegged EEBO account: through friends who are still grad students, advisors, or friends with cushier jobs.

Makes you wonder why we even bother with licensing agreements in the first place? As long as you can get it for free somewhere else that’s all that matters. Just how rampant is this practice? Wish I had a way to do an anonymous poll of faculty, grad students and alums to see how many think it’s all right to provide or take an account to give someone else free access to restricted resources. Based on this post – probably a lot more than we think. So much for setting good examples.