Tag: statistics

Unpacking Assessment

ACRLog welcomes a guest post from Lisa Horowitz, Assessment Librarian at MIT Libraries.

As an assessment librarian, I am always looking for different ways to think about assessment. Most librarians aren’t statisticians, and for some, even the word itself, assessment, is daunting in that its meaning is unclear. Additionally, it’s such a broad topic that many of us are interested in only specific angles: learning outcomes, collection assessment, return on investment, the Value of Academic Libraries, and so on.

So what is assessment, when you come right down to it? Some librarians where I work find that the terms assessment, evaluation, statistics and data seem to be used interchangeably. The most meaningful way for me to approach the topic is to think of assessment as quality control. It is a way to look at your services, your workflows, your teaching — whatever — to determine what works and what can be improved. In that sense, yes, it is also evaluation. I’ve seen explanations that differentiate between assessment and evaluation, but I tend to just use the term assessment.

Statistics that are gathered for whatever reason, for ARL or ACRL, or for accreditation or other purposes, are actually gathered to assess something. Sometimes they are separated from that assessment because often those who gather these statistics are not the ones who do the assessment. About a dozen years ago, I was on a team that was involved in assessing our reference services while a different team was analyzing our reference-statistics-gathering procedures, until we all realized that the procedures we used to gather statistics would really depend on what we were trying to learn about our services; in other words, we needed to know what we were trying to assess in order to determine what statistics would be useful. Statistics should be inextricably tied to what you are assessing.

The use of the word “data” in libraries can be equally confusing. In the case of assessment, data are the actual numbers, or anecdotes even, that are used to assess. The data themselves are not assessment, but the use of those data are. Sometimes collections librarians see their data-gathering as separate from assessment. Sometimes instruction librarians see their evaluations as unrelated to assessment of library services as a whole. Sometimes librarians from different areas will collect different data to represent something (e.g., the number of items in a collection), but because they use different sources, they come up with different numbers. All of this relates to assessment, and ideally, it should all support library planning, resource allocation and project development.

Assessment, if done well, shows how services, workflows, collections, etc., can be improved. At the same time, it also should contribute to the library’s planning efforts. Let’s say that a library has done collection assessment which shows that a particular collection needs to be developed because of a new area of research among the faculty. At the same time, the instruction assessment has shown that students’ learning outcomes could be improved if information literacy training efforts were doubled, while assessment of the workflows at the service desks show that books are getting to the stacks more efficiently but interlibrary loans are taking longer than users expect. The point of assessment is not only to use these results to determine how to improve those particular areas, but they should also contribute to decisions made by senior management about resource allocation and strategic directions. In other words, assessment should help determine priorities by comparing needs uncovered by assessment with strategic goals, and by advocating for resources not only where they are most needed but where they advance the strategic goals of the library.

If you are new to assessment, there are a few articles that you may want to look at.
• Tina E. Chrzastowski (2008): “Assessment 101 for Librarians: A Guidebook,” Science & Technology Libraries 28:1-2, 155-176.
• Lisa R. Horowitz (2009): “Assessing Library Services: A Practical Guide for the Nonexpert,” Library Leadership & Management 23:4, 193-203.

Both of these have bibliographies that may be helpful, as well as links to tools, blogs, and organizations that may be useful.

What does assessment mean to you? What tools do you use? What have you done that helps staff at your library be more comfortable with assessing library services?

Academic Library Data For Prospective Students

One area in which higher education has underperformed is in providing prospective students with fair, balanced and objective information about colleges and universities that could help in the college choice decision-making process. Most students and their parents end up gravitating to U.S. News and World Report rankings for this information, and we know that content is hardly objective and that institutions can – and willingly do – game the system to achieve higher rankings.

Renewed attention on this problem is being prompted by a soon-to-be-released report titled “Grading Higher Education: Giving Consumers the Information They Need.” One of the report’s findings is “that more-organized and better-targeted information will help prospective students and their families make wiser choices — especially if this information is oriented to their needs and concerns.” In a post about the report, its author is quoted as stating “Giving students and their families better information would enable them to avoid unworthy college investments that would leave them with substantial debt and little in the form of skills.”

If the federal government or some other organization, perhaps some coalition of higher education institutions, were to develop a college data source that aided selection – something along the lines of what the report recommends – I doubt there’d be information available about the academic library. This would be unfortunate because, to some extent, we know that the academic library does play a larger-than-believed role in the decision-making process.

What would we want prospective students and their parents to know about our libraries that would help their decision process? Do we even compile the type of data that would help in making a choice? Some of these would be easy to provide, others we’d have to start thinking about how we’d collect them. Here’s what comes to mind:

Number of volumes: While we can certainly provide the data I don’t see how this helps the decision process unless your primary criteria is based on bigger is better. What might be much better is some indicator of collection concentration by subject. If I want to be an architect it might be helpful to rank institutions by how much material they have on architecture. Again, it might only tell me who is bigger, but it might also say something about that institution’s commitment to supporting its researchers and students in that area – and it might also let me know I’ll probably be doing less borrowing from other libraries.

Number of study spaces: Students typically look to the library as a quiet study space. Having the ability to compare libraries based on study spaces could be relevant. If it’s just a number of total spaces that doesn’t necessarily tell the whole story. How many of those spaces are in quiet study rooms? How many are in collaborative work areas? How many are in spots where there’s a computer? Compiling this data isn’t impossible, but it would be more challenging to indicate what type of seating and where it is.

Number of Questions Answered: Most of us already collect this one, and could provide breakdowns on the type of question asked (e.g., reference vs. directional). Could be helpful in letting prospective students have an indicator of demand for assistance. Best perhaps to present it as a ratio, questions per student, so large schools have no distinct advantage over small ones. Of course, providing the number of questions doesn’t tell prospective students anything about the impact of the answers on student success.

Number of librarians: Could be helpful as an indicator of the administration’s commitment to the library. Again, best perhaps as a ratio, as in number of librarians per student. It could be helpful to offer a breakdown of sorts, by total staff, total librarians, total paraprofessional staff. By no means perfect, but let’s give them some numbers they can use for comparisons.

Number of Instruction Sessions: While I like this one in particular, I’m uncertain of how well it would be understood by prospective students and their parents. The terminology could benefit from tweaking. Done correctly, those considering the institution could get a sense of how active the librarians are in contributing to students’ education process.

Librarian-Student Contact Hours: For me this would be a meaningful indicator of the degree to which a prospective student could expect some personalized consultation time with a librarian. I can’t say to what extent prospective students would appreciate knowing this.

Those are just a few possibilities that come to mind. Perhaps there are better, more imaginative indicators that I’m missing (e.g., number of tweets about the library per semester; average time required to find and checkout a book; daily cups of coffee served at the cafe) . I’m getting the sense that there’s a direct relationship between library data indicators and data indicators that prospective students and their parents would truly value: the more valuable the indicator the harder it is to gather, compile and present that data. What indicators, no matter how difficult they might be to create, do you think would be important for academic libraries to serve as a good recruiting tool?

While no such prospective student-oriented library rankings truly exist today (a motivated student could potentially find some comparative data online – such as ACRL or ARL statistical sources – although those could hardly be described as “consumer-friendly”), we should be forward thinking. Knowing there is a likelihood that in the not-too-far future such college choice data will exist, academic librarians should be planning today for compiling the type of data that would make the most sense for prospective students and their families.