Category Archives: Assessment

If At First You Don’t Assess, Try, Try Again

ACRLog welcomes a guest post from Katelyn Tucker & Alyssa Archer, Instruction Librarians at Radford University.

Instruction librarians are always looking for new & flashy ways to engage our students in the classroom. New teaching methods are exciting, but how do we know if they’re working? Here at Radford University, we’ve been flipping and using games for one-shot instruction sessions for a while, and our Assessment Librarian wasn’t going to accept anecdotal evidence of success any longer. We decided that the best way to see if our flipped and gamified lessons were accomplishing our goals was to evaluate the students’ completed assignments. We tried to think of every possible issue in designing the study. Our results, however, had issues that could have been prevented in hindsight. We want you to learn from our mistakes so you are not doomed to repeat them.

Our process

Identifying classes to include in this assessment of flipped versus gamified lessons was a no-brainer for us. A cohort of four sections of the same course that use identical assignment descriptions, assignment sheets, and grading rubrics meant that we had an optimal sample population. All students in the four sections created annotated bibliographies based on these same syllabi and assignment instructions. We randomly assigned two classes to receive flipped information literacy instruction and two to play a library game. After final grades had been submitted for the semester, the teaching faculty members of each section stripped identifying information from their students’ annotated bibliographies and sent them to us. We assigned each bibliography a number and then assigned two librarian coders to each paper. We felt confident that we had a failsafe study design.

Using a basic rubric (see image below, click to enlarge), librarians coded each bibliography for three outcomes using a binary scale. Since our curriculum lists APA documentation style, scholarly source evaluation, and search strategy as outcomes for the program, we coded for competency in these 3 areas. This process took about two months to complete, as coding student work is a time-consuming process.

assessmentchart

The challenges

After two librarians independently coded each bibliography, our assessment librarian ran inter-rater reliability statistics, and… we failed. We had previously used rubrics to code annotated bibliographies for another assessment project, so we didn’t spend any time explaining the process with our experienced coders. As we only hit around 30% agreement between coders, it is obvious that we should have done a better job with training.

Because we had such low agreement between coders, we weren’t confident in our success with each outcome. When we compared the flipped sections to the gamified ones, we didn’t find any significant differences in any of our outcomes. Students who played the game did just as well as those who were part of the flipped sections. However, our low inter-rater reliability threw a wrench in those results.

What we’ve learned

We came to understand the importance of norming, discussing among coders what the rubric means, and incorporating meaningful conversations on how to interpret assessment data into the norming process. Our inter-rater reliability issues could have been avoided with detailed training and discussion. Even though we thought we were safe on this project, because of earlier coding projects, the length of time between assessments created some large inconsistencies.

We haven’t given up on norming: including multiple coders may be time-intensive, but when done well, gives our team confidence in the results. The same applies to qualitative methodologies. As a side part of this project, one librarian looked at research narratives written by some participants, and decided to bravely go it alone on coding the students’ text using Dedoose. While it was an interesting experiment, the key point learned was to bring in more coders! While qualitative software can help identify patterns, it’s nothing compared to a partner looking at the same data and discussing as a team.

We also still believe in assessing output. As librarians, we don’t get too many opportunities to see how students use their information literacy skills in their written work. By assessing student output, we can actually track competency in our learning outcomes. We believe that students’ papers provide the best evidence of success or failure in the library classroom, and we feel lucky that our teaching faculty partners have given us access to graded work for our assessment projects.

Digital Badges for Library Research?

The world of higher education has been abuzz this past year with the idea of digital badges. Many see digital badges as an alternative to higher education’s system of transcripts and post-secondary degrees, which are constantly being critically scrutinized for their value and ability demonstrate that students are ready for a competitive workforce. There have been several articles from the Chronicle of Higher Education discussing this educational trend. One such article is Kevin Carey’s “A Future Full of Badges,” published back in April. In it, Carey describes how UC Davis, a national leader in agriculture, is pioneering a digital open badge program.

UC Davis’s badge system was created specifically for undergraduate students majoring in Sustainable Agriculture and Food Systems. Their innovative system was one of the winners of the Digital Media and Learning Competition (sponsored by Mozilla and the MacArthur Foundation). According to Carey,

Instead of being built around major requirements and grades in standard three-credit courses, the Davis badge system is based on the sustainable-agriculture program’s core competencies—”systems thinking,” for example. It is designed to organize evidence of both formal and informal learning, from within traditional higher education and without.

As opposed to a university transcript, digital badges could provide a well-rounded view of a student’s accomplishments because it could take into account things like conferences attended and specific skills learned. Clearly, we’re not talking about Girl Scout badges.

Carey seems confident that digital badges aren’t simply a higher education fad. He believes that that with time, these types of systems will grow and be recognized by employers. But I’m still a bit skeptical over whether this movement will gain enough momentum to last.

But just for a moment, let’s assume that this open badge system proves to be a fixture in the future of higher education. Does this mean someday a student could get a badge in various areas of library research, such as searching Lexis/Nexis, locating a book by its call number, or correctly citing a source within a paper? Many college and university librarians struggle with getting information competency skills inserted into the curriculum in terms of learning outcomes or core competencies. And even if they are in the curriculum, librarians often struggle when it comes to working with teaching faculty and students to ensure that these skills are effectively being taught and graded. Perhaps badges could be a way for librarians to play a significant role in the development and assessment student information competency skills.

Would potential employers or graduate school admissions departments be impressed with a set of library research badges on someone’s application? I have no idea. But I do know that as the amount of content available via the Internet continues to grow exponentially, the more important it is that students possess the critical thinking skills necessary to search, find, assess, and use information. If digital badges do indeed flourish within higher education, I hope that library research will be a vital part of the badge sash.

Unpacking Assessment

ACRLog welcomes a guest post from Lisa Horowitz, Assessment Librarian at MIT Libraries.

As an assessment librarian, I am always looking for different ways to think about assessment. Most librarians aren’t statisticians, and for some, even the word itself, assessment, is daunting in that its meaning is unclear. Additionally, it’s such a broad topic that many of us are interested in only specific angles: learning outcomes, collection assessment, return on investment, the Value of Academic Libraries, and so on.

So what is assessment, when you come right down to it? Some librarians where I work find that the terms assessment, evaluation, statistics and data seem to be used interchangeably. The most meaningful way for me to approach the topic is to think of assessment as quality control. It is a way to look at your services, your workflows, your teaching — whatever — to determine what works and what can be improved. In that sense, yes, it is also evaluation. I’ve seen explanations that differentiate between assessment and evaluation, but I tend to just use the term assessment.

Statistics that are gathered for whatever reason, for ARL or ACRL, or for accreditation or other purposes, are actually gathered to assess something. Sometimes they are separated from that assessment because often those who gather these statistics are not the ones who do the assessment. About a dozen years ago, I was on a team that was involved in assessing our reference services while a different team was analyzing our reference-statistics-gathering procedures, until we all realized that the procedures we used to gather statistics would really depend on what we were trying to learn about our services; in other words, we needed to know what we were trying to assess in order to determine what statistics would be useful. Statistics should be inextricably tied to what you are assessing.

The use of the word “data” in libraries can be equally confusing. In the case of assessment, data are the actual numbers, or anecdotes even, that are used to assess. The data themselves are not assessment, but the use of those data are. Sometimes collections librarians see their data-gathering as separate from assessment. Sometimes instruction librarians see their evaluations as unrelated to assessment of library services as a whole. Sometimes librarians from different areas will collect different data to represent something (e.g., the number of items in a collection), but because they use different sources, they come up with different numbers. All of this relates to assessment, and ideally, it should all support library planning, resource allocation and project development.

Assessment, if done well, shows how services, workflows, collections, etc., can be improved. At the same time, it also should contribute to the library’s planning efforts. Let’s say that a library has done collection assessment which shows that a particular collection needs to be developed because of a new area of research among the faculty. At the same time, the instruction assessment has shown that students’ learning outcomes could be improved if information literacy training efforts were doubled, while assessment of the workflows at the service desks show that books are getting to the stacks more efficiently but interlibrary loans are taking longer than users expect. The point of assessment is not only to use these results to determine how to improve those particular areas, but they should also contribute to decisions made by senior management about resource allocation and strategic directions. In other words, assessment should help determine priorities by comparing needs uncovered by assessment with strategic goals, and by advocating for resources not only where they are most needed but where they advance the strategic goals of the library.

If you are new to assessment, there are a few articles that you may want to look at.
• Tina E. Chrzastowski (2008): “Assessment 101 for Librarians: A Guidebook,” Science & Technology Libraries 28:1-2, 155-176.
• Lisa R. Horowitz (2009): “Assessing Library Services: A Practical Guide for the Nonexpert,” Library Leadership & Management 23:4, 193-203.

Both of these have bibliographies that may be helpful, as well as links to tools, blogs, and organizations that may be useful.

What does assessment mean to you? What tools do you use? What have you done that helps staff at your library be more comfortable with assessing library services?

Clickers, or Does Technology Really Cure What Ails You?

ACRLog welcomes a guest post from Cori Strickler, Information Literacy Librarian at Bridgewater College.

During idle times at the reference desk, or when the students are gone for a break, I find myself creating instruction “wish lists” of tools or gadgets that I’d love to have for my sessions. One item that has been on my list for a few years now is clickers, or student response systems as they are officially called. In academic classrooms they are used for attendance, quiz taking, or other more informal assessments. For me, I saw clickers as a way to solve one of my basic and most frustrating problems: getting students to be engaged during the sessions. Students have little desire to participate in library sessions and trying to get them to comment on their library experience is like pulling teeth, except that the process is a lot more painful for me than it is for the students.

For those of you who haven’t heard of clickers before, they are little remote control like devices that allow the students to answer multiple choice questions by sending their responses to the computer for real time analysis. They are sort of like the devices they use on Who Wants to Be a Millionaire to poll the audience.

My library doesn’t have the budget for clickers, but this semester through a chance discussion with the director of the health services department, I learned that the college received a grant for 100 TurningPoint clickers and the necessary software. The director rarely needed all of the clickers at the same time, so she offered about fifty for me to use during my instruction sessions.

So, I now have access to a tool that I had coveted for many years, but that was only the easy part. I still have to figure out how to meaningfully integrate this technology into my sessions.

My overall goals are relatively simple. I want to encourage student involvement in any way possible so I would not have to lecture for fifty minutes straight. My voice just can’t handle the pressure. To be successful, though, I need to be purposeful with my inclusion. I can’t just stick a clicker quiz at the beginning of a session and assume that the students will suddenly be overwhelmed with a desire to learn everything there is about the library. Most faculty who schedule a library instruction session have a particular purpose in mind, so I also need to be sure that I fulfill their expectations as well.

After much consideration, I decided not to add the clickers to all my sessions. Instead, I decided to focus on first year students, who hopefully aren’t quite as jaded as the upper classmen, and haven’t already decided that they know everything about research.

For my first clicker experiment, I used them with a quiz to help me gauge the classes’ knowledge of the library. I also decided to use them as an alternative way to administer our session evaluation survey. Ultimately, I had mixed results with the clickers. The students did respond better than before, but I did not get full participation. While this isn’t a big issue with the quiz, this lack of participation was an issue when they were asked to complete the evaluation survey. For most survey questions I lacked responses from five or six students, which was a larger number than when I used the paper surveys and could potentially affect my survey results.

Their lack of participation could be due to a number of reasons. The students claimed they were familiar with the clickers, but they did not seem to be as adept as they claimed. Also, due to my inexperience with the clickers there might have been a malfunction with the devices themselves. Or, maybe the students just didn’t want to engage, especially since there was still no incentive to participate. When I looked back through the survey results, they did not seem to indicate any greater amount of satisfaction regarding the sessions.

This first experience with the clickers left me a bit skeptical, but I decided to try them again. This time, I created brief quizzes related to brainstorming keywords and types of plagiarism. My second class was smaller than the first, and I seemed to receive better engagement. The clickers also seemed to allow them to be more honest with the surveys and they seem more comfortable indicating their disinterest in the information presented, though the results also indicated that they saw the overall value in the information.

I have used the clickers in about twelve sessions this semester, and overall they were well received by the students. However, I am not completely sure that it adds significantly to the engagement. I also have not seen any indication in the surveys that my sessions are better or worse with their inclusion. I have discovered though that there may be some sessions, and topics, that are better suited for clickers than others. Upper level classes where I am trying to show specific resources do not lend themselves initially to clickers, and the time may be better spent with other activities or instruction.

I am still in the process of learning how clickers will fit into my classes, but I would generally call them a success, if only for the fact that is makes the survey process easier. Though, they aren’t the panacea for student engagement for which I had hoped. Activity type and student familiarity are essential variables that appear to affect clicker success.

Unfortunately, the overall nature of one shot instruction seems to be the greatest contributor to student disengagement. Student and faculty buy-in is the necessary component for library instruction success, whether it includes clickers or not.

On Being Valuable: Point-Counterpoint

The POINT: Amy Fry

On Tuesday, September 14, ACRL released Value of Academic Libraries: A Comprehensive Research Review and Report by Dr. Megan Oakleaf. The report lays out the current landscape of academic library assessment and seeks to provide strategies for libraries to demonstrate and quantify their value within the context of institutional missions and goals.

Oakleaf states that internal measures of value, such as use statistics, user satisfaction, and service quality, while interesting to librarians, are less compelling for external stakeholders such as administrators and trustees (11). Instead, she suggests determining externally-focused measures of value such as “library impact” (best measured by observing what users are doing and producing as a result of using the library) and “competing alternatives” (which focuses on defining what users want and how libraries, rather than our competitors, can help them achieve it) (21-22). She suggests ten key areas libraries should try to address in such assessment: enrollment, retention/graduation, student success, student achievement, student learning, student engagement, faculty research productivity, faculty grants, faculty teaching, and institutional reputation (17). Oakleaf also offers strategies for approaching assessment related to each area.

Oakleaf claims that “use-based definitions of value are not compelling to many institutional decision makers and external stakeholders. Furthermore, use is not meaningful, unless that use can be connected to institutional outcomes” (20). In a brief section about e-resources, she explains that usage counts don’t show why a library resource was used or the user’s satisfaction with it (50); she therefore suggests that, rather than collecting and reporting usage data for electronic resources, libraries try to collect qualitative data, like the purpose of the use (using the ARL MINES protocol). She also suggests examining successful grant applications to “examine the degree to which citations impact whether or not faculty are awarded grants.”

The question of how to use e-resources statistics to draw qualitative conclusions about users’ information literacy levels and the effectiveness of electronic collections (or even about the library’s impact on faculty research or student recruitment and retention), is of special interest to me now, as I have just agreed to examine (and hopefully overhaul) my institution’s management of e-resources statistics. However, such questions are overshadowed for me (and for most libraries), by how to effectively gather, merge, and analyze the statistics themselves, what to do with resources that don’t offer statistics at all or don’t offer them in COUNTER format, and when and how to communicate them internally for collection decisions. It is difficult to see arriving at higher-level methods for library assessment that involve overlaying complex demographic data, research output data, cost data, collection data, and use data in order to tell compelling stories about library use and impact when even the most basic systems for managing inputs and outputs have not been implemented.

I understand and even agree with Oakleaf’s characterization of the shortcomings of “use-based definitions of value,” but am not sure that surveying users about the purpose of their information use or linking library collections to successful grant applications truly gives a more compelling picture of the value of electronic resources collections, nor one that is more complete. For example, assessing value by linking library collections to grants funded or patents produced seems like it would discount libraries’ value to humanities research, because humanities scholarship will never approach the sciences in the amount of dollars coming in.

It is true that libraries currently “do not track data that would provide evidence that students who engage in more library instruction are more likely to graduate on time, that faculty who use library services are more likely to be tenured, or that student affairs professionals that integrate library services into their work activities are more likely to be promoted” (13). But that stuff just really seems like no-brainers to me. If we spend a lot of time and energy collecting the data and putting it together to get the numbers that will allow us to make these claims – then what? What’s the payoff for the library? Administrators who don’t think libraries are just black holes for funding? A way to prove to students that they should use the library? If administrators and trustees are not inclined to fund libraries because their backgrounds did not include library use, or students are not inclined to use libraries because they are focused on graduation and employment instead of research, I don’t know that any such externally focused assessment will result in what seems to be, ultimately, the desired outcome – a reassertion of libraries’ relevance to our core constituents. It will, however, be a drain on library staff time and expertise – time and expertise that could be spent on core activities, like collection building, collection access, and public service.

Oakleaf concludes that our focus should be not to prove but to increase value (141). We should not ask, “Are libraries valuable?” but “How valuable are libraries?” she says. What about “How are libraries valuable?” But this is semantics. No matter what our approach to assessment, I’m afraid the answer will still depend less on what data we present than who we ask.

The COUNTERPOINT: Steven Bell

What’s the payoff for the library? That’s an important question when it comes to assessment and efforts to demonstrate the academic library’s value to its own institution and higher education. Amy Fry makes a good point that we could invest considerable time and energy to collect and analyze the data needed to determine our value in any or all of the ten key areas recommended in the ACRL Value Report – but why bother? She states that when it comes to questioning if library instruction sessions can be connected to better grades or students graduating on time, that’s “no brainer” territory.

But can we in fact assume that just because a student attends an instruction session – or because faculty have access to research databases – that they are indeed achieving institutional outcomes? If, as a profession, we thought that was no brainer territory why are there hundreds of research articles in our literature that attempt to prove that students who sit through library instruction sessions are better off than the ones who don’t – we clearly aren’t just assuming they are, we want to prove it – and in doing so prove why we make a difference to our students’ education and learning process.

As Barbara Fister points out in her response to the Report, provosts already acknowledge, anecdotally, that they value their libraries and librarians. And we also know that the library is the heart of the institution, and that libraries are like Mom and apple pie; everyone likes the library. You probably couldn’t find an academic administrator who would go on record trashing the academic library (well, maybe this one). But none of that may stop administrators, when push comes to shove, from taking drastic measures with library services to resolve a budget crisis. Being the heart of the institution didn’t stop Arizona’s Coconino Community College from performing radical heart surgery by outsourcing the library operations to North Arizona University’s (NAU) Cline Library. Admittedly, that’s a rare occurrence, and I can’t say for sure that even the best set of library value data could prevent it from happening. Yet one can’t help but imagine that if Coconino’s librarians had some rock solid assessment data on hand to confirm their value to administrators – be it how the library keeps students retained or helps students to achieve higher GPAs – they’d still have their jobs and be delivering services to their students at their own library (which was largely chopped up and pieced out to other academic units).

And better assessment and demonstration of library value can indeed result in a financial payoff for the institution if awarded government grants and the indirect costs associated with conducting research. Those indirect costs, typically a percentage rate negotiated between the institution and federal government, can make a huge difference in institutional funding for research. Given the size of some grants, just a slight increase – perhaps a percentage point or two – can make a real impact over time. Amy mentions the ARL MINES protocol, which is a process for making a concrete connection between researchers working on grant projects and their use the library resources to conduct that research. Often the contribution of the library is drastically understated, and therefore it is barely reflected in the calculation of the ICR (indirect cost recovery). My own institution is currently conducting a survey similar to MINES so that our “bean counters” (as Barbara likes to refer to them) can more accurately connect the expenditures for library electronic resources to research productivity – and the government’s own bean counters have very rigid rules for calculating increases to the ICR. It can’t be based on anecdotal evidence or simply having researchers state that they use library resources for their research. In this case, asking the users if we provide value doesn’t mean squat. Providing convincing evidence might mean an increase to our ICR of one or two percent –which over time could add up to significant amounts of funding to support research. That is a real payoff, but make no mistake that we have invested considerable time and expense in setting up the survey process.

For many academic librarians, it may be better to, as Amy suggests, focus on the core activities such as collection building and traditional services (reference, instruction, etc.) – and to keep improving on or expanding in those areas. But I like to think that what drives real advancement in our academic libraries is confronting new mysteries that will force us to seek out new answers that could lead to improvements in fundamental library operations. What happens when we fail to seek out new mysteries to explore is that we simply continue to exploit the same existing answers over and over again until we drive them and ourselves into obsolescence (for more on “knowledge funnel” theory read here).

Lately I’ve been advocating that the new mystery for academic librarians should focus on assessment. We need to get much better at answering a simple question that represents this mystery: How can we tell that we are making a difference – and how will we gather the data to quantitatively prove it? From this perspective the question would be neither “How valuable are academic libraries?” or “How are libraries valuable?” but “How are academic libraries making a real difference and how do we prove it?” Perhaps it remains a case of semantics, but any way we approach this new mystery, the road should lead to a better grasp of the value we provide and new ways to communicate it to our communities. Whatever you may think about assessment and the value question, take some time to review the ACRL Value Study. I’ll be at the Library Assessment Conference in DC at the end of October. I’m looking forward to learning more about how academic librarians are approaching the new mystery of assessment, and how we can all do a better job of quantifying and communicating our value proposition.