Category Archives: Teaching

If At First You Don’t Assess, Try, Try Again

ACRLog welcomes a guest post from Katelyn Tucker & Alyssa Archer, Instruction Librarians at Radford University.

Instruction librarians are always looking for new & flashy ways to engage our students in the classroom. New teaching methods are exciting, but how do we know if they’re working? Here at Radford University, we’ve been flipping and using games for one-shot instruction sessions for a while, and our Assessment Librarian wasn’t going to accept anecdotal evidence of success any longer. We decided that the best way to see if our flipped and gamified lessons were accomplishing our goals was to evaluate the students’ completed assignments. We tried to think of every possible issue in designing the study. Our results, however, had issues that could have been prevented in hindsight. We want you to learn from our mistakes so you are not doomed to repeat them.

Our process

Identifying classes to include in this assessment of flipped versus gamified lessons was a no-brainer for us. A cohort of four sections of the same course that use identical assignment descriptions, assignment sheets, and grading rubrics meant that we had an optimal sample population. All students in the four sections created annotated bibliographies based on these same syllabi and assignment instructions. We randomly assigned two classes to receive flipped information literacy instruction and two to play a library game. After final grades had been submitted for the semester, the teaching faculty members of each section stripped identifying information from their students’ annotated bibliographies and sent them to us. We assigned each bibliography a number and then assigned two librarian coders to each paper. We felt confident that we had a failsafe study design.

Using a basic rubric (see image below, click to enlarge), librarians coded each bibliography for three outcomes using a binary scale. Since our curriculum lists APA documentation style, scholarly source evaluation, and search strategy as outcomes for the program, we coded for competency in these 3 areas. This process took about two months to complete, as coding student work is a time-consuming process.


The challenges

After two librarians independently coded each bibliography, our assessment librarian ran inter-rater reliability statistics, and… we failed. We had previously used rubrics to code annotated bibliographies for another assessment project, so we didn’t spend any time explaining the process with our experienced coders. As we only hit around 30% agreement between coders, it is obvious that we should have done a better job with training.

Because we had such low agreement between coders, we weren’t confident in our success with each outcome. When we compared the flipped sections to the gamified ones, we didn’t find any significant differences in any of our outcomes. Students who played the game did just as well as those who were part of the flipped sections. However, our low inter-rater reliability threw a wrench in those results.

What we’ve learned

We came to understand the importance of norming, discussing among coders what the rubric means, and incorporating meaningful conversations on how to interpret assessment data into the norming process. Our inter-rater reliability issues could have been avoided with detailed training and discussion. Even though we thought we were safe on this project, because of earlier coding projects, the length of time between assessments created some large inconsistencies.

We haven’t given up on norming: including multiple coders may be time-intensive, but when done well, gives our team confidence in the results. The same applies to qualitative methodologies. As a side part of this project, one librarian looked at research narratives written by some participants, and decided to bravely go it alone on coding the students’ text using Dedoose. While it was an interesting experiment, the key point learned was to bring in more coders! While qualitative software can help identify patterns, it’s nothing compared to a partner looking at the same data and discussing as a team.

We also still believe in assessing output. As librarians, we don’t get too many opportunities to see how students use their information literacy skills in their written work. By assessing student output, we can actually track competency in our learning outcomes. We believe that students’ papers provide the best evidence of success or failure in the library classroom, and we feel lucky that our teaching faculty partners have given us access to graded work for our assessment projects.

Musings on Outreach as Instruction

Last week, librarians from many branches of our university gathered for a Teaching Librarians Retreat. The retreat was organized and hosted by a few wonderful colleagues, who I cannot thank enough for their efforts and a fantastic event. The goal for the retreat was to promote a community of sharing, peer support, and ongoing learning among UI librarians who teach, and was a chance to reflect on the year and find colleagues with similar interests and concerns about teaching. Making dedicated time for sharing and reflection is especially important in an institution as large and with as many librarians as ours.

We broke out into discussion groups for part of the retreat, and my group gathered to talk about “outreach as instruction.” What struck me first as we each shared our thoughts is that “outreach” can mean so many different things. We had people contributing to the conversation from perspectives of social media, events and programming, marketing, digital badges, special collections, working with student organizations, and outreach to faculty vs. students vs. the community.

My take on “outreach as instruction” and why it matters has to do with the limitations of one-shot sessions and ways we can expand the impact of instruction beyond traditional methods. One-shot sessions are valuable as point-of-need instruction for academic coursework, but relying solely on them is limiting: only a fraction of students receive library instruction, and a number of them may not be particularly interested in the General Education required course that brought them into the library. This is where I think outreach can be powerful – in the many possibilities to connect with students outside of a classroom setting, while still teaching something. Here are a few ideas on how to go about doing that:

  1. Connect over something interest-based, rather than academics-based. For example, I’ve heard of academic libraries having knitting sessions (which is also closely tied with stress-relief activities during finals week), but it could be something else. The draw to participate is something of general interest that can also be connected to research and resources available at the library.
  2. Communicate with student organizations, and let the student leaders know how the library can support their group and members. This can lead to tailored teaching opportunities for students who are involved and invested in a group that may not get this attention and instruction otherwise.
  3. Use the collection creatively. We’ve found ways to do this by using images from the Iowa Digital Library on buttons, postcards, and Valentine cards. Those are all short and simple activities that can naturally lead to learning something new about a variety of resources. (You can see the Valentine’s activities here.)

Those are just a few ideas, which clearly come from my perspective as an Undergraduate Services Librarian (and barely crack the surface of our group discussion at the Teaching Librarians Retreat). For you, “outreach as instruction” could mean building on relationships with faculty, an emphasis on social media, or something else. Outreach itself is a broad concept with multiple definitions, but that also means there are so many variations and opportunities for librarians to engage with their users and community.

When I hear “outreach as instruction,” I think of how we can connect with undergraduates in ways other than in the classroom for a one-shot session, and incorporate what I like to call “nuggets of information literacy.” What does it mean for you and your library?

Beyond Livetweeting: Twitter Chats for Professional Development

This time 11 days ago I was grumpy. It was the last Friday before Spring Break, and I was prepping to teach an English Comp I session the next morning — not just a Saturday class but the Saturday before the break week. Our English Comp I sessions are typically assignment-driven, as we’ve found that to be more useful for students than a library tour or orientation, and having an assignment to work on encourages them to participate during the session.

But this class was different. Just one week prior to the library session, the current professor had taken over this class from another professor who’d fallen ill. The new prof was still getting his bearings with the students and he hadn’t yet assigned the research essay. I’d been thinking for a while now that I need to develop a solid plan for these occasional no-assignment sessions, something interactive and useful to students, but it’s been a busy semester and I haven’t found the time to put to it. So there I was with a class the next morning, not at all sure what I’d do with those students, how I’d keep them awake and engaged on the last day before break, or what I could offer that would be of most use to them when they eventually got to work on their research assignments near the end of the semester.

And then I remembered the most recent #critlib chat. What is #critlib, you ask? It’s a Twitter chat held at 9pm Eastern time every other Tuesday, begun earlier this month by moderators (in alphabetical order by Twitter handle) Jenna Freedman (@barnlib), Annie Pho (@catladylib), Emily Drabinski (@edrabinski), Kelly McElroy (@kellymce), and Nicole Pagowsky (@pumpedlibrarian). The purpose of the chat is to “engage in discussion about what critical pedagogy is and how it can be used in library instruction.” The most recent chat, on April 8, was terrific, with conversation ranging from whether neutrality is possible to strategies for encouraging our students to think critically about information. As Barbara pointed out:

At the time I’d wondered how I could incorporate what I read and learned during the chat into my usual strategy for teaching the English Comp I library session, and the need to create a new strategy for this no-assignment session let me do just that. I was lucky that the professor’s broad topic for the class is the American Dream, which is perhaps more amenable to a critical information literacy lens than many topics. I began by spending time talking with students about creating a more narrow research question from a broad topic, and we used the hypothetical research question “Is the American Dream available to all Americans in 2014?” to generate keywords and synonyms for searching which I wrote on the whiteboard.

Overall the session included more questions and discussion and less time for students to search on their own than the more assignment-driven sessions I teach. We spent lots of time talking about how information is produced and distributed while trying to keep a practical focus on what’s available in the library and what’s available on the open internet. We talked about how search engine results are ranked, and what to consider when choosing which information to use. I asked them to work with a partner to find one library and one internet source; while I love asking students to work together, it can be challenging if each student has chosen a different research topic.

While much of what I did in this session is similar to what I do in the more assignment-driven sessions, reviewing the #critlib chat before planning the session helped me stay mindful of critical approaches to information literacy as I was teaching. There’s always so much to cover in a library session and it can be so easy to charge on through, and I’m grateful that participating in the chat two weeks ago reminded me to look for opportunities to draw in students’ own experiences and to question the information landscape with students.

I’ve used Twitter to catch up on conference livetweeting for a while now, and also get lots of recommendations for professional reading and resources from the folks I follow, but this is the first time I’ve dipped into a Twitter chat for professional development. If you’ve missed the two chats so far, never fear: there’s a terrific cheat sheet/repository of chats and questions with links to a Zotero bibliography and a Storify of each chat. And if you’re interested in critical information literacy, please join in! The next chat is tomorrow, Tuesday April 22nd, at 9pm Eastern. Use the hashtag #critlib to tweet and follow the conversations.

Wondering About Workshops

Like many academic librarians, my colleagues and I teach several drop-in workshops each semester for faculty and staff at the college on topics like citation managers, Google Scholar and other specialized research tools, and instructional web design, among others. I’ve written a couple of times here about these workshops: we consider them to be opportunities for outreach as much as for instruction, though our attendance levels have waxed and waned over the years, leading us to add a workshops by request option for departments or other groups of interested faculty and staff. The latter has been intermittently successful — some semesters we’ve gotten several requests for workshops while others have seen none — though since these workshops can typically be prepped fairly quickly we’ve decided to keep offering them for now.

The past year or so has brought a new twist to our faculty/staff workshops: students! For several of the workshops we’ve offered — most recently one focusing on using ILL and other libraries in New York City to make the most of research beyond our college library — we’ve had one or two students attending as well as faculty and staff. We advertise the workshops on a faculty and staff email list that doesn’t include students, but we also hang posters around campus, which is probably the way students have learned about the workshops (or via our blog or Twitter). We’ve always had plenty of room in the workshops for the students who’ve dropped in and, as far as I know, there haven’t been any problems with the occasional student sitting in on a workshop with faculty and staff.

If there aren’t any problems, what’s to say about it? I keep coming back to thinking about students in the faculty/staff workshops for a couple of reasons. We used to offer drop-in workshops for students, too, but stopped doing so a few years ago because we very rarely had anyone show up. Perhaps it’s time to bring drop-in student workshops (not course-related) back into our instructional mix? One thing to note is that in the past the drop-in student workshops typically covered one resource like Academic Search Complete or LexisNexis, or were much more general workshops on research strategies for students. Maybe the more specific and advanced topics covered in the faculty/staff workshops are more appealing to our students, especially those who’ve already taken English Comp I, which requires a library instruction session?

On the other hand, every workshop requires at least a little bit of prep time, not to mention the time to promote it via email, posters, blogging, and Twitter. Our workshop committee is fairly busy already, so to add workshops that may not be well-attended could be tough.

All of which makes me wonder: if our faculty/staff workshops are not currently overcrowded, and our student workshops were not historically overcrowded, might we consider offering workshops that are open to any member of the college community, faculty, staff, and students alike?

To my knowledge we’ve never done that before. What are the possible ramifications of workshops open to all? Research has shown that interaction between students and faculty outside of the classroom has a positive impact on student engagement (Kuh et al., 2007, Piecing Together the Student Success Puzzle). Could open workshops provide those opportunities? Would faculty be uncomfortable learning something new alongside students, or vice versa? We would probably want to avoid workshop topics focused on developing plagiarism-resistant research assignments or the like, right? Or would there be a benefit to opening up an information literacy workshop pitched at faculty to students, as well?

If you’re offering workshops or other instructional opportunities for faculty, staff, and students to attend together, I’d love to hear about it!

Getting Started with Instruction

This semester marks a significant step for me as I’m finally getting into doing instruction sessions on my own. Throughout last fall, I observed a lot of instruction sessions from several librarians and across a range of subjects. I also co-taught a handful of classes with a colleague, but it wasn’t until this month that I took on my own instruction sessions. I’m really glad I did some co-teaching already, because I was definitely nervous at the time and it’s good to have that out of the way now (for the most part).

In a short span of time I have done a handful of sessions, and not one of them the same. I started writing detailed reflections of all the instruction I have done so far – what I did, what worked, what didn’t work, what I would do differently next time, etc. – and while that is incredibly useful for me personally, I will refrain from posting the entire detailed accounts here! However, I will give a quick run-down:

  • So far I have done one-shots for two sections of Rhetoric, a course that’s required of all undergraduate students, but which can vary a lot depending on the instructor. For one section, their assignment was concept-mapping and researching potential careers based on their majors; the other section needed to find images to use for a visual analysis. Like I said, interesting stuff going on that was fun to work with!
  • I did a workshop in collaboration with TRiO, an organization that works with first-generation students. Part of the goal was to send them out into the stacks in a safe, no-pressure situation, so that they can avoid the “panic moment” later on when they really need to find something. Attendance was pretty low as expected, because it wasn’t required for a course, but some good discussion came out of it nonetheless.
  • Large groups of middle school students visit our library throughout the year to do primary research for the National History Day competition, and on one occasion I gave a 15-minute introduction. I kept it simple with just basic information and demonstrating SmartSearch – it was fun to switch gears for a bit for a much different audience than usual.
  • And most recently I gave an Express Workshop on how to use and make infographics. Express Workshops are weekly 30-minute workshops held in an open area in the Learning Commons, with a different topic and presenter every week.

I’m glad to have such a variety of classes to work with – for one thing, it keeps things interesting, and for another, I think it’s more challenging (in a good way) than if I were repeating basically the same session. However, the planning has been difficult at times.

A lot of the difficulties may come down to time management and figuring out my own process. I planned ahead as much as possible, but often felt like I was really getting prepared when time was down to the wire. I wanted to have lesson plans laid out a good deal ahead of time and prevent the stress of procrastination, but it was difficult for me to focus on future sessions when there were others to take place first – especially since these were my actual first instruction sessions ever. I think my planning problems stem in part from the fact that this is a much busier time of year than I expected it would be!

I can’t wait to get to the point where I’ve done enough instruction that I’m more confident with the whole process, from planning, to delivery, and assessment. When planning a session I consider many possible options and what would be most effective, and then still tend to question my decisions on what to include and how to conduct the session. I already feel a little more confident in my teaching abilities than I did even a month ago, and I know that the rest will take some more time and practice.

Does anyone else have similar concerns? Do you plan ahead, or do you work better under pressure? How much time does it take to plan a session?