If At First You Don’t Assess, Try, Try Again

ACRLog welcomes a guest post from Katelyn Tucker & Alyssa Archer, Instruction Librarians at Radford University.

Instruction librarians are always looking for new & flashy ways to engage our students in the classroom. New teaching methods are exciting, but how do we know if they’re working? Here at Radford University, we’ve been flipping and using games for one-shot instruction sessions for a while, and our Assessment Librarian wasn’t going to accept anecdotal evidence of success any longer. We decided that the best way to see if our flipped and gamified lessons were accomplishing our goals was to evaluate the students’ completed assignments. We tried to think of every possible issue in designing the study. Our results, however, had issues that could have been prevented in hindsight. We want you to learn from our mistakes so you are not doomed to repeat them.

Our process

Identifying classes to include in this assessment of flipped versus gamified lessons was a no-brainer for us. A cohort of four sections of the same course that use identical assignment descriptions, assignment sheets, and grading rubrics meant that we had an optimal sample population. All students in the four sections created annotated bibliographies based on these same syllabi and assignment instructions. We randomly assigned two classes to receive flipped information literacy instruction and two to play a library game. After final grades had been submitted for the semester, the teaching faculty members of each section stripped identifying information from their students’ annotated bibliographies and sent them to us. We assigned each bibliography a number and then assigned two librarian coders to each paper. We felt confident that we had a failsafe study design.

Using a basic rubric (see image below, click to enlarge), librarians coded each bibliography for three outcomes using a binary scale. Since our curriculum lists APA documentation style, scholarly source evaluation, and search strategy as outcomes for the program, we coded for competency in these 3 areas. This process took about two months to complete, as coding student work is a time-consuming process.

assessmentchart

The challenges

After two librarians independently coded each bibliography, our assessment librarian ran inter-rater reliability statistics, and… we failed. We had previously used rubrics to code annotated bibliographies for another assessment project, so we didn’t spend any time explaining the process with our experienced coders. As we only hit around 30% agreement between coders, it is obvious that we should have done a better job with training.

Because we had such low agreement between coders, we weren’t confident in our success with each outcome. When we compared the flipped sections to the gamified ones, we didn’t find any significant differences in any of our outcomes. Students who played the game did just as well as those who were part of the flipped sections. However, our low inter-rater reliability threw a wrench in those results.

What we’ve learned

We came to understand the importance of norming, discussing among coders what the rubric means, and incorporating meaningful conversations on how to interpret assessment data into the norming process. Our inter-rater reliability issues could have been avoided with detailed training and discussion. Even though we thought we were safe on this project, because of earlier coding projects, the length of time between assessments created some large inconsistencies.

We haven’t given up on norming: including multiple coders may be time-intensive, but when done well, gives our team confidence in the results. The same applies to qualitative methodologies. As a side part of this project, one librarian looked at research narratives written by some participants, and decided to bravely go it alone on coding the students’ text using Dedoose. While it was an interesting experiment, the key point learned was to bring in more coders! While qualitative software can help identify patterns, it’s nothing compared to a partner looking at the same data and discussing as a team.

We also still believe in assessing output. As librarians, we don’t get too many opportunities to see how students use their information literacy skills in their written work. By assessing student output, we can actually track competency in our learning outcomes. We believe that students’ papers provide the best evidence of success or failure in the library classroom, and we feel lucky that our teaching faculty partners have given us access to graded work for our assessment projects.

Thinking Tenure Thoughts

Last week Meredith Farkas wrote a thoughtful post on her blog, Information Wants to Be Free, about tenure status for academic librarians. Spirited discussion ensued in Meredith’s blog comments and on libraryland Twitter (much of which Meredith Storified) which has continued to today. The conversation has included many varied perspectives on the advantages and disadvantages of tenure for academic librarians, including preparation for research and scholarship in graduate library programs, the perceptions of status and equality between academic librarians and faculty in other departments, salary parity, academic freedom, and the usefulness and rigor of the library literature.

I support tenure for academic librarians as I do for faculty in other departments primarily because I believe that tenure ensures academic freedom, which is as important in the library as it is in other disciplines. I also have concerns about the tenure system more generally, concerns that many academics in libraries and other departments also voice. One of my big concerns is that the pressure to publish can result in quantity over quality.

This conundrum was raised during the Twitter discussion of Meredith’s post and had me nodding vigorously as I read. I am absolutely in agreement that the tenure system as it currently stands has encouraged the publication of large amounts of scholarship that ranges from the excellent and thought-provoking, to the interesting if somewhat obvious, to the just not very good, to the occasionally completely wrong. Of course, this is a problem not just in academic librarianship but in other disciplines as well. The avalanche of scholarship resulting from the pressures to publish to gain tenure affects libraries and the broader academic enterprise in a variety of ways.

It takes time to write and publish, and time spent on that is less time to spend on doing research or reading the research that others have published, research that might be useful in our jobs as well as our own research. You might remember the article in the Guardian late last year in which Nobel Prize-winning physicist Peter Higgs suggested that he’d be unlikely to get tenure in today’s academic climate because he hasn’t published enough. I try to stay current on what’s being published in a handful of library journals, but like many of us my interests are interdisciplinary and there is no way I can read even a fraction of what’s relevant to my scholarly interests. And the more that’s published, the more difficult it can become to find the good stuff — something we see when we teach students to evaluate sources, but something that can stymie more experienced researchers as well.

There’s also a direct connection between the ever-increasing publication for tenure needs and academic library budgets. Those articles need to go somewhere, and journal publishers have been more than willing to create new journals to fill up with reports of academic research and sell back to libraries. Publishing in open access journals can help, as others including Barbara Fister have suggested.

But I think academic librarians with tenure can make an impact on the quality versus quantity problem, both in the library literature and in scholarly communication more widely. I’m coming up for tenure in the fall, and while I’ve published my research open access, it’s also true that I’ve submitted most of my work for publication in peer reviewed journals, primarily because that’s what “counts” most. I don’t know that I’ve written anything in the past 6 years that I wouldn’t have otherwise, but as Meredith and others noted in the Twitter conversation, without worries about what counts I probably wouldn’t have felt as much pressure to write as much as I have for peer reviewed journals, and might have spread my efforts more evenly between blogging or other forms of publication as well. I’ve also felt torn spending time on other work that I know isn’t as highly regarded as traditional scholarly publishing — work like conference organizing and article reviewing and blogging, for example.

I’m looking forward to coming up for tenure in part because I’d like to help work toward expanding the definition of scholarly productivity to include alternatives to peer-reviewed publication in journals, and to focus on quality over quantity. Some of this is work that librarians are already doing — work in promoting open access, for example, among faculty in other departments who may not realize that there are peer-reviewed, highly-regarded OA journals. As academic librarians we have a view of the scholarly publishing landscape that other faculty may not share, and I hope we can use this position to advocate for tenure requirements that take into account more of the possibilities for contributing to the creation and propagation of knowledge than peer review and impact factor alone.

Publishing!?

Scholarship and publishing: both the blessing and the curse of a tenure-track academic position. Of the three requirements for tenure and promotion, scholarship seems to be the most stressful requirement for many tenure-track librarians to meet. It is a frequent topic of conversation among younger faculty, reflective of the stress associated with this requirement. I feel that I’ve at least been able to keep up with this requirement (though others are far more active than I am) and so I’d like to share some of the things that have worked well for me. Furthermore, I want to follow my three points with some things I struggle with and am working on as a young librarian, as well as include a call to hear advice from readers.

First, write about that which interests you. There is so much literature in the library and information science profession that is just not of interest to many people. Some of the best and most interesting writing comes from authors who are interested in and excited by their work – be one of those writers! Perhaps it’s some topic which is interesting, or an innovation or novel process that you are implementing at your library – whatever it is, write about it and find a good journal for it. Chances are that you enjoy your career as a librarian, but have a wide variety of other interests (we are a profession with broad passions and curiosities). Where do your professional and personal interests align? Last month, I attended a panel at which Jessica Pigza spoke about her book BiblioCraft – which is an example of what great things can happen when you find the intersections between your professional and personal interests – and then write about it!

Second, though being sole author of articles is important for tenure and promotion, equally important to me has been collaboration on articles and other peer reviewed work. I feel very fortunate to have worked with some great folks on publications and presentations – and that collaboration made the finished product far stronger. Much of the work I’ve done collaboratively really could not have been done alone. Thinking about my first point, being passionate about what you write, it occurs to me that you might have some great, big idea piece you’d like to write. Part of it is very much in your area of expertise, but much of it is not. This is a perfect opportunity for collaboration. Invite the person that can speak to the areas where you feel less knowledgeable to be a co-author. I’ve met some great people this way, and have vastly expanded my own knowledge. It’s also a great way for a new professional to get one’s name on a wider stage.

Third, managing my writing and publication has been key for me. It’s important to always be writing, and to show progress, so that at the end of the tenure clock you aren’t trying to write a multitude of articles and hoping they will be accepted and published. For me, this means I am trying always to be active on three tasks: waiting on review of articles I have submitted, actively writing an article, and developing ideas and collaborations. Having some kind of sequencing like that is helpful, but presentations and articles don’t write themselves, and don’t appear magically out of the air. Perhaps sharing how I write and craft new ideas will be helpful to you, reader, and will also prompt you to share ideas with me (and the wider audience) that will improve our writing processes.

For me, writing begins with a very rough idea. It might come from something I read, an art work, a presentation, or even a movie. Always being open to blending things in and outside libraries has really expanded the pale of what I write about. Drawing parallels between libraries, and say, the work of Wes Anderson (I’d love to read that article) for example. When I have an idea, though, I need to write it down before it vanishes from my mind. Next for me is refining that idea by talking informally with knowledgeable people in and outside of the library. What things are interesting to people? What things work? Listen, and be willing to adjust your original idea – or abandon it altogether. When I feel like I have an idea a bit more refined, I am usually very excited to start writing – and that’s exactly what I do. Riding those waves of inspiration and excitement gets the majority of my first draft finished – but deadlines for draft submission (and tenure requirements) help. Blocking out time when you are writing is essential to me (as is having a clean workspace). When the writing is finished, my first draft is typically awful – and I try to step away from it. Give it to some people you trust to look at – for me revision is key in refining my flow and points.  I try to listen earnestly to the feedback, and swallow my pride and address the comments – even if that means a very thorough revision. Repeat the process of review and revision a few times, and something approaching a finished article or presentation is the result.

As I mentioned above, it’s not all quite dancing and happy times when I am writing (and I am certainly no Gene Kelly). My first point above was about finding where your interests overlap, which seems fairly simple prima facie. However, finding the area of overlap between your personal interests and the expectations for areas to publish about in scholarly journals is a bit more difficult for me. It’s especially hard for me – blending the tenure expectations of my job with writing for my PhD and then finding where that small area overlaps with my personal interests is very difficult. I would be really interested to hear what ideas you readers might have about that, and more broadly where, and how, you find positive overlap for scholarship.

Beyond finding that intersection of personal and professional interests, follow-through is also a problem for me. I have a hard time after the initial blush of interest starting to write. How do you all bridge the gap from idea to actually writing?

Finally, I frequently have ideas for scholarship that are broad (and inspired by people that inspired me) that I need to invite collaborators. It can be hard for me to swallow my pride and ask people who I deeply respect (and am a bit in awe of) to work with me. Do you all have any strategies on asking people to collaborate, and doing that collaboration in the best way?

I’d like to conclude with an invitation to you, reader, to share what works well for you (and what doesn’t) in the scholarship arena! Perhaps together we can reduce the stress we all feel about this area of tenure and promotion!

The Varied Life of an Academic Librarian

Earlier this week, I represented the UI Libraries and Learning Commons at an Orientation Information Fair. It’s your standard “tabling” event where we put out a variety of handouts and other materials (one might say “swag”…pens, post-its, buttons, even squishy brain-shaped stress-relief balls) and chat with incoming freshman and their families.

In the lulls between waves of people coming in, I got to talking with the volunteer at the Student Legal Services table, which is next to ours. We compared swag and strategies for engaging the orientation attendees, and more. She’s a student assistant going into her senior year, and wants to go to law school and specialize in immigration law. I told her that I have a Master’s degree in Library and Information Science, but that unlike her, I found myself in grad school after having absolutely no plan of action following my college graduation. She actually seemed pretty interested in the “library talk” parts of our conversation, and asked what it was like being a librarian.

The answer to that, of course, is that every day is different. Sometimes I’m teaching, or alone at my desk (often doing the planning for teaching), in meetings or collaborating with colleagues in the library, at an event or training session where I get to learn from people outside the library, or attending events like the Orientation Information Fair we were at that day. I’d say the variety is definitely something I like the most about this career. Here’s a quick snapshot of the various projects I’ve got going on at the moment:

  • I’m collaborating with our Learning Commons Coordinator and others to develop and manage a digital badges program which will pilot in the fall. There was a great session at ALA that helped me think through our process a little differently – if you are thinking about digital badges, here are some resources to check out from that session. Our goal at UI is to encourage and reward engagement with the Learning Commons and library resources.
  • This fall semester I’ll be teaching an online section of a one credit-hour course offered through the Libraries, called “Library Research in Context: Find the Good Stuff Fast.” I’m pretty excited about this but also a little nervous about the time commitment, since I’ve never taught a semester-long course before. I’m a bit behind on planning, but fortunately have several experienced librarian colleagues as resources and support.
  • Another new project coming up for me is that I’ll be coordinating our Personal Librarian Program, where librarians are matched with Living Learning Communities. I’m taking this on in the absence of a co-worker who is leaving soon for another position, and am currently getting introduced to her contacts. While I don’t expect the actual coordinating to be a huge time commitment, continuing the libraries’ relationship with campus partners and communicating with people in Residence Education is a great way to stay in touch with what’s happening on campus outside the library and get other perspectives on student success.
  • All new international students who are undergraduates are required to take an online course that serves as a continuing orientation to the university. A colleague and I reached out to the instructor and recently completed a mini-module on basic library information for this course. Previously, none of the modules in the course covered (or mentioned) anything about the libraries. This is part of the libraries’ ongoing efforts to improve services and resources for international students.
  • One last thing for now – I’m chairing a committee for ALA’s New Members Round Table. Although I was a member of the same committee last year, it’s my first experience chairing a committee and already I definitely feel the big difference in responsibility from “member” to “chair.”

I’ve heard of “summer slowdown,” and although it felt much slower and quieter for the first few weeks after classes let out, that is definitely not the case anymore! That seems to be true for others, as well. If anyone has similar projects going on, it would be great to hear about it and share ideas!

Something I’m thinking about right now, is that with so much variety and flexibility in my day to day, I haven’t really taken the time to think about what parts of my job I really enjoy the most. What kinds of things do I want to spend my time on, now and into the future? The legal studies student I talked with was so passionate and knew the kind of law she wanted to go into, because it fascinated her and she cared about it. Right now, I don’t think I have that clear of a vision for any particular area – rather, I’m happy to do whatever needs to be done and help the best that I can wherever it’s needed.

Now, I know our situations aren’t exactly parallel, and I do like doing a little bit of everything, but it would be an advantage to figure out where I’m more personally motivated within librarianship. What do I enjoy the most? What are my strengths? How can I leverage those two? Within this wide variety of projects going on, I’m going to try to start paying attention to what I enjoy doing the most in my job.

Professional Conference Lurker No More!

Hello there. My name is Chloe. Long time conference lurker, first time participant.

In the language of the internet, a ‘lurker’ is someone who observes online forums or communities without actively participating.  This is the way I have approached conferences until recently…hovering at the fringes, without much direction or purpose.

In June, I attended the Canadian Learning Commons Conference in Sherbrooke, Quebec Canada.  CLCC is a relatively small conference, attended by US and Canadian delegates who work in the specific niche of Learning Commons (or, in our case Research Commons) library spaces. Attendees are not only librarians, but also writing center directors, IT help desk coordinators, and space designers.  The smaller scale and specific focus of this conference allowed my boss (Research Commons Librarian, Lauren Ray) and I to dial in on some very specific aspects of our service model for a presentation that we delivered, and to get some very granular advice about best practices from our colleagues.

Large, student-created statue, seen in the Library at Bishop's University (our conference sponsor).
Large, student-created statue, seen in the Library at Bishop’s University (our conference sponsor)

The last time I participated in planning and delivering content for a conference, I was still an MLIS student.  But It’s really nice to feel that I have something to offer in terms of professional practice, rather than student research alone.  Another difference is that, since I am not currently job-seeking, I could allow my interactions with the other delegates to be more relaxed and natural, rather than tinged with desperation.  It was nice to know that I might have something to offer THEM (like a valuable contact, or idea for a best practice) rather than just the other way around.

With that in mind, I feel like my conference impressions bear some special weight this time around, as I was in a much more receptive state of mind to receive them.  Here are a few selections:

Pre-Conference:

I got very lucky here, because the pre-conference was directly relevant to my professional duties. The topic was “Training and Mentoring Peer Learning Assistants, Peer Tutors and Learning Commons Student Assistants,” presented by Nathalie Soini and Caleigh Minshall from Queen’s University in Kingston, Ontario.  The presenters gave a lot of practical advice as to how to foster engagement in our student workers.  The session gave me lots of ideas, and was a good reminder of what an important job student workers have to do, and that we literally cannot function without them.

Our Presentation:

Overall, I think that Lauren and I did a great job with our presentation. Again, it was nice that our audience already understood the Research Commons concept, so that we could get right to the meat of our presentation without too much exposition.  We carried the 45 minutes we were allocated fairly well, and received positive audience feedback. In preparing the presentation, I really came to understand the value of Lauren’s mentorship. She has given lots of conference talks, and has a very structured approach.  While I am certainly capable of organizing 45 minutes worth of thoughts into a coherent presentation, Lauren’s sense of time management around the project was invaluable, as was her commitment to making the final product polished and clear. Before the conference, we were required to submit an abstract for our presentation.  We worked hard to refine this, and it expressed what we wanted to say pretty concisely. One important thing that Lauren reminded me to do, was to look frequently (whenever we added new slides, or ad-libbed new language as we practiced the presentation) back at the abstract we had written, to make sure that we were staying on track. It would be very disappointing for the audience, we reasoned,  if they made a decision to forgo a concurrent talk and attend ours, only to find that our presentation was only loosely related to what we had promised in the abstract (and who hasn’t been to a conference session like that, frankly.)

Other Presentations:

I attended a wide variety of other presentations over the course of the three day conference.  One highlight was a keynote by David Woodbury from the Hunt Library at North Carolina State University. NCSU Libraries are really innovative, and it is was great to get some ideas from their practices.

Another nice thing about this conference…probably due to its size and supportive character, was that a few presenters gave talks that included detailed information about “failures,” challenges, and things that had generally Not Gone Well at their libraries. While it requires bravery to deliver this sort of a presentation, it was so much more valuable for the audience to hear them!

For the curious, all of the presentation abstracts and many slides (including ours) from the conference can be viewed here.