Developing an Evidence Synthesis Training Program

Last May, when I was preparing for my in-person interview for my current position, I was lucky to have the opportunity to talk with librarians at my graduate institution about questions I might be asked and topics that might come up during my interview. One of the main themes that came up in those preparation conversations was evidence synthesis, sometimes also known as knowledge synthesis.

If, like me a year ago, you’ve never heard that term before, it refers to systematically searching existing literature to find evidence and draw conclusions to answer a specific question. There are several types of evidence synthesis – some of the most common are systematic reviews, meta-analyses, and scoping reviews. This type of research started in the medical fields and has branched out from there into other areas, like education.

The librarians I talked to were right, and evidence synthesis was mentioned in my interview. I had done my homework and was able to speak to it enough, but one thing that had been clear from the research I did is that there would be a lot more for me to learn about this topic.

Teaching Evidence Synthesis (To My Fellow Librarians)

Fast forward to January 2026, and my library director sat down with me to talk about evidence synthesis work. One librarian on our team had been doing this work for several years and is very well-versed in it, but the rest of us were much newer to it and had varying degrees of knowledge. My library director explained that she wanted to us to be able to tell our departments that we could help with evidence synthesis projects starting in the Fall 2026 semester. That meant that we had about six or seven months to learn enough to feel comfortable guiding our students and faculty through the process. My director asked me to take on the project of developing a training program to teach all of us about evidence synthesis.

At that point, I had two main sources of knowledge about evidence synthesis: I had taken the University of Minnesota Libraries’ Evidence Synthesis Institute, which had been very helpful but was also, honestly, overwhelming; and I had joined a review project with some other librarians at Vanderbilt. While the knowledge from those things helped, I knew I was only scratching the surface of this topic – so essentially, I needed to come up with a way to teach my fellow librarians, along with myself, how to do evidence synthesis.

Six months later, I think our training program has been an overall success, although the main thing I’ve learned might be that there will always be more to learn about evidence synthesis. I don’t think our training program will ever be truly over – it may just be less intense in the coming months. But as we start to wrap up the first half of this year and look towards the fall semester, here are a few key takeaways that I think helped to make the training program work.

Get Feedback and Plan Ahead

Before I did anything else, I sent around an anonymous survey to the librarians at my library to gauge how much they already knew about evidence synthesis and how comfortable they thought they would be if a student walked into their office at that moment and asked for help with a systematic review. Unsurprisingly, our answers were varied, but a common theme that emerged was that while most of us had some knowledge of evidence synthesis, we were less confident about being able to put that knowledge into practice.

As a result, I set up our training program to be partially learning and information intake, partially exchanges with other librarians (more on that a few sections down), and partially hands-on work. In the external trainings and conversations about evidence synthesis that I’ve attended since January, being unsure of how to put what you’ve learned into practice has also been a common theme for librarians. If I hadn’t gotten this feedback at the beginning and planned out all of our biweekly meetings to make sure we had practice time in July, I might not have thought to include time for actual hands-on, practice learning.

Document and Save Everything

Evidence synthesis and systematic reviews are complicated forms of research. I knew that it was very possible our group of librarians could have a conversation about something, and then a year later, I might remember that conversation and want to go back and look at exactly what we discussed. As a result, from the very beginning of the training program, I made sure to create agendas and schedules in advance, record our meetings and discussions (with my colleagues’ knowledge and consent), and take meticulous notes during each training program interaction. All of those things are saved in an organized folder that’s shared with our team, and after every meeting, I re-share the folder with team members to ensure they have access to everything they might need.

Ask for Support When Needed

From the beginning, I knew that I wouldn’t be able to teach my colleagues all of this information on my own. As a result, I leaned on the experience of people around me to help when I needed to. I asked the librarian on our team who has the most experience with systematic reviews to lead a few of our sessions. He talked us through how to use Covidence and how to design a search string, which are both invaluable parts of evidence synthesis. I also asked the librarians at Vanderbilt’s Biomedical Library, several of whom have been doing evidence synthesis for 5 or 10 years, to do a Q&A session with us about their experiences and advice. That Q&A was also one of the most helpful things we did, and I’m so grateful to the Biomedical Library team for meeting with us. Finally, I also looked for external opportunities that we could use to enhance our training. In mid-June, most of us participated in the online Library Evidence Synthesis Services Symposium (LESSS), and gained a lot of valuable knowledge from those sessions.

I also want to add that doing this type of training program wouldn’t have been possible without the direction that I’ve received from my library director and the support I have from my colleagues. If my fellow librarians weren’t willing to show up, be dedicated, and learn a lot, our training program wouldn’t have made it past the first few weeks.

In July, I’ll re-survey my colleagues and figure out how we’re all currently feeling about evidence synthesis, what we think we can put into place for the fall, and where we need to continue learning. This will definitely be an ongoing learning situation – a “yes, and” training program instead of a checked box saying that our learning has been completed. But even though I haven’t officially asked yet, I feel good about saying that we all know a lot more than we did when the training program started at the beginning of this year. I look forward to what the next steps will be in all of us continuing to learn and grow with evidence synthesis.

Questioning the Evidence-Based Pyramid

As a first year health sciences librarian, I have not yet conducted a systematic review. However, as a speech-language pathologist, I learned about evidence-based medicine and the importance of clinical expertise combined with clinical evidence and patient values. As a librarian, I’m now able to combine these experiences, allowing me to view see evidence-based medicine more holistically.

In the past month, I attended two professional development courses. The first was a Systematic Review Workshop held by the University of Pittsburgh. The second was an Edward Tufte course titled “Presenting Data and Information”. While these are two seemingly unrelated subjects, I left both reconsidering how we literally and figuratively view evidence-based medicine.

One of my biggest takeaways from the Systematic Review workshop was that a purpose of  systematic reviews is to search for evidence on a specific topic in order limit bias. This is done by searching multiple databases, reviewing grey literature, and having multiple team members  to screen papers and resolve disputes. One of my biggest takeaways from the Tufte course was that space should be used well to effectively arrange information and that displayed content should have integrity. In his book Visual Explanations, Tufte poses the following questions to test the integrity of information design (p. 70):

  • Is the display revealing the truth?
  • Is the representation accurate?
  • Are the data carefully documented?
  • Do the methods of display avoid spurious readings of the data?
  • Are appropriate comparisons and contexts shown?

When I think about visualization of evidence-based medicine, the evidence-based pyramid immediately comes to mind. It is an image used in many presentations related to evidence-based medicine:

EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.

While there is a lot of information in this image, I don’t think it is very clear. I have spoken to librarians (in the health sciences and not in the health sciences) that agree. I think this is a problem. I don’t think all librarians need to immediately know what cohort studies are, but I do think they should understand its context within the visual.

From what I have gathered and discussed with other professionals, quality of evidence/limited bias increases as you go up the pyramid. The pyramid is often explained in a hierarchical way; systematic reviews are considered highest standard of evidence, which is why it is at the top. There are usually fewer systematic reviews (since they take a long time and gather all the available literature about one topic), so the apex also indicates the least quantity. So let’s take a look each of the integrity questions about information design and investigate this further:

Is the display revealing the truth?

Is it? How do we know if this truthfully represent the quantity of each type of study/information? I believe that systematic reviews are probably the least in quantity and expert opinion are the most in quantity. That makes logical sense given the level of difficulty to produce and disperse this type of information. However, what about the types of research in between? Also, is one type of evidence inherently less biased than the ones below? Several studies suggest that systematic reviews may be systematic, but are not always transparent or completely reported and are outdated. This includes systematic reviews published in Cochrane, the highest standard of systematic reviews. While there are standards, they are very frequently not followed. However, following these standards can be very challenging and paradoxical. It’s very possible that a cohort study can be designed in a way that is much more systematic and informed than even a systematic review.

Is the representation accurate?

When I see the word “representation”, I am thinking about visual representation – the pyramid shape itself. There is an assumed hierarchy not just in terms of evidence, but also superiority here. This is a simplistic and elitist way of thinking about this information rather than being informative and useful. If you think about it, a systematic review cannot be conducted without having supporting RCT’s or case reports, etc. Research had to start somewhere. It this was seen as more of a scholarly conversation, I wonder if there would be a place for hierarchy.

I have learned that the slices of the pyramid represent the quantity of publications of each level of evidence. However, this is not something that can be easily understood by looking at this visual alone. Also, if the sizes of the slices represent quantity, why so? Quality is indicated in this version with the arrow going up the pyramid. This helps to represent idea of quality and quantity. However, if evidence-based medicine wants to prioritize quality, maybe the sizes of the slices should represent the quality, not quantity, of evidence. If it is viewed from that perspective, the systematic review slice should be the biggest because it is ideally the highest quality. Or, should the slices represent the amount of bias? This is all quite unclear.

Are the data carefully documented? Do the methods of display avoid spurious readings of the data?

I don’t believe that any data is actually represented here. Moreso, it feels like it’s being told to us so we believe it. I understand this is a visual model, but this image has been floating around so much that it is taken as the truth. I don’t think one can avoid spurious readings of the data because data aren’t represented here.

Are appropriate comparisons and contexts shown?

I do think that this pyramid provides visual way to compare information, however, I don’t think contexts are shown. Again, should the amount of each level of evidence referring quantity or quality? Is the context meant to indicate research superiority? If not, perhaps a pyramid isn’t the best shape. By virtue of its definition, a pyramid has an apex at the top, indicating superiority. Maybe a different shape or representation can provide alternate contexts.

So, how should evidence-based medicine be represented?

I have presented my own perceptions sprinkled with perceptions from others. I’m a new librarian, and my opinion has value. However, I also think this concept needs to be re-envisioned collectively with healthcare practitioners, researchers, librarians, and patients.

Another visualization that has been proposed is the Health Care Literature Wedge. It would look like  a triangle with the apex facing right indicating progressive research stages. I do think there are other shapes or concepts to consider. Perhaps concentric circles? Perhaps this can be a sort of spectrum? 3D maybe? I really don’t know. Another concept to consider is that systematic reviews are intended to reduce bias pertaining to a research question. Instead of reducing bias, maybe we can look at systematic reviews as having increased perspectives? How could this change the way evidence-based medicine is visualized?

I think the questions posed by Tufte can help to guide this. And I’m sure there are other questions and models than can also help. I would love to hear other epistemologies and/or models, so please share!

References

  1. Chang, S. M., Bass, E. B., Berkman, N., Carey, T. S., Kane, R. L., Lau, J., & Ratichek, S. (2013). Challenges in implementing The Institute of Medicine systematic review standards. Systematic Reviews, 2, 69. http://doi.org/10.1186/2046-4053-2-69
  2. Garritty, C., Tsertsvadze, A., Tricco, A. C., Sampson, M., & Moher, D. (2010). Updating Systematic Reviews: An International Survey. PLoS ONE, 5(4), e9914. http://doi.org/10.1371/journal.pone.0009914
  3. IOM (Institute of Medicine). (2011). Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press.) Retrieved from http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx
  4. McKibbon, K. A. (1998). Evidence-based practice. Bulletin of the Medical Library Association, 86(3), 396–401.
  5. The PLoS Medicine Editors. (2007). Many Reviews Are Systematic but Some Are More Transparent and Completely Reported than Others. PLoS Medicine, 4(3), e147. http://doi.org/10.1371/journal.pmed.0040147
  6. Tufte, E. R. (1997). Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, CT: Graphics Press.