Meta-Analysis Tools: What I'm Using for My Dissertation
When it comes to research work, having the right tools can make a big difference in efficiency, communication, and just general quality of life for the researcher. For my dissertation I have been working on a large meta-analysis, which has many different parts. Over the past several months I have tried various applications and ways of collaborating, and have learned a lot in the process. Now that I've reached a point where things are working smoothly, I wanted to share the tools I have found most helpful (for meta-analyses and research more broadly). While doing so, I also want to highlight the workflow that I've found to work well for my meta-analysis.
Before jumping into the tools/applications, I just want to briefly summarize what I'm doing for my project and the general steps of a meta-analysis. My dissertation is looking at executive functioning (EF) in pediatric youth. While some others have looked at EF in specific groups (e.g., children with diabetes), I want to look at it more broadly. We have some early results that are interesting and we hope to share at SPPAC 2017.
So what is a meta-analysis? Meta-analyses take a group of published (and, ideally, unpublished) results from other studies and combine them into a dataset. Analyses are then conducted to see if, based on all of the other results, there are significant findings or not. They're not perfect, which is a whole separate topic, but they are a good way of summarizing a large collection of studies.
In order to conduct a meta-analysis, there is a series of general steps that needs to be followed. This is just a very broad list, but that's all we need for the purposes of this post:
- Literature search
- Collection of articles
- Screening of articles
- Entry of relevant data into a database
Again, the actual list of steps is much longer and more detailed. But rather than focus on how to conduct a meta-analysis, I just want to highlight applications/tools I've found helpful for each of the above steps.
(Side note: Some of the tools I discuss below are listed on my resources page, along with other tools I recommend outside of meta-analysis work.)
Literature Search: Google Scholar
Anybody in research/academia is likely familiar with Google Scholar. It's Google's search engine that's specific to scholarly materials (mainly published research articles, but also book chapters and other similar resources). It's helpful because it searches a lot of sources simultaneously, and can link to your institution's access page to download the articles.
For my project, it's the primary tool we're using to conduct the literature searches. However, it does have its drawbacks. First, I recently found out it doesn't handle boolean terms (e.g., AND, OR) in the way you'd likely expect. In general it seems to ignore them and lumps everything into the "with at least one of the words" box in the advanced search. That can make it hard to refine your searches.
Second, it can be hard to determine relevant similar searches. Some databases have tools to allow identifying similar keywords that are likely to return relevant articles. Without that, you could miss a group of articles that are very relevant to your work just by simply using a different keyword.
So Scholar isn't enough on its own, but it gets the majority of the work done. For my dissertation we've been using it as the primary search tool, followed by additional searches on official databases (and by looking through references, emailing prominent authors, etc.) just to be sure our searches are comprehensive.
Collection/Screening of Articles: Mendeley/EndNote
Once the articles have been found, they need to be collected, organized, and screened. Reference managers are applications designed for this very purpose. Overall, my favorite application to use is Mendeley. It's a free program (you pay for additional online backup/syncing storage if needed, but I haven't used up the free amount yet) that has desktop and mobile applications. It's actively being updated and is overall a great service (though it doesn't currently handle high DPI screens well on Windows; everything looks really small, but it sounds like they're working to fix that).
I've been using Mendeley for the general writing of my dissertation (it plugs into Microsoft Word, so it can handle the in-text citations and references section for me). However, I have not been using it for the actual management of the articles that are for the meta-analysis itself.
Instead, the program I've been using is EndNote. It's a relatively popular program that many in academia are likely familiar with. EndNote is paid (though I get it for free through my school), and I've found it to be relatively slow compared to Mendeley. However, there are two reasons I've been using it for managing the articles for the meta-analysis: 1) our research assistants are more familiar with it, and 2) it allows star ratings for articles (which, to my knowledge, is not included in Mendeley).
The first point isn't a big problem overall, as all reference managers work more-or-less the same way. But the star ratings have proven useful, as it allows an easy/visible way to indicate the status of each article. For example, 3 stars can mean it's ready for entry, 4 stars means it's entered once, and 5 stars means it has been verified for accuracy.
It's not perfect, but it has worked well for our purposes. I just download the articles to a folder, import that folder into EndNote, then move the imported articles within EndNote to the appropriate folder. There are folders set-up for eligibility status (i.e., All, Eligible, Ineligible) for each condition we're searching for. I've shared the library with the research assistants helping with the project, so everyone can see the articles and can update them, which is then synced to everyone else.
It's a really helpful system that makes organizing hundreds of articles relatively simple.
Database Entry: Microsoft Access/Sharepoint
Oh Access. Anyone who has used Access before probably has mixed feelings about it, and I'm no exception. Setting up an Access database can take a lot of time, and there are a lot of quirks you need to know about. However, once it's set-up it can be more user-friendly (and enable more consistency throughout data entry) for research assistants. For example, you can have drop-downs to select options, which then save the number code for the selection (e.g., Male = 1, Female = 2). That way nobody needs to memorize the number codes for every variable, but you get the numbers for analyses (which are easier to work with).
However, there are two limitations to using the standard desktop version of Access: 1) most people don't seem to have Access installed on their own computers (it's not included in most Office packages), and 2) it's difficult to work with remotely/collaboratively (it might work if the database is saved on something like Dropbox, but I didn't get around to testing that because of point 1).
To overcome those limitations, I have moved my dissertation to use the web version of Access. It is much more limited than the desktop version (you can't customize it nearly as much), and it requires a SharePoint account (which has a monthly fee, though it's relatively cheap). But the advantage is that anyone you share it with can pull it up in a modern web browser, without needing to have Access installed. (It does require the desktop version to create and edit the web version, but at least that's only one copy instead of everyone needing a copy).
It was a little tricky to set-up at first, but the web version's UI is fairly simple and straightforward. Now I can share the database with a research assistant, along with the EndNote library, and all searching/organization/screening/entry can happen remotely. That's true for me as well, so it makes it much easier to work whenever/wherever is most convenient for everyone involved (assuming internet access). It also makes simultaneous work a little easier and reduces the chances of conflicting file versions. After entry is done, you can just download the data files and open them using something like Excel.
Analyses: Comprehensive Meta-Analysis (CMA)
Once everything is all entered and verified, it's time to actually conduct the meta-analysis. I won't spend long on this section, as it's very specific to meta-analyses, but I want to make a few points. First, I chose the CMA program because it's created by the authors who wrote the book I found most helpful for understanding meta-analyses (the Introduction to Meta-Analysis book found here), making it easier to pick up and learn. Second, it's available from my school in a graduate student lab, so I didn't need to find any alternatives. It's a powerful program, but I didn't do much exploration into alternatives, so I'm not going to claim it's the best program out there.
With that in mind, it's definitely a quirky program. The only additional point I will mention is that anyone planning to use CMA should become familiar with it before starting data entry. Data needs to be copy/pasted into CMA (it doesn't really have an import option; I export from Access to an Excel file, then copy/paste from that). As part of that, you'll want to make sure your data matches the formatting CMA will expect so the transition will be as smooth as possible. I luckily did this early in the entry process, and definitely needed to tweak Access to make things fit more closely with CMA.
Write-Up: Microsoft Word
During my undergrad years, I tried to use free alternatives to the Office suite when I could. For writing basic assignment papers, Word is more powerful than what you need (though it works well for those purposes too). But as my writing has gotten longer and needed more complicated formatting (e.g., table of contents), I've learned to appreciate the tools within Word that help to automate a lot of the process.
I won't go through all of the tools (e.g., tab stops, tab leaders, table of contents, track changes). Instead, I just want to highlight that Word likely has automated formatting options for any formatting things you may need (e.g., lining things up, sections). For great videos demonstrating a lot of the most helpful tools, I recommend looking at Scott Hanselman's videos, including the one below:
Before moving on, I want to make a brief mention of Scrivener. This application has been praised by many writers, including academic writers. I've considered trying it, but haven't felt a need for it yet. For anyone doing long pieces of writing (e.g. lengthy book chapters, novels), it may prove to be helpful. Details about the program can be found here. In short, it allows breaking down writing to small portions of text that can then be reorganized and combined easily. It's both an organization and a writing tool.
All of the applications/tools I've mentioned thus far make up the "core" tools I would recommend for someone conducting a meta-analysis (or even just research more broadly). But there are additional tools I have found to be very helpful, even if they aren't required.
Task Tracking: Todoist
To keep track of to-dos, I have long been a fan of Todoist. It simply fits my workstyle best, and has a wonderful UI as well. I've mentioned it before when discussing how I recommend using to-do lists. There are obviously other great apps out there as well depending on your needs (e.g., Wunderlist). Whatever application/system you ultimately use, I highly recommend having a to-do list of some kind.
Project Tracking: Trello
While EndNote is allowing me to keep track of the status of individual articles, it doesn't do a great job of helping to track higher-level things (e.g., where each group of articles is within the meta-analysis process I listed at the top of this post). To do that, something like Trello works perfectly. I have a list of each step in the process, and I move the "cards" (which contain the names of each group) through the steps as they are completed. I can also add comments, due dates, and so forth to help keep track of those details. It's something that is hard to capture in a to-do list, as it's more of a tracking issue rather than a task list issue.
For about 8 years, since originally starting my undergraduate work, I struggled to find a good note-taking system. I always prefered hand-writing notes, but then I would eventually lose or recycle them, and I didn't always have them with me when I needed them. Eventually I scanned those notes, but it's pretty rare that I go through and try to find scanned ones.
Then, about a year ago, I decided to try OneNote. It's certainly not perfect, but it's been the best system I've found so far. There's enough flexibility to it that it can fairly easily replace hand-written notes (it even allows writing with a stylus, which I sometimes do to draw models). But at the same time it's more neatly organized, is synced and available more easily, and I can always scan in any paper notes which I can have embedded with my digital notes.
Others may find other services like Evernote to better fit their workflow, but I've been impressed with OneNote. Whatever the system, I think it's helpful to have a note-taking system to log meetings, save hand-drawn models, and so forth.
Backup: Google Drive/Dropbox
I can't stress this point enough: backup everything. Many other sites related to technology make this point as well, but it's something that is easy to forget. Computers crash, files go corrupt, and other bad things still happen (even as software gets more and more stable over time).
When it comes to research, I save new versions of files every day that I work on them (with the date appended to the end of the file name). That way I have local backups in case anything was accidentally deleted at some point, or a file became corrupt, or I wanted to review an older draft of my writing.
But separate from having multiple local versions of a file, it's important to save everything in at least two locations. That will include your computer, but should also include a cloud service (e.g., Google Drive, Dropbox, OneDrive) and/or an external HDD/SSD. I try to keep backups in all three (I use the Windows File History feature to keep backups on an HDD/SSD, and have my work files sync to a cloud service).
If you use a cloud service, just make sure it's secure. Use a unique and strong password, and enable two-factor authentication.
That's a general summary of the tools I have found to be most helpful for conducting my own meta-analysis. Again, there is a lot more to the meta-analysis process than I cover here, but those portions (e.g., contacting authors for unpublished results) don't really need their own applications/tools.
These tools are also not limited only to use for a meta-analysis, so hopefully the list is helpful for anyone who is conducting research. For some additional suggested applications/tools, you can look at my resources page.
Know of a good research application/tool I didn't mention? Have questions about what applications/tools would be helpful for a specific research-related task? Let me know in the comments!