The international Alliance of Digital Humanities Organizations’ Digital Humanities conference can be quite different from other academic conferences. Each field and each geographical area has its own conventions for how papers are submitted, accepted, and presented at their conferences. As a field comprising practitioners from a range of disciplines and from all around the world, the DH conference draws from humanities, social sciences, computer sciences, and information sciences.
In many North American humanities disciplines, papers are not peer reviewed by a pool of reviewers; abstracts are, instead, vetted by a program committee who determines what should be presented at the conference. In this model, abstract lengths are often short (sometimes only 300-500 words) and participants do not get any direct feedback on their submission except to learn if it has been accepted or not. For many computer science conferences, deadline extensions are not the norm, whereas humanities scholars can be accustomed to these extensions.
Continuing our efforts to make the DH2020 conference planning process more transparent, we would like to share with you an overview of how the peer review assignment process work. For DH2020, papers are vetted by a pool of reviewers as well as a program committee.
Assembling the Reviewer Pool
Peer review assignments begin with generating a list of qualified peer reviews for the conference. Names are generated through three mechanisms laid out in the Conference Protocol:
- A list maintained by the ADHO Conference Coordinating Committee of individuals who reviewed for a previous Digital Humanities Conferences
- An open call soliciting members of the community to nominate themselves or someone else as a potential reviewer
- Names suggested by members of the Program Committee from their digital humanities networks.
Each reviewer must meet at least two of the following four criteria:
- Be currently associated directly with or have been associated professionally with an institutional DH program, project, or initiative within the last three years.
- Hold at least a Master’s degree or equivalent in a relevant discipline (including but not limited to Humanities, Science(s), Library Science, Fine Arts).
- Have presented at an ADHO-, CO- or SIG-organized conference, symposium, colloquium, or equivalent. [CO is constituent organizations, such as the Canadian Society for Digital Humanities. [[SIGs]] are Special Interest Groups–you might consider joining one in your field!]
- Have published a DH-specific article or essay in a scholarly journal or volume (either print or electronic).
The Chairs of the Program Committee verified that open call, program committee nominations, and individuals who created Conftool account met the criteria.This involved culling publicly available information including personal websites, digital humanities conference programs, digital humanities journals, etc. All individual names were forwarded to the Chair of the Conference Coordinating Committee for vetting by the ADHO CCC and Executive Board. From this vetting process, the reviewer designation was applied to individuals deemed eligible by ADHO. (We’ll be writing in the future about how these criteria shape particular aspects of the conference, but for now, just know that it would assist future program committees to have this type of information available publicly about yourself.) All qualified reviewers were invited to join the DH2020 Conftool as a reviewer.
Reviews themselves
Once we received the names of approved reviewers and the submission window closes, our next task was to prompt individuals who’ve agreed to serve as reviewers to complete their list of priority topics. Priority topics are intended to identify what languages, geographies, temporal periods, methods, and disciplines/fields of study best describe the submission and are used by reviewers to signal their areas of expertise.
How Many?
The first step of assigning reviews is to determine the optimal number of reviews needed per submission. ADHO requires a minimum of three on every submission; however, with recusals due to conflicts of interests and/or reviewers who fail to complete their assignments, our first task is to ballpark how many reviews we need completed per submission. We heard from our colleagues handling DH2019 that there were significant numbers of reviews not completed. We determined to assign 6 reviewers per submission to account for this attribution.
Languages?
The underlying Conftool algorithm matches the areas ascribed to the submission with the areas identified by the reviewer. The problem though for Digital Humanities Conferences is that we handle submissions in multiple languages and Conftool cannot weight topical selections. So our second step is to manually assign any submissions that are non-English. For DH2020, this meant we went in and manually assigned roughly three dozen non-English submissions. ADHO has five official languages in which participants can submit: English, Spanish, French, German, and Italian; for the 2020 conference, we added the option of Anishinabeg (Algonquin) to reflect the conference’s location in Ottawa.
Manually assigning reviews involved looking at the percentage match between the submission and reviewer cross-indexing for ability to review in that particular language. A couple of hours later and voila, the non-English submissions have their assigned reviewers.
The Algorithm
Once we’ve cleared and assigned language-driven submissions we can then leverage the Conftool Algorithm to automatically assign the rest of the reviewers to the appropriate submission. If all reviewers completed their priority topics then the system works perfectly. However, because for DH2020 we were using a new set of ontologies and accounts, roughly 65-70% of the reviewers completed their priority selection. We sent reminders and picked up another couple dozen reviewer selections of priority topics.
Step three involved running the automated assignment process and then spot checking to ensure that it was working. Those who selected priority topics saw four to six submissions being assigned to them; this was because the algorithm had more preferences to work from. Importantly, the first process of the automation is to identify conflict of interests based on email addresses and institutional affiliations. It works well when people use institutional email addresses; It doesn’t work well with generic mailing programs like gmail or hotmail as it can’t distinguish between the potential contact of someone in the same institution versus the millions using gmail.
Those who didn’t select priorities were assigned just two submissions per person. In some cases, this meant that individuals were assigned to topical areas that they have no experience in. As those refusals to review come in, we reassign them manually using priority topics.
Monitoring
Similarly, as reviewers identified conflicts of interest that Conftool wasn’t able to account for, we reassign new reviewers to that submission. We increased the potential ways to signal a conflict because of our use of the open review process. This year, you can signal that the review would negatively impact one’s career or personal life. We ask no questions of that conflict; it is immediately reassigned.
Daily, the PC co-Chairs read every review as it is completed and answer every question that comes to the PC2020@adho.org email address. This allows us to flag potential submissions and reviews that may need additional attention. It also allows us to quickly turn over manual reassignments. The goal of the program chairs is to ensure that the reviews abide by the code of conduct and are productive for authors.
What’s the Program Committee doing?
DH2020 has a twenty person [[program committee]] made up of 2 representatives per ADHO association. These are individuals who’ve agreed to devote extensive time and energy to ensuring the success of the conference.
As part of this service, the program committee has been split up into groups which each group being assigned to read every single submission in that track. This allows them to not only look at all reviews of a given submission but also to get a sense of the track itself. They can then make recommendations about which submissions should be accepted and which need further work.
Author Responses
After DH2020 submitters receive their feedback, there will be a window where authors can write responses and reviewers will be able to change their evaluations of proposals. These are brief dialogues that let authors respond to reviewer concerns. It also allows the authors to give the program committee additional information that they should consider as they weigh admission to the conference.
Program Committee Discussions and Decisions
The final decisions about the DH2020 program are made by the program committee in consultation with the conference coordinating committee as needed. We look at all reviews including the one completed by a member of the program committee. We also read the author’s response and notes to the program committee.
What can you do?
There are many ways to be involved in making decisions about upcoming digital humanities conferences:
- Serving on the conference coordinating committee
- Serving on the program committee (contact your ADHO constituent organization to see how they choose their program committee member(s))
- Serving on the local organizing committee (or proposing to host a future conference as local organizers)
- Joining and organizing ADHO SIGs (Special Interest Groups), which create programming at each ADHO conference
- Submitting proposals
- Reviewing proposals
The conference is also supported by the ADHO multilingualism and multiculturalism committee, the Constituent Organization Board, and Executive Board.
DH2020 is not your average conference–but we have teams of people working to make it above average. Hope to see you in Ottawa!