ACH Guidelines for Assessment of Digital Scholarship in Tenure and Promotion

This document was prepared by the Executive Board of the ACH during the Fall and Spring of 2015-16. Feedback from the community was solicited in the summer of 2016, and incorporated over the following months.

Introduction

Following the lead of scholarly societies in some of the disciplines most actively represented in digital humanities work, the Association for Computers and the Humanities is putting forth some draft guidelines for the evaluation of digital scholarship in tenure and promotion processes. We are grateful to the Modern Language Association, the American Historical Association, Emory University, University of Southern California, Texas A&M, University of Nebraska-Lincoln, the NINES Institute on Evaluating Digital Scholarship, and our many colleagues for doing the work on which we base our guidelines.

General

• Expectations about any individual scholar’s tenure and promotion criteria should, as much as possible, be made clear at the time of hiring, reviewed at intermediate stages, and reflect a sincere commitment on the part of both the scholar and the institution to assess the scholarly contribution fairly.
• Scholarship should be considered for its contribution to the scholar’s discipline, and, where relevant, additional fields including the field of digital humanities.
• In addition to traditional scholarly work, digital platforms, tools, and other resources may be considered scholarly products with potential to contribute to the field of inquiry. They should be reviewed on the basis of their originality, use value, sustainability, and other contextual aspects of quality outlined below.
• Digital contributions might constitute all or part of a project (e.g. metadata might make a scholarly contribution) if it adds to knowledge in the discipline.

Assessment process

• Materials must be reviewed in their native format and medium so that the contribution is assessed for its merits as a digital project. Explanations of the technical or design features can accompany the project to assist reviewers unfamiliar with digital work, but these should not substitute for the project in its native format.
• The following aspects of digital scholarly products should be reviewed independently of each other by knowledgeable parties:
  • technical implementation details
  • design
  • content
  • data (content and design)
  • sustainability/projected lifespan
  • intellectual argument and impact to the home discipline or the field of digital humanities

Some metrics for assessing tools, projects, and platforms

 Digital projects take a variety of forms. These include the production and/or publication of data sets, results of analysis, inquiry into existing corpora through digital means, archives, metadata, as well as the creation of tools and platforms.

Overall Evaluation       

• Intellectual contribution and substantive research contribution in the humanities
• User experience as purposeful in ways that add value to the profession
• Evidence of platform/tool use by communities of practice 

Impact and dissemination

• Peer-Review of the project including internal or external funding, pre-publication review, and post-publication review
• Links to the project from other projects
• Citation in other projects, blogs, tweets, social media, reports, media coverage
• Presentation and/or discussion in conference presentations, published articles, and other vehicles of scholarly communication

Candidate responsibilities

• Ask for clarification of assessment criteria at the time of hiring
• Negotiate roles/responsibilities to distinguish research from service
• Make the scholarly contribution clear
  • When multi-authored works are presented for review (most digital tools, platforms, and other projects fall under this category), make the individual scholarly contribution clear
• Make the teaching contribution clear
  • Demonstrate clear goals and learning objectives for incorporating digital media, tools, activities into classroom use;
  • Demonstrate the value of courses focused on learning digital methods.
• Document the prestige rankings of online publication sites, in particular, the use of peer review and acceptance rates; distinguish self-published and/or repository publication from peer-reviewed publication;
• The relationship of design, content, and medium should be documented  
  • The intellectual contribution of each work should be explicitly presented
  • Authors of digital tools, platforms, and other projects should subject their work to peer review prior to tenure and promotion review, where possible
  • Long term viability of the project should be considered
  • Provide explanatory narratives of project development and value
  • Document technical competence as applicable:
    • Knowledge and use of technical standards (TEI, XML, GIS, statistical standards, etc.)
    • Solidity of database design (metadata standards, fields etc.)
    • Interoperability with existing resources (as appropriate)
    • Show how the project fulfills its goals by using the digital media
    • State where the project lives and who is responsible for its maintenance
  • Document collaboration: roles, responsibilities, and contributions for collaborative work
  • Document and explain the contribution of the work
  • Document grant successes, network building
  • Bring colleagues into the work early on
  • Document prizes and other recognition
  • Document user assessment and testing
  • Make target audience clear and demonstrate connections
  • Demonstrate long-term sustainability

Departmental responsibilities

• The expectation that digital tools, platforms, and other projects of sufficient depth are considered scholarship should be built into the hiring process as appropriate, but also, the recognition that digital projects can be constituted as data, metadata, archives, repositories, and scholarly materials in digital format;
• The relationship of design, content, and medium should be documented  
  • Request assessment letters from experts who have knowledge of digital projects as part of the dossier
  • Make the criteria for assessment explicit at hiring
  • Recognize the often on-going, iterative, or open-ended nature of digital projects and establish criteria for assessing work in progress
  • Consider relationship of design, content, and medium (why is this digital?)
  • Assess digital editions and some projects as “curation” (selection, organization, editing, and critique)
  • Consider how digital format contributes to the scholarship
  • Consider conference presentations on the project
  • Consider print publications about the project

As in all tenure and promotion reviews, candidates’ work should be assessed by scholars in relevant fields of study, not only in digital humanities.

Professional association responsibilities

• Create working groups for digital scholarship to keep current trends in tenure and promotion, peer review, and assessment up to date
• Foster conversations about digital scholarship
• Sustain a curated gallery of projects with assessment criteria
• Promote reviews of digital scholarship
• Sustain a curated list of digital scholars willing to serve as external evaluators