QTD Process

Jump to:

The Qualitative Transparency Deliberations have proceeded through a three-stage process, guided by a Steering Committee appointed by the QMMR Section President, Peter Hall. The first stage of the process focused on identifying the dimensions along which we must differentiate when discussing research transparency. The second stage involved extended substantive deliberations over research transparency and related issues of research integrity, openness, or explicitness, differentiated along the dimensions identified during the first stage. The second stage has been driven forward by a set of working groups, constituted based on the input received during the first stage, each dedicated to considering issues of research transparency and explicitness as they arise for a particular form of research. Each working group has been comprised of a diverse group of political scientists with experience in the aspects of qualitative and multi-method research under discussion, and each group will invite input from, and consult with, a broad range of scholars.

Preliminary Stage: Appointment of the Steering Committee

The members of the Steering Committee, initially appointed for 1-year terms by the QMMR Section President in consultation with other section officers, between them have research experience using a diverse set of qualitative methods—such as process tracing, ethnography, interpretive methods, comparative historical analysis, interview-based research, archival work, and algorithmic forms of qualitative analysis. Individual members, however, do not serve as representatives for particular constituencies; instead, the Steering Committee endeavors to be broadly representative of QMMR scholarship. The Steering Committee was also to include at least one member who is junior (pre-tenure) and one who is teaching at an institution that is not a Ph.D.-granting research university. The Steering Committee seeks to make decisions by consensus but if necessary can make decisions with a vote of a majority of its members.

The Steering Committee's primary responsibilities are to:

  • structure and moderate the at-large consultations;
  • select, delineate, and coordinate the substantive foci of the Working Groups, based on the input received during the at-large consultation;
  • appoint Working Group members;
  • provide an initial set of common structuring questions and thus a common framework to help guide and coordinate Working Group deliberations;
  • oversee the timely progress of the Working Groups;
  • facilitate conversation, where helpful, across Working Groups on issues of common interest; and
  • provide feedback on interim drafts of Qualitative Transparency Statements developed by the Working Groups.

Stage 1: At-Large Consultations

During Stage 1, QMMR scholars were invited to participate in an at-large, online consultation over how to structure the substantive deliberations. While comments submitted during this stage could in principle address any aspect of research transparency in political science, the Steering Committee at this stage primarily sought comments about the forms and aspects of qualitative research that require separate substantive discussions. The categories used to structure the symposium about research transparency in the Spring 2015 issue of Qualitative and Multi-Method Research—types of evidence, forms of analysis, and social settings in which empirical engagement unfolds—served as initial categories for this consultation, but the process during this stage was open to any and all suggestions for structuring the substantive discussion. The Steering Committee also invited scholars at this stage to articulate dilemmas or challenges that qualitative researchers face in pursuing transparency and that warrant substantive discussion.

The QTD online platform, through which most of the at-large deliberations took place, was designed to allow anyone to submit a comment, and the Steering Committee actively encouraged participation by anyone interested in QMM research, regardless of section membership. The discussion boards, moreover, were set up to allow comments by registered users (logged in at the time of their posts) to appear on the QTD web portal without delay; anonymous posts were also possible for users who preferred not to have their identity displayed with the forum post; these posts appear online after a brief delay for moderator review but without edits, provided that comments observe norms of civility and professionalism. The QTD online platform was created with the help of the IT staff at Duke University's Social Science Research Institute, with support from the U.S. National Science Foundation under Political Science Program grant # 1644757; since summer 2017, the Hochschule für Politik/Bavarian School of Public Policy at the Technical University of Munich, Germany has provided additional IT maintenance support

Interim Stage: Selection of Working-Group Foci

After the conclusion of Stage 1, the Steering Committee reviewed the input received through the at-large consultations and, informed by these comments, selected the foci for the substantive deliberations. These foci were selected to reflect the different dimensions along which qualitative research projects vary. For each identified focus, the Steering Committee appointed a working group (WG) to initiate and moderate the online deliberations, as well as consult and deliberate with a broad range of scholars and amongst themselves about research transparency and explicitness as they relate to the kind or aspect of research that is the focus of each WG's work. In total, 13 WGs were formed in 4 clusters, with each WG comprised of 3-4 scholars who regularly engage in the kind of research that is the WG’s focus. In appointing the members of the WGs, the Steering Committee aimed to achieve diversity in scholars' professional circumstances, including career stage and type of institution.

The Steering Committee provided the working groups with an initial set of common guiding questions. These questions were not intended to be exhaustive of the issues that a WG might consider, but designed to ensure a degree of commonality and comparability across the deliberations, while respecting the WGs' differentiated mandates. Guiding questions, for instance, encouraged the WGs to identify the features of research (of the kind that a given WG is examining) about which it is most important to be explicit; to identify current prevailing or "best" practices; to point to possible ways in which current practices might be improved; and to identify important benefits, costs, risks, and tradeoffs confronted by researchers conducting this form of research in pursuing transparency or explicitness.

Initial questions to open each WG's deliberations were discussed and refined at workshop meeting of all the working groups on the day before the 2016 APSA annual meeting in Philadelphia, made possible by crucial support from the U.S. National Science Foundation under Political Science Program grant # 1644757.

Stage 2: Working Group-Based Differentiated Deliberations

In addressing the issues that they have been tasked with considering, the WGs have consulted with a wide variety of scholars in their research communities. For several months, the WGs gathered broad input from relevant research communities, focused on the guiding questions as well as other issues identified by each WG as warranting discussion.

Working group consultations with interested research communities primarily took two forms: open online deliberation via the QTD platform, similar in form to the consultations carried out during the first stage (but now on separate discussion boards for each Working Group); and the active solicitation of input from colleagues with experience in the use of the qualitative method or in dealing with the particular issues that are the focus of a particular working group. All scholars were invited to participate in substantive discussions of research transparency and explicitness within differentiated research communities.

Throughout the deliberations, Working Groups reviewed the input received. Working remotely, they began to identify areas of agreement, areas of disagreement, key considerations, tradeoffs, and open questions that surfaced during their consultations. The Steering Committee and each working groups (separately) also continued to meet virtually to identify and discuss areas of agreement, areas of tension, and common themes emerging within and across WG deliberations.

Initial Output: Draft Working Group Reports

While the Stage 2 discussion boards remained and remain open, each working group began in January 2017 to draft a report crystallizing the key issues, insights, and perspectives arising from the respective group's consultations and deliberations. While structure and content of the reports differ, especially for the reports of the WGs on cross-cutting issues in clusters 1 and 2, the reports generally try to

  • articulate and clarify the (possibly multiple) understandings of "transparency" – as well as related concepts such as explicitness and openness – common among scholars undertaking a given form of qualitative research;
  • identify--for that particular form or aspect of qualitative research--the perceived needs for, and benefits of, researcher explicitness about her sources, methods, inferential or interpretive logic, etc.;
  • point to prevailing and "best" practices within the relevant research communities;
  • discuss the costs, risks, tradeoffs, and practicalities that the pursuit of transparency or explicitness involves in the WG's research domain;
  • and, in some cases, examine limitations of the concept of "transparency" as an intellectual value or operating principle for certain logics of inquiry.

Each Working Group has ultimate discretion over the content of its report. Reflecting extended and inclusive deliberations among diverse qualitative research communities, these texts vary in their degree of prescriptiveness, depending on the nature of the working groups' deliberations and degree of consensus: Some reports take the form of specific standards or guidelines; others raise considerations or questions that scholars undertaking or assessing a given type of research might ask, including costs and benefits of being explicit about specific aspects of the kind of research under consideration, or of being transparent in particular ways. Other reports delineate tradeoffs, ambiguities, and lines of disagreement over the meaning and practicalities of transparency for a given mode of inquiry. The reports also discuss the degree of heterogeneity of views among the participants in a given working group's deliberations. Each WG also strives to produce a text that will be a useful resource for designing and conducting research, for teaching graduate students, and for evaluating research outputs. At the same time, the texts produced through this process surely will ultimately be tentative in nature, and may later be further revised as understandings, norms, practices, and debates evolve.

Stage 3: Posting and Commenting on Draft Working Group Reports

The draft reports of the working groups were posted on the QTD website just prior to the 2017 Annual Meeting of the APSA, where they were also presented on an APSA theme panel and at the Qualitative and Multi-Methods Research section business meetings. The QTD Steering Committee and working groups sought to spark further engagement by holding a lunch-time drop-in session, at which Annual Meeting attendees could discuss draft reports with representatives from the WGs. The Steering Committee and the working groups now invite online comments on the draft reports. Based on this feedback and their own ongoing deliberations, the working groups will then revise their report in late 2017.

Posting and Dissemination of Revised, Final Reports

Revised, final reports are expected to be posted in early 2018. The Steering Committee is currently considering options for publication and broader dissemination of the deliberations' key findings.