Standards for conducting and reporting consensus and recommendation documents: European Society of Cardiovascular Radiology policy from the Guidelines Committee

Abstract

Cardiovascular imaging is exponentially increasing in the diagnosis, risk stratification, and therapeutic management of patients with cardiovascular disease. The European Society of Cardiovascular Radiology (ESCR) is a non-profit scientific medical society dedicated to promoting and coordinating activities in cardiovascular imaging. The purpose of this paper, written by ESCR committees and Executive board members and approved by the ESCR Executive Board and Guidelines committee, is to codify a standardized approach to creating ESCR scientific documents. Indeed, consensus development methods must be adopted to ensure transparent decision-making that optimizes national and global health and reaches a certain scientific credibility. ESCR consensus documents developed based on a rigorous methodology will improve their scientific impact on the management of patients with cardiac involvement.

Critical relevance statement

This document aims to codify the methodology for producing consensus documents of the ESCR. These ESCR indications will broaden the scientific quality and credibility of further publications and, consequently, the impact on the diagnostic management of patients with cardiac involvement.

Key Points

• Cardiovascular imaging is exponentially increasing for diagnosis, risk stratification, and therapeutic management.

• The ESCR is committed to promoting cardiovascular imaging.

• A rigorous methodology for ESCR consensus documents will improve their scientific impact.

Background and objective

The European Society of Cardiovascular Radiology (ESCR) is a non-profit scientific medical society dedicated to promoting and coordinating activities in cardiovascular imaging [1]. Imaging plays a strategic role in the diagnosis, risk stratification, and therapeutic management of patients with cardiovascular disease. Moreover, the current clinical guidelines drive an exponentially increasing demand for non-invasive cardiovascular imaging [2].

Various scientific documents have been developed or endorsed so far by the ESCR to present recommendations or expert consensus on specific topics [3,4,5,6]. Moreover, the ESCR is called upon to produce or endorse the following scientific documents according to the level of evidence provided:

• Clinical practice guidelines: evidence-based recommendations, based on a systematic review of the literature as assessed by expert panels and organizations, to offer standard approaches to clinical decision-making;

• Consensus documents: agreement or the consensus between experts on a specific topic based on critical appraisal of the literature together with expert opinion;

• Position papers on a specific issue, based on expert consensus without requiring a systematic review of the literature, to guide decision-making or inform the public; position papers provide expert recommendations on evolving issues, often based on limited evidence, and may be initiated by a specific group;

• White papers present a problem or explore a specific topic, providing an in-depth issue analysis, proposing an innovative approach, or discussing the implications of a new diagnostic or therapeutic approach. White papers aim to share ESCR’s opinion on a specific topic and should report and discuss the best available evidence.

Position and white papers follow the same preparation and approval procedures as for consensus documents, although a systematic literature review is not mandatory.

ESCR devised a standardized approach to creating such documents to streamline the process and avoid significant drawbacks, like the dominance of specific individuals, scientifically unsupported choices by strong opinions, and oversight of complex issues due to an unstructured process. Consensus development methods must be adopted to ensure transparent decision-making that optimizes national and global health and reaches a certain scientific credibility. Of note, reaching a formal consensus is established by utilizing already available information rather than meaning that the knowledge on a specific topic is implemented [7, 8].

The purpose of this document, written by ESCR committees and Executive board members and approved by the ESCR Executive Board and Guidelines committee, is to codify this standardized approach and detail the general rules for compiling and writing ESCR scientific documents.

Clinical practical guidelines are often also developed by clinicians, but the ESCR, through its delegates, could be called to endorse this type of document; thus, if imaging indications, techniques, or interpretation are part of a guideline, the inclusion of radiologists in the writing group and the application of the following indications is strongly advised. The following approach can also be applied to position and white papers except for a systematic literature search and review (SLR). Nevertheless, the ESCR highly recommends consensus documents as they align closely with the most stringent types of scientific literature utilized in radiology.

The target audience of this document is identified in guidelines committees, writing groups, collaborating organizations, and staff involved in developing scientific documents on cardiovascular imaging.

Methodology

There may be unsolicited applications from an author or solicited by the ESCR Executive Board or Guideline Committee for a specific person to lead a writing group. The chair (corresponding author) submits the topic proposal to the ESCR Guidelines Committee through a specific form (office@escr.org) (Fig. 1). The ESCR Guidelines Committee evaluates the proposal and reserves the right to integrate it to guarantee full compliance with the current scientific objectives of the ESCR and the maximum clinical impact.

Fig. 1

Overview of the steps for ESCR scientific documents

The writing group members are required to declare any conflicts of interest (COI).

Panel selection and composition

The selection of a core group of panel members is crucial as the panel’s composition may significantly impact the document results. Member characteristics and experience determine their preferences and likely determine consensus ratings. Heterogenous panels have been shown to provide more diversity in opinions, knowledge, and perspectives and perform better than homogenous groups [9, 10]. Panel members are typically selected because of their knowledge and experience, stakeholder group representation (e.g., physicians, researchers, multidisciplinary, steering committee), affiliation to a specific profession, current board memberships, being a recognized authority in the field, or their willingness to participate [11]. Heterogeneity should also be sought regarding age, gender, region, and socio-economic background. Moreover, the involvement as co-authors of a patient indicated by a representative patients’ organization is strongly recommended, if applicable. Furthermore, including an evidence-based medicine specialist to guarantee a rigorous methodology is advisable.

No specific criteria exist for defining an ‘expert’, and the experience of panel members often remains unreported. Nevertheless, selection criteria should be predefined, the reasons behind such requirements should be clear and transparent and the selection process should be documented comprehensively. Including societies or specific individuals from a society in the panel is advised. COI of panel members should be defined, reported, and appropriately managed [12]. The involvement of stakeholders should be addressed when not being a part of the expert panel [12].

The number of panel members depends on the chosen consensus methodology. It typically includes a minimum of five to a hundred with Delphi and ‘modified’ Delphi methods, and this number can vary per round. In contrast, five to ten panelists seem optimal based on the nominal group technique (NGT). Combinations of consensus methods, the so-called multi-methods, are also gaining popularity and will be discussed below [13].

The selection and tasks of the external review group will be discussed at the end of this document. Based on the need for a multidisciplinary patient-centered approach, it is encouraged to invite other societies and organizations for formal endorsement to spread the impact of the document further.

The ESCR strongly recommended applying the following criteria in the panel selection and composition:

• proven expertise in the field: H-index > 15 for authors ≥ 40 years, H-index > 8 for authors < 40 years old, with at least two publications in the specific field of the document,

• gender balance: at least 30% of representation for each category (male/female),

• age balance: at least 30% of representation for each category (≥ 40/< 40 years old),

• Economic geographic balance: at least one representative from upper-middle, lower-middle- or low-income country groups [https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups].

The chair will be the one to apply the topic proposal to the ESCR Guideline Committee.

The chair (corresponding author) will also indicate the co-chair (first author/last author). The chair is required to have an H-index > 25 and at least five publications on the specific topic. Nevertheless, the ESCR Guideline Committee has the right to evaluate the proposal and proposed panel members and propose adaptations.

Procedure of the writing group

The panel group is encouraged to organize a brainstorming session to clearly define the specific clinical question addressed in the scientific document. This is important, as a straightforward question will ascertain finding an answer and increase the results’ reliability. The objective and rationale for the consensus process should be clear. The population to which the clinical problem applies should be clearly defined [14]. Predefining the consensus document’s target audience and potential users is essential, as its content should be adjusted accordingly. It is also recommended to define expected results and their use and impact on this audience. Last, the panel should also specify all outcomes that may be relevant for clinical decision-making.

Expert panel members must be current with current guidelines, standards, and recent publications to achieve reliable consensus results. Hence, all expert panel group members should receive a literature-based summary on the specific topic or clinical problem, as discussed in the following section.

Evidence

A systematic literature search and review (SLR) is recommended to support reported scientific evidence (Fig. 1) [15].

The SLR on the specific issue could be an independent preliminary publication or addendum/summary as part of the main document, coordinated by two authors of the primary scientific document and up to six young researchers (< 40 years old) gender balanced. A PhD degree or a PhD training (or equivalent formal demonstration of scientific/academic work excellence) is considered a qualifying criterion to be selected in the SLR as a young researcher. Moreover, young researchers did not necessarily need to be involved in the primary scientific document.

Keywords should be carefully defined by the writing group and the search strategy described in the document, as well as any language or time interval restriction applied. An Evidence-Based Medicine specialist could be consulted to ensure the correct search methodology.

The proposed approach involves two independent young investigators using two or three databases (e.g., PubMed, Scopus, Web of Science). A citation manager (e.g., Mendeley, Endnote) could be used to collect and manage all results and automatically remove duplication. An automatic email search alert to identify newly published studies could be set up for each database.

Eligibility criteria should be defined and described in the document.

Up to six young researchers, in couples, proceed with SLR by screening the titles and abstracts of the records. The full text of the documents considered eligible is then evaluated for data extraction; any disagreement should be resolved by consensus, and cross-review on a limited sample should be considered to establish consistency.

For data synthesis, the best methodological/statistical practice should be followed.

Consensus development and definition

The choice of the method for developing the consensus is critical to producing documents that effectively impact the clinical management of the patients. Several methods are available such as consensus development panel or conferences, NGT, Delphi, and RAND/UCLA appropriateness method.

The ESCR recommends using NGT or Delphi to represent a good balance between rigor and feasibility, particularly considering the specific imaging field.

NGT will be used to reach a formal consensus [13]. NGT has the advantage of allowing for the generation of more ideas and the possibility of discussing ideas among the participants. Face-to-face meetings could also be held online using common platforms.

NGT will be articulated in the following stages: (1) preparation and presentation of the items to be discussed; (2) silent idea generation (each member independently generates ideas or solutions to the topic); (3) sharing ideas in a round-robin fashion (one idea at a time); (4) clarification and discussion (each idea is debated in order every member understands it); (5) private ranking; (6) feedback to the participants. If consensus is not reached, a further discussion and private voting session can be carried out. A mediator should be present for each group to ensure every participant can share opinions and vote.

A Delphi method to reach a formal consensus can be adopted as well [13]. Compared to NGT, Delphi has the advantage of ensuring the anonymity of participants (they never meet or interact), enabling balanced ideas among panelists, reducing individual bias, and including a higher number of participants [16]. Also, for Delphi, we have different stages: (1) identification of the topic to be discussed; (2) literature research; (3) preparation of the questionnaire; (4) iterative round of responses (answers are always anonymized and shared with other participants); (5) individual or group feedback between rounds; (6) final report. A facilitator must be present to ensure anonymity.

Feedback to participants after each round should be given as a statistical analysis (in the case of quantitative parameters), including new or modified items. There will be no need for further discussion in case of immediate agreement on a specific item, and it should be removed from the next rounds [11].

The ranking should be given using a 9-point Likert scale [11]. Reaching consensus is defined when ≥ 70% of participants converge on a score ≥ 7/9 [11, 13]. When using the Delphi method, the iterative process is restarted until the predefined threshold for consensus is reached or three rounds are completed [17].

Reporting

Appropriate and transparent reporting of writing group and expert panel processes holds similar importance to that of reporting scientific documents. The report provides the reader with information on the quality of the approach, content, conclusions, and generalizability to other populations or settings. Unfortunately, systematic literature studies have shown a lack of appropriate reporting of consensus building [10, 11, 13, 18, 19]. The EQUATOR (enhancing the quality and transparency of health research) Network aims to improve reporting quality. However, guidelines for reporting consensus documents have not been available. (https://www.equator-network.org/) Recently, van Zuuren et al presented a potential checklist outlining elements to integrate into guidelines on reporting consensus papers [18]. Their guideline project (ACCORD; accurate consensus reporting document) has just been published [20].

Reporting of ESCR consensus documents

Reports of consensus projects should adhere to a structured and standardized format to ensure a comprehensive and transparent description of each step and to provide an explanation for the decisions taken, thereby adding credibility, and reducing bias. In Table 1, such a structured and standardized format is provided as a checklist. In short, (1) the rationale and aim of the consensus document should be clearly defined, (2) the selection, appointment, and background of the expert panelists need to be reported, (3) PRISMA 2020 (Preferred reporting items for systematic reviews and meta-analyses) guidelines [15] should be followed, (4) consensus methods and consensus process and decisions should be thoroughly reported to ensure reproducibility, including a clear definition of consensus (5) it is encouraged to report results on both the consensus process and its content, (6) study results and (non)-consensus need to be interpreted and put in perspective (7) the report should include a discussion of the strengths and limitations and their impact on applicability and generalizability. The inclusion of figures such as flow diagrams, tables, and supplemental material is highly recommended to enhance transparency and efficacy and provide further insights. Standardized terminology is recommended to minimize disparities and misinterpretation [21, 22]. When standardized terminology is lacking, further, more detailed specification is required. This is especially important for the term “modified” Delphi, as various methodological adaptations have been grouped under this label [11, 23].

Table 1 Format for structured and standardized reporting of consensus documents

Internal review

The internal peer review process ensures scientific research consensus documents’ quality, reliability, and credibility. To maintain objectivity, this review process should be performed by ESCR and should include three to five peer reviewers. The chair of the writing group chooses reviewers according to the chairperson of the Guidelines Committee.

As a first step, it is essential to select suitable reviewers, considering the following points: (1) Expertise: internal reviewers should be experts in the same or at least a closely related field as the consensus topic, with a broad knowledge of the subject matter, including relevant methods and current standards. The selection regarding a reviewer’s expertise should also consider their scientific impact. Thus, an H-index > 15 with at least three publications in the specific field of the document is recommended. (2) Diversity: ideally, external reviewers should have diverse backgrounds (sex, age, economic geographic area, and multidisciplinarity) to ensure a comprehensive assessment and reduce bias. (3) Independence: COI with the authors of the consensus document should be avoided to guarantee an independent external review process.

In addition to independent internal reviewers, recommendations from the editorial board can also be sought.

All internal reviewers should receive clear instructions on reviewing the consensus document. These criteria should include aspects such as appropriateness of the panel selection, clarity of consensus, validity of scientific methods, topic-related evidence, rigor of conclusions, and adherence to ethical standards.

All comments from internal peer reviewers must be seriously considered. When the scientific document is published, peer reviewer names and disclosures will also be published with the scientific document.

Once submitted to a scientific journal for dissemination, the document will be reviewed as foreseen in the regular journal’s evaluation process.

Conclusions—GRADE/interpretation and SWOT analysis

Since consensus documents aim to provide recommendations by synthesizing diverse viewpoints and establishing a shared agreement among experts within a specific field, it is essential to include a compelling conclusion at the end of the document. Based on the document’s goal and target audience, this conclusion should summarize the key findings, highlight implications, and discuss strengths and limitations.

To clearly understand the recommendations’ significance, it is recommended to rate the certainty of evidence transparently. This should preferably be done using the GRADE (grading of recommendations assessment, development, and evaluation) system [24,25,26], as it reduces the multiplicity of conflicting systems for grading evidence and recommendations by unifying several aspects into a single system. It is a systematic approach to assess the quality of evidence and the strength of recommendations, classified into four and two levels, respectively. The levels of evidence are “high”, “moderate”, “low”, and “very low”. Recommendations are either defined as “strong” (benefits outweigh the risks) or “weak/conditional” (the balance between benefits and risks is less confident). This can help to provide a clear and concise summary of the main findings.

In addition to summarizing the key findings, discussing strengths, weaknesses, opportunities, and threats (SWOT) related to the consensus topic is crucial. This approach provides a comprehensive assessment of consensus recommendations by highlighting positive aspects (strengths and opportunities) while identifying potential challenges (weaknesses and threats) that may hinder implementation.

Publication/availability/updates

The scientific documents will be preferentially submitted to the official reference societies journals, which will be applied for an external review as previously described. The publications should be open source and/or available on the official ESCR website. When necessary, the ESCR Guidelines Committee reviews scientific documents based on the opinion of Committee members or the presence of new evidence.

The Guidelines Committee reviews scientific documents at least every two years after publication to verify their currency and validity and eventually proposes an update to optimize patient management (Fig. 1).