Thoracic ultrasound competence for ultrasound-guided pleural procedures

Australasian practice

In 2017, the Thoracic Society of Australia and New Zealand published a position paper on recognition of TUS competence. This was produced by an expert panel and consists of completion of an introductory course, a logbook of at least 40 scans, two formative assessments after five to 10 scans and again after 20 scans, followed by a barrier assessment after 40 scans carried out by an independent assessor, with a minimum pass mark of 90% [42].

Formative and summative assessments in this pathway are all undertaken using the ultrasound-guided thoracentesis skills and tasks assessment test (UG-STAT). This tool assesses 100 points over 11 domains related to the British Thoracic Society guidelines, including familiarity with basic ultrasound parameters, recognition of ultrasound images illustrative of common pleural effusion appearances, as well as the ability to perform a TUS using a model or mannequin and to subsequently mark an appropriate site for pleural drainage. It has been validated to discriminate between novice, intermediate and advanced users [43].

This pathway would certainly appear more robust than current UK practice, and addresses each aspect of Miller's pyramid, but a number of limitations remain. The Thoracic Society of Australia and New Zealand recognise the inadequacy of using the number of scans to define competence; however, still concluded that a minimum of 40 scans was desirable based on expert consensus and comparison with other accepted pathways. The timing and necessity of formative assessment in this context was not investigated but it should be noted that assigned mentors may identify areas of deficiency in trainees in order that they may be addressed prior to formal assessment. One of the most significant areas of limitation appears to be in the definition of a pass mark. The original validation study demonstrated mean scores of 49.3 in the novice group, 73.0 in the intermediate group, and 91.5 for advanced practitioners. To obtain recognition of competence by the Thoracic Society of Australia and New Zealand, a score of 90% is required. This may not be achievable for some users, may not reflect the minimum standard required to safely undertake a procedure and fails to recognise the gradations of skill or competence between basic and advanced practitioners [42, 43].

European practice

No specific European TUS training programme exists at present, although the ERS does provide TUS training courses that conform to EFSUMB recommendations. They consist of pre-course online learning with assessment, followed by a 2-day course, including lectures and practical supervised ultrasound experience. The course culminates with a 15–30 min assessment consisting of a supervised ultrasound scan. The method used for this assessment is unclear, as is the subsequent definition of competence and the minimum requirements to achieve it.

Within European practice, however, a Danish group have developed a similar assessment tool to the UG-STAT, although this encompasses a much broader curriculum than pleural intervention guidance and includes diagnostic lung ultrasound, such as the identification of the interstitial syndrome. A comprehensive assessment such as this has its advantages but is less likely to be applicable at all levels, principally the most basic level which augments procedural competence. Although validation included assessment of novice, intermediate and expert groups, the clinical application in defining competence was not addressed in the development of this tool [44].

Endobronchial ultrasound

Endobronchial ultrasound (EBUS) represents another area in respiratory medicine in which the introduction of ultrasound has revolutionised practice. This is due to a similar diagnostic yield and reduced side-effect profile in comparison with mediastinoscopy. There is also an increased diagnostic yield when compared with conventional transbronchial biopsy, along with a reduced risk of inadvertent sampling of the myocardium, oesophagus or great vessels [45].

Issues surrounding the achievement and assessment of competence have become apparent and are similar to those evident in the assessment of TUS competence. One systematic review of 27 studies suggested that to achieve an accuracy of 80%, trainees should undertake a minimum of between 37 and 44 procedures. Although the definition of competence was somewhat unclear, the number of procedures required to achieve “competence” varied between 10 and 529; however, using a cumulative sum analysis this varied between 10 and 100. This serves to prove previous data that minimum numbers of procedures when taken in isolation are unreliable as a marker of competence, although expert consensus continues to differ [46–48].

In this field, a three-step approach has therefore been suggested comprising of theoretical learning followed by simulator-based learning prior to supervised practice involving patients. The first stage is based on the intuitive premise that theoretical knowledge is required before practical application. The second step has been applied because the EBUS assessment tool (EBUSAT) study demonstrated that training using a simulator produced higher scores using an objective scoring system when compared with those who undertook traditional apprenticeship style training. Finally, supervised practice on genuine patients is required to improve validity. This study resulted in the development of an objective EBUS assessment tool called the EBUSAT and subsequently a further assessment tool has been developed in the USA called the EBUS-STAT. These tools have been proven to be reliable and valid in their ability to discriminate between novice and expert operators [49, 50].

Therefore, the current ERS training programme has been developed to reflect this and contains a well-defined curriculum. It incorporates the three-stage approach of theoretical learning, which is assessed using multiple choice questions, a second stage of structured simulation training, followed by a portfolio of 20 cases, of which three must be submitted by video for assessment using the EBUSAT tool, although the justification for using 20 cases as a minimum requirement is unclear [51].

Echocardiography

Traditional assessment of competence in echocardiography is similarly based on recording a minimum number of scans presented in a logbook. Recently, this has developed somewhat further to incorporate other formal objective measurements. The European Association of Cardiovascular Imaging is part of the European Society of Cardiology and provides accreditation in transthoracic echocardiography. Requirements include an intensive theory examination followed by completion of a logbook of at least 250 cases. The British Society of Echocardiography accreditation has the same requirements; however, it is possible to achieve the more basic level one accreditation following completion of a logbook with 75 cases over a 12-month period, followed by an assessment, including review of the logbook, a live assessed scan and interpretation of video clips. Accreditation information on transthoracic echocardiography can be accessed from the society websites (www.escardio.org and www.bsecho.org). Similar problems have been identified with regard to the use of a logbook alone to define competence in this cardiology context. Conflicting evidence has been published, although overall it would appear that logbooks in isolation are insufficient, as has already been suggested. This has resulted in the development of alternative methods such as objective structured assessments, including the use of simulators. Despite the development of such assessments, they have yet to be widely adopted [52–54].

Focused intensive care echocardiography training has been developed to facilitate more widespread integration of echocardiography in a critical care context. The current pathway requires engagement with an initial teaching day, including both didactic and practical sessions followed by completion of a logbook including 10 directly observed scans, 40 further scans and a final observed assessment. The rationale for setting a total of 50 scans as an acceptable benchmark is based on a solitary study that demonstrated an acceptable level of performance from six trainees who completed an average of 33 scans each. The study therefore postulates that 30 scans may be a reasonable number to achieve competence, as performance was improved when compared with a previous pilot study with a mean number of 15 scans. It is not clear whether attributing these improvements to increasing experience represents a true causal relationship or whether curriculum modification and improved initial teaching with the inclusion of interactive clinical cases acted as a confounding factor. No direct measurement of competence with increasing scan volume was undertaken. As a result of these findings, the Intensive Care Society recommends a minimum of 50 scans to allow for differing levels of skill acquisition [55, 56].

Future directions for TUS competence

It is clear that ultrasound has become a crucial component of thoracic medicine by improving the safety profile of frequently undertaken interventions, as well as providing a valuable diagnostic tool. Its uptake continues to increase both within this context and also in other specialist areas. Assessment of competence is, therefore, critical to ensure patients receive high-quality, effective and safe care; however, current methods allow for significant bias and do not appear to be robust enough to provide adequate rigour.

Current medical education thinking would suggest that competence requires knowledge, technical and nontechnical skills, as well as clinical reasoning and its application into individual scenarios. An international multispecialty group has suggested that when this is applied to ultrasound competence, the criteria required to demonstrate competence include recognition of the indication for the examination, applied knowledge of equipment, an ability to obtain and optimise an image and performance of a systematic examination, as well as appropriate interpretation and documentation of the images, leading to sound medical decision making. This has led to the formulation of a generic objective structured assessment of technical skills [57]. In the simplest terms this would suggest that assessment should ensure the ability to produce a good image with appropriate interpretation of that image leading to a correct clinical decision.

Although attempts have been made to improve assessment methods in TUS, at present UG-STAT and the Danish assessment tool based on objective structured assessment of technical skills provide the only current validated assessment tools; however, each of these are associated with a number of limitations. One significant limitation with UG-STAT is that the pass mark has been validated to correspond with advanced practice and as such, fails to recognise the gradations of competence present in clinical practice and ignores that a more basic standard may be satisfactory for practice in certain straightforward scenarios [43]. The Danish tool encompasses a much wider definition of TUS consistent with the prevailing European perspective and, in particular, integrates the role of TUS as a diagnostic tool in the assessment of a wide range of intrathoracic pathologies, including pneumothorax or the interstitial syndrome, as well as pleural effusion and pleural procedural guidance. This may not be necessary for those undertaking basic TUS training but may be suitable for incorporation within the more advanced levels [44, 57].

To practically apply the EFSUMB and British Medical Ultrasound Society recommendations for tiers or levels of competence to TUS, we would suggest that level one corresponds to the ability to produce an adequate image, identify a large simple effusion and make the correct decision to proceed to intervention. This level may be more appropriate and achievable for those whose pleural experience is more limited, such as junior respiratory trainees or trainees from other specialties such as acute internal medicine. Level two may correspond to the ability to identify and intervene in smaller, more complex effusions and is likely to represent a level achievable by more senior respiratory trainees. Subsequently, level three would then represent those with a subspecialist interest in pleural disease who can undertake more advanced procedures, such as the use of live in-line ultrasound guidance, motion mode or perform image-guided pleural biopsies of pleural thickening. The inverse should also be considered, such that in level one, the identification of a more complex effusion or pathology not previously seen should result in no procedure being undertaken but instead input sought from a level two or three practitioner. This hypothesis for a new definition in the gradations in competence is not dissimilar to a suggested definition provided by the recent UK expert consensus presented in table 3. Although the standards of practice described could be directly adopted, assessment methods would need to be adapted further as definition of these competency levels is primarily based on minimum number of scans and logbooks. Directly observed assessments have been added to provide a measure of objectivity but are unlikely to be as objective or robust as a bespoke multimodal assessment tool [24, 26, 27].

Therefore, the currently available validated assessment tools do not appear to be sensitive or specific enough to apply to this context of pleural procedural guidance, nor does their current clinical application adequately discriminate between gradations of competence.

Assessment for these new levels of competence must be multimodal. A written component is likely the best method to assess the most basic level of Miller's pyramid's to “know”. Multiple choice questions or extended matching questions appear to produce the best reliability. It could be suggested that this need only be tested at the basic level one as this provides the foundation from which other skills are developed. The “knows how” and “shows how” categories are most likely best assessed using objective structured clinical examinations; however, in contrast, this is likely to require repeat assessment as each level progresses. The requirement for real patients with genuine findings may be the most valid assessment medium, although this is unlikely to be practical on the large scale required. High-fidelity simulators have been proven to be valid and reliable in a number of examination settings, including ultrasound and in particular, a validated TUS simulator test has been recently developed. The final assessment of “does” is likely best measured using either portfolios with ongoing summative and formative assessments or logbooks with supervision; however, minimum numbers of procedures or scans not only remain undefined but may not be a valuable metric given the uncertainty around the validity and the variations in learning curves [31, 33, 39, 41, 43, 54, 58].

As a result, further work is required in a number of areas. A valid and reliable assessment tool is required, and this could either take the form of a solitary tool that discriminates between levels one and three or a number of separate assessment tools instead separately validated for each individual level. It may also be possible to measure the mean number of ultrasound scans associated with the achievement of competence at each level; however, any studies are likely to be difficult to accurately interpret given the wide variations in individual learning styles as well as other factors such as the degree of supervision or complexity of cases encountered. Therefore, it would seem most prudent that future research should primarily concentrate on the development of valid and reliable assessment tools that define and adequately discriminate between the various gradations of competence. It may also be necessary to undertake a similar approach toward the development of a separate assessment tool to independently address procedural competence of the invasive pleural procedure, such as thoracocentesis. Following the introduction of UG-STAT into clinical practice, the same Australian group has already developed one such example [59].