Annex A > Chapter 19 - Statistics Relating to the Clinical Performance of Paediatric Cardiac Surgical Services in Bristol Compared with Other Specialist Centres during the Period 1984 to 1995 > Section three: the principal conclusions of the Inquiry's Experts on statistics > The principal conclusions in greater detail
<< previous | next >>
The principal conclusions in greater detail
100 In the main body of their Report, Dr Spiegelhalter et al. set out their principal conclusions in greater detail. As regards the statistical evidence relating to activity [145] and mortality in Bristol derived from the data produced by Bristol, they concluded:
`There are clear limitations to all sources, and none is subject to defined procedures for data collection, follow-up and validation. It would be fair to say that none is held in high regard as a source of reliable evidence for clinical audit. However, Evans (1999) concludes that where direct comparison is sensible, the pattern is similar and there are no startling discrepancies. Although there is no gold standard for comparison, the Bristol PAS system appears of reasonable quality, and hence this lends confidence to Bristol returns to the national HES database. Our overall comparison suggests that the different sources agree well on the open operations in general and for many specific procedures.
`The main findings of interest concern mortality rate [sic] for open surgery in under 1s. Overall, sources agree that the mortality rate was around 25-30% during the period under scrutiny, although with considerable variability between different procedures.' [146]
101 As regards the statistical evidence on activity and mortality in Bristol compared with other specialist centres, as derived from UKCSR and HES data, the experts concluded:
`Although the [UK]CSR data report statistically significant excess mortality for Bristol in over 1s during 1988-1990, the primary finding from both [UK]CSR and HES is of excess mortality from 1991-1995 in open operations in under 1s, in which the mortality rate in Bristol was around double that in other centres. This difference is retained after stratifying for operative group, which is the available determinant for case-mix. There is no evidence for excess mortality in closed operations, or for open operations in over 1s from 1991-1995. Reported mortality for open operations in under 1s fell in other centres from 21% in 1984-1987 to 12% in 1991-1995. Bristol appears not to have followed that pattern of improvement. There is no evidence of excess mortality in Bristol during Epoch 4, [147] although activity in Bristol was too small to draw any firm conclusion.
`We emphasise that the estimated total excess deaths for HES depends on the age-stratification used: the excess risk is greater in younger children: for all open operations in epoch 3 [148] the total is 30.1 when dividing only into under and over 1s (Table 6.1) and 34.3 when including a < 90 day category (Aylin et al., 1999).' [149]
`HES identifies excess mortality with 95% confidence for switches (G3), AVSD (G5), ASD (G6) [150], open operations stratified for case-mix, (G1 to G11), and all open operations taken together.' [151]
`The [UK]CSR results show that each year between 1988 and 1994 (with the exception of 1990), Bristol had either the highest or near the highest mortality rate for open surgery in under 1s. This is reinforced by the HES data between 1991 and 1994. It is clear that Bristol's activity was consistently below the median in the country ...' [152]
102 Referring to the national sources of data, the Experts concluded:
`The two national sources, HES and the CSR, are admittedly imperfect. Both suffer considerably from lack of agreed operating procedures for ensuring completeness and accuracy of activity, coding and outcome results. Both the OPCS4 coding scheme and the use of non-clinical coders lead HES to be viewed with suspicion by clinicians. There are also strong concerns about variability between centres in the [UK]CSR's coding procedures and recording of mortality. Even if they were meticulously completed, agreement between the two sources could not be expected due to their different criteria. However, HES was found to be surprisingly accurate in its recording of in-hospital mortality and, with certain clear exceptions, the sources described the same broad picture.' [153]
103 Referring to the local sources of data, they concluded:
`The local sources were found to provide good agreement on activity and overall mortality, although comparison at a finer level was sensitive to the coding conventions used. Nevertheless, the six sources on Bristol's activity and outcome agree well for open operations in general and, to a lesser but still reasonable extent, for finer consensus procedure groups of interest. Where there is disagreement, then there are clear reasons, usually resulting in transfer of operations between two groups.' [154]
104 Dr Spiegelhalter et al. set out their detailed conclusions concerning the evidence of divergent performance in Bristol:
`There is no evidence of excess mortality in closed operations carried out in Bristol, and limited evidence in open operations on children aged over 1 year. However, there is strong and consistent evidence of excess mortality in open operations in children less than 1 year old at operation. It is estimated from HES data that in the period 1991-1995, 24.1 (95% confidence interval 12 to 34) of 41 recorded deaths are in excess of that expected were Bristol a "typical" centre: finer age-stratification increases the estimated excess mortality. [UK]CSR data suggest the excess mortality dates back at least to 1988. Open procedures on children aged less than 1 that can be identified with reasonable consistency as having excess mortality include "switches", operations for TAPVD, AVSD and, although rare in this age group, ASD. [155] It is to be expected that excess mortality is easier to detect in higher risk groups.
`The excess mortality was not just restricted to AVSDs and switch operations, and the conclusions are robust to admissions with missing outcomes. National mortality rates were comparable to those in the international literature. One other centre had a consistent pattern of excess mortality in open operations in children over 1 year, [156] but there were no other centres with consistently divergent raised mortality in the younger age group.' [157]
105 The Experts stated, as their overall conclusions:
`The single most compelling aspect of the data is the magnitude of the discrepancy between the outcomes observed at Bristol and those observed elsewhere. For children aged under one year undergoing open surgery between 1988 and 1994, the observed mortality rate at Bristol was roughly double that observed elsewhere in 5 out of 7 years. While the national trend over this period was for mortality rates to fall substantially, no such trend was seen in the Bristol results. In spite of the many flaws in the data sources, we do not believe that statistical variation or any systematic bias in data collection can explain a divergence of this magnitude. We therefore conclude that there is strong evidence of divergent performance at Bristol in the areas identified above, and we believe that the imperfections of the data do not cast serious doubt on these conclusions.' [158]
`Given the many flaws that have been identified in existing data sources, it is clear that only gross divergence could have been identified with any degree of confidence. If, for example, the mortality rate for open operations in under 1s observed at Bristol had been 50% higher than elsewhere rather than 100% higher, it would have been very difficult to exclude the possibility that the difference had arisen through a combination of differences in case mix, in the coding of operative procedures, and in the thoroughness of achieving follow-up data.' [159]
<< previous | next >> | back to top
Footnotes
[145] At INQ 0045 0014, Dr Spiegelhalter et al. stated: `An event has to be identified that measures activity and hence forms the basis for the denominator in any calculated mortality rate. The primary analysis focused on the number of admissions/spells as the basis for comparison, although some of the data sources use operations as their measure of activity. ... There is normally only one operation per admission and so there is limited difference according to which is chosen.'
[147] April to December 1995
[148] 1991 to March 1995
[150] Group 6 includes more complex procedures such as closure of persistent ostium primum, and sinus venous atrial septal defects, as well as simple atrial septal defects
[155] Group 6 includes more complex procedures such as closure of persistent ostium primum, and sinus venous atrial septal defects, as well as simple atrial septal defects
[156] Leading Counsel to the Inquiry announced in the oral hearing on 3 November 1999 that this centre was Harefield Hospital. In their Overview Report to the Inquiry, Dr Spiegelhalter et al. stated: `This finding must be treated with caution. Harefield has been an innovative centre for transplant surgery and these operations are included in the CSR (although not in the HES open category), and it also has a reputation for taking difficult cases from abroad.' INQ 0045 0026
[157] INQ 0045 0036; see Chapter 3 for an explanation of these clinical terms