Annex A > Chapter 19 - Statistics Relating to the Clinical Performance of Paediatric Cardiac Surgical Services in Bristol Compared with Other Specialist Centres during the Period 1984 to 1995 > Comments of the clinicians at Bristol (and others) on the evidence of the Experts on statistics received by the Inquiry, and the Experts' responses > The national sources of statistics used to compare clinical performance
<< previous | next >>
The national sources of statistics used to compare clinical performance
144 Mr Wisheart expressed a variety of concerns about the reliability and validity of the UKCSR as a source for comparing performance between centres:
`I now believe that there are substantial limitations upon the reliability and the validity of the UKCSR. Therefore its value to this Inquiry as a comparator must be in doubt.
`... Reservations about the reliability and validity of the Register stem from the possible under-reporting of mortality, the lack of information from some individual centres, other missing data, intra-centre variability and the use of differing definitions. In addition, the absence of information about the range of results obtained by individual centres or surgeons, and the absence of any allowance for risk stratification limit further the value of the available data.' [205]
`The reliability of data published in the U.K.C.S.R. has always been questioned, as the figures were never validated. Errors could have been made at the source of entry, as junior members of staff could have been entrusted with form filling, with no arrangements for double checking the figures. The 30-day mortality figures may not represent the true picture, as at some institutions patients are transferred back to referring hospitals after a few days and therefore the follow-up information for this group for the first 30 days may not have been complete. We were also aware that a centre of excellence with a large volume of cases might mask the true mortality figures of smaller centres. The probable deficiencies in data collection and lack of comparison of like with like, meant the resulting average figures given in the Register were not perceived as being statistically accurate, and this limited its value as a tool in providing adequate comparators.' [206]
146 Referring to the UKCSR, Mr Bruce Keogh, Secretary of the Society of Cardiothoracic Surgeons of Great Britain and Ireland, told the Inquiry:
`I think it is incumbent upon me to doubt the reliability, otherwise I would not be doing my job properly. I have less reason to doubt the activity data, but I do sometimes feel that operative mortalities that are reported may be a bit low.' [207]
147 Mr Wisheart expressed concern as to the reliability of the categories used in the UKCSR as a means of recording surgical activity. Referring to the reliability of data derived from the UKCSR's diagnostic category `Transposition of the Great Arteries', he observed:
`The handling of Transposition by the Inquiry's experts with regard to the type of operation carried out has not been successful. I do not believe that there is any evidence that paediatric cardiac surgeons have ever consistently classified the Mustard or Senning operation as palliative, in making returns to the UKCSR.' [208]
148 As for the `over-1' age category in the UKCSR, he stated:
`There is a problem in using the UKCSR as a comparator for the Inquiry's children aged 1-15. The UKCSR's category of "over 1" includes older teenagers and adults having open heart surgery for congenital abnormalities; there is no cut-off point in the UKCSR figures at age 15, until the mid-nineties.' [209]
149 As regards the reliability of the national data as a basis for comparing mortality at Bristol and elsewhere, Mr Wisheart submitted:
`Although Dr Spiegelhalter feels it is unlikely that Bristol has produced good quality data whilst other centres have produced unreliable data, serious doubt about the reliability of the data from the other centres has been expressed by a number of experts. The other data is of two types; first, Hospital Episode Statistics (HES) which was collected for administrative purposes and not for the clinical purposes for which it is now being used, and secondly, the United Kingdom Cardiac Surgical Register (UKCSR), the shortcomings of which have been repeatedly rehearsed. The reliability of the UKCSR is most dramatically questioned by the observation that of twelve centres reporting their results for 1988-1991 both to the UKCSR and to a Working Party of the Supra Regional Services Advisory Group of the Department of Health, only one returned the same figures to both - and that one was Bristol. Other questions about the comparator data are:
- `There is thought to be under-reporting of death.
- `There are believed to be variations in the definition of death which have been used by different centres and surgeons.
- `Survival status is not known in some HES and some UKCSR data.
`Unless there is a high degree of confidence in the data both from Bristol and from elsewhere, there cannot be confidence in the comparison.' [210]
150 Mr Wisheart stated further:
`... the figures [in the Experts' Overview Report] are based upon HES and [UK]CSR which are not high quality, and importantly, do not agree with each other. Indeed the disagreement between HES and [UK]CSR is striking in terms of numbers of deaths, death rates and excess deaths. For example, the number of excess deaths in open operations in children under one, between 1991 and 1995, by case mix stratification, is estimated by [UK]CSR to be 12.9 and by HES to be more than double that figure at 27.2. There is no agreement about the total number of deaths in these databases, therefore I believe that these discrepancies should be examined and resolved. I had hoped that they would have been resolved much earlier but as the end of the Inquiry approaches, there remains uncertainty.' [211]
151 Mr Jaroslav Stark wrote to the Inquiry:
`The quality of the data available from HES and the UKCSR for a period 1984-1995 is of great concern. This raises serious doubts about the validity of any conclusions based on the analysis of this data. The use of such unreliable data for the assessment of the performance of the paediatric cardiac unit at Bristol may have wider implications. It may set a precedent for the future.' [212], [213]
152 In their report to the Inquiry, Ms Audrey Lawrence [214] and Professor Gordon Murray advised:
`... surgeons have unanimously more confidence in the data they have provided to the [UKCSR] than in that provided by the hospital administration system (HES), in terms of both procedures and deaths.' [215]
153 Referring to the data returned by Bristol to HES and UKCSR, Mr Wisheart stated:
`I believe that the data available from Bristol is usable, in that both Mr Dhasmana's and my surgeon's logs are reliable sources of information. ... The information returned to the UKCSR was based on the data in our two logs.' [216]
`The data will be very accurate for Bristol, but there is no knowledge of the accuracy from other centres.' [217]
155 Referring to the CCR, Mr Wisheart stated:
`The CCR provides a reliable standard against which the local Bristol data can be judged.' [218]
`For all the Bristol patients a Coded Clinical Record was created from the case notes and this must be regarded as being of extremely high quality, almost certainly the highest quality database which exists within the Inquiry. Data of comparable quality has not been created for any other centre.' [219]
`If the analysis [advanced by the Experts] was based on the CCR, which is clearly a very high quality database, then it might well be that this argument [the conclusions reached by the statistical Experts] would carry great weight.' [220]
156 In response to these expressions of concern over the reliability of HES and UKCSR as sources of data from which to draw comparisons, Dr Spiegelhalter, Professor Evans, Dr Aylin and Professor Murray stated:
`The Submission [of Mr Wisheart] expresses concern about under-reporting and varying definitions of deaths in other centres. There is always the possibility, although it does not seem especially plausible, that Bristol has produced good-quality data, while the bulk of the rest of the country were systematically under-reporting mortality.' [221]
`Further investigation of the accuracy of the mortality rates derived from the HES data has shown that over 95% of 30-day deaths following open surgery are recorded in HES, and that Bristol's accuracy is typical.' [222]
`When no data source is a gold-standard, corroboration between reasonably independent sources reinforces the conclusions from both.' [223]
157 In their Overview Report to the Inquiry, Dr Spiegelhalter et al. observed:
`The reasonably consistent patterns ... lend added weight to the HES evidence, as do the KP70 [224] and linkage exercises carried out to assess the quality of the recorded activity and outcomes in HES. There is no evidence that Bristol was at variance with the national pattern in HES reporting. The [UK]CSR data must be treated with great caution at the level of individual procedure groups. The crucial issue is whether the undoubted inaccuracies are sufficient to cast doubt on any observed divergent performance.' [225]
`A possible marker of data quality is the ratio of episodes recorded by HES to those on KP70 (paper returns to the DoH). Aylin et al. (2000, INQ 0030 0017) found that there was excellent agreement both in Bristol and elsewhere for cardiothoracic surgery as a whole, but were unable to compare for paediatric cardiac surgery.' [226]
`Although using different definitions and arising from relatively independent sources, HES and [UK]CSR data showed reasonable consistency at an aggregated level, although considerably poorer for individual procedure groups ... The crucial issue is not whether HES or [UK]CSR precisely measure activity and outcome, but the extent to which feasible data inadequacies could explain any observed divergent performance ...' [227]
<< previous | next >> | back to top
Footnotes
[205] WIT 0120 0294 Mr Wisheart
[206] WIT 0084 0052 Mr Dhasmana
[208] INQ 0012 0066; `A review by Mr Wisheart of the evidence offered by Professor Evans, Dr Aylin, Professor Murray, and Dr Spiegelhalter'
[209] INQ 0045 0081; `Review of data sources and statistical methods, available to the Public Inquiry for discussion', 23 September 1999, Mr Wisheart
[210] SUB 0009 0023 - 0024 Mr Wisheart
[211] INQ 0045 0100; `The response of Mr Wisheart to the Overview of statistical evidence concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995'
[212] WIT 0567 0010; `Comments on statistical analysis and review of outcomes of paediatric cardiac surgical services at Bristol and other specialist centres', 7 April 2000, Mr J Stark
[213] Professor John Yates, Director of Inter-Authority Comparisons and Consultancy, Health Services Management Centre, University of Birmingham, expressed the view in a submission to the Inquiry that HES data was of value for comparative analysis (see WIT 0568
0027 - 0043), although aspects of his statistical methodology were criticised in peer review reports commissioned by the Inquiry (see for example INQ 0036 0001 - 0013 , Professor Stephen Gallivan)
[214] Research management consultant, Lawrence Research
[215] INQ 0033 0003 Ms Lawrence and Professor Murray
[216] WIT 0120 0299 Mr Wisheart
[217] INQ 0045 0081; `Review of data sources and statistical methods available to the Inquiry for discussion', 23 September 1999, Mr Wisheart
[218] WIT 0120 0471 Mr Wisheart
[219] INQ 0045 0092 Mr Wisheart
[220] INQ 0045 0100 Mr Wisheart
[221] INQ 0034 0006; `A Response to Submissions on behalf of Mr JD Wisheart, Appendix 2, The Inquiry's Statistical Analysis', May 2000, Dr Spiegelhalter et al. See also WIT 0567 0004 Mr Stark
[222] INQ 0034 0002; `A Response to Submissions on behalf of Mr JD Wisheart, Appendix 2, The Inquiry's Statistical Analysis', May 2000, Dr Spiegelhalter et al.
[223] INQ 0034 0002; `A Response to Submissions on behalf of Mr JD Wisheart, Appendix 2, The Inquiry's Statistical Analysis', May 2000, Dr Spiegelhalter et al.
[224] Mr Richard Willmer, a chief statistician at the Department of Health, describes the nature and purpose of KP70 returns (Körner Patient aggregated return no 70) in his supplementary written statement, WIT 0189 0133 - 0134
[225] INQ 0045 0021; `Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995', September 2000, Dr Spiegelhalter et al. (emphasis added)
[226] INQ 0045 0018; `Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995', September 2000, Dr Spiegelhalter et al.
[227] INQ 0045 0003; `Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995', September 2000, Dr Spiegelhalter et al.