Final Report > Chapter 20: Understanding and Assessing the Quality of Clinical Care in Bristol > Audit and information
<< previous | next >>
Audit and information
4 To assess outcome, there need first to be standards. Then, there needs to be a system of audit, both at local and national level, which can allow the assessment of performance and outcome in relation to these standards. We have already described how audit developed in the 1980s and 1990s. We are led to the view that, during the period of our Terms of Reference, although there were pockets of activity, and although the Department of Health's (DoH) thinking about audit was developing into a search for mechanisms of quality assurance, no clear national standards of care emerged against which clinicians could confidently expect to compare their performance, and that, even by the end of the period, it is unlikely that any clinician would be expecting to do so.
5 Turning to Bristol, Dr Roylance took what can be seen now to be a modern and far-sighted view of audit. He did not regard the activities which went by the name of audit as being `truly' audit. Rather, they merely involved the intermittent collection of data. For Dr Roylance, such data could only have meaning, and become part of a true process of audit, if, first of all, there were standards of care against which to evaluate current practice and so give meaning to the data. Then, secondly, he recognised that it was necessary to `close the audit loop' by improving those aspects of care where standards were not being met. Regrettably, however, despite his being clearly ahead of his time in general terms, his decision to delegate responsibility for audit to the directorates was effectively counter-productive. It reflected his overall approach to management, on which we have commented earlier. It had the effect again of channelling activities into separate and distinct compartments which did not effectively communicate with each other (which we have called the `silo' effect).
6 The consequence of the arrangements for audit was a lack of coherent organisation and co-ordination. In particular the role of the Medical Audit Committee (MAC) was uncertain. For example, following a visit to the UBHT in March 1994, the Regional Audit Team produced a report which noted that power in relation to audit lay with the directorates, whose directors were not members of the MAC. The Regional Audit Team report said that the MAC was bypassed by managers when they addressed audit issues and that there was `confusion for the Audit Committee over its role'. [1] Because audit was assigned to directorates, funding for audit went to them too, leaving the MAC without resources to do its job. Moreover, Dr Roylance said that he considered it to be the responsibility of the Chair of the MAC to be satisfied that the process of audit was being carried out [2] and that he should be told by the Chairman if any management action was needed. [3] But he also said that if a unit failed to carry out the audit process that would not be a management issue which would involve him. [4]
7 The net effect of these various arrangements was that they clearly militated against the development of a strong body of information and analysis that would have enabled healthcare professionals to look across the boundaries of the various specialties to assess the care provided by multidisciplinary teams, of which the PCS service was a particularly obvious example. [5] Senior management, which could have helped to bring this about, stood back from involvement in audit, in keeping with Dr Roylance's commitment to the autonomy of the clinicians and the need for senior management to avoid `interference'. But this reluctance on the part of senior management to be involved was common and accepted practice until the early 1990s. [6] It may well have hindered the development of any effective `audit loop', but this is a criticism of audit as it was introduced and operated, and of the general absence of standards, rather than a particular comment on Bristol. [7] In Bristol, there was no survey of the clinical work of the hospital as a whole, and thus no way to gauge its performance in the complete range of its activities. The Annual Reports of the UBHT's Medical (later Clinical) Audit Committee, while they were sent to the Region and to the Chief Executive, did not go to the District, nor (before late 1995) to the UBHT Board. Moreover, the audit process addressed only a small selection of topics, all of which were chosen by the clinicians. But this was a product of the system and not something peculiar to Bristol. Audit in the 1990s had simply become a system principally driven by concerns to maintain funding and hampered by the imperatives of the internal market. It is no surprise that the current Chief Medical Officer (CMO) for England commented in a paper in 1998:
`Although the concept of peer review is well established in the United Kingdom, the implementation of clinical audit in the NHS is not a complete success. Concerns have focused on the failure of audit processes to detect and moderate significant clinical failure; on incomplete participation... on the lack of connection and flow of information to those responsible for managing services; substantial decline to the amount of regional audit; and on the value for money for what amounts to a significant annual investment.' [8]
8 It will be recalled that in Bristol topics for audit were selected by the clinicians. There was no mention of the PCS service in the Annual Reports of the Trust's Audit Committee of 1992 and 1993/94. In other words, the PCS service was not identified by the relevant directorate as a topic to be included in the Trust's report of its audit activity. Nor was there pressure from the District, since it disclaimed responsibility for requiring audit of paediatric cardiac surgery (PCS) on the under-1s, as this care was funded by the Supra Regional Advisory Service (SRSAG) until 1994. [9]
9 As regards the monitoring of the quality of performance of PCS on children under 1, we have seen that there was a significant confusion or lack of clarity about which, if any, body was responsible, which left a hole right at the centre of any efforts at quality assurance. In keeping with the assumptions of the time, no questions were raised about the ability of the Bristol Unit to undertake the surgery. It was simply assumed that the volume and throughput of cases would increase to some appropriate level, and that the quality of care would correspondingly improve. This was because there was a prevailing, although unproven, assumption that quality was directly related to volume. Retrospective statistical analysis commissioned by the Inquiry indicates that this assumption was not entirely unwarranted in that there is an association between mortality rates and volume. For open-heart operations on the under-1s, and for arterial Switches and Atrio-Ventricular Septal Defect (AVSD) operations in particular, centres undertaking a higher volume of operations had lower rates of mortality in the period 1991 to March 1995. [10]
10 Notwithstanding the lack of audit generally and of any systematic mechanism for monitoring performance, there was information about PCS available to the clinicians in Bristol. There was the data from the UK Cardiac Surgical Register (UKCSR). There were the surgeons' logs and the data collected by the cardiologists (the South West Congenital Heart Register). Meetings were regularly held: the Clinical Pathology Conference to review the care of any patient who had died; the Surgical Audit meetings to review data on mortality and morbidity; the Paediatric Club to discuss detailed statistical summaries; and the Paediatric Cardiology meetings where results in PCS generally and in specific procedures were discussed. But all the data were seen in isolation. First, there were no agreed standards. Secondly, while data were compared with the UKCSR, as we have seen this was regarded as of limited value as a tool for benchmarking. [11] Furthermore, crucially as regards the neo-natal Switch operation, the UKCSR used categories based on diagnosis rather than procedure, so that meaningful comparisons were difficult.
11 Moreover, the difficulty in obtaining any reliable data from which conclusions could be drawn was further compounded by the small number of paediatric open-heart operations carried out in Bristol in each of the various diagnostic categories. This meant that, to obtain a large enough series of cases for statistical testing, results from a number of years had to be run together. But by doing this it could be claimed for some considerable time that no trend in outcome could be discerned.
12 The inevitable conclusion, therefore, is that, while there may have been a considerable amount of data around, indeed as we have said that Bristol was awash with data, these data offered little by way of real information whereby the BRI surgeons could assess their performance. Certainly, it would have been difficult at the time unequivocally to assess their performance against national figures. It follows that it would have been difficult, purely on the basis of statistical evidence, to reach an unequivocal view on the adequacy of the PCS service. It is true that, as our Experts comment, [12] by applying 3-year rolling chi-squared tests [13] to their own data on open-heart surgery on the under-1s and comparing it to the UK figures, the Bristol clinicians would have identified significant statistical differences had they run together their data into groupings covering 3 years. Such aggregation would have been necessary as the figures for any one year would not have been sufficient to allow any conclusion to be drawn. But they did not have the mindset to undertake such analysis, preferring to believe that things would get better. The Unit also produced a series of three Annual Reports (for 1987, 1988 and 1989-1990) on paediatric cardiology and cardiac surgery at Bristol and made them widely available within the UBH, and, at least as regards the 1987 Report, within the District. The Reports contained figures on open and closed procedures and deaths at Bristol, broken down by age group. The 1989-1990 Report also gave the UK figures for 1988, derived from the reports in the UKCSR, for comparison. The figures on mortality for open-heart surgery in under-1s contained in the 1989-1990 Report, when compared with the national data, showed Bristol's performance to be noticeably worse.
13 We asked a group of our Experts to advise us about the Annual Reports of 1987, 1988 and 1989-1990. We asked specifically: `What, if any, course(s) of action would you reasonably expect a clinician at Bristol to have taken in the light of these clinical audit data and statistical analyses?' Mr Leslie Hamilton in his reply [14] made the point that `... no clinician would have used the mortality figure from the UK Register as being an accurate model, recognising that deaths are likely to be underreported.' He stated that while he would not have expected the clinicians in Bristol to focus solely on `the absolute mortality rate ... I would expect that the team would look at the management of the patients who died particularly carefully and see if any particular diagnostic group was implicated.' As regards any course of action to be taken, he stated that the `... most realistic option is for the surgeon (and other staff) to go and visit another unit to observe practice and hopefully pick up "tips" that can be used to improve practice.' He continued that `... in 1999 a unit whose results did not improve with these attempts would stop carrying out that procedure. However in the late 80s I believe that the mindset in all units would have been to persevere in making changes in the hope that things would improve.' Dr Bull [15] advised us that `... once the 1989 report was published it could have been formally discerned that the mortality rate for open-heart surgery in infancy in 1988 was excessive in comparison to UKCSR.' This, in her view, meant that `... further exploration of the numbers was necessary internally within the department.' Dr Macrae [16] took the view that `... the failure of the Bristol annual reports to demonstrate an improving mortality should have raised concerns and led to discussions within the unit about the reason for this, and any necessary remedial action, given that in the wider UK context (I note that the UK figures are appended to the 1988 - '90 reports) mortality for open heart surgery under one year of age was approximately half of that reported in Bristol.' Dr Houston agreed with Mr Leslie Hamilton that the individual types of procedure should have been examined and that any further response should have depended on the outcome.
14 It could be said that the clinicians in Bristol did take action of the kind suggested. In time, Mr Dhasmana, as we have seen, went to Birmingham with his anaesthetists to observe Mr Brawn's surgery. Meetings were held at which results generally and in specific procedures were discussed. Both Mr Wisheart and Mr Dhasmana ultimately, albeit much later, ceased to carry out particular procedures (AVSD in Mr Wisheart's case and neonatal Switch in Mr Dhasmana's). But the Unit continued to carry out PCS. We come back to mindset, which Mr Leslie Hamilton stressed. The data even when examined still needed a judgment to be taken. That judgment, as Mr Hamilton indicated, was to behave according to the mindset of the time, `to persevere ... in the hope that things would improve.' [17]
15 The Bristol clinicians might also have compared their work with that reported in scientific meetings and publications (for example, Professor de Leval told us that, at a conference in Boston in 1991, he learned that intervention at an earlier age than had previously been the practice in cases of AVSD increased the likelihood of a successful outcome. Subsequently, he changed his practice at Great Ormond Street Hospital and the mortality rate for AVSD fell dramatically). But it was well known that professional journals and meetings tended to report the good results from the larger centres, such that these statistics might not give a true picture of the range of results encountered in practice. One publication in a journal, however, which was relevant to the PCS service was the paper written by two pathologists actually in Bristol. They had identified problems in surgery associated with cardiac anomalies not diagnosed prior to surgery. Their work appeared to go unremarked. [18] Professional meetings did, however, serve other purposes. It was at a meeting of the British Paediatric Cardiac Association (BPCA) in November 1992 that Mr Dhasmana mentioned his problems in relation to the neonatal Switch operation to one of the paediatric cardiologists at the meeting who recommended Mr William Brawn at Birmingham as someone who might help him. Mr Brawn and Mr Babulal Sethia were also at the meeting; Mr Dhasmana spoke to them both there and subsequently went to watch Mr Brawn operating at Birmingham.
16 A further source of data throughout the period of our Terms of Reference should be mentioned. This was the hospital administrative system, Patient Administration System (PAS), from which data were fed into the national system, Hospital Episode Statistics (HES). PAS was seen by clinicians generally as a system produced by clerical staff for administrative purposes. The clinicians thought that it was unreliable, not least since the coding [19] was carried out by administrators rather than clinicians. Clinicians only trusted their own data. Thus it did not appear to occur to clinicians to use the HES data as a means of developing views on standards and performance. That said, it would have been difficult to do in any event, since access to the HES data was controlled by the DoH, and the use of large databases such as HES requires specialist skills which most clinicians would not have had. But, as our Experts point out, these data could have served as a good source of information. [20]
17 In the light of our examination of audit generally and of the data available at that time to the clinicians in Bristol, we reach the following view. The surgeons were working in a relatively new and developing field of highly complex surgery. They were dealing with small numbers of disparate congenital cardiac anomalies. Perhaps unsurprisingly, they tended to turn to their own logs of operations as the most detailed, relevant and reliable sources of data. In these logs they saw a pattern of complex cases. In this hard-pressed service, which was attempting to offer the full range of specialist care to these children, as well as meeting all the other needs of a cardiac surgical unit, the poor results achieved were believed then, and are still believed, by Mr Wisheart to be the result of this pattern of complex cases, the result of caring for an unusually high proportion of unusually difficult cases. [21]
18 Despite what has been said, it could be argued that, by around 1990, simple statistical comparisons with the available national data for 1988-1989 and 1989-1990 might have suggested, at the very least, that the clinicians in the Bristol Unit should have been asking themselves questions about the Bristol Unit's performance in open-heart surgery on the under-1s. An analysis could have suggested a pattern of less than adequate performance. This was certainly the case as regards the comparison between the Annual Report on PCS for 1989-1990 and the UK figures. But, given the time lag in the distribution of this national data and in the preparation of the Annual Reports on PCS, the national data would have reached Bristol towards the end of 1990. This was just the time when the first sense of the results in Bristol for 1990 suggested that things were improving: that a corner had been turned. This was confirmed when the results in Bristol became known in mid-1991. As our Experts advise in their `Statistical Overview':
`... it is not until the data for 1988 were included that the divergence from the national rates became statistically significant, and this was reinforced by the data for 1989. Given that there was a delay of the order of 18 months before the UKCSR data were fed back to centres, it would have been 1990 before the data from the UKCSR might have given any reason for concern, and the independent reinforcement for the 1989 data, which would become available during 1991, would have heightened this concern. However, the data for 1990 then came back into line with national figures ... which might have been taken as reassurance that any problems which might have existed previously had been resolved.' [22]
Thus concerns arising in 1990 were understandably assuaged by the good results for that year. Sadly, these good results turned out to be the exception. But even thereafter, other explanations, particularly low numbers from which statistically significant conclusions could not be drawn and what was thought to be a run of unusually complex cases, were called upon.
19 We point out again that the PCS service in Bristol was subjected to review. Data were available and they were discussed. They lent themselves, however, to a variety of interpretations, not all of which pointed to poor performance. It is a failure in the system which existed in Bristol, as well as a matter of the greatest regret, that the data were rarely considered by all members of the team together. This could well have led to a more rounded assessment of whether there were problems and where they lay. Moreover, there was a reluctance to challenge interpretations of the data and to look for alternative explanations, both of which attitudes militated against any better understanding of what was happening. This reluctance, while at one level human and understandable, at another level constitutes a further factor contributing to make the PCS service less than adequate. It highlights the fact that there were no effective systematic mechanisms within the UBH/T as a whole, or beyond Bristol, to evaluate and assess performance. Thus the clinicians, having satisfied themselves that matters were improving, or that there were what to them were good reasons why they were not improving, were not accountable to anyone else.
<< previous | next >> | back to top
Footnotes
[1] UBHT 0024 0076. Dr Thomas rejected the idea that there was confusion; T62 p.111 Dr Thomas
[5] WIT 0108 0019 Dr Roylance
[6] T28 p.98 Sir Barry Jackson
[8] Scally G, Donaldson LJ. `Clinical Governance and the Drive for Quality Improvement in the New NHS in England' `BMJ' l998; 317:61-65
[11] WIT 0085 0052 Mr Dhasmana
[12] See Annex B, 4a: Spiegelhalter D, et al. `Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995'. September 2000
[13] The chi-squared test is a standard procedure for comparing two proportions, for example mortality rates at two different units. It is possible to assess how likely it is that any observed difference between the two proportions could have arisen by chance, in a situation where the true underlying rates are identical. Once chance has been eliminated then conventionally this is interpreted as evidence of a genuine difference between the true underlying proportions
[14] Hamilton L. `Expert Comment on the Bristol PCS Annual Reports for 1987'; 1988 and 1989/90', Annex B, 10f
[15] Bull C. `Expert Comment on the Bristol PCS Annual Reports for 1987'; 1988 and 1989/90', Annex B, 10i
[16] Macrae D. `Expert Comment on the Bristol PCS Annual Reports for 1987'; 1988 and 1989/90', Annex B, 10g
[17] Hamilton L. `Expert Comment on the Bristol PCS Annual Reports for 1987'; 1988 and 1989/90', Annex B , 10f
[18] Russell GA, Berry PJ Postmortem audit in a paediatric cardiology unit. J Clin Pathol 1989; 42: 912-918.
[19] Coding is a procedure whereby information from patients' clinical records is recorded according to agreed codes
[20] See Annex B, 6a: Evans SJW. `A report on local data relating to children who received cardiac surgery under the terms of reference of the Bristol Royal Infirmary Inquiry', October 1999
[21] WIT 0120 0298; T92 p.104-5 Mr Wisheart
[22] See Annex B, 4a: Spiegelhalter D, et al. `Overview of statistical evidence presented to the Bristol Royal Infirmary Inquiry concerning the nature and outcomes of paediatric cardiac surgical services at Bristol relative to other specialist centres from 1984 to 1995', September 2000,