Verifying quality and safety in health informatics services
BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7312.552 (Published 08 September 2001) Cite this as: BMJ 2001;323:552
- Michael Rigby, senior lecturer (m.j.rigby{at}keele.ac.uk)a,
- Jari Forsström, directorb,
- Ruth Roberts, lecturerc,
- a Centre for Health Planning and Management, Keele University, Keele ST5 5BG
- b Medical Informatics Research Centre in Turku, University of Turku, FIN-20520 Turku, Finland
- c School of Postgraduate Studies in Medical and Health Care, University of Wales Swansea, Swansea SA2 8PP
- d Knowledge Management Unit, School of Public Policy, University College London, London WC1H 9QU
- Correspondence to: M Rigby
Information and its handling and transmission form an essential part of health care and are reflected in professional standards. Automated information systems in health care—health informatics services—will improve these functions and bring new opportunities through the harnessing of modern information and communications technologies. Thus, computer support is now essential in many parts of medicine, the US Institute of Medicine has long espoused the value of computerised patient records,1 and many countries have developed strategies on this topic, and there are countless health related internet sites.
However, as new information and communication technologies in health bring new opportunities, they also bring new risks. Emphasis has rightly been placed on ensuring appropriate levels of confidentiality in electronic information systems—to the point that the highly exacting requirements being demanded by independent commentators and professional bodies2 are difficult to satisfy without jeopardising the functioning of core services 3 4 or the interests of the most vulnerable groups.5 In contrast, much less thought has been given so far to ensuring the appropriateness of the design and integrity of functioning of health informatics services.
Summary points
Like drugs 40 years ago, products in health informatics are unregulated with regard to safety and efficacy
A European project has now recommended ways of accrediting healthcare related software, telemedicine, and internet sites
A scheme like CE marking of electrical goods is recommended for software, national regulatory bodies should be identified for telemedicine, and a European certification of integrity scheme developed for websites
Importance of quality assurance of health informatics systems
If informatics systems are increasingly essential in the delivery of health care then their integrity and quality must be of equal importance, but this has been scarcely recognised to date. In 1963 the then UK secretary of state for health stated to the House of Commons: “The House and the public suddenly woke up to the fact that any … manufacturer could market any product, however inadequately t sted, however dangerous, without having to satisfy any independent body as to its efficacy and safety and the public was almost uniquely unprotected in this respect.”6 That statement related to drugs, being triggered by the thalidomide disaster, and the situation was changed rapidly. However, the same situation applies today with regard to electronic health informatics products and services, which are now the most important unregulated healthcare resource—in sharp contrast to drugs, medical devices, and licensed health professionals.
When errors and failures have occurred it has generally been in the interests of suppliers, provider organisations, and clinicians to quietly rectify or remove the flawed systems rather than draw attention to them. This, however, allows for unidentified and thus unquantified errors to be dispersed, with potential risk to patient health. Box 1 gives published examples of such health threatening errors in computer software. In a modern consumerist environment, however, this situation is unacceptable, as shown by the public furore over the software that miscalculated the risk of Down's syndrome in pregnancies.10
Box 1: Examples of health-threatening software errors
Errors in updated embedded clinical coding software giving false plain language representation of diagnoses, United Kingdom7
Errors in reference database calculation of Down's syndrome screening, giving false negatives8
Age cohort of women omitted from call up for cervical screening, Grampian Region, Scotland9
Error in software calculating risk of Down's syndrome led to falsely low calculation of risk for 150 women, Sheffield10
The TEAC-Health project
Recently, a European project—towards European accreditation and certification of telematics services in health (TEAC-Health)—was conducted to investigate the issues, and we report its core findings here. The findings outlined in the project report11 have recently been formally accepted by the European Commission, which intends to examine in detail the steps required for their implementation (Jean-Claude Healy, head of health applications unit, Information Society Directorate-General, personal communication, 2000).
The project arose from an expert conference at Turku University in 1997 organised by JF, which resulted in several published articles.12–14 The work of the project was undertaken by representatives of five European countries; details of the membership and working reports can be found on the Multimedica website.15
Classification of health informatics services
For the project, we classified health informatics services into three categories—software and related services, telemedicine, and internet sites. Although many services combine more than one of these elements, the quality assurance and regulatory components for each need to be considered separately as the issues are quite distinct. We also felt it inappropriate to consider processes of quality assurance and verification solely in the health sector and therefore looked at commercial approaches such as regulation in the financial sector and at other areas of public risk such as air traffic control and food safety.
In the health sector, precedents have been set in the regulation of drugs and medical devices, but neither of these is directly applicable to health informatics services. Safety control of new drugs now depends largely on controlled trials, which are neither feasible nor affordable as a mandatory control for clinical software or internet sites. Regulation of medical devices has several similarities, but key differences are the much wider range of user proficiency and circumstances of use of informatics systems compared with medical devices and the difficulties of ensuring structured user training and education.
A taxonomy of risk assessment
We next considered how best to categorise risk in health informatics services, as it is only by identifying risk that appropriate control methods can be identified. For medical devices, the regulations are clear and helpful: they require that a device's manufacturer or supplier identifies the risk level as determined by the type of product and how life critical are the circumstances of its use.16 We concluded that risk in health informatics services depends on a combination of type of user, circumstances of use, type of use, and nature of the system. For example, a failure in an automated appointments system can have serious consequences by passing undetected, whereas an experienced clinician may filter out spurious results from a diagnostic support tool used merely as an aide memoire. The table shows the different levels of risk associated with different health informatics services.
Levels of risk in the use of health telematic systems
Quantification of the problem
We sought to identify and quantify the risks attributable to informatics services, and the degree of concern they produced. A comprehensive literature search and a small targeted survey of European opinion leaders from health and consumer domains showed that the problem was, if anything, greater than anticipated.15
Clinical software
Many of the problems identified when using clinical software are resolved between supplier and user on condition that there is no publicity, while the problems that are not identified cannot, by definition, be reported. Thus the literature will substantially underestimate these problems, but some errors have been reported (see box 1), as has the adverse outcome of software upgrades producing erroneous printed interpretations of previously recorded diagnostic data.17
Telemedicine
Less has been published about the risks of telemedicine services because of their comparative newness. However, we identified concerns about authenticity and risks in telemedicine services, including email consultations, other than those within a single provider organisation or on a closed, point to point basis.15 There are indications that a quarter of those offering telemedicine consultations directly to the general public do not hold the qualifications they claim (S Schanz, personal communication, 2000), and others may be offering advice beyond their qualifications. Studies have shown there is wide variation in the quality of advice provided, and, although guidance may generally be sound, the occurrence of so many outliers is an unacceptable and avoidable risk. 18 19
Internet sites
Services on the world wide web are the most obvious risk, as anyone can publish any information they like. Much of this information is valuable and the internet allows freedom of expression for patient support groups and leaders in alternative therapies, but studies have shown that both misleading and life threatening advice is readily available. 20 21 A figure of 1400 “suspicious” websites was reported by the coordinator of a study for the G8 group of countries, with a 21% increase in that number annually,22 and a recent US study found errors and contradictions even within sites.23 Yet, by its very nature, the internet cannot be controlled or censored.
Project survey
Our survey of opinion leaders, for which we used a “snowball sample” method, yielded 54 respondents, of whom 36 (67%) indicated that they had experienced one or more problems with health telematics services. Of the 74 problems reported, 10 adversely affected patient safety, four adversely affected optimum treatment of a patient, and 31 adversely affected the health professional's duty of care to a patient. Of all the respondents, 19 were “very concerned” about the current lack of quality assurance of telematics services and a further 22 had some concerns, giving a total of 41 (76%) “concerned.”
TEAC-Health recommendations for clinical software
In view of the need to avoid identified risks to the public, and the professional opinion in favour of some form of regulation, we concluded that specially crafted regulation was needed based on existing European experience with product control and monitoring health risks. The components suggested are as follows.
CE marking
Applying this publicly understood and reliable mark on approved goods is a well established process in Europe based on clear regulation24 and with variants for medical devices.16 However, further research is needed on the specific criteria to accommodate clinical software. This will require a “notified body” to have overall responsibility and to identify and monitor essential requirements for these products and services. As concurrent verification of design and quality is far more effective than retrospective testing, the necessary identification of control measures for production and quality assurance will itself yield invaluable standards for clinical software developers.
Labelling
A legally underpinned requirement for accurate and detailed labelling is a key element of our proposed solution, as this will enable purchasing organisations and clinical users to know much more about the software product. Identification of named responsible individuals will also substantially increase the commitment to ensure quality of design and manufacture. The exact requirements will need further discussion and definition, but box 2 shows a suggested list.
Box 2: Suggested labelling requirements for clinical software
Country of origin
Identity of legal person or company responsible
Intended purpose (such as clinical advice, decision support, prescribing advice)
Competence of intended end user (such as general practitioner, endocrinology specialist, triage nurse)
Assumed knowledge of user (such as specific clinical qualification)
Identity and registration body of health professional responsible for supervising the clinical element of the design
Key sources of clinical logic or knowledge (such as citation of published material, authorship of in house clinical design)
Extent of previous use or in house testing of this version
“Hotline” telephone number for postmarketing surveillance
“Hotline” for postmarketing surveillance
An essential part of CE marking is postmarketing surveillance, in particular the requirement that the supplier provides a “hotline” telephone number to which any problem or concern can be reported. It is also a statutory requirement of CE marking that all serious incidents are reported by the supplier to a “competent authority,” and this process is liable to unannounced audit on site.
National hotlines and monitoring organisations
Based broadly on existing models for drug products and medical devices, national hotlines and monitoring organisations are necessary for clinical software to ensure that problems such as adverse interactions between different products (see box 1) can be identified speedily. They are of proved benefit for other clinical products and already apply to health software in Sweden.
In house software and informatics services
Software and services developed by particular healthcare organisations for their own use cannot readily be subjected to compulsory CE marking as they are not marketed products. However, our proposed regulation would bring two safeguards. Firstly, the identification of professional standards would form a yardstick for identifying reasonable practice and duty of care should there be a formal complaint or litigation. Secondly, in house products could be submitted voluntarily to the verification process.
TEAC-Health recommendations for telemedicine
Telemedicine presents an entirely different situation because telecommunications based services that cross legislative boundaries are almost free of regulation. Thus, providers of healthcare services could escape regulation, particularly when moving to the internet. Since this leaves patients at risk, some control mechanisms are needed. In principle, legislation should be independent of the communication medium used—namely, the same ethical principles and liabilities should apply to telemedicine as to conventional patient care. Because telemedicine services can readily cross international boundaries, international coordination or coregulation is needed in Europe and beyond. Similarly, in countries such as the United States regulation is at the state level, leading to complex and unwieldy situations that hamper legitimate national providers and thus also patients.
Key elements of regulating telemedicine services should include international agreement as to whether such services are delivered under the law of the supplier or that of the consumer. The European Permanent Committee of Physicians (EPCP) now favours accepting European law that it is the supplier's legal system that applies (Ä Markku, chairman, EPCP, personal communication, 2000). Secondly, labelling (as above) with legal sanctions should be required, linked to a code of conduct, which needs to be developed. Box 3 shows proposed key elements.
Box 3 : Key elements of proposed labelling requirements and code of conduct for telemedicine
Healthcare professionals should state their full name and qualifications
The professional body responsible for monitoring clinical practice must be identified
Records must be kept to an agreed standard, with the database maintained and protected according to European standards for data protection
Telemedicine traffic should be strongly encrypted
Telemedicine service providers should be required to register with a national agency for the provision of the services, related to international standards and qualifications which need to be developed
Services should be provided in accordance with stated technical standards (including those for equipment, telecommunication, and data interchange) together with stated practice standards (such as for image labelling and agreed terminology)
Global regulation
A global regulatory framework is also important. There are clear and effective global conventions and supervisory organisations for both civil aviation and food standards, both of which operate on an evidence based principle, obtaining and interpreting emergent scientific evidence in order to formulate new standards that then become the basis for universally agreed international regulation. Delivery of telemedicine services internationally puts individual patients at risk of injury or death through incompetent or malicious unregulated providers, but, because the transactions are individual and confidential, adverse outcomes are not as conspicuous as in domains such as civil aviation. The same situation applied to pharmaceutical products until regulation.6 The global risk to personal health continues unabated in the absence of international agreement on regulation, liability, and control. We consider international telemedicine to deserve at least the same level of regulation as the civil aviation and food sectors. This could also aid the development of national frameworks, especially in countries with largely independent states or provinces.
TEAC-Health recommendations for internet sites
We believe that the cost of developing a system solely to verify the quality of health internet sites would be high and that it would be impractical. The Health on the Net Foundation (HON) has for some time been promoting a voluntary code of conduct, and there have been several overlapping initiatives in the United States (see box 4), but their main drawback is that there is no external verification and so the system is open to abuse and, indeed, offers false security.
Box 4: Voluntary initiatives for codes of conduct for health internet sites
Health on the Net Foundation (HON). www.hon.ch
Swiss based organisation, European focus
Internet Health Coalition. www.ihealthcoalition.com
US based organisation, European input
American Medical Association
Principles governing AMA publications websites. pubs.ama-assn.org/ama_web.html
JAMA special communication. Guidelines for medical and health information sites on the internet. jama.ama-assn.org/issues/v283n12/ffull/jsc00054.html
Health Internet Ethics. www.hiethics.com
US based organisation
Quackwatch—Your Guide to Health Fraud, Quackery, and Intelligent Decisions. www.quackwatch.com
US based initiative
All sites accessed 20 June 2001
However, the need for independently verified sites is common to many other internet activities, including retailing.25 As with CE marking and other recognised quality standards, the power of effective regulation depends on the universality of use leading to public recognition. We studied earlier attempts to identify high quality sites to the public, the best known being filtering mechanisms and rating systems.11 Both have drawbacks.
Most filtering excludes inappropriate items but also excludes many relevant sites, as it is difficult to develop a 100% specific yet sensitive filter that does not filter out required material. For example, a filter designed to protect against pornography will exclude sites with the word “breast,” but it will also filter out important medical sites. Such “heuristic” filtering depends on finding and interpreting key words. The alternative, “filtering in,” requires the site to undertake self rating honestly and accurately.
Rating systems depend on third parties such as informed users to provide a rating and score for each individual site, but this raises questions of ensuring objectivity, impartiality, and common clinical and cultural values to the extent that there are now proposals for rating the raters. Moreover, this leaves most sites unrated. Clearly, these methods are not feasible to aid general public users, nor indeed most health professional users unfamiliar with the intricacies of the internet. Box 5 summarises the issues.
Box 5: Impediments to voluntary quality assurance for websites
Voluntary codes
No closed industrial or commercial grouping
Voluntary initiatives may reflect sponsors' interests or values
Enforcement and sanctions are difficult to apply
Consumer confusion with numerous initiatives
Filtering
Undiscriminating
May exclude relevant sites
Rating
Requires major expert resources
Imposes values of raters
Slow to cover new sites
Sites can change rapidly after rating
Monitoring or reporting apparently adverse sites
Cannot be comprehensive
Based on personal values
No action
Allows inaccurate (and malevolent) sites to remain unchallenged
Consumers continue to be at risk
The EuroSeal proposal
We have therefore proposed development of a new European system and standard, entitled the EuroSeal. 12 15 This would be a seal supplied to a website by an accredited agency (the approach fundamental to CE marking). Once attached to the site, its integrity would be verified by secure single socket layer or similar secure software, as currently happens with secure trading sites. The seal would be provided at two levels, the higher of which would require independent onsite verification (for a higher fee). The verification processes would be open and transparent—by clicking on the EuroSeal symbol, visitors to the site would see details of the site inspections, drawn in real time from the records of the accrediting body (as applies with current secure links for web commerce), as well as the code(s) of conduct to which the site adhered.
Codes of conduct
These are an important element of the EuroSeal approach, as they would form the basis on which the third party assessed a site provider's claims and decided whether to award the EuroSeal. Each health professional body would be able to devise its own codes of conduct and standards, and viewers would know against which code the EuroSeal had been applied. This approach would also allow special interest groups—such as ethnic groups, those with particular religious beliefs, and advocates of alternative medicine—to devise their own codes of conduct. Patient support groups could also devise codes of conduct, provided they met a prescribed framework and standard for codes.
Thus, the EuroSeal approach would not only provide a simple, clear, and universal public safeguard without seeking censorship but would also be socially progressive, enabling positive support and selection for special interest groups and minorities. As a mark of high integrity, it would be sought after by sites and looked for by search or filter by viewers.
Conclusions
Health informatics systems are invaluable to aid health care. Moreover, they bring intrinsic advantages, such as electronic records being more accessible than paper ones and, if properly protected and encrypted, being more secure from damage or prying. However, this is no excuse not to address current known and avoidable risks.
The TEAC-Health project has clearly shown that public safety and professional integrity are threatened by the lack of regulation of health informatics services. These risks will increase rapidly as health informatics services expand and as telecommunications and globalisation radically change attitudes to and delivery of health care. 26 27 Initiatives to date have been based on restricted research, lacked consideration of overall feasibility and other issues, or depend on the (usually unpublished) integrity and values of a secondary service provider. The strategic proposals we describe, which have now been welcomed by the European Commission, form an evidence based solution.
Footnotes
-
Competing interests JW has a small part of the equity of Medix, an internet service provider for doctors, and receives research and consultancy funding from various commercial sources.