What opportunities does digitalization offer for better measures of output quality in the public sector? This question is the focus of what follows – the second in a series examining the extent to which digitalization can improve performance measurement in government.
Let’s first recap the discussion so far. Digitalization means the use of digital tools to support or automate government business processes, including service delivery. Digitalization makes administrative data – that is, data on activities carried out and other information routinely collected during service delivery – available in a digital form that can be readily drawn upon for performance measurement purposes. This can considerably facilitate performance measurement, particularly in the development of output and intermediate service indicators. The ability of digitalization to enhance the measurement of the effectiveness of government is, however, limited by the fact that information on many important outcomes is not collected as part of service delivery and is therefore not part of the administrative data the accessibility of which is greatly improved by digitalization.
Output quality indicators become important at this point because they also shed light on the effectiveness of government services. We therefore need to ask whether digitalization can help deliver better output quality indicators. The answer given to this question here is “yes” – to a certain extent. More specifically, there is a significant group of public services for which digitalization facilitates the measurement of certain dimensions of output quality. However, good output quality measurement cannot rely only upon digitized administrative data. Other data sources and analytic methods must also play an important role.
In discussing this matter, it is essential to avoid the widespread confusion about the meaning of “output quality.” Output quality is not the same as outcomes. For example, the medical treatment offered to a car accident victim may be of the highest quality, but the patient may still die. Similarly, teaching may be high quality, but some students in the class may still fail their exams. Delivering a high-quality output does not, in other words, guarantee that the intended outcome will be achieved. There is, nevertheless, a direct relationship between quality and outcomes: namely, the higher quality the output, the more likely it is that the intended outcomes will be achieved. Quality directly increases the effectiveness of government services. This makes output quality indicators a very important complement to outcome indicators when gauging the effectiveness of government services.
There are three key areas where digitalization can assist in the measurement of output quality. These are the provision of timeliness indicators, compliance-with-standards indicators, and client satisfaction indicators.
Timeliness indicators are measures of how promptly a time-sensitive service is delivered: for example, how long it takes ambulances on average to arrive after being called, or how long patients are obliged to wait before receiving needed hospital treatment. Digitalization makes timeliness indicators easily and quickly available.
Compliance-with-standards indicators require a little more explanation. There is a subset of government services for which there are clearly defined standards of what constitutes a satisfactory service – more specifically, of the activities that should be carried out as part of the service and which, if not carried out, mean that the quality of the service was poor or inadequate. For example, the failure to administer anticoagulant drugs to patients after surgery is clearly-recognized bad practice. In the domain of antenatal care, there are widely-accepted standards which identify activities which should be carried out during antenatal care consultations (e.g. testing urine for proteinuria, measuring blood pressure) — activities which, if omitted, unambiguously mean that the care was of poor quality.
For services where there are clearly-defined best-practice standards specifying activities which should be carried out, a valuable performance indicator is the percentage of cases in which these activities were carried out. In principle, the indicator value should be 100%. In practice, it is often less than this, and if this is the case it is important to be aware of the problem. Digitalization can help greatly in the provision of compliance-with-standards indicators. If the service delivery staff are required to record digitally the activities they carry out when delivering the relevant outputs, the data required to report compliance-with-standards indicators becomes immediately available.
Finally, there are client satisfaction indicators – indicators which can, depending on how they are framed, provide information on either or both output quality and outcomes. Digitalization helps us measure client satisfaction by making it easy to carry out digital client satisfaction surveys (e.g. via email) – something which is today widely done in both the public and private sectors.
We must not, however, make the mistake of thinking that facilitated access to administrative data through digitalization gives us everything we need to measure output quality. Far from it.
There are, in the first place, major limitations to the extent to which output quality in the public sector can be measured via client satisfaction indicators. There are many government outputs which are not delivered directly to specific clients who can be asked how satisfied they are with the service. Moreover, for services which are delivered to specific clients, it is in significant instances the case that the client does not have the necessary knowledge to fully assess output quality. This is, for example, the case for much medical care, where the patient often has limited ability to assess the quality of the treatment they receive. Patient satisfaction surveys remain useful but provide only a partial perspective on the quality of medical treatment.
This is why expert quality assessments should play an important role in measuring quality in many areas of government. In a medical context, for example, peer assessments of the quality of treatment received by a sample of patients can be used to generate valuable quality performance indicators. The same approach has applications in many other areas – for example, tax administration, where expert review of a sample of tax files can form the basis of quality indicators. New York City sends inspectors out to city parks to do formal ratings of the quality of the park maintenance work done by city staff and contractors – and publishes indicators based on these ratings.
Even with respect to customer satisfaction, placing too much reliance on digital tools would be a major mistake. Client surveys – particularly online surveys – have well-recognized limitations, and it is well-known that to establish what clients really think it is often important to use other instruments such as structured interviews and even focus groups. Useful client satisfaction indicators may be derived from these data sources as well.
The benefits derived from digitalization are also limited by the facts that (1) that compliance-with-standards indicators are not relevant for many government outputs, and (2) timeliness indicators are considerably less relevant for some services (the less time-sensitive ones) than others.
Measuring output quality is something which many governments have not been good at. Digitalization provides one means of helping them do better. However, much of the data which is needed to do a good job of assessing output quality is not administrative data that can be accessed through digitalization.
This points to a clear conclusion on the potential contribution of digitalization to the measurement of government effectiveness, whether in the form of outcome indicators or output quality indicators. This is that, although digitalization can indeed help in a number of ways, its potential contribution has major limits. Measuring effectiveness adequately requires that we go well beyond administrative data.