What is Accuracy?

As Arbiter Systems^®, Inc. has evolved from a metrology company a decade ago into a company providing measurement and control equipment to the electric utility industry, we have become aware of a large gulf of misunderstanding about the meaning of the term 'accuracy.' In its most fundamental definition, accuracy is simply how close to a true value an instrument will read when measuring a signal or other physical quantity.

However, that is just the tip of the iceberg, and lots of other issues start to appear when you look more closely into this topic. We have observed that many competitor's products (as reported by our customers) do not meet their accuracy specifications, at least not all of the time. This is because of the methodology used to determine or state the accuracy specifications.

Guaranteed vs. Typical

One of the first misconceptions many customers have is that any number which appears on a data sheet is a guaranteed specification. Some people even believe that this should be required! The fact is that there are two different types of "specification": guaranteed and typical.

A guaranteed, or warranted, specification is a number that (hopefully) the manufacturer has determined with a complete error analysis, accounting for all of the sources of uncertainty which can affect performance (see below). This is usually desired for the main specifications of a product. The methods used to calculate the error analysis are discussed in more detail below. Normally, each of these characteristics of a product is measured in the production process to verify compliance. In some cases, lot testing, "guaranteed by design," or derived data (calculated from other measured values) can be used, provided that a justification is made demonstrating why this method is valid.

For guaranteed specification, you would expect to measure actual numbers well within the specified range almost all the time, under any conditions within the specified operating range. Nothing is absolute, however; even a six-sigma specification will theoretically be exceeded occasionally. In practice, however, measurements will typically be within a small fraction of the specification. Such a specification is also sometimes referred to as a worst-case specification: even with the worst case combination of parameters which can be expected to occur, the measurement will still be within the specification.

A typical specification is just that. In rough terms, it is an average expected value, based on a number of measurements or units of the product. You would expect to measure actual numbers above and below this specification each roughly half of the time.

A well-specified product clearly states which numbers are guaranteed and which are typical. It is possible to have all of the numbers either typical or guaranteed, but this is not likely. Furthermore, once you know which are typical and which are guaranteed specifications, you can apply the laws of probability and statistics in a meaningful way to make predictions about expected system performance.

Sources of Uncertainty

Metrologists, that is, people engaged in the science of metrology or measurement, generally use the term "uncertainty" rather than "error" when talking about a measurement process. "Error" means "mistake," i.e. something done wrong. "Uncertainty" means the amount by which a measured value deviates from the true value.

Measurement uncertainty is caused by a number of factors. Some of these are: initial calibration uncertainty, temperature sensitivity, power supply variations, sensitivity to signal conditions, measurement noise, and drifts over time. Properly determined, a guaranteed specification, which takes all of this into account, will ensure measurement results which are within a desired window. The designer of the equipment or system must take all of these factors into account when designing the product to maximize its performance. The results of the analysis done by the designer form the basis to determine the guaranteed specifications at a later date.

Error Analysis

Error analysis is the fundamental tool of an engineer or metrologist when (s)he is considering measurement uncertainty. (In keeping with the definitions above, it should probably be called "uncertainty analysis," but the term "error analysis" has been used for so long that it is difficult to change.) Error analysis involves two steps.

First, one must identify and quantify the sources of uncertainty in the system being designed. Every component can in theory have an effect on performance, although in practice with a well-designed system or product, the number of factors to be included is not too large. Small variations in the value of a pull-up resistor, for example, would have a negligible effect on most systems and can normally be safely ignored. Many components contribute uncertainty from more than one mechanism. A voltage reference, for example, contributes to initial setting uncertainty, noise, temperature drifts, and drifts over time.

It is possible by calibrating the product or system after manufacture to eliminate some of these uncertainties. In this example, measuring the voltage of the reference (either directly or indirectly) can eliminate the initial setting uncertainty from the equation. Put another way, if there is a means to make the necessary corrections, a voltage reference can have a fairly large setting uncertainty without affecting overall performance. This can be done with a hardware adjustment, although more recently, software calibration has come into vogue, largely because it is "free," but also because mechanical adjustments can change over time, particularly when subjected to mechanical shock and vibration.

Second, you have to determine how to combine the various uncertainties to get an estimate of expected performance. There are different ways to do this, each of which might be appropriate under different circumstances. Two common methods are discussed in the following sections.

Worst-Case Analysis

Worst-case analysis consists of simply adding up all of the individual uncertainties to get the expected level of performance. This is the most conservative approach, but it also gives the largest possible expected uncertainty values. In most circumstances, a more sophisticated approach such as RSS, discussed next, gives a more realistic estimate. There are conditions, however, where worst-case analysis is appropriate. These include when you are working with typical, rather than guaranteed, specifications, and when you suspect that many of the uncertainties may be correlated (see discussion below).

Root-Sum-of-Squares (RSS) Analysis

In RSS analysis, each of the errors is squared. Then, they are added together, and the square root is taken. This (the square root of the sum of the squares) is the estimated uncertainty of the measurement. RSS analysis is appropriate when: (1) there are several sources of uncertainty; (2) no one source dominates; and (3) the uncertainties are not correlated (see below).

This method is based on the fact that when you add together two distributions of random (normally-distributed or Gaussian) measurements, the standard deviation of the resulting distribution is equal to the square root of the sum of the squares of the standard deviations of the initial distributions. Or, the variance (which is the square of the standard deviation) is equal to the sum of the variances of the initial distributions. This is a well-proven characteristic of normally-distributed (random), independent (uncorrelated) data sets. It does not hold true in the general sense if either of these conditions are violated, although it is possible to perform a more sophisticated statistical analysis to determine the expected uncertainty in these circumstances. In many cases as it turns out, non-normal distributions (one-sided distributions, in most cases) can in fact be treated pretty much the same as in the ideal case.

RSS analysis is most appropriate for use with guaranteed, worst-case specifications. Experience has shown that in these cases (with allowances for correlated terms) the RSS method yields the most reasonable result.

Correlation - What Is It?

Sources of uncertainty are said to be correlated if they track each other. What this means is that if one is going up, then the other is most likely to be going up (or down) too. Statisticians have developed very sophisticated methods (called analysis of variance, sometimes abbreviated ANOVA) to figure out just how correlated two different quantities are. This analysis ends up with a number, called the correlation coefficient. The correlation coefficient can be any number between 1 and 0. A correlation coefficient of 1 implies perfect correlation, or in other words, whenever one quantity moves up the other moves up or down in an exact proportional relationship. An example of two quantities which are perfectly correlated is the voltage and current in a resistive circuit. A correlation coefficient of zero implies that there is no relationship between the two quantities, at least as far as can be determined from the data analyzed.

Correlation does not necessarily imply causality. This is actually of much more interest in the social sciences, where public policy decisions are sometimes based in error on the simplistic assumption of causality when correlation is identified. An example might be where a study showed that there was a fairly strong correlation between the budget of a school district and the performance of students. One could leap to the (possibly incorrect) conclusion that spending more money will improve schools. A more comprehensive study, however, might show that there was an even stronger correlation to parental involvement in the school system, and that both of these (school budgets and parental involvement) were in turn strongly correlated to some factor like community income. Social scientists formulate very sophisticated studies in an attempt to determine causality among numerous correlated factors. They might find, for example, that the factor which really affects student performance is parental involvement, not school budgets or community income. This would be good news, because increasing community income or school budgets in an attempt to increase student performance would not only be politically difficult, but would fail! Increasing parental involvement, however, is something that could be implemented in any community.

In the physical sciences, and in metrology in particular, causality is usually more easily determined. Unlike human behavior, which is incredibly complex and often poorly understood, it is an engineer's job to understand the factors which affect a system's performance. It is rare to have correlation without causality, although not impossible. The most common cause of correlation is where a single physical quantity (let's say voltage reference accuracy) affects two or more of the uncertainties in a measured quantity (for example, voltage measurement uncertainty and current measurement uncertainty, both part of power measurement uncertainty). In this case, if one was measuring a power quantity (watts, VARs, Wh, etc.) the voltage reference uncertainly would affect both the current and voltage component of the power measurement in exactly the same way! Fortunately, the existence of such a situation does not preclude the use of the RSS method. See the next paragraph. (Interestingly, if a measurement is the ratio of voltage and current, for example impedance, the voltage reference uncertainty will cancel. Taking advantage of such cancellations wherever possible is an important part of high-performance system design.)

To proceed when you have correlated quantities, do the following: first add the correlated uncertainties together before squaring. Only those items correlated to each other should be added together before squaring. If there are two groups of correlated items, i.e. A and B are correlated to each other, and C and D are correlated to each other, but neither A nor B is correlated to C or D, then you would add A and B together, square, and add to the sum, and then add C and D together, square, and add to the sum.

What About Accuracy Class?

In the power industry, it has become common to assemble metering and instrumentation systems using components of a certain "accuracy class." One then draws the (unjustified) conclusion that the metering accuracy is equal to the accuracy class. This is probably wrong. The only way to know for sure is to perform a detailed error analysis, as described above. In general, however, if you have several components in a metering setup that are of the same class, the overall performance would be expected to be worse than the specified accuracy class due to the contributions from more than one component affecting uncertainty as described above. If, for example, there are three items affecting the uncertainty (watthour meter, PT, and CT), then you might expect the overall uncertainty to be equal to the accuracy class multiplied by the square root of three. If you have a class 0.2 installation, then the expected uncertainty would be 0.35 %.

Example Calculation

Example error analysis: Model 1133A Power Sentinel^™ Power Measurement Uncertainty

Temperature uncertainties (0 °C to 50 °C, relative to 25 °C):

Source of Uncertainty	Specified Value	Extended	Squared
Current input resistor	2.5 ppm/K	62.5 ppm	3906
Current transformer	1 ppm/K	25 ppm	625
Voltage input resistor network	2 ppm/K	50 ppm	2500
Voltage reference	1 ppm/K (*2)Note #1	50 ppm	2500
Cal. source resistor network	0.5 ppm/K (*2)Note #1	25 ppm	625

Aging uncertainties (for one year):

Source of Uncertainty	Specified Value	Extended	Squared
Current input resistor	25 ppm/year	25 ppm	625
Voltage input resistor network	<20 ppm/year	20 ppm	400
Voltage reference	6 ppm/(khr)½ (*2)Note #1	36 ppm	1296
Cal. source resistor network	<20 ppm/year (*2)Note #1	40 ppm	1600

Calibration artifact (Rotek MSB-001):

Source of Uncertainty	Specified Value	Extended	Squared
Transfer accuracy	50 ppm	50 ppm	2500
NIST traceability	50 ppm	50 ppm	2500

Other factors:

Source of Uncertainty	Specified Value	Extended	Squared
Measurement noise	10 ppm	10 ppm	100
Range linearity	20 ppm	20 ppm	400

Overall uncertainty:

Source of Uncertainty	Specified Value	Extended	Squared
Overall uncertainty:		140 ppm	19577
Specification		250 ppm (0.025 %)
Margin		110 ppm

^{Note #1} Correlated uncertainty affecting both current and voltage measurement

« Back to Advancing the Technology Index