October 2005

Here’s a question that I get asked again and again – What’s the difference between the quality of a test and the performance of a method? Aren’t they the same thing?

In general, test quality has to do with the correctness of a test result, whereas method performance has to do with the expected variability of a test result. While the two are related, they are not the same. Here’s why and also what you should do about it!

Examine the assumptions!

Method performance characteristics, such as precision and accuracy, are determined with experimental studies, such as a replication experiment and a comparison of methods experiment. These experiments are usually carried out under conditions where the method is performing optimally, meaning that there are not supposed to be any problems occurring during the collection of these data. If problems do occur, then the data during those runs are discarded because they no longer represent the stable performance of the method.

A manufacturer’s claims for precision and accuracy represent the method’s performance when it is working properly, i.e., under conditions of stable operation.

Test quality, on the other hand, depends on method performance under routine operating conditions where problems may occur and cause errors in the test results. To guard against possible errors during routine operation, the laboratory applies statistical QC to monitor the stability of the method and hopefully detect any changes from earlier observations of the stable performance of the method. Therefore, test quality depends on method performance AND laboratory QC.

In short, a manufacturer’s claims for method performance reflect stable operating conditions, whereas test quality may be affected by unstable operating conditions. According to current FDA guidelines for product approval, manufacturers need only make claims for precision and accuracy, not for QC, and not for test quality. Therefore, a manufacturer’s claims for method performance may not reflect the actual quality of the test results produced during the routine operation of the method.

Consider the truth about quality!

I often use the analogy that quality is like truth, which is a complex variable that has multiple dimensions. We know that the standard for evidence in a courtroom is to “tell the truth, the whole truth, and nothing but the truth.” That’s also good advice for a laboratory test if the results are to provide reliable evidence for patient diagnoses and treatment.

The truth analogy can also help us understand the relationship between quality, method performance, and QC. Think about “truth” as quality, the “whole truth” as method performance, and “nothing but the truth” as QC. Now we have a good model for understanding the relationship between quality, method performance, and quality control. For a test to provide reliable evidence, we must be concerned with the quality under routine operating conditions, which depends on the expected performance of the method, and the capability of the QC procedure to detect any cases of unstable performance.

If we want to be sure that laboratory tests won’t mislead the physician, we must be able to detect conditions of unstable performance. That requires that we perform the right QC to be able to detect medically important errors. If we don’t, then it is possible that a test result might be incorrect, be misinterpreted, and cause mistreatment of the patient. QC prevents confounding conditions that may cause erroneous and misleading results.

Recognize who’s responsible!

According to the CLIA regulations, the laboratory is responsible for having a QC procedure to monitor a method’s precision and accuracy and to detect immediate errors that may be caused by operator performance, adverse environmental conditions, and test system failures. Manufacturers are off the hook here, since they generally do not provide detailed guidance for the QC procedure. Instead, they default to the CLIA minimum requirement of running at least two controls every 24 hours.

That leaves the laboratory with the responsibility for test quality – like it or not! Manufacturers sometimes demean the importance of QC because it only detects problems rather than prevents problems (which would improve quality rather than just controlling quality). Actually, corrective actions should follow detection, and should lead to prevention of future problems, but laboratories usually have very limited capabilities to implement preventive changes with today’s highly automated analytic systems. Test quality depends on the performance capabilities provided by the manufacturer, plus the manufacturer’s diligence in making improvements that prevent problems from occurring. The laboratory’s role is to manage the quality of the available analytic systems, which means being responsible for QC, taking the corrective actions that are possible, and alerting manufacturers when improvements are needed.

Identify a process for selecting appropriate QC procedures!

To be sure that test quality is adequate for patient care, the starting point is to define the quality that must be achieved, then assess the precision and accuracy available from the method, and finally determine the control rules and number of control measurements needed to assure the desired quality will be achieved in routine operation. Here's a brief outline of this process (for more detail, see here):

For the quality requirement, begin with the CLIA criteria for acceptable performance in proficiency testing, which define the allowable total error (TE_a) for 70 to 80 different tests. These are the default user requirements for quality and represent the minimum level of quality that must be achieved in US laboratories to stay in business and serve their customers.


Precision, in the form of a CV, can be estimated initially from a replication experiment; bias can be estimated initially from a comparison of methods experiment. Once a method is in routine service, the CV can be estimated from current QC data and bias from current peer-comparison data. (see Method Validation process and procedures)


Calculate the Sigma metric for the testing process, as follows: Sigma = (TE_a – Bias)/CV, e.g., for a cholesterol testing process where TE_a = 10.0%, CV = 1.5%, and Bias = 1.0%, Sigma would be 6.0 [(10.0-1.0)/1.5]; if CV = 2.0% and Bias = 0.0%, Sigma would be 5.0; if CV = 2.0% and Bias = 2.0%, Sigma would be 4.0.


Use the calculated Sigma metric to determine the appropriate QC, with the aid of available QC planning tools.

Get the tools to do the job!

The performance of a QC procedure can be described by its probability for rejecting analytical runs having different sizes of errors. Such information is available in the literature in the form of “power curves” or “power function graphs,” such as shown in the graphic below:

The idea here is to locate the Sigma value on the x-scale at the top of the graph. Note that the x-scale at the bottom of the graph shows the size of “systematic error”, thus the Sigma of a method is also related to the size of error that needs to be detected by the QC procedure. Drop a vertical line to see where it intersects the power curves for the different QC procedures, which are identified in the key at the right side (where the lines top to bottom match the power curves shown top to bottom on the graph). Try to identify a QC procedure that gives a probability of 0.90 for error detection. Try also to keep the probability of false rejection as low as possible, which is shown by the y-intercept of the power curve.

• Note that for the cholesterol 6.0 Sigma testing process, a single-rule such as 1_3s with N=2 would provide appropriate QC; for the cholesterol 5.0 sigma process, 1_2.5s single rule with N=2 is preferred; for the 4.0 Sigma process, a multirule QC procedure with N=4 would be appropriate. N here is the total number of control measurements, e.g., an N of 2 can represent 1 control measurement on each of two different levels of materials; and an N of 4 could represent 2 measurements on each of 2 levels of control materials.
  
• Note that if a 1_3s rule with N=2 were employed for these 3 methods, the probability for error detection would be 1.00 for the 6-Sigma method, 0.87 for the 5-Sigma method, and 0.48 for the 4-Sigma method.
  

For laboratories that want computerized support, check out the EZ Rules 3 program that provides automatic QC selection plus complete documentation of the QC Design process.

Improve your laboratory quality system!

You can’t manage quality unless you can measure it! That’s why it is important to apply the concepts of Six Sigma Quality Management to the analytical testing process and measue quality on the sigma-scale. While there is an extensive literature on Six Sigma, as well as many training programs and belt certifications, there are few applications to a measurement process, other than found on this website and in our educational and training materials. Sigma metrics will help you make quality measureable and manageable.

You can improve the quality systems in your laboratory by implementing a QC planning process, which in turn will improve the management of test quality in your laboratory. Statistical QC is the right approach for this kind of quality management because it can be widely applied to many different tests and analytical systems. Since manufacturers are not required to make any claim for quality, nor to provide detailed QC instructions, the laboratory must exercise its responsibility for doing the right QC right.

James O. Westgard, PhD, is a professor of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.

Past Essays: