FROM METHOD VALIDATION TO SIX SIGMA: FOUR POC BLOOD GLUCOSE MONITORING SYSTEMSFROM METHOD VALIDATION TO SIX SIGMA: FOUR POC BLOOD GLUCOSE MONITORING SYSTEMS |
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.] |  |
One of the early Six Sigma applications presented on Westgard Web was an analysis of a POC chemistry analyzer. The results of that were a little stunning. Not only were many of the tests on that system below 3.0 Sigma - below the minimum threshold of stability that is considered acceptable level in other industries - many of the tests had in fact negative Sigma metrics. The variation described by the instrument manufacturer simply exceeded the quality requirements specified by the government. The metrics were so bad on that instrument we made it an anonymous study.
One unspoken question left by that analysis was: was this POC device typical of other POC devices, or was it a "bad apple"? With this article, we hope to provide some more evidence of other POC device peroformance. Through the magic of Google, we were able to find a comparison of 4 POC blood glucose monitoring systems:
- Accu-Check Advantage GTS (Roche Diagnostics/Boehringer Mannheim)
- Precision G (Medisense)
- One Touch II Hospital (Lifescan)
- Encore QA+ (Bayer)
"Precision and Accuracy Evaluation on Four Hospital Blood Glucose Monitoring Systems" was a Roche Diagnostics sponsored poster presentation at the 1998 AACC convention. It is available on the web at http://us.labsystems.roche.com/products/98artic/hosp.pdf
As always, it's a good idea to take a look at the actual paper and its data. There are two studies performed: a within-run replication study to estimate imprecision and comparison of methods study to estimate inaccuracy(bias). We should note that the Accu-Chek was compared to a different reference method than the other three methods, which disrupts the comparison of performance somewhat.
The paper shows that there are multiple levels for each method included in the replication study.
Meter System Mean SD CV(%)
44.9 1.8 4 Accu-Chek 121 2.9 2.4
266.9 2.6 0.97
42.4 2.6 6 Precision G 82.4 3.5 4.3
285.1 7.8 2.7
46.6 1.2 2.5 One Touch II 280.8 7.8 2.8
32.9 1.2 3.6 Encore QA+ 89.8 4.2 4.7
416.7 14.9 3.6 If we use all of these different levels and values, we will end up with multiple Sigma metrics for each method, and that will only cause confusion. In order to provide a single comparison, we need to choose just one level and use that as our basis for comparison.
For this case, we're going to use 125 mg/dL as the critical level of interest. Now, most of the methods don't have an imprecision value right at 125. So for each method, we are going to use the CV estimate for the level that is closest to 125. It would be ideal if each method had a control at that level, but the data isn't there, so we have to be flexible.
(Why 125? Because it's at the high end of a range recommended in the 2002 Diabetes recommendations from the from the American Diabetes Association (ADA) and U. S. Health and Human Services (HSS). Values above 125 are to be considered indicative of hyperglycemia. So you don't want a 124 becoming a 126. See the related article on Westgard Web).
Now that we have selected a level, we can calculate a bias at that level, using the comparison of methods study data in the paper. We simply use the slope and y-intercept values to calculate the new value. The paper presents the following results:
Instrument Accu-Chek Advantage Precision G One Touch II Encore QA+ Thus, for the Accu-Check method, our calculations are as follows
Bias = (1.03 * 125) - 1.8
= 128.75 - 1.8
= 126.95
There is a difference between the Accu-Check result and its reference method of 1.95 at the level of 125. Now we convert that difference into a percentage: 1.95/125 = 1.56%
Here are the calculated biases for all the methods:
Instrument Accu-Chek Advantage Precision G One Touch II Encore QA+
CLIA provides a clear quality requirement for glucose: 10%. However this is for tests subject to proficiency testing. As a POC device not subject to proficiency testing, there is a different quality requirement applied. The "Clarke Error Grid" is often used with comparison of methods studies to evaluate home monitoring and POC devices. The quality requirement used in this grid is 20%.
Now that we have a quality requirement, a CV and a bias (all expressed in percentages), we can calculate the Six Sigma metric. The equation has been explained elsewhere, but we'll restate it here for clarity:
Sigma Metric = (TEa - biasmeas)/smeas
where TEa is the allowable total error (quality requirement), biasmeas is the inaccuracy of the method, and smeas is the observed imprecision of the method.
For Accu-Check, here is our calculation:
Sigma Metric = (20 - 1.56) / 2.4 = 7.68
Here is the final tally of Sigma metrics:
Instrument Accu-Chek Advantage Precision G One Touch II Encore QA+
From a QC Design perspective, some of these Sigma metrics are world class, but others are raising an alarm. The Accu-Check and One Touch II have achieved Six Sigma, but the other two methods are around 3 Sigma, which is the minimum standard for acceptability acceptable metric for business and industry processes.
For the methods with Six Sigma or higher, you can QC with wide limits and just a few controls. For the methods at 3 Sigma, you need to do as much QC as you can afford. Use "Westgard Rules" with as many controls the method allows you to run. And even then, try to find ways to provide additional checks, preventive maintenance, etc. These are problematic methods and they need extra attention.
1. POC performance is not uniform. We've some great methods and some not-so-great, and without the Sigma metrics, it wasn't so obvious which was which.
One must also remember that these calculations are based on CV figures derived from a within-run estimate of imprecision. It is very likely that a within-day (or between-run or total imprecision) estimate of imprecision will be higher, and thus the Sigma values will be even lower for these methods.
Finally, as always, we must note that these Sigma metrics are based on method validation studies performed by someone not in your laboratory. (This poster was presented in 1998, so hopefully performance has improved since then). If you were going to make a truly informed decision about these methods, you would want to have method validation studies that reflect the performance of the methods in your laboratory with your staff. That would give you the most accurate picture of the Sigma metrics of these instruments.
2. For some methods, CLIA minimums are not enough. Below Six Sigma, methods are not good candidates for "equivalent QC procedures"
Running just 2 controls per run would be sufficient for the Accu-Chek and the One Touch II (in fact, you could run just one control and achieve the necessary error detection). The other two methods need a lot of QC. Using "Westgard Rules" and only 2 controls on a 3.0 Sigma method would only provide approximately 14% error detection (ARL of 7 runs before you would detect the problem). Some of these instruments even look like they need 6, 8, or more controls per run!
If we were to apply the 10% quality requirement to these methods, the results would be less positive:
Instrument Accu-Chek Advantage Precision G One Touch II Encore QA+ When held to the standard that the central lab chemistries must achieve, POC devices fall short, very short.
- Normalized OPSpecs charts are available as a free download from the website.
- The OPSpecs Manual, Expanded Edition, offers a low cost, manual QC Design technique.
- EZ Rules® is a software program that automates the rule selection process for QC Design
- QC Validator® is a software program that gives fine-grained, custom control over automatic QC Selection
