Dual vision procedure for analysing the agreement

The agreement between two diagnostic methods is the comparison of a new method with the old one. When the new method agrees sufficiently welll with the old, the old one may be replaced. The dual vision procedure is based on a main assumption:

Two diagnostic methods agree when they have the same sensitivity and specificity

This is true, if and only if b = c

Agreement Table

 

         Metodh 1

 

    Metodh 2

(+)

(-)

Total

(+)

a

b

a + b

(-)

c

d

c + d

Total

a + c

b + d

   N

The dual vision procedure has two steps:

1) Statistical vision: Verify that sensitivity and specificity are similar in both methods by using the G-test. The null hypothesis is Ho: b = c. For testing this Ho there are three statistical tests a) McNemar's test (with Yates correction). b) Cochran Q-test. and c) McNemar's G-test (likelihood) with Williams correction. The last is the more powerful, however if b or c is equal to 0, G is not finite, then Q-test should be used.   

The adjusted value Gadj is compared with the critical value of Chi-square test for 95% of confidence and one degree of freedom. If Gadj > 3.841 there is a statistical proof for rejecting Ho, i.e. the methods have different values of sensitivity and specificity. If Gadj < 3.841 there is no evidence for thinking that both methods cannot be interchanged. 

But this is not enough, because the sample size (N) and the raw agreement l = (a+b)/N are not taken into account. This fact provokes clinical paradoxes, tehrefore a second step is needed:

2) Clinical vision: Verify the condition: The level of the agreement l (raw) should be suffinciently large for having an acceptable agreement from a clinical viewpoint. For example,

The agreement is acceptable when there is no more than 10 disagreements in 100 cases

The Discordance Odds index (DO) can be used.  DO = (b + c) / (a + b) = 10 /90, i.e DOcritical is 1 : 9 Clinically, the agreement should be acceptable when there will be 1 diasagreement and 9 agreements in 10 cases - (lcritical = 90% can be used too).

With the measured data of the experiment the sample value of DO can be calculated; and its 95% confidence interval too.  If the critical value lies into the interval, or if it is greater than the upper limit of the interval, then there will be enough agreement from a clinical viewpoint. In the opposite case, the methods cannot be interchanged.

When both steps are tested the clinical decision can be adopted.

Reference:  Azzimonti Renzo JC, The agreement between two diagnostic methods in binary cases: a proposal, Scand J Clin Lab Invest 2002, 62: 391-398

For non-commercial use only. Originally published in Scandinavian Journal of Clinical & Laboratory Investigation (www.tandf.no/sjcli), 2002, 62(5): 391-398.