Confidence interval for test results
Posted: Tue Nov 10, 2020 9:19 pm
Confidence Interval for Efficiency Index (result of testing with pre and test period and test and control group)
Background:
Basis is the result of a test.
Example: You tested the effectiveness of advertising. You have a test grop of 1500 households and a control group of 1500 households. You are measuring in a pre period and test period. There is no advertising in the pre period. In the test period a product advertising is given to the households of the test groups, not to the households of the control group.
Results are:
Shares of households buying the product
Pre period (PP) /Test period (TP)
Test group (TG) 7.7% / 10.8%
Control group (CG) 6.9% / 7.3%
Efficiency index of advertising: (0.108/0.077)/(0.073/0.069) = 1.326 (+32.6% effect)
Question: Is this significant?
Theory:
We have a fraction in the nominator and a fraction in the denominator. According to R.C.Geary (JRSS (1930), series A 93, pp. 442-446) the standard deviation of a fraction Q = mean(X)/mean(Y) is
Sqrt(s²(Y)*Q²-2*r*s(X)*s(Y)*Q+s²(X))/(mean(Y).
R is the correlation coefficient between X and Y.
This formula is first applied to the nominator, then to the denominator and finally to the total fraction. Since the fractions are (under some conditions) approximately normal distributed we can use normal distribution for testing.
Example 1:
Given are the shares of households buying the product from above. Correlation in test group between pre period and test period is 0.5, in control group 0.6.
Note: Correlation coefficient between test group and control group always is zero, since there are no common elements.
You want to test with 95% significance.
Key strokes:
XEQ “CIEX”
SHARE (0,1)? – 1 R/S (you have shares, so you type 1)
> N TG? – 1500 R/S (sample size of test group)
> X TG PP? – 0.077 R/S (asks for the first value of 4-field-table, test group in pre period)
> X TG TP? – 0.108 R/S (asks for the test group in test period)
> R TG? – 0.5 R/S (Correlation coefficient between pre period and test period in test group)
> N CG? – 1500 R/S (sample size of control group)
> X CG PP? – 0.069 R/S (control group in pre period)
> X CG TP? – 0.073 R/S (control group in test period)
> R CG? – 0.6 (correlation between pre period and test period in control group)
> ND LIMIT? – 1.96 R/S (normal distribution limit, e.g. for 95% confidence level 1.96)
Result:
T: 0.1560 (std. dev. of efficiency index)
Z: 1.0199 (lower limit of efficiency index)
Y: 1.6316 (upper limit of efficiency index)
X: 1,3257 (efficiency index)
Comment: Since 1 is outside the interval between lower and upper limit, the efficiency index is significant.
Example 2:
The following table is given: Mean, (Std. deviation)
Pre period / Test period
Test group: 4.00 (0.42) / 5.99 (0.84) Corr.coeff.: -0.23
Control group: 4.01 (0.40) / 4.80 (0.35) Corr.coeff.: 0.58
Keystrokes:
XEQ CIEX
> SHARE (0,1)? – 0 R/S (we do not have shares but means and std. deviations, so we type 0)
> x TG PP? – 4.00 R/S (mean of test group in pre period)
> x TG TP? – 5.99 R/S (mean of test group in test period)
> r TG? – -0.23 R/S (correlation coefficient in test group between pre period and test period)
> s TG PP? – 0.42 R/S (std. deviation within test group and pre period)
> s TG TP? – 0.84 R/S (std. deviation within test group in test period)
> x CG PP? – 4.01 R/S (mean of control group in pre period)
> x CG TP? – 4.80 R/S (mean of contro group in test period)
> r CG? – 0.58 R/S (correlation coefficient in control group between pre period and test period)
> s CG PP? – 0.40 R/S (std. deviation of control group in pre period)
> s CG TP? – 0.35 R/S (std. deviation of control group in test period)
> ND LIMIT? – 1.96 (again we take the 95%-limit of normal distribution)
Result:
T: 0.2633 (std. dev. Of efficiency index)
Z: 0.7350 (lower limit of efficiency index)
Y: 1.7671 (upper limit of efficiency indes)
X: 1.2510 (efficiency index)
Comment: 1 is between lower and upper limit, so the index is not significant.
Use of registers / flags:
PP TP r n
TG: 01 02 03 04
CG: 05 06 07 08
For computation: Denominator y: 09
Nominator x: 10
Sy: 11
Sx: 12
r: 13
n: 14
s(nominator): 15
s(denominator) 16
s(fraction) 17
Index 18
Use of flags: Flag 01 and Flag 02 used.
Background:
Basis is the result of a test.
Example: You tested the effectiveness of advertising. You have a test grop of 1500 households and a control group of 1500 households. You are measuring in a pre period and test period. There is no advertising in the pre period. In the test period a product advertising is given to the households of the test groups, not to the households of the control group.
Results are:
Shares of households buying the product
Pre period (PP) /Test period (TP)
Test group (TG) 7.7% / 10.8%
Control group (CG) 6.9% / 7.3%
Efficiency index of advertising: (0.108/0.077)/(0.073/0.069) = 1.326 (+32.6% effect)
Question: Is this significant?
Theory:
We have a fraction in the nominator and a fraction in the denominator. According to R.C.Geary (JRSS (1930), series A 93, pp. 442-446) the standard deviation of a fraction Q = mean(X)/mean(Y) is
Sqrt(s²(Y)*Q²-2*r*s(X)*s(Y)*Q+s²(X))/(mean(Y).
R is the correlation coefficient between X and Y.
This formula is first applied to the nominator, then to the denominator and finally to the total fraction. Since the fractions are (under some conditions) approximately normal distributed we can use normal distribution for testing.
Example 1:
Given are the shares of households buying the product from above. Correlation in test group between pre period and test period is 0.5, in control group 0.6.
Note: Correlation coefficient between test group and control group always is zero, since there are no common elements.
You want to test with 95% significance.
Key strokes:
XEQ “CIEX”
SHARE (0,1)? – 1 R/S (you have shares, so you type 1)
> N TG? – 1500 R/S (sample size of test group)
> X TG PP? – 0.077 R/S (asks for the first value of 4-field-table, test group in pre period)
> X TG TP? – 0.108 R/S (asks for the test group in test period)
> R TG? – 0.5 R/S (Correlation coefficient between pre period and test period in test group)
> N CG? – 1500 R/S (sample size of control group)
> X CG PP? – 0.069 R/S (control group in pre period)
> X CG TP? – 0.073 R/S (control group in test period)
> R CG? – 0.6 (correlation between pre period and test period in control group)
> ND LIMIT? – 1.96 R/S (normal distribution limit, e.g. for 95% confidence level 1.96)
Result:
T: 0.1560 (std. dev. of efficiency index)
Z: 1.0199 (lower limit of efficiency index)
Y: 1.6316 (upper limit of efficiency index)
X: 1,3257 (efficiency index)
Comment: Since 1 is outside the interval between lower and upper limit, the efficiency index is significant.
Example 2:
The following table is given: Mean, (Std. deviation)
Pre period / Test period
Test group: 4.00 (0.42) / 5.99 (0.84) Corr.coeff.: -0.23
Control group: 4.01 (0.40) / 4.80 (0.35) Corr.coeff.: 0.58
Keystrokes:
XEQ CIEX
> SHARE (0,1)? – 0 R/S (we do not have shares but means and std. deviations, so we type 0)
> x TG PP? – 4.00 R/S (mean of test group in pre period)
> x TG TP? – 5.99 R/S (mean of test group in test period)
> r TG? – -0.23 R/S (correlation coefficient in test group between pre period and test period)
> s TG PP? – 0.42 R/S (std. deviation within test group and pre period)
> s TG TP? – 0.84 R/S (std. deviation within test group in test period)
> x CG PP? – 4.01 R/S (mean of control group in pre period)
> x CG TP? – 4.80 R/S (mean of contro group in test period)
> r CG? – 0.58 R/S (correlation coefficient in control group between pre period and test period)
> s CG PP? – 0.40 R/S (std. deviation of control group in pre period)
> s CG TP? – 0.35 R/S (std. deviation of control group in test period)
> ND LIMIT? – 1.96 (again we take the 95%-limit of normal distribution)
Result:
T: 0.2633 (std. dev. Of efficiency index)
Z: 0.7350 (lower limit of efficiency index)
Y: 1.7671 (upper limit of efficiency indes)
X: 1.2510 (efficiency index)
Comment: 1 is between lower and upper limit, so the index is not significant.
Use of registers / flags:
PP TP r n
TG: 01 02 03 04
CG: 05 06 07 08
For computation: Denominator y: 09
Nominator x: 10
Sy: 11
Sx: 12
r: 13
n: 14
s(nominator): 15
s(denominator) 16
s(fraction) 17
Index 18
Use of flags: Flag 01 and Flag 02 used.
Code: Select all
1 LBL CIEX
2 CF 01
3 SF 02
4 "SHARE 0,1?
5 PROMPT
6 X=0?
7 GTO 01
8 SF 01
9 "N TG?
10 PROMPT
11 STO 04
12 STO 14
13 LBL 01
14 "X TG PP?
15 PROMPT
16 STO 01
17 STO 09
18 "X TG TP?
19 PROMPT
20 STO 02
21 STO 10
22 "R TG?
23 PROMPT
24 STO 03
25 STO 13
26 FS? 01
27 XEQ 11
28 FC? 01
29 XEQ 12
30 XEQ 14
31 STO 15
32 STO 11
33 CF 02
34 FC? 01
35 GTO 02
36 "N CG?
37 PROMPT
38 STO 08
39 STO 14
40 LBL 02
41 "X CG PP?
42 PROMPT
43 STO 05
44 STO 09
45 "X CG TP?
46 PROMPT
47 STO 06
48 STO 10
49 "R CG?
50 PROMPT
51 STO 07
52 STO 13
53 FS? 01
54 XEQ 11
55 FC? 01
56 XEQ 12
57 XEQ 14
58 STO 16
59 STO 11
60 RCL 02
61 RCL 01
62 /
63 STO 10
64 RCL 06
65 RCL 05
66 /
67 STO 09
68 /
69 STO 18
70 0
71 STO 13
72 RCL 15
73 STO 12
74 XEQ 14
75 STO 17
76 "ND LIMIT?
77 PROMPT
78 *
79 RCL 18
80 x<>y
81 -
82 LAST x
83 RCL 18
84 +
85 RCL 18
86 RCL 17
87 RDN
88 CF 01
89 RTN
90 LBL 11
91 RCL 09
92 XEQ 13
93 STO 11
94 RCL 10
95 XEQ 13
96 STO 12
97 RTN
98 LBL 12
99 "S CG PP?
100 FS? 02
101 "S TG PP?
102 PROMPT
103 STO 11
104 "S CG TP?
105 FS? 02
106 "S TG TP?
107 PROMPT
108 STO 12
109 RTN
110 LBL 13
111 1
112 x<>y
113 -
114 LAST x
115 *
116 RCL 14
117 /
118 SQRT
119 RTN
120 LBL 14
121 RCL 13
122 RCL 11
123 *
124 RCL 12
125 *
126 2
127 *
128 CHS
129 RCL 10
130 RCL 09
131 /
132 *
133 LAST x
134 RCL 11
135 *
136 x²
137 +
138 RCL 12
139 x²
140 +
141 SQRT
142 RCL 09
143 /
144 RTN
145 END