Confidence interval for test results

Post Reply
rawi
Posts: 21
Joined: Sat Dec 28, 2019 4:50 am

Confidence interval for test results

Post by rawi »

Confidence Interval for Efficiency Index (result of testing with pre and test period and test and control group)
Background:
Basis is the result of a test.
Example: You tested the effectiveness of advertising. You have a test grop of 1500 households and a control group of 1500 households. You are measuring in a pre period and test period. There is no advertising in the pre period. In the test period a product advertising is given to the households of the test groups, not to the households of the control group.
Results are:
Shares of households buying the product
Pre period (PP) /Test period (TP)
Test group (TG) 7.7% / 10.8%
Control group (CG) 6.9% / 7.3%
Efficiency index of advertising: (0.108/0.077)/(0.073/0.069) = 1.326 (+32.6% effect)
Question: Is this significant?
Theory:
We have a fraction in the nominator and a fraction in the denominator. According to R.C.Geary (JRSS (1930), series A 93, pp. 442-446) the standard deviation of a fraction Q = mean(X)/mean(Y) is
Sqrt(s²(Y)*Q²-2*r*s(X)*s(Y)*Q+s²(X))/(mean(Y).
R is the correlation coefficient between X and Y.
This formula is first applied to the nominator, then to the denominator and finally to the total fraction. Since the fractions are (under some conditions) approximately normal distributed we can use normal distribution for testing.

Example 1:
Given are the shares of households buying the product from above. Correlation in test group between pre period and test period is 0.5, in control group 0.6.
Note: Correlation coefficient between test group and control group always is zero, since there are no common elements.
You want to test with 95% significance.
Key strokes:
XEQ “CIEX”
SHARE (0,1)? – 1 R/S (you have shares, so you type 1)
> N TG? – 1500 R/S (sample size of test group)
> X TG PP? – 0.077 R/S (asks for the first value of 4-field-table, test group in pre period)
> X TG TP? – 0.108 R/S (asks for the test group in test period)
> R TG? – 0.5 R/S (Correlation coefficient between pre period and test period in test group)
> N CG? – 1500 R/S (sample size of control group)
> X CG PP? – 0.069 R/S (control group in pre period)
> X CG TP? – 0.073 R/S (control group in test period)
> R CG? – 0.6 (correlation between pre period and test period in control group)
> ND LIMIT? – 1.96 R/S (normal distribution limit, e.g. for 95% confidence level 1.96)
Result:
T: 0.1560 (std. dev. of efficiency index)
Z: 1.0199 (lower limit of efficiency index)
Y: 1.6316 (upper limit of efficiency index)
X: 1,3257 (efficiency index)
Comment: Since 1 is outside the interval between lower and upper limit, the efficiency index is significant.
Example 2:
The following table is given: Mean, (Std. deviation)
Pre period / Test period
Test group: 4.00 (0.42) / 5.99 (0.84) Corr.coeff.: -0.23
Control group: 4.01 (0.40) / 4.80 (0.35) Corr.coeff.: 0.58
Keystrokes:
XEQ CIEX
> SHARE (0,1)? – 0 R/S (we do not have shares but means and std. deviations, so we type 0)
> x TG PP? – 4.00 R/S (mean of test group in pre period)
> x TG TP? – 5.99 R/S (mean of test group in test period)
> r TG? – -0.23 R/S (correlation coefficient in test group between pre period and test period)
> s TG PP? – 0.42 R/S (std. deviation within test group and pre period)
> s TG TP? – 0.84 R/S (std. deviation within test group in test period)
> x CG PP? – 4.01 R/S (mean of control group in pre period)
> x CG TP? – 4.80 R/S (mean of contro group in test period)
> r CG? – 0.58 R/S (correlation coefficient in control group between pre period and test period)
> s CG PP? – 0.40 R/S (std. deviation of control group in pre period)
> s CG TP? – 0.35 R/S (std. deviation of control group in test period)
> ND LIMIT? – 1.96 (again we take the 95%-limit of normal distribution)
Result:
T: 0.2633 (std. dev. Of efficiency index)
Z: 0.7350 (lower limit of efficiency index)
Y: 1.7671 (upper limit of efficiency indes)
X: 1.2510 (efficiency index)
Comment: 1 is between lower and upper limit, so the index is not significant.


Use of registers / flags:
PP TP r n
TG: 01 02 03 04
CG: 05 06 07 08
For computation: Denominator y: 09
Nominator x: 10
Sy: 11
Sx: 12
r: 13
n: 14
s(nominator): 15
s(denominator) 16
s(fraction) 17
Index 18
Use of flags: Flag 01 and Flag 02 used.

Code: Select all

1	LBL CIEX
2	CF 01
3	SF 02
4	"SHARE 0,1?
5	PROMPT
6	X=0?
7	GTO 01
8	SF 01
9	"N TG?
10	PROMPT
11	STO 04
12	STO 14
13	LBL 01
14	"X TG PP?
15	PROMPT
16	STO 01
17	STO 09
18	"X TG TP?
19	PROMPT
20	STO 02
21	STO 10
22	"R TG?
23	PROMPT
24	STO 03
25	STO 13
26	FS? 01
27	XEQ 11
28	FC? 01
29	XEQ 12
30	XEQ 14
31	STO 15
32	STO 11
33	CF 02
34	FC? 01
35	GTO 02
36	"N CG?
37	PROMPT
38	STO 08
39	STO 14
40	LBL 02
41	"X CG PP?
42	PROMPT
43	STO 05
44	STO 09
45	"X CG TP?
46	PROMPT
47	STO 06
48	STO 10
49	"R CG?
50	PROMPT
51	STO 07
52	STO 13
53	FS? 01
54	XEQ 11
55	FC? 01
56	XEQ 12
57	XEQ 14
58	STO 16
59	STO 11
60	RCL 02
61	RCL 01
62	/
63	STO 10
64	RCL 06
65	RCL 05
66	/
67	STO 09
68	/
69	STO 18
70	0
71	STO 13
72	RCL 15
73	STO 12
74	XEQ 14
75	STO 17
76	"ND LIMIT?
77	PROMPT
78	*
79	RCL 18
80	x<>y
81	-
82	LAST x
83	RCL 18
84	+
85	RCL 18
86	RCL 17
87	RDN
88	CF 01
89	RTN
90	LBL 11
91	RCL 09
92	XEQ 13
93	STO 11
94	RCL 10
95	XEQ 13
96	STO 12
97	RTN
98	LBL 12
99	"S CG PP?
100	FS? 02
101	"S TG PP?
102	PROMPT
103	STO 11
104	"S CG TP?
105	FS? 02
106	"S TG TP?
107	PROMPT
108	STO 12
109	RTN
110	LBL 13
111	1
112	x<>y
113	-
114	LAST x
115	*
116	RCL 14
117	/
118	SQRT
119	RTN
120	LBL 14
121	RCL 13
122	RCL 11
123	*
124	RCL 12
125	*
126	2
127	*
128	CHS
129	RCL 10
130	RCL 09
131	/
132	*
133	LAST x
134	RCL 11
135	*
136	x²
137	+
138	RCL 12
139	x²
140	+
141	SQRT
142	RCL 09
143	/
144	RTN 
145	END

Post Reply