Sample Size Matters

Is my Test response B significantly different to my Control response A?

Enter the mailing volumes and response rates achieved for each of a Control and Test sample.

You can also use this calculator to detemine if any two test responses (A and B) are significantly different, one to the other.

We should at this point try and offer some advice as to “next steps”…

Test Sample response is significantly different to the Control Sample *

You have discovered hopefully a defining test element that impacts on your in-the-mail performance, either in a good way or a bad way, from which you can learn. But remember that our *asterisk above denotes that this is likely to be the case here in 95 times out of 100 and depends on all other things being equal!

You would be well advised therefore to pursue your test, perhaps with a further validation, perhaps at a larger volume. As we know, larger sample sizes improve our confidence in forecasting future performance and a second result might remove (or introduce) some external influence that impacted on our first result.

Test Sample response is not significantly different to the Control Sample *

Here we have possibly discovered that a subjective or “gut feel” opinion on what might work much better is actually not delivering anything particularly earth-shattering and a new approach to variable testing might be called for.

For example, if we were testing “Free P&P” versus “10% Voucher” and no significant difference was observed, or indeed if a significant difference was observed, we might want to go back and re-test these elements focussing on different audience segments – based on average order values for example. What would be most appropriate, of course, depends on your data availability and your business objectives.

The caveat…

BaseData can accept no responsibility for results obtained from the Sample Size Matters calculator or any interpretation based on these outcomes.

It is as well to remember that our ready-reckoner is based on a 95% confidence level and that all signficance calculations depend on “all other factors remaining the same”.

Direct Marketing & Data Specialists

Example

A Control sample of 5.000 records and a Test sample of 10,000 records achieved responses of 2.50% and 2.30% respectively. This produces a “limit of error” (of 0.52% either side of the difference between the Control and Test sample response rates) and a range of response that is not significantly different to the Control: 2.18% — 3.22%.

This means that the Test sample response is only significantly different to the Control sample if the response rate falls outside this range. In this example, therefore, the Test sample response of 2.30% is not significantly different to the Control.