Traditional testing methodology tells us to 1) test only one thing at a time, 2) always test head to head and 3) always test statistically valid samples. This works fine when you have lots of time. But what if you need to know what will work for your product and target audience right away? And what if your mailing universe is constrained to 100,000 names or less, as many business-to- business or geographically constrained direct marketers are? With a limited number of names to mail, conventional testing may limit the number of test cells to fewer than 10, even if you tested the entire file. Often, that's just not enough testing, particularly if there is no control package and little or no time to find one.
The objective of a PowerTest® is to develop a campaign control (medium, offer, creative approach and more) as fast as possible. With a properly run PowerTest, you can usually find a winner - or several winners - in the first test, and almost always by the second test event. This unique multivariate testing approach shows how to find a winning marketing strategy in a hurry.
Hacker Group ®
PowerTest Strategy
How to find a control fast
1 2 1 5 4 t h A v e n u e S u i t e 2 1 0 0 Seattle, Washington 98161-1018 t 206.805.1500 f 206.805.1599h t t p : / / w w w. h a c k e r g r o u p . c o m®Hacker Group PowerTest Strategy
How to Find a Control Fast
The Right Way for ALL Mailers to Establish a Control Package . . . Fast!
Traditional testing methodology tells us to 1) only test one thing at a time; 2) always test head-to-head; and 3) always test statistically valid samples. This works fine when you have lots of time and you mail in high volume so you can afford to allocate 5,000 or more names to each test cell. For example, a mailer dropping 1,000,000 packages at a time can test 40 cells of 5,000 each and still mail the control package to 80% of the file.
But what if your mailing universe is constrained to 100,000 names or less, as many business-to-business or geographically constrained direct marketers are? With a limited number of names to mail, conventional testing may limit the number of test cells to 5-10, even if you tested the entire file. Often, that's just not enough testing, particularly if there is no control package and there is little or no time to find one.
Recently, one of our clients gave us the following opportunity. If we could test in October and find a control package that hit their economic target, they would allocate $2,500,000 to a rollout in January. Due to a very tight time constraint, we had to find a winner on the first drop. ®Since the client had no mailing history with this product, we had to conduct a PowerTest because it was the only way to find a winner in time to hit the January rollout date. In this case, the test matrix contained 32 different offers and 288 discrete test cells distributed over about 200,000 pieces of mail.
®How to Do a PowerTest
The objective of a PowerTest is to develop a control package as fast as possible. With a properly run PowerTest, you can usually find a winner - or several winners - in the first test, and almost always by the second test event. Here's how it works.
Phase 1
In a PowerTest situation, there is often no control. There has been no testing - or failed testing - and there is no meaningful performance data for lists, packages or offers. Therefore, the objective of Phase 1 is to determine these key success drivers:? Discover which lists will work for the offer? Determine which offers will work for the lists
To show you how it works, let's look at a simple 90,000 piece PowerTest. The key assumptions are:? There are two package formats? There are three offer splits? There are ten lists being tested at 9,000 each
®Hacker Group PowerTest Strategy 1Based on the above test assumptions, the test matrix would look like this: Test Matrix Offer A Offer B Offer CPackage Package Package Package Package Package X Y X Y X YList 1 1500 1500 1500 1500 1500 1500List 2 1500 1500 1500 1500 1500 1500List 3 1500 1500 1500 1500 1500 1500List 4 1500 1500 1500 1500 1500 1500List 5 1500 1500 1500 1500 1500 1500List 6 1500 1500 1500 1500 1500 1500List 7 1500 1500 1500 1500 1500 1500List 8 1500 1500 1500 1500 1500 1500List 9 1500 1500 1500 1500 1500 1500List 10 1500 1500 1500 1500 1500 1500
In this matrix, there are 60 test cells, each with 1,500 names. The results from each cell are not statistically valid. To generate a statistically valid sample, this test would have to be about 300,000 pieces, which is often too risky, too expensive or both. TMHowever, the results are still highly indicative. You will see strong trends and HotZones that tell you what to do next. Let's assume that the test generated the following response rates:
Response Rate Matrix Offer A Offer B Offer CPackage Package Package Package Package Package X Y X Y X YList 1 4.5% 3.8% 6.4% 4.6% 2.5% 1.9%List 2 3.6% 2.8% 4.4% 3.9% 2.7% 1.8%List 3 1.6% 1.3% 3.2% 2.8% 1.6% 1.4%List 4 1.7% 1.6% 2.2% 1.6% 4.5% 3.2%List 5 1.4% 1.5% 1.5% 2.1% 1.4% 1.4%List 6 2.7% 1.9% 3.7% 2.4% 2.4% 1.9%List 7 1.6% 1.3% 2.1% 1.9% 1.3% 1.5% List 8 3.4% 2.2% 4.2% 2.5% 2.1% 1.7%List 9 1.5% 1.4% 1.6% 1.8% 1.4% 1.5%List 10 2.3% 1.8% 2.6% 2.4% 1.8% 1.6%
In this example, it is safe to conclude the following, assuming we need a response rate of 3% or more to hit the economic target:? Package X is a clear winner over Package Y? Offer A worked ... [download for more]
