Creative A/B testing: decide what the result can answer
Decide what you're testing before making the ads. Keep track of the change so the results can help you choose what to make next.
2 openings, one intended difference
This is a travel-pouch production plan. A begins with the pouch closed. B begins with its 3 compartments open. After 2 seconds, the footage, spoken line, captions, offer and destination are intended to match. Neither variant is a real advertiser campaign.
- Decision
- Should the next production round develop the open-pouch opening?
- Primary outcome
- Verified product-page visits per impression. Define and validate that event before launch. Ad clicks are not automatically verified visits.
- Hypothesis
- Making the layout visible immediately may help people understand the product. This is a proposal, not an established effect.
- Controls
- Same product version, named items, remaining footage, duration, message, offer, destination, audience eligibility, placement and measurement definitions.
- Assignment
- Use the platform’s supported experiment setup and verify randomization and non-overlapping groups. A normal campaign with unequal delivery is not automatically an A/B test.
- Before launch
- Choose the smallest useful effect, sample-size basis, duration and stopping rule with the experiment owner. Record the plan before seeing results. There is no universal ad count or spend threshold here.
TikTok’s official Split Testing guide, checked 14 October 2026, describes separate groups, keeping other variables stable and supported test variables. Follow the current setup and statistical reporting of the platform you actually use.
An A/B test compares groups randomly assigned to different versions. For creative, decide what you’re changing, what stays the same and which result will guide the next decision. The made-up example below shows why a few extra clicks aren’t enough to choose a winner.
Calculate the readout before interpreting it
These are made-up numbers in USD. No campaign ran. Both versions have the same number of impressions, which makes the ratios easy to compare. That alone doesn’t tell us whether the groups were randomly assigned, had similar audiences or were large enough for the test.
| Variant | Impressions | Clicks | Purchases | Spend | Revenue | CTR | CPM | ROAS |
|---|---|---|---|---|---|---|---|---|
| A · closed-pouch opening | 10,000 | 120 | 5 | $180.00 | $250.00 | 1.20% | $18.00 | 1.39× |
| B · open-pouch opening | 10,000 | 126 | 4 | $180.00 | $200.00 | 1.26% | $18.00 | 1.11× |
CTR = clicks ÷ impressions × 100. CPM = spend ÷ impressions × 1,000. ROAS = attributed revenue ÷ spend. B has 6 more clicks: a 0.06 percentage-point CTR difference, or 5% relative to A’s 1.20%. Those are different ways of describing the same arithmetic, not proof of a causal effect.
Why this decision is inconclusive
We don’t have verified counts for the main outcome, a record of group assignments, a sample-size plan, attribution settings or an uncertainty analysis. Those extra clicks and few attributed purchases aren’t enough to call a winner, a statistically significant difference or a profitable ad.
- What is observed
- In this example, B’s click ratio is slightly higher and its attributed purchase count and ROAS are lower.
- What is unknown
- Whether clicks became verified visits, whether groups were properly assigned, whether tracking/deduplication matched, and what uncertainty surrounds a difference.
- Decision
- Inconclusive. Do not scale a supposed winner from this table. Complete the design and measurement checks before a real launch.
- Next action
- Ask the buyer to verify experiment assignment and delivery, the analyst to validate primary events/attribution, and the creative team to preserve the intended single difference.
More clicks don’t necessarily mean more purchases. Decide which result matters before the test. Don’t switch to whichever metric makes a version look better afterward.
Check the confounds in an ordinary ad comparison
- Unequal audiences or placements: record the assignment method and actual delivery before attributing a difference to the creative.
- Different offers or destinations: the comparison conditions changed, so an opening-only conclusion is unsupported.
- Tracking or attribution changes: verify event definitions, window, deduplication and missing data for both cells.
- Early stopping or repeated winner hunting: use the planned review and the platform’s current statistical method. A favorable snapshot is not a stopping rule.
- Several changed creative elements: describe it as a whole-concept comparison. It cannot isolate one crop, line or shot.
The creative brief’s worksheet plans a whole-concept comparison. This resource adds the separate work of checking numeric inputs, missing evidence and a readout decision. Keep both records linked.
Take the working document with you
Start with the blank worksheet, or read the filled-in example to see how to use it. You can copy either one.