A/B testing

An A/B test shows different versions of one banner to different visitors. Each version is called an arm. CookieBeam records which arm each visitor saw and compares how visitors answered.

A/B testing is included in the Business plan and above.

When to use it

Use a test when you want evidence before changing a banner for everyone. Typical questions:

• Does a different layout or color scheme change how many visitors make a choice?
• Does a different consent model, tested in a limited region, change the outcome?

Do not use a test to find out whether a banner is lawful. A test measures visitor behaviour. It does not tell you whether a version meets your legal obligations.

What an arm can change

Each arm uses one of the banner's themes. A theme controls the banner's look. Create the themes first, in Design (see Appearance and themes). A test needs at least two themes.

These settings are shared by every arm and cannot differ between arms:

• regional rules and the fallback, unless an arm has a regional override (below),
• copy and translations,
• categories and cookies,
• integrations and privacy signals.

An arm can also override the banner's consent model, on the core runtime only. The options are:

• Inherit (regional decision) — use the consent model the visitor's regional rule sets. This is the default.
• Opt-in — nothing non-essential runs until the visitor accepts.
• Opt-out — categories run until the visitor opts out.
• Notice — a transparency notice with no blocking.

The baseline arm is served by your normal banner. It cannot carry a consent-model or regional override.

Overriding the consent model changes which scripts run before the visitor decides. An arm with such an override cannot start until someone with Publish permission on that banner records a legal acknowledgement. Changing the override afterwards clears the acknowledgement, so it must be given again.

An arm can also carry a regional override, on the core runtime only, which changes the regional behaviour for visitors in that arm. It needs the same legal acknowledgement.

A visitor whose browser sends a Global Privacy Control or Do Not Track signal never gets an arm's consent-model override. They get the consent model their regional rule sets.

Create and start a test

Creating, starting, pausing, stopping and viewing tests needs Admin permission for the Banners section.

1. Open the banner and choose Insights → A/B testing.
2. Click Create New Test.
3. Enter a Test name and, optionally, a description of what you are testing.
4. Under Region targeting, choose the countries or regions that enter the test. Leave it empty to include every visitor.
5. Add arms. Each arm needs a name and a theme. A test has between 2 and 10 arms.
6. Select the baseline arm. Results compare every other arm with it.
7. Set the traffic split. The weights must add up to 100. Use Distribute evenly to split traffic equally.
8. Optionally set a consent-model override on a non-baseline arm. If you do, complete the legal acknowledgement step.
9. Create the test. It is saved as a draft on the Drafts tab.
10. Click Start Test.

A banner can run one active test at a time. Stop the current test before you start another.

Starting a test builds a script for each non-baseline arm from the banner's published configuration, then rebuilds the loader. Publish the banner before you start a test, so the arms use the configuration you expect.

How visitors are assigned

Each visitor is assigned by a stable hash of their consent ID. On a first visit the loader creates the consent ID before assignment, so the visitor keeps the same arm on later visits. A visitor outside the test's target regions sees the normal banner and is not counted.

The loader does not set an extra cookie for the test. The arm a visitor saw is recorded on their consent records, and you can see it as A/B variant in consent logs.

Pause, stop and delete

• Pause stops new assignments and serves only the baseline. Data collected so far is kept. Start the test again to resume.
• Stop Test ends the test and marks it completed. The loader goes back to serving the normal banner.
• You can delete a draft or completed test. Stop an active or paused test first.

You can only edit an arm's overrides while the test is a draft or paused.

Read the results

Results appear on the Current Test tab and, after the test ends, on the History tab.

• Visitors are counted once per arm. Each consent ID counts once, using the visitor's latest choice. A default written at page load counts as an exposure, not as a decision.
• Opt-in rate is based on explicit choices. On an arm with an opt-out override, a visitor who ran under that override counts as a success unless they explicitly opted out.
• Automated traffic is excluded by default. The page shows how many automated visitors were excluded.

Result readiness shows two checks:

• Sample floor. Every arm needs at least 100 exposures before a leading arm can be named. Directional confidence needs more; the page shows the target for your test.
• Duration. The test must run at least 7 full days, so it covers weekday and weekend traffic.

When both checks pass and an arm is ahead of the baseline with a statistically significant difference, the page names it as leading. Significance is adjusted for the number of arms you compare. A leading arm is the current front-runner. It is not a declared winner, and more traffic can change it.

If you see Sample ratio mismatch detected, traffic did not split as configured. Common causes are blocked, cached or failed arm scripts. Check your deployment before you trust the numbers.

Troubleshooting

The create form says you need more themes. A test compares at least two themes. Duplicate your default theme in Design, change it, and try again.

Start is refused with "legal acknowledgement required". An arm has a consent-model or regional override. Someone with Publish permission on the banner must record the acknowledgement.

Start is refused because the banner already has an active test. Stop the active test first.

An arm looks out of date. Arms are built from the published configuration when the test starts. Publish the banner, then pause and start the test again.

A warning says the CDN cache purge failed. The test started, but some visitors may get the previous loader until the cache expires. If it persists, pause and start the test again.