---
title: "A/B testing"
description: "Show visitors different versions of a banner, compare how they answer, and read the results without over-reading them."
url: "https://cookiebeam.com/docs/configure/ab-testing"
lang: "en"
---

# A/B testing

Show visitors different versions of a banner, compare how they answer, and read the results without over-reading them.

An A/B test shows different versions of one banner to different visitors. Each version is called an **arm**. CookieBeam records which arm each visitor saw and compares how visitors answered.

A/B testing is included in the **Business** plan and above.

## When to use it

Use a test when you want evidence before changing a banner for everyone. Typical questions:

- Does a different layout or color scheme change how many visitors make a choice?
- Does a different consent model, tested in a limited region, change the outcome?

Do not use a test to find out whether a banner is lawful. A test measures visitor behaviour. It does not tell you whether a version meets your legal obligations.

## What an arm can change

Each arm uses one of the banner's **themes**. A theme controls the banner's look. Create the themes first, in **Design** (see [Appearance and themes](https://cookiebeam.com/docs/configure/appearance)). A test needs at least two themes.

These settings are shared by every arm and cannot differ between arms:

- regional rules and the fallback, unless an arm has a regional override (below),
- copy and translations,
- categories and cookies,
- integrations and privacy signals.

An arm can also override the banner's **consent model**, on the core runtime only. The options are:

- **Inherit (regional decision)** — use the consent model the visitor's regional rule sets. This is the default.
- **Opt-in** — nothing non-essential runs until the visitor accepts.
- **Opt-out** — categories run until the visitor opts out.
- **Notice** — a transparency notice with no blocking.

The baseline arm is served by your normal banner. It cannot carry a consent-model or regional override.

Overriding the consent model changes which scripts run before the visitor decides. An arm with such an override cannot start until someone with **Publish** permission on that banner records a legal acknowledgement. Changing the override afterwards clears the acknowledgement, so it must be given again.

An arm can also carry a **regional override**, on the core runtime only, which changes the regional behaviour for visitors in that arm. It needs the same legal acknowledgement.

A visitor whose browser sends a Global Privacy Control or Do Not Track signal never gets an arm's consent-model override. They get the consent model their regional rule sets.

## Create and start a test

Creating, starting, pausing, stopping and viewing tests needs **Admin** permission for the Banners section.

1. Open the banner and choose **Insights → A/B testing**.
2. Click **Create New Test**.
3. Enter a **Test name** and, optionally, a description of what you are testing.
4. Under **Region targeting**, choose the countries or regions that enter the test. Leave it empty to include every visitor.
5. Add arms. Each arm needs a name and a theme. A test has between 2 and 10 arms.
6. Select the **baseline** arm. Results compare every other arm with it.
7. Set the traffic split. The weights must add up to 100. Use **Distribute evenly** to split traffic equally.
8. Optionally set a consent-model override on a non-baseline arm. If you do, complete the legal acknowledgement step.
9. Create the test. It is saved as a draft on the **Drafts** tab.
10. Click **Start Test**.

A banner can run one active test at a time. Stop the current test before you start another.

Starting a test builds a script for each non-baseline arm from the banner's published configuration, then rebuilds the loader. Publish the banner before you start a test, so the arms use the configuration you expect.

## How visitors are assigned

Each visitor is assigned by a stable hash of their consent ID. On a first visit the loader creates the consent ID before assignment, so the visitor keeps the same arm on later visits. A visitor outside the test's target regions sees the normal banner and is not counted.

The loader does not set an extra cookie for the test. The arm a visitor saw is recorded on their consent records, and you can see it as **A/B variant** in [consent logs](https://cookiebeam.com/docs/compliance/consent-records).

## Pause, stop and delete

- **Pause** stops new assignments and serves only the baseline. Data collected so far is kept. Start the test again to resume.
- **Stop Test** ends the test and marks it completed. The loader goes back to serving the normal banner.
- You can delete a draft or completed test. Stop an active or paused test first.

You can only edit an arm's overrides while the test is a draft or paused.

## Read the results

Results appear on the **Current Test** tab and, after the test ends, on the **History** tab.

- **Visitors are counted once per arm.** Each consent ID counts once, using the visitor's latest choice. A default written at page load counts as an exposure, not as a decision.
- **Opt-in rate** is based on explicit choices. On an arm with an opt-out override, a visitor who ran under that override counts as a success unless they explicitly opted out.
- **Automated traffic** is excluded by default. The page shows how many automated visitors were excluded.

**Result readiness** shows two checks:

- **Sample floor.** Every arm needs at least 100 exposures before a leading arm can be named. Directional confidence needs more; the page shows the target for your test.
- **Duration.** The test must run at least 7 full days, so it covers weekday and weekend traffic.

When both checks pass and an arm is ahead of the baseline with a statistically significant difference, the page names it as **leading**. Significance is adjusted for the number of arms you compare. A leading arm is the current front-runner. It is not a declared winner, and more traffic can change it.

If you see **Sample ratio mismatch detected**, traffic did not split as configured. Common causes are blocked, cached or failed arm scripts. Check your deployment before you trust the numbers.

## Troubleshooting

**The create form says you need more themes.** A test compares at least two themes. Duplicate your default theme in **Design**, change it, and try again.

**Start is refused with "legal acknowledgement required".** An arm has a consent-model or regional override. Someone with Publish permission on the banner must record the acknowledgement.

**Start is refused because the banner already has an active test.** Stop the active test first.

**An arm looks out of date.** Arms are built from the published configuration when the test starts. Publish the banner, then pause and start the test again.

**A warning says the CDN cache purge failed.** The test started, but some visitors may get the previous loader until the cache expires. If it persists, pause and start the test again.