How Cookie Scanners Work: Deep Scanning, Live Monitoring, and Why Most CMPs Only Do Half the Job

Why Cookie Scanning Matters More Than Ever

Under GDPR and the ePrivacy Directive, every cookie your website sets must be disclosed and, unless it is strictly necessary, consented to before it reaches the visitor's browser. That applies to cookies you set deliberately and to cookies set by third-party scripts you may not even know about: analytics platforms, ad pixels, chat widgets, video embeds, and A/B testing tools all drop cookies on your domain the moment their JavaScript executes.

The enforcement landscape has sharpened considerably. In September 2025, CNIL fined SHEIN €150 million for installing cookies before consent was given. The UK's ICO surveyed 1,000 websites in early 2025 and issued 134 enforcement warnings for consent violations including pre-ticked boxes and missing reject buttons. Across the EU, regulators now actively scan websites rather than waiting for complaints, meaning undisclosed cookies are a ticking fine.

A cookie scanner is the tool that tells you exactly what your website sets, when, and from which vendor. Without one, you are declaring cookies in your cookie policy based on guesswork. With one, your consent banner and cookie declaration stay accurate and legally defensible.

But not all cookie scanners are equal, and most have a critical blind spot.

The Problem with Scan-Only Approaches

The standard approach to cookie detection is straightforward: a crawler visits your website in a headless browser, records which cookies appear, and gives you a report. OneTrust, Cookiebot, Osano, Termly, every major CMP offers some version of this.

These scans typically run on a schedule: weekly, fortnightly, or monthly. Some only run when you manually trigger them. Between scans, your website is unmonitored.

Here is why that matters:

• Marketing installs a new tracking pixel on Tuesday. Your next scan runs on Sunday. For five days, an undisclosed cookie is being set on every visitor, a compliance violation you do not know about.
• A WordPress plugin auto-updates and adds a new cookie. You will not find out until the next scan, or until a regulator does.
• A developer deploys a new feature with an embedded iframe. The iframe loads third-party scripts that set cookies outside your declared list.
• An ad network changes their cookie names. Your cookie declaration now lists cookies that no longer exist, while new ones go undisclosed.

Periodic scanning captures a snapshot. Compliance requires continuous visibility. This is the gap that most CMPs ignore.

How Cookie Scanners Work: The Deep Scan

A proper cookie scanner does more than check document.cookie in a browser tab. Here is what happens under the hood in a modern headless-browser scanner, using CookieBeam's deep scanner as the reference implementation.

Headless Chrome and the Chrome DevTools Protocol

CookieBeam's scanner runs on AWS Lambda with Puppeteer and a full Chromium browser. The key technology is the Chrome DevTools Protocol (CDP), the same debugging interface that powers Chrome DevTools.

Why CDP matters: the JavaScript API document.cookie cannot see cookies marked as httpOnly. These are cookies explicitly flagged to be invisible to client-side JavaScript, a security best practice used by virtually every server-side session cookie. If your scanner relies on document.cookie, it misses them entirely. CDP's Network.getAllCookies() method returns every cookie the browser holds, including httpOnly cookies, secure cookies, and cookies set by third-party domains in iframes.

For each cookie, the scanner captures: name, domain, path, expiry, httpOnly flag, secure flag, sameSite attribute, vendor, and category.

Beyond Cookies: Scripts, Network Connections, and Client-Side Storage

Modern tracking doesn't stop at cookies. CookieBeam's deep scanner captures three additional categories:

• Scripts (external URLs and inline content): every script that executes during the scan, whether loaded from an external URL or embedded inline in the page
• Network connections (via CDP Network.requestWillBeSent): every outbound HTTP request the page makes, including API calls, tracking pixels, beacon payloads, and iframe-loaded resources. This is a distinct capture layer from scripts. A page might load one analytics script that fires dozens of network connections to different endpoints, and the scanner records each one.
• localStorage and sessionStorage items: persistent and per-tab storage used by analytics platforms, A/B testing tools, and fingerprinting libraries

This matters because the ePrivacy Directive covers all client-side storage and tracking mechanisms, not just cookies. A localStorage entry used for tracking requires the same consent as a tracking cookie, and an undisclosed network connection to an ad server is equally non-compliant.

The Scan Flow: Sitemap to Convergence

Scanning a website is not as simple as visiting the homepage. Cookies vary by page, a checkout page sets different cookies from a blog post. CookieBeam's scanner uses a multi-phase approach:

Phase 1, Seed URL with Consent. The scanner loads your homepage and detects your consent banner. It programmatically accepts consent (supporting CookieBeam, CookieYes, OneTrust, Cookiebot, Complianz, Iubenda, Quantcast, and generic banners) so that all categories of cookies activate, giving a true picture of what a consented visitor would see. It then scrolls the page to trigger lazy-loaded scripts and waits for network activity to settle.

Phase 2, Page Discovery. The scanner fetches your robots.txt, locates your sitemap (including nested sitemaps, something Osano does not support), and builds a page list. If no sitemap is found, it falls back to extracting internal links from the DOM, prioritising high-value paths like /shop, /cart, /blog, /pricing, and /login.

Phase 3, Parallel Group Scanning with Convergence. Pages are grouped by URL path prefix (/shop/*, /blog/*, etc.) and scanned in parallel using 2–10 browser tabs, dynamically adjusted based on memory pressure. Within each group, the scanner stops after three consecutive pages with zero new cookies, a convergence heuristic that avoids scanning hundreds of identical blog posts while ensuring thorough coverage where cookies differ.

There is no hard page cap. Osano limits scans to 500 pages and pauses at 1,000 unclassified items. CookieBeam scans until convergence is reached or the Lambda timeout approaches, whichever comes first.

Layer 2: Live Drift Monitoring for Cookies, Scripts, and Connections

This is where CookieBeam diverges from every other CMP on the market. The consent banner script, the same lightweight script already loaded on every page for displaying the cookie banner, does triple duty as a real-time drift monitor for three categories:

• Cookie drift: unknown cookies appearing between scans
• Script drift: new scripts loading that weren't in the scanner inventory
• Connection drift: new network connections to unknown third parties (detected by wrapping fetch(), XMLHttpRequest.open(), and navigator.sendBeacon())

This is continuous, real-time monitoring running in every visitor's browser, not periodic scanning.

How It Works

When the banner script initializes, it receives baselines from the most recent deep scan: known cookie names, known script URLs, and known connection endpoints. On every page load, it compares what's active on the page against these baselines. Anything not in the known lists gets reported.

For cookies, the script checks at five strategically timed moments:

1. Page load, catches cookies set by inline scripts and server-side responses
2. 500 milliseconds after consent is granted, catches cookies set by scripts that activate immediately after the user accepts the banner
3. 2 seconds after page load, catches slowly-loading third-party scripts (ad networks, social embeds)
4. 10 seconds after page load, catches deferred analytics and advertising scripts that initialize late
5. When the browser tab regains focus, catches cookies set by background processes or scripts triggered by visibility changes

Script and connection drift detection works differently: it wraps the browser APIs that load scripts and make network requests, catching new activity as it happens rather than polling at intervals.

What Gets Reported, And What Does Not

The drift monitor sends minimal data. For cookies: only the cookie name (no values). For scripts: the script URL. For connections: the endpoint URL. No IP addresses. No user agents. No browsing behavior. Each payload contains: the identifier (name or URL), the hostname, which check triggered the detection, and a timestamp.

This is a deliberate privacy-by-design decision. The system is architecturally closer to a Content Security Policy (CSP) violation reporter than a tracking system. It detects compliance violations, not user behavior. Under GDPR, this processing is classifiable as strictly necessary, you are detecting whether your own consent mechanism is working correctly.

Deduplication and Rate Limiting

Without safeguards, a drift monitor on a high-traffic site would generate millions of redundant reports. CookieBeam handles this at multiple levels:

Client-side deduplication. Each browser session tracks which cookie names it has already reported using sessionStorage. If a visitor navigates across ten pages and the same unknown cookie appears on each, it is reported once, not ten times.

Batching. Up to 20 unknown cookie names are combined into a single API request, minimising network overhead.

Server-side rate limiting. The drift reporting endpoint accepts a maximum of 200 reports per minute per IP address. Beyond that, requests receive a 429 Too Many Requests response.

Origin validation. The API verifies that the reporting request comes from a domain registered in the banner's configuration. Reports from unauthorised domains are logged as security events and discarded.

Anti-abuse caps. Each banner is limited to 500 unique cookie-hostname combinations in the drift table. This prevents a malicious script from flooding the system with fabricated cookie names.

Auto-Promotion: The Self-Maintaining Baseline

When the same unknown cookie, script, or connection is reported by 5 unique sessions, CookieBeam automatically promotes it to the known baseline. The pattern is identical for all three drift types:

1. The drift report's hit count reaches 5.
2. The item is inserted into the corresponding detected table (cookies, scripts, or connections), merging it with the scanner baseline.
3. A CDN script rebuild is triggered automatically.
4. The next time the banner script is served, the promoted item is included in the known list.
5. Visitors stop reporting it.

This creates a self-correcting loop: new cookies, scripts, and connections are detected by real visitors in real time, verified by volume (5 independent reports filters out one-off noise), and absorbed into the baseline without manual intervention. Each banner can auto-promote up to 50 items per type. Beyond that, manual review is required, preventing runaway automation.

The result is an inventory that stays accurate between scans, automatically, with zero human effort for the most common scenario: a new legitimate tracker added by a third-party script update.

The Dashboard: Managing Scan Results and Drift Alerts

Both data sources (deep scanner results and live drift reports) feed into a unified dashboard with separate tabs for cookies, scripts, and connections.

The Cookie Details tab displays every cookie detected by either method. The Scripts tab shows detected scripts with their classified vendors and categories. The Connections tab shows all captured network connections. Each tab has a source filter to distinguish scanner-discovered items from drift-detected ones.

The Drift Alerts panel surfaces items detected by the live monitor across all three types. Each alert shows the identifier (cookie name, script URL, or connection endpoint), hostname, hit count, timestamps, and a status indicator. The status lifecycle is:

• New, just detected, not yet reviewed
• Reviewed, acknowledged and under investigation
• Dismissed, investigated and determined to be acceptable
• Auto-promoted, reached the 5-report threshold and merged into the baseline automatically

You can update individual alerts or use bulk actions to transition all alerts of one status to another.

The Banner Overview panel shows an alert badge when new drift items have been detected since the last scan, giving you an at-a-glance compliance health indicator.

CookieBeam vs Osano vs OneTrust vs Cookiebot: Cookie Detection Compared

The following table compares how each platform handles cookie detection. This comparison is based on publicly documented features and verified behavior as of early 2026.

The Osano Privacy Difference

Osano's approach to live cookie detection deserves specific attention because it is architecturally opposite to CookieBeam's. The osano.js script reports all cookies from every visitor back to Osano's servers, not just unknown ones. This is a surveillance-style telemetry model: Osano sees your full cookie inventory on every page load from every user.

CookieBeam's drift monitor reports only the delta: cookies that are not in the known baseline. Once a cookie is known (either from a scan or via auto-promotion), it is never reported again. The vast majority of page loads generate zero drift reports because most cookies are already in the baseline.

The practical difference: CookieBeam's approach transmits orders of magnitude less data, stores no cookie values, and is privacy-minimal by design. If you are evaluating CMPs and privacy engineering matters to your DPO, this distinction is significant.

From Scanner to Blocking Engine: How Scan Results Drive Runtime Enforcement

Most CMPs treat scanning and blocking as separate features. CookieBeam connects them directly: the scanner inventory feeds the script map that the blocking engine uses at runtime.

Here's the chain: the scanner detects a script, classifies it (via the 4-layer pipeline), and assigns it a consent category. When you publish your banner, that classification data is compiled into a script map embedded in the CDN bundle. At runtime, CookieBeam's 5-layer blocking engine checks every script and connection attempt against this map before allowing execution.

The 5 blocking layers work in sequence: document.createElement interception (catches script, iframe, img, and embed elements), DOM insertion hooks (appendChild, insertBefore, replaceChild), MutationObserver fallback (catches anything the first two layers missed), Worker/SharedWorker blocking, and anti-tamper protection (locks neutralized scripts so they can't be re-enabled by other code).

Connection-Level Blocking: A Separate Enforcement Layer

Beyond script blocking, CookieBeam offers connection-level blocking as a separate toggle. This wraps five browser APIs: fetch() returns a 204 response with an X-CookieBeam-Blocked header, XMLHttpRequest fires an error event without making the request, navigator.sendBeacon() returns false, and WebSocket/EventSource constructors return stubs that immediately error or close.

This catches tracking that happens via API calls, not script loading. A marketing script might be allowed to run (it's in the Analytics category and the visitor consented to Analytics), but a specific network connection it makes to an ad server (Marketing category, no consent) gets blocked at the connection level. The blocking is category-aware per connection, with first-party bypass for same-site requests and infrastructure host exclusion.

Few if any CMPs offer connection-level consent blocking.

Privacy by Design: What CookieBeam Does Not Collect

Here's what the drift monitoring system deliberately excludes, since these are common concerns when evaluating any client-side reporting mechanism:

• No cookie values are transmitted. The drift monitor reads document.cookie to extract names, then discards the values. Only the name string is sent to the API.
• No IP addresses are stored. The rate limiter uses the IP for throttling, but drift report rows don't contain IP addresses.
• No user agents or device fingerprints are collected. The drift payload contains the identifier (cookie name, script URL, or connection endpoint), hostname, trigger type, and timestamp. Nothing else.
• No browsing behavior is tracked. The system doesn't record which pages a visitor views, how long they stay, or where they navigate.
• No cross-site tracking. Reports are scoped to a single banner's domain configuration. There's no mechanism to correlate visitors across sites.

This design means the drift monitor itself doesn't require cookie consent. It's a compliance monitoring tool, analogous to a CSP violation reporter, that detects whether your consent mechanism covers all active cookies, scripts, and connections. Under both GDPR and the ePrivacy Directive, monitoring the correctness of your own consent implementation is a legitimate interest and, in most interpretations, a strictly necessary function.

How to Run Your First Cookie Scan with CookieBeam

If you have already set up CookieBeam, running a scan takes three steps:

1. Open your banner dashboard and navigate to the Scanner tab.
2. Click Scan Now. CookieBeam launches a headless Chrome instance that visits your site via the sitemap, accepts consent, and records every cookie, script, network connection, and storage item it finds.
3. Review the results. Auto-classified items appear with their vendor and category pre-filled across the Cookies, Scripts, and Connections tabs. Unclassified items are flagged for manual review. Assign them a category, and they'll feed into both your cookie declaration and the blocking engine's script map on the next publish.

Scans also run automatically on a monthly schedule. If you need more frequent scanning, for example, during a major site redesign or before a regulatory audit, you can trigger ad-hoc scans at any time.

Once your baseline is established, the live drift monitor activates on the next script build. From that point, any new cookie, script, or network connection that appears on your site is detected in real time, reported to your dashboard, and (if it reaches 5 session reports) automatically added to your baseline.

Frequently Asked Questions

Can a cookie scanner detect httpOnly cookies?

Only if it uses the Chrome DevTools Protocol (CDP) or equivalent browser-level APIs. JavaScript-based scanners that read document.cookie can't see httpOnly cookies; the browser hides them from client-side code by design. CookieBeam's deep scanner uses CDP.Network.getAllCookies(), which returns all cookies regardless of the httpOnly flag.

What does CookieBeam's live monitoring detect?

The drift monitor covers three categories: cookie names (via document.cookie), script URLs (by intercepting script loading), and network connections (by wrapping fetch(), XMLHttpRequest, and sendBeacon()). localStorage and sessionStorage monitoring isn't included in drift detection yet. The deep scanner captures both storage types during scheduled scans.

How quickly does live monitoring detect a new cookie?

Within seconds. The drift monitor checks at page load, 500ms after consent, 2 seconds, 10 seconds, and on tab focus. A new cookie set by a third-party script will typically be detected on the 2-second or 10-second check, the first visitor to encounter it triggers a report.

Will the drift monitor slow down my website?

No. The drift check reads document.cookie (a synchronous property access that takes microseconds), compares names against a hash set (O(1) lookup per cookie), and conditionally fires a single fetch request with keepalive: true. On a page with no unknown cookies, which is the typical case, it adds zero network requests.

What happens if an attacker floods the drift endpoint?

Three layers of protection: IP-based rate limiting (200 requests per minute), origin validation (only registered domains accepted), and per-banner caps (500 unique cookie-hostname entries, 50 auto-promotions). Beyond these limits, requests are silently accepted but discarded, no error messages that could aid enumeration.

Does the drift monitor require cookie consent to run?

No. The drift monitor doesn't set any cookies, doesn't store any personal data, and doesn't track user behavior. It monitors whether your consent mechanism is working correctly, a strictly necessary compliance function. This is comparable to how a Content Security Policy reporter doesn't require consent.