State of Consent Compliance — quarterly index methodology

Owner: TagPatrol (Agent Galactica Pte. Ltd.). Cadence: quarterly, first edition Q4 2026 (crawl in week 7–8 of the V1 build plan; publish within 10 working days of crawl close). Status: v1.1 methodology, 11 Sep 2026 (see the change log in Section 10). Purpose: the recurring PR and backlink asset for TagPatrol; a public, reproducible benchmark that agencies and press can cite.

Every number in the report must be reproducible from the stored crawl artefacts (HAR, screenshots, cookie jars, Consent Mode signals). The methodology is published in full alongside each edition; no proprietary scoring hidden from the reader.

1. Sample design

1.1 Markets and size

Three markets, 500 sites each, 1,500 sites per edition.

Expansion candidates for 2027: Spain (AEPD), Italy (Garante), Netherlands (AP), then US/California (opt-out regime; requires different metrics).

1.2 Source lists

Rank by traffic, not by revenue or brand, so that the sample is what residents actually visit.

1. Primary: Cloudflare Radar top domains per country (free, public, monthly; API available). Take the top 700 per country to allow for exclusions.
2. Cross-check: Tranco list (research-grade, combines multiple providers; global, filtered by ccTLD + known local brands). Used to flag anomalies in Radar (CDN and infrastructure domains).
3. Fallback: Similarweb top-websites-by-country page (public top 50 per category) only to sanity-check the top of the list; not licensed for bulk use.

Record the exact list version, date and rank for every domain in index/data/<edition>/sample.csv. The same rank cut-off is used every quarter; domains that drop out of the top 500 are retained for one edition as a "tracked cohort" so quarter-on-quarter comparisons are like-for-like (report both "top 500 this quarter" and "cohort of Q1 sites").

1.3 Exclusions (applied in order, logged with a reason code)

• Infrastructure / CDN / API-only domains (no HTML homepage for a human visitor).
• Adult content, gambling and any site requiring age gate before the homepage renders.
• Government (.gov.uk, .gouv.fr, .bund.de, etc.) and educational domains: reported separately in an annex, excluded from the headline score so the commercial benchmark is clean.
• Sites that refuse our audit runner are not excluded. They are recorded with the product that refused them and reported as the refusal rate (Section 3.6), and they are held out of the scored metrics with the denominator stated on every table. Excluding them silently would be the single biggest unstated bias in a study like this: the sites most likely to run aggressive bot management are the large commercial sites the benchmark is most about, so dropping them would quietly make the sample smaller, richer and better-behaved than the web it claims to describe.
• Duplicates by owning organisation (e.g. amazon.de and amazon.co.uk both remain because they are market-specific properties; google.com and google.de collapse to the market-local one).

Refill from the ranked list until each market has 500 scored sites.

1.4 Pages per site

Three pages: homepage; one deep page discovered from the homepage (first internal link matching product/article/category patterns); and one form or account page if discoverable within one click (login, newsletter, contact). Consent behaviour differs by template; homepage-only studies under-count.

1.5 Vantage points

In-market egress, because CMPs and Consent Mode defaults are geo-scoped by visitor IP and not by browser locale: a UK banner is only observable from the UK. The crawl therefore runs from our own workers in the market being scored, from published static addresses, with locale headers set to en-GB, de-DE, fr-FR. Standard desktop Chromium, 1366×768, no ad-blocker. A mobile-viewport run (390×844) is done on the homepage only and reported as an annex, because the reject-button prominence question is different on mobile.

Not residential proxies, and not by accident. v1.0 of this methodology specified residential-class IPs. That is the industry default and it would measurably raise the sample size, because residential addresses are refused far less often than data-centre ones. It is dropped in v1.1 for two reasons and the trade-off is stated here rather than buried:

1. It contradicts what the crawler tells sites about itself. The runner declares its identity in its User-Agent, publishes its addresses at tagpatrol.app/bot, and obeys a robots.txt group that names it. A crawler that announces itself at the front door and then comes in through a residential address it rents is not declared, whatever its User-Agent says. CLAUDE.md §4 puts that outside the lines, and a compliance benchmark that quietly evades access controls has no standing to report on anyone else's.
2. The refusal itself is a finding. "What share of a market's top sites will not let an independent party measure their consent behaviour?" is a real, citable question that no other index answers, and routing around it destroys the only data that answers it (Section 3.6).

What it costs us, stated plainly. The scored sample will be smaller than v1.0 assumed — on the 20-site engine canary, 8 of 20 large sites refused a data-centre address on 11 Sep 2026. Every headline number therefore carries its denominator and the refusal rate beside it, and a market with a refusal rate above 40% is reported with an explicit caveat that its scored sample is skewed toward sites that permit measurement. Sites that want to be measured can allowlist the runner in one rule; named sites are told how in their correction notice (Section 7).

2. Consent states replayed

For each page, three fresh-profile sessions:

If no banner is detected in S0, the site is scored as "no CMP" and S1/S2 are recorded as not applicable (the site is still scored on pre-consent behaviour).

The reject recipe records click depth: the number of clicks from banner to a completed rejection, and whether a reject control was present on the first layer.

3. Metrics

All metrics are computed per site (any of the three pages failing counts as a site failure unless stated), then aggregated per market and overall. Every metric has a binary site-level flag plus supporting counts.

3.1 Pre-consent marketing rate (headline)

Share of sites where, in S0, at least one request is made to a domain classified as marketing/advertising in the TagPatrol classifier (Meta pixel, Google Ads/DoubleClick conversion or remarketing endpoints, TikTok, LinkedIn Insight, Microsoft UET, Criteo, Taboola/Outbrain, affiliate networks, retargeting DSPs), or a cookie with a known marketing identifier (_fbp, _gcl_au, _gcl_aw, IDE, _ttp, _uetsid, li_fat_id, criteo_*) is set. Reported separately:

• pre-consent analytics rate (GA4 /g/collect with gcs not equal to G100, Adobe, Matomo cloud, Hotjar, Clarity, FullStory);
• pre-consent any third-party rate (broader; includes fonts, CDNs, embedded media; context only).

Classification of unknown domains: deterministic list first (TagPatrol tracker DB, seeded from public blocklists such as EasyPrivacy and DuckDuckGo Tracker Radar), then a Claude Haiku classifier for unknowns, with every model classification stored and human-sampled at 5% per edition.

3.2 Decorative-reject rate

Among sites with a detected banner, share where S1 shows at least one marketing request or marketing cookie set (i) after the reject click on the same page, (ii) on reload, or (iii) on pages 2–3. Reported with sub-rates for each of (i)–(iii), because "leaks on navigation" is a distinct engineering failure from "ignores the click".

A stricter "reject changes nothing" sub-metric: the set of marketing domains contacted in S1 equals the set in S2 (Jaccard similarity ≥ 0.9).

3.3 Consent Mode v2 adoption and correctness

Among sites making any Google tag request:

• Adoption: a gcd parameter is present on Google requests (v2 signals implemented). Also report gcs-only (v1) share.
• Default-granted: first Google hit in S0 carries gcs in {G110, G101, G111} or a gcd string with a granted marker in any of the four signal positions.
• Reject not propagated: S1 Google hits after the reject click still carry any granted marker.
• Implementation type: Basic (no Google measurement hits in S0) vs Advanced (denied cookieless pings in S0) vs Misconfigured (granted in S0). Sites with no Google tags are excluded from the denominator.

Decoding follows the public gcs/gcd scheme (Ahava); the decoder is open-sourced with the report.

3.4 Dark-pattern rate

Among sites with a banner, share exhibiting at least one of the following, each reported individually:

Each pattern has a screenshot saved as evidence. Prominence measurement is automated but every positive is human-reviewed before publication (Section 6).

3.5 Supporting metrics (reported, not scored)

CMP market share by vendor and market; share with a server-side tagging endpoint on a first-party subdomain; median number of third-party domains in S0/S1/S2; share honouring Global Privacy Control header (UK/DE/FR context only, not a legal requirement in these markets).

3.6 Refusal rate — "blocks automated auditing"

Share of sampled sites that would not serve a page to a declared, identified audit runner from a published address. This is a headline number in its own right, not a footnote: it is the coverage caveat on every other metric in the edition, and it is the one statistic here that no other consent index publishes.

A site counts as refusing when the untouched-state load produced no measurement and the response carries the fingerprint of a bot-management product or a blocking status — an interstitial, a 4xx or 5xx, a 404 on the site's own root, or a 200 with a stub body. The detection and its evidence are the same code path the product uses on a customer's own scan (engine/src/coverage.js, engine/src/waf.js), so a refusal in the index is reproducible by the site itself from its own tagpatrol.app scan.

Reported per market:

• refusal rate, and the scored denominator it implies;
• attribution by product (Cloudflare, Akamai, Imperva, DataDome, HUMAN, AWS WAF, Azure, …), which doubles as a bot-management market-share statistic nobody else publishes;
• refusal rate by sector, because the pattern is expected to be strongest in travel, ticketing and finance;
• quarter-on-quarter movement on the fixed cohort, which is the number that shows whether the web is becoming more or less measurable by independent parties.

Opt-outs are counted separately and are never called refusals. A site whose robots.txt names our runner has made a deliberate, legitimate choice; it is excluded from the sample with the reason code opted-out, the count is published, and the site is not named anywhere in the edition.

4. Scoring

4.1 Site score (0–100)

Deterministic, published, no weighting hidden.

Grades: A ≥ 90, B 75–89, C 60–74, D 40–59, F < 40. Grades are for readability; the published table carries the raw score and each component.

4.2 Market and overall index

Market index = mean site score across the 500 scored sites. Overall = mean of the three market indices (equal weight, not traffic-weighted, so one market's platforms do not dominate). Report the median and the distribution as well as the mean; the mean alone hides a bimodal spread.

4.3 Quarter-on-quarter

Report the change on the fixed cohort (sites present in both editions) and on the full top-500 separately. Movement of ± 2 points on the cohort is within crawl noise (established from a re-crawl of 100 sites 48 hours apart at each edition; the observed test-retest agreement is published).

5. Publication format

Per edition:

1. Headline report (web page + PDF, ~2,500 words): five headline numbers, one chart per metric, three-market comparison, top-10 and bottom-10 by sector (not by name in the bottom 10; see Section 7), CMP-vendor comparison, what changed since last quarter, methodology summary, link to full methodology.
2. Data release: index/data/<edition>/scores.csv (domain, market, rank, CMP vendor, each metric flag, score, grade) under CC BY 4.0, plus the decoder and classifier lists on GitHub. HAR files are not published (they may contain third-party identifiers); they are retained privately for 12 months for the correction process.
3. Market one-pagers (UK, DE, FR) in English plus a German and French translation of the one-pager, for local press.
4. Sector cuts on request: retail, media/publishing, travel, finance, SaaS, public sector annex.
5. Newsletter edition and a thread of five charts for social.

Fixed naming: State of Consent Compliance — Q4 2026 (UK · DE · FR). URL pattern tagpatrol.app/state-of-consent/2026-q4. All prior editions stay online.

Every page and PDF carries the disclosure line required by os/policies/disclosure-and-brand.md: built and operated by AI agents under human direction; and the standard "Written by TagPatrol's AI content team and reviewed under human direction."

6. Quality control before publication

1. Automated sanity checks: CMP detection rate compared to the previous edition (± 5 points triggers review); no metric moves > 10 points quarter-on-quarter without a named cause; refusal rate (3.6) compared to the previous edition, where a move of ± 5 points triggers a review of our own egress before it is reported as a change in the web. A low scored denominator is not a reason to delay publication — it is a result — but a market whose refusal rate exceeds 40% is published with the caveat named in 1.5.
2. Human review sample: 5% of sites (75) plus 100% of sites flagged for any dark pattern and 100% of the named top-10 lists, reviewed in a 60-minute session by the board (Ivan) using the stored screenshots and HARs. Reviewer can overturn any flag; overturns are logged and feed the recipe/classifier fixes.
3. Re-crawl of any site whose score changed by > 25 points from the previous edition, before publication.
4. Legal read of the headline report against Section 7 by the operator, with the board signing off the named lists.

7. Legal and ethical notes

Public pages only, as a declared crawler. The crawler visits publicly reachable pages without logging in, creating accounts, submitting forms or bypassing any access control. Its User-Agent carries the TagPatrolBot token and a +URL, every request carries a From contact header, and what it is, where it comes from and how to stop it are documented at tagpatrol.app/bot. Rate limit: one session per site per state, three states, three pages, no more than nine page loads per site per edition, spaced ≥ 2 s. This is below the load of a single human visit with a modern browser.

No evasion, and the index is measured with that constraint accepted. No residential or rotating proxies, no fingerprint or navigator.webdriver spoofing, no CAPTCHA or challenge solving, no attempt to look like a browser we are not. Where a site's bot management refuses us, that is recorded and published (Section 3.6) rather than routed around. A benchmark of other people's compliance that itself evades access controls would not deserve to be cited, and this constraint is the reason the refusal rate exists as a metric at all.

The opt-out is real and it is honoured. A robots.txt group naming TagPatrolBot stops the crawl for that site, including for a page somebody submitted by hand, and the site is dropped from the edition with the reason code opted-out. A blanket User-agent: * disallow is honoured for pages the crawler discovers by itself; it is deliberately not read as a refusal of a scan a site owner requested of their own site, because it is a directive aimed at search indexers. The distinction is documented publicly at tagpatrol.app/bot, not only here.

No personal data. Sessions use fresh, synthetic browser profiles; no real user is observed and no user data is collected. Requests and cookies captured are those a site sends to an anonymous first-time visitor. HAR files can contain third-party identifiers assigned to the synthetic profile; they are stored encrypted, retained for 12 months for the correction process, and never published. The published dataset contains domains, flags and scores only. Where a site's third-party requests reveal the identity of an individual (e.g. a personal blog in the top 500), the site is excluded from named lists.

Statements of observed fact, not law. The report describes what a browser observed ("marketing requests were made before any interaction"). It does not state that any named site is in breach of any law; whether a given behaviour is unlawful depends on facts the crawl cannot see (a valid exemption, a legitimate-interest assessment, a contractual arrangement). The headline report uses "pre-consent", "after reject" and "non-compliant with [regulator]'s published guidance" and never "illegal" or "in breach". Comparative statements about CMP vendors cite the observed configuration on customer sites, not vendor capability, and say so.

Naming. Top-10 lists name sites (positive). Bottom-10 lists are reported by sector and score band, not by name, in the headline report; the full CSV includes every domain with its flags, because a benchmark that hides its data is not reproducible. Any site named in prose gets a copy of its evidence pack (screenshots, request list, timestamps) by email to its published privacy or DPO contact at least 10 working days before publication, with an invitation to correct or comment.

Right to correction. Any site can request its evidence pack at any time via tagpatrol.app/state-of-consent/correction. If the site shows the observation was a crawl error (recipe failure, geo mismatch, transient third-party) or has since fixed the issue, the operator re-crawls within 5 working days and, if the finding does not reproduce, amends the published CSV and report with a dated correction note. Fixed-after-publication sites are marked "remediated" in the next edition rather than deleted, so history stays honest. Disputes the operator cannot resolve go to the board; a formal legal notice is handled per CLAUDE.md §3 (board only).

No commercial pressure. A site's TagPatrol customer status has no effect on its score, its naming, or the timing of any correction. Customers are not identified in the report. The operator does not contact sites in the sample with sales outreach off the back of the index (this is the "no scraped outreach" rule in the V3 README); the correction contact is the only unsolicited email, and it goes only to sites that will be named.

Competitor and vendor references cite public pricing/documentation pages with the date accessed, per the disclosure policy.

Jurisdictional note. Agent Galactica is a Singapore company; the crawl runs from our own workers with published static addresses in the market being scored. Singapore's PDPA does not regulate cookies; the index is an observational study of public web behaviour and processes no personal data of identifiable individuals, so no DPIA is triggered, but a short record of processing is kept in os/policies/ for the crawler.

8. PR plan

Goal per edition: 5+ press mentions in marketing/adtech or privacy trade press per market, 30+ referring domains, 1,000 newsletter sign-ups over the 8 weeks after launch, and inbound free-scan volume of 500+ in launch week.

T−15 days: crawl closes; QC (Section 6); correction notices sent to named sites. T−7 days: embargoed report to 20 named journalists and analysts per market (privacy trade: IAPP Daily Dashboard, Privacy Laws & Business; marketing trade: Marketing Week, The Drum, Horizont, Journal du Net, CB News, Digiday EU; adtech: AdExchanger, ExchangeWire; newsletters: Simo Ahava, MeasureCamp community, Consent-related Substacks). One-to-one, named emails with the market one-pager and an offer of the full CSV; no mass send (V3 rules). Regulators are not pitched. The ICO, CNIL and German DPAs receive a courtesy copy on publication day only, with no ask. T0: publish report, CSV, GitHub repo; newsletter issue; social thread; Product Hunt/Indie Hackers post for the free scanner tied to the index; LinkedIn post from the TagPatrol page (AI-operated label in bio). T+1 to T+14: two follow-up articles per market ("What the UK top 500 get wrong about Consent Mode"; "Les 500 premiers sites français et le bouton Tout refuser"); guest-post offers to two CMP-neutral communities per market; comparison piece on CMP vendors (with data, cited, dated). T+30: "what changed" note for sites that remediated after being contacted; this is the softest and most effective PR beat because it is positive. Ongoing: every edition adds a quarter to the trend line, which is the asset; by Q3 2027 the four-quarter trend is the story.

Assets per edition: 5 charts (PNG + SVG, brand-neutral palette per dataviz conventions), market one-pagers, an embeddable score badge for sites scoring A (drives backlinks; badge links to the site's public score page), a press kit page with methodology summary and contact.

Measurement: referring domains (Ahrefs free/Search Console), press mentions (Google Alerts + manual log), newsletter sign-ups attributed via UTM, free-scan starts from index pages, agency sign-ups mentioning the index. Logged to os/ledger/ weekly per CLAUDE.md §5.

9. Costs and effort (per edition)

• Crawl: 1,500 sites × 9 page loads ≈ 13,500 loads; on the V1 engine (Playwright on OCI ARM / Fly) at ~20 s per load with 4 workers ≈ 19 hours wall-clock, ~$0 marginal compute.
• In-market egress: one worker region per market with a static egress address, at ~US$6.50/mo per machine plus US$3.60/mo per static address — roughly US$30/mo for three markets, continuous, versus the US$200–320 per edition v1.0 budgeted for residential proxy bandwidth. Dropping the proxies makes the index cheaper as well as defensible; the constraint it buys is a smaller scored sample, quantified in 1.5 and published in 3.6.
• Classification of unknown domains: ~2,000 unknowns × ~400 tokens on Haiku ≈ $1.
• Report drafting and editing: Sonnet draft + Opus edit ≈ $5–10 (see pipeline.md).
• Human review: one 60-minute board session.
• Correction handling: operator, ad hoc.

10. Change log

• v1.0 (8 Sep 2026): initial methodology. To be frozen 14 days before the Q4 2026 crawl.
• v1.1 (11 Sep 2026): residential proxies dropped in favour of declared, published in-market egress (1.5, 7, 9); sites that refuse the runner are no longer an exclusion but a published metric with vendor attribution (1.3, 3.6); opt-outs counted and named separately from refusals; the reachability ≥ 95% QC gate replaced with a refusal-rate comparison, since a high refusal rate is now a result rather than a fault (6). Prompted by the engine canary on 11 Sep 2026, where 8 of 20 large sites refused a data-centre address — a number large enough that the v1.0 design would have silently dropped a large, non-random slice of every market. Frozen 14 days before the Q4 2026 crawl; changes after that go to v1.2 with a note in the report.