How ad comparisons work

What the percentage measures

We use the smaller eligible impression count as the size of each arm in a hypothetical equal-sized test. Extra impressions for the larger arm do not compensate for the smaller one. A standard two-proportion normal approximation estimates detection power. The value is rounded down to a whole percent and capped at 99%, even when the calculation approaches one. This cap is a display policy, not a statistical threshold. We show no percentage below 1,000 eligible impressions per ad: our approximation guard requires at least ten expected clicks at the reference baseline.

Where the reference target comes from

Method v1 assumes a click-rate change from 1% to 1.5% (a 50% relative increase), a two-sided 5% significance threshold and independent random assignment. About 7,750 observations per arm give 80% power under this model. The method is established; the CTR assumptions are an illustrative Serge planning reference, not an official advertising benchmark, measured campaign baseline or guarantee. They remain fixed rather than being chosen after seeing which ad looks better. Different baselines or worthwhile improvements require a different sample plan.

Which impressions count

Select exactly two published ads. We count only dates inside the displayed report window with reported positive impressions for both ads, before the date the report was fetched in the ad account’s time zone. The fetch day is excluded because it may be incomplete. Missing dates are unknown, not zero. The panel’s totals can therefore be higher than the eligible counts. The window moves with reporting, so progress can decrease; it is not a lifetime counter.

The same landing page and creative

We compare saved destination URLs after removing the five standard UTM parameters and the OpenAI oppref tracking reference. Other query parameters, URL fragments, subdomains and paths remain distinct. This does not verify redirects or personalized page content. If a destination is missing, the pages differ, or stored history combines published creative versions, we show an explanation instead of a percentage. For a clean new comparison, create fresh ads and keep their creative, destination and delivery settings stable.

Why this is not significance or certainty

OpenAI selects delivery using relevance and expected outcomes. The public controls reviewed do not establish random assignment. Audiences can differ; repeated impressions are not independent people, and shared dates do not prove simultaneous exposure. We cannot correct these issues with aggregate counts. The percentage describes only the hypothetical reference model, not the actual experiment’s power, a p-value, a probability of winning or causal lift. High reference power can coexist with no observed difference. No winner is inferred.

How to use the other results

Click rate describes clicks divided by impressions. Spend describes media cost. Provider conversions follow the provider’s attribution and reporting rules; they are not automatically verified purchases or qualified leads. Website visits and actions are observed through captured landing tags and available consented tracking. Missing tags, blocked tracking and delayed conversions affect coverage. These measures are related and use different denominators, so adding them into one confidence score would double-count evidence. Choose the business outcome you care about and read these results separately.

What to do next

Use one clear creative hypothesis and review observed results alongside lead quality, spending and tracking coverage. A higher click rate need not mean more customers. No score triggers a budget change, and an inconclusive comparison is a valid outcome. Do not keep spending just to raise this reference estimate. A controlled experiment also needs verified random assignment, a chosen outcome, an appropriate sample plan and stopping rules.

Why higher percentages need more data

The curve comes from the power calculation, not a hand-shaped progress bar. These rounded planning points assume both arms have at least the stated number of eligible impressions. A lower observed CTR, smaller true improvement, repeated exposure or missing tracking can make these assumptions unsuitable. Below the approximation guard we show a status, not a percentage.

Reproducible calculation

Let n be the smaller eligible impression count, a = 0.01, b = 0.015, p = (a + b) / 2 and Φ the standard normal cumulative distribution. Use the pooled variance under the null and the nonpooled variance under the alternative, with z = 1.959963984540054. This is the normal-approximation calculation documented by statsmodels for two independent proportions. It does not use the observed difference between the ads.

s₀ = √(2p(1 − p) / n)
s₁ = √((a(1 − a) + b(1 − b)) / n)
δ = b − a
power = Φ((δ − z s₀) / s₁) + Φ((−δ − z s₀) / s₁)

Methodology references

• statsmodels: power for two independent proportions
• NIST: sample size and power assumptions
• NIST: comparing two proportions
• OpenAI: ad selection and delivery
• OpenAI: comparing messaging variations
• Microsoft Research: trustworthy experimentation