Chrome Extension A/B Testing: What Manifest V3 Allows and How to Measure It
6 min readModerok team
Chrome extension A/B testing in MV3: the Web Store policy that permits it, where the variant assignment lives, and what to send to read the result.
Yes, you can run A/B testing in a Chrome extension, and the Chrome Web Store policy names the pattern: among acceptable uses of remote resources it names "Fetching a remote configuration file for A/B testing or determining enabled features", on the condition that "all logic for the functionality is contained within the extension package". The Manifest V3 constraint is not the experiment, it is where the code lives: both variants ship in your bundle, and remote data only chooses between them. The hard part is the rest: assigning an install once, keeping that assignment across worker restarts, and getting two numbers worth comparing.
TL;DR: Ship both variants in the package. Assign each install once, write the assignment to
chrome.storage.local, and never re-roll it on a later service worker wake. Then send two guarded events per arm, one when the install reaches the tested flow and one when it finishes, with the arm in the event name. Guarding makes each count approximate that arm's installs rather than its clicks, and both arms carry the same bias, so the ratio is worth reading.
Is A/B testing allowed in a Chrome extension?
Chrome's migration guidance on extension security states that "In Manifest V3, all of your extension's logic must be part of the extension package" and that "You can no longer load and execute remotely hosted files according to Chrome Web Store policy," while also saying that "You can still load data remotely at runtime" and offering the pattern "Your extension loads and caches a remote configuration (for example a JSON file) at runtime."
The store's Manifest V3 requirements draw the same line: external resources "must not contain any logic," but "The extension may reference and load data and other information sources that are external to the extension". So {"experiment_enabled": true, "split": 0.5} is data and switching on it is fine; a JSON file carrying a function body you eval is logic, and is not.
Do you need a remote config at all?
A config file buys one thing: switching the experiment off without shipping a version. Chrome's network requests guide says a service worker script can reach servers outside its own origin when the extension requests host permissions, and that a request to another origin is otherwise treated as cross-origin. Host permissions are an exemption from CORS, not the only route: serving the config yourself lets you send the CORS headers instead and declare no new host, a distinction host_permissions vs permissions in Manifest V3 works through. Or put the split in the bundle, which for a first experiment is usually the right trade.
Assign once, then stop deciding
This is where extension experiments break. The background service worker is not a long-lived page. The lifecycle docs say Chrome terminates it "After 30 seconds of inactivity," when a request "takes longer than 5 minutes to process," and when "a fetch() response takes more than 30 seconds to arrive," and are blunt: "Any global variables you set will be lost if the service worker shuts down."
So a variant held in a module-level variable gets re-rolled every time the worker wakes, and one install drifts between A and B all week. Persist it on the first decision, and do the assignment and the event in one function:
import { Moderok } from "@moderok/sdk";
Moderok.init({ appKey: "mk_your_app_key_here" });
const ASSIGNMENT_KEY = "exp_onboarding_v1";
// Serializes storage reads and writes within one worker lifetime.
let chain = Promise.resolve();
function serialize(fn) {
chain = chain.then(() => fn(), () => fn());
return chain;
}
function recordOnce(stage) {
return serialize(async () => {
const stored = await chrome.storage.local.get(ASSIGNMENT_KEY);
let variant = stored[ASSIGNMENT_KEY];
if (typeof variant !== "string") {
variant = Math.random() < 0.5 ? "a" : "b";
await chrome.storage.local.set({ [ASSIGNMENT_KEY]: variant });
}
const eventName = `${stage}_${variant}`;
const flag = `sent_${eventName}`;
const seen = await chrome.storage.local.get(flag);
if (seen[flag]) return variant;
await chrome.storage.local.set({ [flag]: true });
Moderok.track(eventName);
return variant;
});
}
// await recordOnce("exp_onboarding_v1_shown") / ("..._done")
Moderok.init() runs at the top level, before any await, because the SDK registers chrome.runtime.onInstalled inside init() and the service worker events guide requires handlers to "be declared in the global scope" so "they are registered synchronously on initial script execution."
Four things about recordOnce are deliberate. The stage string carries the experiment version, so event names and stored sent_ flags roll over together when you run a v2 on the same flow. There is one serialized function rather than two, because chain would deadlock if a serialized call awaited another. The chain covers only the current worker lifetime, since globals die with the worker, so it removes the read-then-write race between concurrent callers in one wake, not between wakes. And the flag is written before track(), so a worker killed in between loses that event rather than sending it twice. Await the call or attach a .catch: an unhandled rejection from chrome.storage is itself captured as an __error event by default.
Use chrome.storage.local, not chrome.storage.sync: the storage reference says that if the user enables syncing, sync data "syncs with every Chrome browser that the user is logged into," turning a per-install coin into a per-person one. For the same reason, do not derive the variant from an analytics id. No method on the Moderok SDK returns its anonymous id (API reference), and that id may be recovered through sync.
Four event names, each guarded
An outcome count alone cannot be compared: you do not know how many installs in each arm had the chance. The denominator has to be exposure, not assignment, since an install bucketed but never shown the flow belongs in neither number.
Then there is the constraint most A/B testing advice skips. An event chart counts events, not people, so a user who completes the flow four times contributes four. Moderok's user-level charts cover actives, engaged users and retention, but custom events are counted per event name. The sent_ flag keeps that count close to a count of installs:
| Event | When it fires | Role |
|---|---|---|
exp_onboarding_v1_shown_a | Arm A first reaches the flow | Denominator, arm A |
exp_onboarding_v1_shown_b | Arm B first reaches the flow | Denominator, arm B |
exp_onboarding_v1_done_a | Arm A first completes it | Numerator, arm A |
exp_onboarding_v1_done_b | Arm B first completes it | Numerator, arm B |
The arm goes in the name because custom events are charted per name, not by property. Add { variant: "a" } as a property too if you want it on the raw event; properties must be flat (string, number or boolean), and nested values are dropped, as the events and properties guide documents.
Close is not exact. A guarded event can go missing (over the size limit, a full queue, a fatal 4xx, a shutdown before the queue persists), and a retried batch can arrive twice, so a count estimates installs rather than equalling them. The comparison holds because both arms run the same code and carry the same bias, so read the ratio, not the totals. The same discipline makes feature adoption readable in general, covered in Chrome extension usage tracking.
Three things that quietly ruin the result
An update mid-experiment. chrome.runtime.onInstalled fires with reason "update" on every version bump, so installs that predate the experiment join at a different point in their lifetime than new ones. If you are testing onboarding, enroll only installs where onInstalled fired with reason "install".
Uninstalls counted as neutral. Someone who removes the extension on day two never fires the outcome event, so they look identical to someone who stayed and did not convert. If one arm drives removals, the rates hide it.
Calling it too early. Extension experiments are slow: the population is whoever opens your extension, with no page load accumulating samples. Decide the stopping point before you start, and run the result through a significance calculator.
Getting the numbers back
Nothing above needs an experimentation platform. It needs a stable assignment in storage, four guarded event names, and somewhere to count them. If you want the counting handled, Moderok is an analytics SDK for MV3 service workers: 5.7 kB gzipped, zero runtime dependencies, no DOM globals in its source, no remotely hosted code, and installs, updates, daily actives and errors arriving without a track() call. Start with the getting started guide, or see what the dashboard shows.