Architecture

How LayerSift works end to end, and why it is built the way it is. This is the one document to read before changing anything; the other docs describe the surfaces, this one describes the machine.

The design rule

A client report is a claim, not a fact. Every layer of the system exists to move evidence from "the client says so" toward "someone else can vouch for it":

Evidence class	Example	Who vouches
Claims	UA string, screen size, collected hashes	the client — adjustable by definition
Observations	TLS fingerprint, header order, ASN	the client's network stack — forgeable only below the app
Testimony	Web Bot Auth, Play Integrity, App Attest	a third party's signature — unforgeable by the client
History	velocity, ASN profile, payload reuse, baselines	our own records — requires time, not permission
Labels	the integrator's `suspect` verdicts	you — ground truth for the learned model

Detectors report evidence and the server decides, so thresholds retune without shipping a client through an app-store review. Scores are sums of documented weights plus one inspectable fitted model, so every decision can be explained to the customer it was made about.

The pipeline

An identification moves through six stages, each of which can only add suspicion or context — none can clear a name:

agent ──collect──▶ transport ──observe──▶ identify ──score──▶ rules ──▶ advice
  │                  │                       │
  └ integrity key ◀──┘                       ├ replay / payload reuse
  └ HMAC-signed payload                      ├ velocity / device / ASN history
                                             ├ per-environment baselines
                                             └ learned model (if trained)

Collect. The browser agent gathers ~30 entropy sources plus evidence detectors (automation, headless, tamper, VM, remote control, devtools, incognito, privacy), a DOM census, navigation timing, an API-call census correlated against trusted gestures, a pointer trajectory, and the honeypot state. Mobile SDKs collect the platform equivalents plus anti-tamper evidence and, optionally, a Play Integrity verdict token (Android) or an App Attest assertion over the request's canonical string (iOS).
Transport. Behind the first-party proxy the edge contributes what the client cannot edit after the fact: header order, ALPN, TLS version, a JA4 fingerprint. The proxy also serves the agent and the API under your own domain, so blockers have nothing to match.
Integrity. Before identifying, the agent fetches a single-use HMAC key (/api/v4/integrity, five-minute life) and signs its payload. A captured payload resubmitted — or an edited one — fails at the door. The canonical string, the HMAC itself and a per-deploy salt live in a small embedded WASM module (agent/wasm/security.wasm), not in the JavaScript: the build stamps a fresh { buildId, salt } on every deploy (web/.agent-builds.json, server keeps the latest two), so a solver reverse engineered from one deploy stops verifying after the next. The same module solves the proof-of-work, times the engine (an interpreted-wasm/JITed-JS split flags emulation), and derives the key that seals the payload: on the WASM path the body is AES-256-GCM ciphertext, so a proxy log shows an envelope of routing fields and nothing else. The bundle itself rotates too — the build obfuscates several independently seeded variants and the serving route hands out a different one each hour. And each key comes with a program written for it alone (lib/vm): random opcodes, random questions for the browser — a box laid out, a method's native code, a CSS declaration — and, from a datacentre's address or in attack mode, a proof of work. The server writes it again from the key's id, runs it on the answers the agent reports, and reads an answer they do not give as forged, and answers no browser gives as an emulated DOM. Each build writes its programs in a dialect of its own, drawn when it is built. Independent of that, a SHA-256 over the payload catches a replayed result: the same bytes arriving again under a different identity.
Score. The server fans out history lookups (velocity, device stats, the visitor's own ASN profile, per-environment Welford baselines), computes the signal set, and weights it into the 0–100 suspect score. If the environment has trained its model, the model speaks last: up to twenty points of probability mass on top of the heuristics.
Rules. The finished signal set feeds the rules engine. First match in priority order decides among allow, flag, challenge, block.
Advice by default, enforcement where you mount it. The response carries suspectScore, signals, decision, and a signed sessionToken (ab1.…, ten-minute HMAC) so your backend can verify what was decided without another round trip. The token's claims name the signals that fired (sig) and carry a unique id (jti) a replay cache can spend for single-use endpoints. Sessions can also be ended mid-flight: a session_revocations row (Management API) blocks the visitor's next identification, withholds new tokens, and is pollable so an offline verifier can enforce it between identifications. The first-party proxy can go further: with enforcement configured it verifies the _abid receipt at the edge and 403s condemned verdicts before they reach your origin — enforcement you switch on per path, with the token's verdict as the policy, never the proxy's guess. Without that configuration LayerSift never blocks your visitor — only you know what the request was trying to do.

The pulse

Identification scores a moment; the session pulse scores the time between moments. While a session token is live, the agent beacons a small telemetry frame every few seconds (POST /api/v4/pulse, or /_ab/pulse through the proxy): interaction deltas, attended-vs-hidden time in the window, SPA navigations, and runtime drift flags probed against the values the page loaded with. The token is the beacon's only credential — by design, because a stolen token pulsing alongside the real page interleaves sequence numbers and exposes itself.

The server keeps one row per live token (session_states) and re-scores it per beacon with rules in both directions:

damning: runtime drift (UA, platform, webdriver, screen, cores, language — sticky once raised), per-tab sequence regression (replays and token-sharing clones), metronome cadence (three byte-identical activity windows, or per-window motion statistics no hand produces: typing whose flight time barely varies, a pointer gliding at constant speed), script-dispatched events dominating, attended-but-inert windows (capped, and never charged to hidden tabs), a GPU in WebGL that WebGPU's adapters, read after identification, say the machine does not have;
forgiving, a little: coherent human activity lowers the score by a few points, floored near the minted score, and stops the moment any sticky flag exists — a session cannot behave its way out of being caught.

The action only ever hardens: flag at 30, challenge at 50, block at 70, never softer than the token's own. On block the engine writes a token revocation (which every offline verifier picks up on its next poll) and the agent clears _abid and stops pulsing — an edge in deny posture refuses the session from then on. Every beacon lands in session_pulses with its delta and notes, so the Management API can show not just the standing but the exact evidence that produced it (GET /sessions/live/{sessionId}). Rows die with the token and are swept a day later; the pulse keeps no history it does not need.

The waiting room

The queue module turns the verdict into a scarcity primitive. A spot in line (queue_entries) exists only behind a live session token, and the (room, visitor) unique index makes "one spot per visitor" a constraint rather than a policy: a duplicated tab or a second profile behind the same fingerprint is handed its own position back. Verdicts gate the door — sig containing bot is refused outright; a score at or above the room's holdScore takes a ticket the drain never reaches (held). Fairness is pure math (lib/intelligence/queue.ts): arrivals before the opening are shuffled by a per-opening seed (early earns a lottery ticket, not a place), arrivals after queue FIFO by ticket, and the drain is the clock — target = elapsed × ratePerMinute, applied lazily on every enter/poll, so there is no worker to lag and a pause resumes from the clock rather than a burst. Admission is a session token with three queue claims, verifiable offline with the same keyring as verdicts: q names the room (a plain verdict token has none, so it cannot be spent as an admission), qj names the exact session that earned the spot (a scalped ticket presented beside any other session refuses — the ticket-resale brake), and qi tags the entering network when the room binds IPs. Polls re-check revocation, so a session condemned mid-queue loses its place; an admission outliving its token lapses and the line gets the place back; a pause freezes the drain clock so resuming never bursts. Polling presents a signed ticket token (qt1.…), so the sequential ticket number is a display value, not a credential. Rooms are per environment, operated through the Management API (queues:* scopes, audited), and a hosted page at /queue/[room] runs the whole flow through the public agent. The edge enforcer checks admissions natively: ANTIBOT_ENFORCE_QUEUE=<slug> makes protected paths require session + bound admission, refusing or redirecting to the waiting room otherwise.

The challenge

When the verdict is "probably a bot but worth one more test", the press-and-hold flow imposes two costs at once. A proof-of-work (SHA-256 leading-zero-bits, difficulty scaled by the visitor's current suspect score: 14 bits for clean traffic, 17 for a suspected farm) makes each attempt cost CPU, turning "free to try a million times" into "expensive at scale". Hold telemetry — duration, movement, pressure variation, event trustworthiness — asks the solver to also look like a hand. Be clear what that half is: the numbers are reported by the client. The agent only counts trusted events, and the server refuses a short hold, synthetic events and a perfectly straight path, but a caller speaking to the API directly can write plausible numbers itself. The hold raises the price of a lazy script; it does not authenticate a person, and the compute half is what a farm actually pays for.

Challenges are database rows with an atomic single-use claim, so they work across instances and their verdict history doubles as analytics. A passed challenge mints a session token with the ch flag set — but only when the challenge was issued on an identification. Issuance takes the session token the visitor's browser holds and checks it against the requesting client; the resulting challenge is bound to that visitor, session and client, and the solve has to come back from the same client. The token a pass earns carries on what the first said of the browser — its signals (sig) and its handshake (tf) — so a solve lifts a verdict without washing out a conviction. A visitor id alone is not read, not even for the price: the proof-of-work is milliseconds of hashing, so letting a name typed into a request pick who the signed token vouches for would make a solve a free pardon, or an identity for anyone who knew an id, and the price it answered would tell how suspect that id stands. A pass lifts a flag or challenge verdict; it cannot overturn a block (verdictStands, counting the pulse engine's block on that session too) or vouch for a visitor the environment never identified (unknownVisitor).

The learned layer

Two mechanisms learn from the environment's own traffic, deliberately small and inspectable:

Baselines (metric_baselines): Welford accumulators for behavioural timings. Only the fast tail is suspicious — slow is just humans — and a baseline needs a few hundred events before it may speak, so a young site convicts nobody.
Behavioral profiles (behavior_profiles): the same Welford moments, per visitor, of the session's motion statistics — typing cadence, click hold, pointer gait. Trusted sessions alone train it (allowed, low-score, untampered, unreplayed, unflagged — an attacker must not teach the profile their own cadence), three sessions before it may speak, and then a session three sigmas out on two or more features is the account- takeover tell: same device, different hands. The moments describe cadence only — no keystroke identity or pointer coordinate is in them, nor kept anywhere (an identification's trajectory, its last 128 pointer positions, is read for its shape and not stored), so the profile is not biometric identity in the GDPR sense anyone needs to lose sleep over, though it deserves a line in a privacy notice all the same.
The credential corpus (credential_digests): SHA-256 digests of normalized username:password pairs, per environment, sourced from the integrator's breach lists ("feed") and from the engine's own convictions ("observed" — a digest presented by proven automation is a credential being stuffed). Lookups are k-anonymous — the caller sends a 40-bit prefix and compares locally — and a presented digest is stored only when it matched or a bot presented it; a clean session's unmatched digest never touches disk. The match surfaces as ato_credential, corroborative by design: a breached password marks a target, not an attacker.
The model (models): logistic regression over the boolean signal outcomes, trained on your suspect labels, retrained in the background as labels accumulate, exposed through GET /api/management/v1/model with coefficients you can read. It answers the question the heuristics cannot: how often did sessions shaped like this turn out to be bots on this site?

All three follow the same rule as the rest of the system: explainable or absent.

Where the design came from

The capability set was benchmarked against PerimeterX/HUMAN (as deployed on large retail sites) from public sources — their patents and third-party write-ups — and rebuilt from documented behaviour. Two deliberate divergences:

API-call monitoring uses gesture correlation, not script attribution. The PX patents (US 10,951,627 and family) claim attributing API calls to the script that made them. Attribution is also the weaker engineering: renamed scripts, eval and service workers all defeat it. Counting calls that no trusted gesture preceded is a property of the session itself and survives all of that. This is a design note, not legal advice — have counsel review before commercial shipping.
Enforcement stays on the integrator's side. PX blocks at the edge; LayerSift returns a decision and a signed token. The integrator knows what the request was for, so the integrator decides. This makes the product strictly safer to adopt — a false positive shows up as a step-up, never as a locked-out customer LayerSift chose to block.

Where enforcement runs

The decisions — verify the receipt, refuse a condemned one, ask the risk call about the rest, answer with a refusal, the interstitial or a ChallengeRequired, read a sign-in's outcome — are written twice, and everything else is a port of one of the two:

At the edge, proxy/src/enforce.ts and core.ts, written against the Fetch API. The CDN adapters run it in Cloudflare, Fastly, Azure and CloudFront. The forward-auth service runs it beside a reverse proxy the customer operates — nginx, Traefik, Caddy, Envoy — which asks it about each request.
In the application, sdk/node/src/guard.ts. It backs the Express middleware and the Fetch-API guard (Next.js, SvelteKit, Hono). The Python, Go, .NET, PHP and Java SDKs carry ports of it. Their refusal page, ChallengeRequired JSON and interstitial were checked byte for byte against the Node guard's.

Where a request is enforced decides what can be seen of it. Only an edge sees the TLS handshake, and only the edges that hold the page can place the agent on it. Header order survives only where nothing in between kept the headers in a map. A caller that knows it lost the order says so (headersOrdered: false), and the order is then not read.

Watching it run

/api/health answers liveness and asks nothing of the database. /api/ready answers only while the database replies. /api/metrics exposes verdicts, latencies and deliveries to Prometheus, behind a token. Log streams send every assessment and identification, as they happen, to Splunk, Datadog or an HTTPS collector. They are batched per instance, tried twice, and capped in memory. Only counts and the last error are kept.

Data model in one paragraph

Events are the ledger: every identification, with signals, score, and the integrator's suspect label once set. Visitors are derived from events. Session keys, integrity keys and challenges are short-lived secrets and one- time tickets. queue_rooms and queue_entries are the waiting rooms — one row per room, one row per spot in line. metric_baselines and models are the learned state — both rebuildable from events, so losing them costs a retrain, not data. IPs can be pseudonymized to keyed digests per organization; signals that need an address then report unavailable rather than pretend.

Where the system trusts something it did not produce

Every defect found by review in this codebase has had one shape: a value that came from outside was believed because it was there. The rule that follows is short — believe a value only if the party that supplied it could not have chosen it — and these are the places it is applied. Each names the module, so a change near one can be checked against it.

What arrives	Who can choose it	What the system does	Where
The client's address	Anyone, by sending a forwarding header	Believes exactly one header, the one the platform in front of it overwrites (`ANTIBOT_CLIENT_IP_HEADER`); `x-forwarded-for` is read from the right; anything that is not an IP is "unknown"; a header it has been told to ignore is never a fallback	`lib/intelligence/client-ip.ts`
A URL to fetch — a webhook, a log stream, a Web Bot Auth key directory named by an anonymous request	Whoever typed it or sent the header	Refuses private, loopback, link-local and metadata addresses where the connection is made, against every address a name resolves to; never follows a redirect; caps the answer; the anonymous path never opens to private destinations outside development	`lib/net/safe-fetch.ts`, `lib/intelligence/web-bot-auth.ts`
The `User-Agent` header	The client	Never the only evidence. Compared with the engine that actually ran the agent (family and version), with Client Hints, with the Worker's and a fresh frame's `navigator`, and with the transport; a session token is bound to the one it was minted for	`lib/intelligence/engine.ts`, `network.ts`, `lib/api/session-token.ts`
A visitor id in a request body	Anyone who has seen one (they are returned to the page)	Names who a challenge is priced for and nothing else. What a signed token speaks for comes from the stored row of a challenge issued on a verified session token, from the same client	`lib/api/challenge.ts`
Hold telemetry (duration, movement, pressure, trust ratio)	A caller who writes the JSON	Used, and refused when contradictory — but never the only thing: the proof-of-work is the cost, the server's own clock bounds how long a hold could have taken, and the telemetry is documented as the client's own report	`lib/api/challenge.ts`
An evidence section that failed to collect	A page that can make a detector throw or hide the fresh frame	Reported (`tamper.evidenceUnavailable`), never read as "clean", and never a verdict on its own	`agent/src/detectors.ts`, `signals.ts`
Whether an account exists	Anyone, by timing the sign-in	The same work either way	`lib/password.ts`
A forward-auth question — the visitor's address, the request it describes, the TLS headers	Anyone who can reach the service	Believed only with `Antibot-Forward-Secret`; without it every request is refused. The visitor's own `X-Forwarded-For` entries are never the address, only the last one the proxy appended or the address it names	`proxy/src/forward-auth.ts`
`headersOrdered: false` — "the order is not the visitor's"	A caller holding a secret key	Turns off the header-order reading for that request and nothing else; a visitor cannot send it	`app/api/v4/assess/route.ts`, `lib/intelligence/network.ts`
`Upgrade: websocket`	The client	Lets a handshake that offers HTTP/1.1 alone fit its family, which is what a browser's WebSocket connection offers. The cipher and extension counts must still be the family's	`lib/intelligence/ja4.ts`, `proxy/src/ja4.ts`
A verdict's authority	The holder of a token	A `block` is never lifted by a solve; a token minted by solving a challenge cannot start another; a session the pulse engine blocked cannot earn a pardon	`lib/api/challenge.ts`, `session-pulse.ts`

Two limits belong with the table. The hold telemetry and any user-agent string are still reports: the system cannot make them true, only make lying cost something, and it says so wherever they are used. And "believe what the platform overwrites" needs an operator who has said what the platform is, which is why the production checklist leads with it.

Threat model, honestly

What this stops: commodity bot frameworks, replayed payloads, scripted clickers, emulator farms, cloned apps, forged attestation, and the long tail of "the page made a hundred API calls while nobody touched it".

What it does not: a human solving challenges on behalf of a bot farm (that is what farms charge for — the challenge prices their time, which is the point), an attacker with a hardware device farm running your real app (Play Integrity narrows this to genuinely enrolled devices), or an adversary reading this document. Security here is economics: every layer raises the attacker's cost per attempt, and the documentation of the weights is what lets you keep raising it.