LayerSift
Log inStart free

← All docs

Deployment architecture: regions, availability, cost

Launch, 2026-11-10: to keep the bill small, the service starts on one server behind Cloudflare. This page describes what it grows into.

How the hosted service is laid out, and why. Decided 2026-10-26: AWS, one whole stack per data region (United States and European Union at launch; Asia-Pacific later, from the same infrastructure code), Postgres for what the service keeps, Valkey for what it counts, ClickHouse for the identifications themselves.

This replaces the earlier plan (Vercel + Turso, one SQLite-compatible database per region). That plan ran the code unchanged, but its database is one writer per region in one zone, shared by every tenant, with no published SLA — and a database outage is not invisible to customers: one whose server SDK or edge runs with missing: "challenge" (docs/first-party.md) sends a new visitor to the interstitial, and the interstitial needs the API. Built once, properly, the service does not have to be moved later.

What it has to do

  • Data stays in the region the customer chose. A customer picks a region when they sign up — United States or European Union at launch, Asia-Pacific later — as Fingerprint offers (US in Northern Virginia, EU in Frankfurt, each with its own API host, chosen once and not changed afterward). An EU customer's events, visitors, keys and audit log are stored and processed in the EU and nowhere else.
  • The public API is close and always answers. The browser agent and the edge call it on every identification and assessment; a few milliseconds of CPU each, one to three rows written.
  • No single machine, zone or provider incident takes customers' sites down, and a region's own loss is recovered in minutes, inside the same jurisdiction.
  • It scales by configuration, not by a rewrite: more tasks, more capacity units, more shards — never another database.

The property that makes availability easier

Most of a customer's traffic does not reach the API at all. A receipt is verified offline — at the customer's edge (Cloudflare, Fastly, Azure, CloudFront, the forward-auth service) and in every server SDK — with the organization's session keys; the API is asked only about requests that carry none, and every enforcer fails open when it cannot reach it (ANTIBOT_ASSESS_TIMEOUT_MS). With the default missing: "allow", an API outage degrades detection for its length and blocks no one. With missing: "challenge", a visitor arriving without a receipt meets the interstitial, which does need the API: for those customers the API's availability is their site's, which is why every layer below is replicated.

Topology, per region

                         Cloudflare: DNS, TLS, WAF, the agent's script cached at the edge
                                                   │
                     ┌─────────────────────────────┴─────────────────────────────┐
                     │  Application Load Balancer, across three zones             │
                     └─────────────────────────────┬─────────────────────────────┘
                                                   │
                     ECS on Fargate: the app (Next.js), ≥ 2 tasks per zone in use,
                     scaled on CPU and on requests per task
                        │                 │                    │
           ┌────────────┘                 │                    └───────────────┐
           ▼                              ▼                                    ▼
   Aurora PostgreSQL               ElastiCache Serverless              SQS queue → consumer
   Serverless v2: a writer and     (Valkey), multi-zone:               → ClickHouse Cloud,
   a reader in another zone,       rate limits, spent nonces,          replicated across zones:
   failover to it automatic;       velocity counters, hot copies       every identification and
   Global Database secondary in                                        assessment, the dashboards'
   a second region of the same                                         breakdowns, search, usage
   jurisdiction
           │
           ▼
   S3 (same jurisdiction): backups, events past the hot window
Region Primary Disaster recovery (same jurisdiction) API host
United States us-east-1 (N. Virginia) us-west-2 (Oregon) api.antibot.dev
European Union eu-central-1 (Frankfurt) eu-west-1 (Ireland) eu.api.antibot.dev
Asia-Pacific (later) ap-south-1 (Mumbai) ap-southeast-1 (Singapore) ap.api.antibot.dev
  • A region is a whole stack. The same image, deployed once per region, with its own database, cache, queue and event store. Everything an organization has — memberships, keys, events, visitors, rules, webhooks, audit log — lives in its region. No row of customer data is replicated across jurisdictions; the disaster-recovery copy stays inside the same one.
  • What lives where, by how it is used:
    • Postgres — what the service keeps: organizations, users, keys, environments, rules, webhooks, the audit log, and the identity graph (visitors, their component hashes, behavior profiles, sessions). Small rows, read by key, written by the request that owns them, with transactions.
    • Valkey — what the service counts: rate-limit windows, spent nonces and integrity keys, per-address and per-device velocity, attack-mode state, the hot copies every instance now keeps for a few seconds. Many small writes a second, each worth nothing after minutes.
    • ClickHouse — what the service saw: every identification and assessment, appended once, read in bulk by the dashboards, search and usage. Written off the response path through the queue, so a slow event store never slows an answer.
  • The region is fixed at sign-up, as Fingerprint's is. Moving an organization later is an export and an import, done by us, not a setting.
  • API hosts are regional: api.antibot.dev (US), eu.api.antibot.dev, ap.api.antibot.dev — the hosts the agent's region option already uses. The dashboard's integration page renders its own region's host into every snippet.
  • Intelligence shared between customers (network reputation, public blocklists) is kept per region: an address seen misbehaving on an EU site is personal data, and is not sent to the US stack.

The directory: the one piece every region shares

app.antibot.dev (sign-up and the sign-in router) and every stack consult a small directory that holds no customer data: a keyed digest (HMAC) of each account's email address with its region, each organization's id with its region, and a keyed digest of each email domain an organization proved for single sign-on. It is a Postgres database of its own in us-east-1 (ANTIBOT_DIRECTORY_URL), which the EU stack reaches across regions over TLS; it is asked at sign-up, sign-in and single sign-on, never on an API call. Losing it stops new sign-ups and the sign-in router, nothing else — each region's own sign-in page, API and dashboards keep working.

The digests are keyed, so a copy of the directory is not a list of addresses one can check guesses against; they are still pseudonymous data of EU users held in the US, which the DPA names (section 5, [DIRECTORY REGION]) for the legal review before launch to confirm.

Availability

Failure What happens What is lost
A task, or a zone's tasks The load balancer stops sending to it; ECS starts replacements in the remaining zones Nothing
The database's writer With a reader (aurora_readers = 1, the HA shape in infra/aws/README.md): Aurora promotes it in another zone; AWS states failover typically completes within 30 seconds, and service is restored in under 60. On the lean defaults there is no reader: Aurora replaces the writer, which takes minutes. The app's pool reconnects either way (lib/db/index.ts) Writes for that half minute — for those minutes without a reader: identifications without a receipt wait or fail open
The cache ElastiCache Serverless is replicated across zones by the service. Out of reach, each request is answered without it, and readiness stays 200 (saying valkey: "unavailable"), so no task is taken out or replaced for it Nothing; at worst counters restart their window. Out of reach: velocity, for as long as it lasts
The event store Events wait in SQS (kept up to 14 days) and are written when it is back Dashboards lag; nothing is lost
The region Fail the Global Database over to the second region of the same jurisdiction, and run the stack there from the same infrastructure code. AWS's guide puts an unplanned failover's data loss at seconds (the replication lag at the moment) and its recovery in the order of minutes The last seconds of writes; detection without receipts for the switch; customers on missing: "allow" unaffected
The directory New sign-ups and the sign-in router stop Nothing else

What it costs

At launch, two regions (US and EU), each with the smallest highly available shape — a writer and a reader at Aurora Serverless v2's minimum, Valkey serverless, two app tasks per zone in use, a load balancer, the queue, and ClickHouse Cloud's smallest service — and the directory: an estimated $1,400–1,800 a month, read from the providers' public price pages on 2026-10-26. It grows with traffic mostly through ClickHouse storage and Aurora capacity units; the per-service figures are to be confirmed in the AWS Pricing Calculator, and the Global Database secondaries' cost decided, before ordering (Phase 4 below).

What the code does for it

Built:

  1. Postgres (Phase 1): the schema (lib/db/schema.ts, migrations in drizzle/) runs on Postgres through a pool per instance (DATABASE_URL postgres://…?sslmode=verify-full, ANTIBOT_DB_POOL connections, 10 by default). A request waits at most ANTIBOT_DB_CONNECT_TIMEOUT_MS (5,000) for a connection, a statement runs at most ANTIBOT_DB_STATEMENT_TIMEOUT_MS (15,000) and a transaction left idle holds one at most ANTIBOT_DB_IDLE_IN_TRANSACTION_MS (30,000), so a database that slows down — a failover, a lock held too long — fails requests fast, and readiness says why, instead of queueing every one behind it. A connection on which nothing comes back at all — an instance gone in a failover without closing it — is given up on five seconds past the statement's limit, and the pool opens another; idle connections are probed (TCP keepalive) after ten seconds. Migrations run on a connection of their own, held to neither. Every instance that starts after a deploy takes the same advisory lock before migrating, so they migrate one after another. A developer's machine and the tests run the same SQL on PGlite (Postgres compiled to WebAssembly, in-process): .data/pglite by default, DATABASE_URL=pglite:memory in the tests. The database must compare text byte by byte (LC_COLLATE 'C') — a migration refuses any other — because id ranges and keyset pages assume it.

  2. The region a stack serves: ANTIBOT_REGION (us, eu, ap), and ANTIBOT_REGIONS, every region's public URLs as JSON — {"us":{"app":"https://app.antibot.dev","api":"https://api.antibot.dev"},"eu":{"app":"https://eu.app.antibot.dev","api":"https://eu.api.antibot.dev"},…}. The integration page and its snippets give the stack's own API host. Unset, there is one stack and nothing below happens.

  3. Choosing the region at sign-up: the sign-up page shows where the data will live, this region selected, the others a link to their own sign-up — a region signs its own accounts up. An address is claimed in the directory before the account is made, so one address has one account across every region; while the directory cannot be reached, sign-up waits ("try again shortly") rather than risk two.

  4. The sign-in router: an address with no account in this region, held by another, is sent to that region's sign-in with the address filled in (the password is typed again; it is never carried between regions). Every region's sign-in works on its own.

  5. The directory (lib/directory.ts): ANTIBOT_DIRECTORY_URL (a Postgres every stack shares) and ANTIBOT_DIRECTORY_KEY (the same in every stack, at least 32 characters; the directory keeps a check of the first stack's key and refuses any other, since a stack with another key would miss every lookup): an HMAC of each address, never the address, with its region, and each organization's id with its region. Deleting an account frees its address; deleting an organization forgets it. A sign-up's claim is unproven until its inbox follows a link (verification, signed in to the account; a reset; a mailed invitation); an account made by single sign-on or SCIM claims its address proven, and takes over an unproven claim held in another region, so an address someone registered elsewhere without reading its mail does not keep its owner out of their company's sign-in. The inbox behind an address ([email protected] is [email protected]'s) is claimed too: one free trial per inbox across every region, as within one.

  6. Single sign-on across regions: a domain an organization proves is claimed in the directory, so it is one organization's in every region, as it is within one; the directory also knows whether that organization requires single sign-on. /login/sso in any region sends an address on that domain to the organization's region, filled in; a sign-up anywhere is refused for an address whose organization requires single sign-on; an account made by single sign-on or SCIM claims its address like a sign-up, and is refused when the address has a proven account in another region.

    Known limits, by design: an organization's required single sign-on is enforced in its own region; an account on its domain that lives in another region (made before the domain was proved) still signs in there with its password — it holds nothing of that organization, whose memberships are in its region, and refusing it would leave its owner with no way in anywhere. The sign-in router tells whether an address has an account in another region, as the sign-up form tells whether it has one here. And every stack holding the directory's credentials can write any row: the directory trusts the stacks it is shared with.

    What a deletion tells it, when it is out of reach. A deletion goes on whether the directory answers or not, and logs what it could not tell it. An organization deleted then is forgotten later: the timers (or /api/cron, its directory job) find this region's organizations that hold a domain in the directory and are gone from its database, and forget them (forgetDeletedOrganizations) — left, a domain it required single sign-on of stayed refused to password sign-up in every region, and no other organization could prove it. An account deleted then — or a sign-up that failed after claiming its address, or an account erased again after a restore — leaves its address and inbox held by its region, with a keyed digest of a deleted person's address kept. The timers find them too, once an hour an instance (forgetDeletedAccounts): every account of the region is read and digested, and the region's claims that no account has, made more than an hour before, are let go. Nothing is replayed: an address signed up again in the meantime has an account and stays held, and a claim made again is dated anew. A sweep that would let go of more than a tenth of the region's claims (and more than twenty) lets go of none and logs it — an emptied or restored database looks like every account deleted.

    Opening a second region after running one. A stack that ran alone kept no directory, so the directory starts out knowing none of its accounts: until it does, an address with an account there could be given a second one in the new region. Set ANTIBOT_REGION, ANTIBOT_REGIONS and the directory on the first stack before the second takes sign-ups; its timers (or /api/cron) then record what it holds (lib/directory-backfill.ts) — every account's address, proven where it was proved, its inbox, the trial of an account that made an organization, every organization, every proved domain and whether it requires single sign-on — 500 rows a run, going on where the last stopped. It logs antibot: directory: region <r> recorded in full when it is done; open the new region's sign-ups after that line. Anything another region already held is left to it and logged by this region's ids. Trials of accounts deleted before the directory existed are not carried over (the region kept only a digest of their inboxes, under a key of its own).

  7. Deleting what others refer to: every delete of a row others refer to applies the schema's rules itself (lib/db/cascade.ts, test:cascade), and an environment is emptied in batches, table by table, before its own row goes: a large environment's millions of identifications never hold the database in one statement, and the caller learns what was destroyed.

  8. Every query served by an index at production's size: test:query-plans and test:identify plan each statement the code ran as Postgres would on a full database (scripts/lib/plans.ts), and fail on one that reads a growing table whole or walks a whole tenant to find a few rows. They ask PGlite, which is Postgres 18 and runs one statement at a time; production is 17, statements side by side. So test:query-plans-postgres asks a real Postgres 17's planner the same, and test:suites-postgres runs the suites written for PGlite each on a real Postgres through production's driver — which found a test that raced two owners without the lock the app takes.

  9. Short-lived state in Valkey (Phase 2, begun): the shared rate-limit windows, spent nonces and integrity keys — written on every identification and sign-in — go through one small store (lib/hot/index.ts: count, claim, put, take). With ANTIBOT_VALKEY_URL it is Valkey (ANTIBOT_VALKEY_CLUSTER for ElastiCache Serverless's cluster endpoint; its keys begin ANTIBOT_VALKEY_PREFIX, antibot: by default, for a cache shared with something else); without it, the database's hot_state table, so one instance, a developer's machine and the tests need nothing more. While a Valkey that is configured does not answer — a failover, a node replaced — that table stands in for it: the first call that fails goes to the database, every call after goes straight there rather than waiting out Valkey's one-second timeout past an edge's deadline, and a ping every five seconds hands the calls back once Valkey answers. A key issued or a nonce spent meanwhile is still found in the table after, by every instance — one whose own calls never failed too, since a failover reaches instances one at a time: whether the table holds anything alive is asked of it once a second, and with Valkey nothing else writes it. test:hot-store holds both to the same answers (Valkey when ANTIBOT_TEST_VALKEY_URL names one: CI's Valkey service, and Garnet on a developer's machine).

  10. Identification without reading the log (Phase 3, begun): what an identification asks of the ones before it is kept beside them as they are written (lib/intelligence/identifications.ts), not read back from them — the visitor's latest verdict and place, the visitors an address had, the networks a visitor came from (Postgres), and the device's and the address's velocity and the last hour's payloads (series in the short-lived store: a sorted set each in Valkey). Trial limits and usage count calls from the hourly counters. So the log can be written off the response path without an identification missing the seconds before it. And the log is read and written in one place (lib/event-log: search, histories, the dashboards, usage, training, erasure, retention), so that ClickHouse keeps it by changing that place alone; a lint rule keeps the rest out.

Still to do, in order:

  1. Valkey, the rest (Phase 2, when the load tests ask for it): the velocity counters (*_sightings, trap_hits, the content bands) and the session pulses' state. Each counts one sighting along several dimensions at once (an address's accounts, an account's addresses, a device's accounts), which in Valkey is a structure per dimension kept in step by script; and all of it lives ten minutes, so moving it later moves no data. Decided by measurement: if Phase 4's load tests show these writes holding Aurora back, they move; until then they stay in Postgres, indexed and swept.
  2. The queue and ClickHouse (Phase 3, the rest): identifications and assessments written to SQS and from there to ClickHouse; the dashboards, search and usage read ClickHouse; an identification kept by id in Valkey for the minutes before ClickHouse has it (a server asks for it moments after its browser was identified); Postgres keeps the identity graph only.
  3. Infrastructure as code, and running it (Phase 4): OpenTofu for each region (network, Aurora with its Global Database secondary, ElastiCache, SQS, ECS, the load balancer, S3, secrets, alarms) and for the directory; the database created with LC_COLLATE 'C'; the RDS certificate bundle in NODE_EXTRA_CA_CERTS for sslmode=verify-full; CI that builds one image and deploys it region by region; load tests (k6) at the target rates; runbooks for a writer failover, a region switch and a restore. docs/launch.md is rewritten for it then. Begun: one region's stack — the network, Aurora (its database made C by a setup task, psql inside the VPC), ElastiCache, ECS behind the load balancer, ECR, the app's secret, alarms — and the image (web/Dockerfile, the RDS bundle in it) are in infra/aws, written and checked offline (it parses, it is formatted, every reference resolves) but not yet through tofu validate or applied; the Global Database secondary, the directory, SQS and S3, and the pipeline are still to come. Of the runbooks, the restore is written (infra/aws/README.md, "Restoring the database"), its reconcile tested between two real databases and the rest to rehearse; the writer failover is Aurora's own, and the region switch waits for the second region.

Why the app must sit next to its database

Every statement is a network round trip, and some of a call's statements wait for others. Measured on the in-process server with every database call made to wait a fixed time (2026-10-26, before the move to Postgres; the number of statements per call is what matters, and it is the same):

Round trip Assessment (navigation, p50) integrity identify A browser's identification (both)
0 ms (in-process) 7 ms 11 ms 68 ms ~80 ms
2 ms 16 ms 19 ms 78 ms ~97 ms
10 ms 48 ms 46 ms 166 ms ~212 ms

An assessment waits on about four round trips one after another, integrity four, identify about ten (it reads and writes the visitor, its history and the event). So the tasks run in the same region as Aurora — well under a millisecond a round trip inside a region — and an edge's assessment (it waits 250 ms by default, then lets the request through) has its time to spare. The directory is the exception that can be far: it is asked only at sign-up and sign-in.

What one instance holds

Measured on a developer's Windows machine — one in-process server (NODE_ENV development, TypeScript compiled as it loads), Postgres and Valkey (Garnet) on the same machine, address lookups off — with assessments sent as an edge or a guard sends them, 8 at a time, on 2026-10-28 after what the rest of this section describes:

Traffic Assessments a second p50 / p99 Statements each
People browsing (many addresses, a page each) 555 14 / 21 ms 3.5
A credential-stuffing flood (50 addresses, a new account each) 315 25 / 35 ms 7.8
One scraping address 668 12 / 20 ms 2.5
GraphQL sign-ins batched as aliases 527 15 / 22 ms 3.2

First measured the day before: browsing 311 a second with 7 statements each (339 at 32 at a time, 94 / 118 ms), the flood 220–242 with 11, one scraping address 405 with 5, GraphQL sign-ins 340 with 6.

The process, not the database, is what fills: a pool of 30 connections instead of 10 changed nothing (339 → 346 a second), and past 8 at a time only the wait grows. Of the process's own time, about a quarter was the ORM building each statement's SQL anew; the hot path's statements that no cache answers — the key, the address's record, the traps, the reputation — are prepared now (lib/db/prepared.ts), built once and parsed once per connection: browsing went from 317 to 370–402 a second, one scraping address from 367 to about 430, the flood from 231 to 243–261 (8 at a time, the same machine, back to back). Then, on 2026-10-28: the copies each process keeps of an environment's rules, request filters and threat feed (lib/api/hot-cache.ts) had kept the query, not its rows — a Drizzle query runs each time it is awaited — so every request ran them anyway, and a miss twice. Kept as rows, a browsing assessment takes 3.5 statements, not 7: browsing went from 396 to 529 a second, one scraping address from 436 to 681, the flood from 258 to 312, GraphQL sign-ins from 387 to 502 (back to back, as above). A flood's cost to the database grew with the flood: each sign-in counted every account its address had tried in the ten minutes, so from five addresses a minute in, an assessment read 1,541 rows on average and rising (toward 36,000 at a ten-minute window's end). Each count now reads at most 1,000 of them (VELOCITY_SCAN_CAP, far above any threshold): 762 on average over the same minute, and no more after it. So an instance is sized by CPU — about 500 assessments a second for each vCPU on this measure, a lower bound for a production build on Linux — and tasks are added as the traffic grows; Aurora is not what an assessment waits on. A pooler in transaction mode between the tasks and Aurora (PgBouncer; RDS Proxy, which would pin every connection) would need ANTIBOT_DB_NAMED_STATEMENTS=off: the SQL is still built once, and Postgres parses it each time (330 a second). Taken with pnpm load:assess (web/scripts/load-assess.mjs: the in-process server on a fresh Postgres).

Identifications, measured 2026-10-28 the same way with pnpm load:identify (web/scripts/load-identify.mjs): a real agent's payload, 13 KB, captured once from the served agent in headless Chrome in its cleartext form, then sent with a fresh nonce each time and a device of its own (a first visit) or one of three hundred returning ones:

Traffic Identifications a second p50 / p99 Statements each
First visits, 8 at a time 141 55 / 93 ms 17
Returning visitors, 8 at a time 157 50 / 76 ms 15
First visits, 32 at a time 140 223 / 403 ms 17

(The first two rows as they went, on 2026-10-28: 106 and 114 a second, with 23 and 21 statements; 116 and 125, 20 and 18, once the copies above kept rows; 126 and 134, 17 and 15, once they held an environment's webhooks and baselines and a host's custom domain too, each forgotten by what writes it; and these, once the visitor's own lookups were prepared. The third row was 100 a second, 309 / 472 ms, before all of it.)

Since 2026-11-02 a first visit asks nothing of the visitor's past: its id was drawn a moment before, so its step-up, revocation, networks, learned cadence, holds and last place were six statements answered with nothing. Counted by pnpm test:statements — which holds a first visit, a returning one and a sign-in to a budget of statements — a first visit runs 21 where it ran 27 without Valkey, 13 with it (Valkey takes the other 8). A returning visit reads that past whole in one statement (lib/intelligence/visitor-past.ts), six lookups by the same indexes, each a column of one row: 17 where it ran 22 without Valkey, 8 where it ran 13 with it. And a sign-in assessed in a flood reads what its address, account, inbox, device and history have seen in one statement (lib/intelligence/account-velocity.ts), where it was four or five: 10 where it ran 13 without Valkey, 5 where it ran 8 with it. None is yet measured under load.

The process again: a pool of 30 connections (109 a second) or commits that do not wait for the disk (114) change little, and past 8 at a time only the wait grows. The identification's lookups are prepared as the assessment's are; its writes are not — the largest, the identification's own row, is about a tenth of the process's busy time, and that is mostly compressing its components and signals, which a prepared insert does as well: one prepared measured no faster (143 and 144 a second). A level of compression below the default saves little either: they are mostly hashes.

A statement prepared by name is planned, by default, for the values of its first five runs, and after that Postgres may keep one plan for any values. For a visitor's exact match — strict_hash = $3 ORDER BY last_seen_at LIMIT 1 — that plan walked the environment's visitors newest first: about 6,000 rows read for every new device of the load test, where 11 were. Each connection the app opens therefore asks for a plan for each run's values (plan_cache_mode=force_custom_plan, lib/db/index.ts; pnpm test:db-pool). So about 140 identifications a second for each vCPU, a lower bound as above — and a page identifies once a visit, where every protected request is assessed.

What each call writes, which is what Aurora bills writes by (both load tests print it, from the write-ahead log's position): an identification about 10 KB of log (a first visit 10.4, a returning visitor 9.4), a browsing assessment none, an assessment in a credential-stuffing flood about 1 KB. Aurora counts a write I/O per 4 KB of log and writes no full pages, so an identification is about three write I/Os. What one keeps: about 6 KB a real browser's identification — its components compressed (3.7 KB), its signals, its row, its share of the table's indexes — where the probes' 21,000 stored identifications, many of them synthetic, average 3.1 KB.

Sources (read 2026-10-26)