Architecture decision records

An ADR records a decision that was expensive to make and would be expensive to reverse. Each one states the situation that forced a choice, the choice, and what living with it costs. The consequences section is the important half — it is where the bill comes due.

If you are about to do something these records rule out, that is a new ADR superseding an old one, not a pull request.

Index

#	Decision	Status
ADR-01	SvelteKit on Bun; Rust owns business logic	Accepted
ADR-02	Modular monolith, catalog split out	Accepted
ADR-03	Multi-tenancy via tenant_id + Postgres RLS	Accepted
ADR-04	The vendor catalog is global	Accepted
ADR-05	Opaque session tokens, not JWT	Accepted
ADR-06	Job queue in Postgres, not Valkey	Accepted
ADR-07	No Kubernetes yet	Accepted
ADR-08	Long syncs are cheap to kill, not hard to kill	Accepted
ADR-09	Envelope encryption for vendor credentials	Accepted

---

ADR-01 — SvelteKit on Bun; Rust owns business logic

Context

Platyn needs server-rendered HTML with fast, form-heavy interactions — quoting screens, art approval queues, production boards — and it needs the correctness properties of a typed, compiled language for pricing, tenancy, and authorization. Those two needs pull in opposite directions.

Putting everything in TypeScript makes the UI pleasant and the money math nervous. Putting everything in Rust makes the money math solid and the UI a chore of hand-rolled templates and Set-Cookie headers.

The failure mode we are steering away from is specific and observed: a Node BFF that starts as a proxy, then “just this once” queries the database directly for a dropdown, then owns a second set of models, and eventually there are two places that decide who can see a price.

Decision

Two processes, with a hard line between them.

SvelteKit on Bun is the only internet-facing process. It renders HTML, owns the session cookie and CSRF, handles redirects and form actions, and calls the Rust API for everything else.

Rust on Axum owns every piece of business logic, every SQL statement, and every authorization decision.

Four invariants make the line enforceable rather than aspirational:

1. Bun never opens a Postgres connection. No Postgres driver appears in web/package.json.
2. Bun never makes a consequential authorization decision. It may hide a button using the permission set the API returned. It may not be the thing that stops a request.
3. Rust never renders HTML and never sets a cookie. JSON in, JSON out; RFC 9457 problem documents for errors.
4. The Rust API is not internet-facing. No ingress, no gateway, no browser-origin CORS.

Bun, rather than Node, because SvelteKit’s adapter-node runs on it unmodified, the container is smaller, cold start is faster, and the toolchain — install, test runner, TypeScript — collapses to one binary.

Consequences

• Every screen costs an API route. A field the UI needs is a change in Rust, then a change in Svelte. This is friction, and it is the point: it makes “just query the DB from the BFF” the expensive option rather than the cheap one.
• One authorization implementation. There is exactly one place to audit and exactly one place to get wrong.
• An extra network hop per request. Sub-millisecond on the internal network, and it buys the API the ability to be called by future clients (a mobile app, a partner integration) without reimplementing anything.
• Two languages to hire for and two toolchains in CI. Accepted deliberately.
• A tempting escape hatch stays closed. When a page needs data the API does not expose, the answer is always “add the endpoint”, never “reach around”.

The invariant that gets broken first

Invariant 2. It breaks quietly — someone checks session.role === 'admin' in a +page.server.ts to decide whether to perform an action rather than whether to render a button. Guard against it in review: in web/, a permission check may only influence markup.

---

ADR-02 — Modular monolith, with the catalog split out

Context

Platyn has six domains that all touch the same customer records inside the same transaction. A deal references a company, a customer, catalog variants, and artwork; changing a deal’s line items and recomputing its totals must be atomic. Microservices would turn that into a distributed transaction to solve an organizational problem this team does not have.

But one part genuinely does not fit the same shape. The catalog sync is a long-running, IO-bound, rate-limited crawl of vendor APIs that produces millions of rows and cares about none of the tenant tables. Its resource profile, deploy cadence, and blast radius are all different.

Decision

A modular monolith: one Cargo workspace, one binary serving the API, with domains as crates that depend on each other only through explicit public interfaces.

api/crates/
  platyn-core       shared types, error taxonomy
  platyn-config     figment config loading
  platyn-telemetry  tracing setup
  platyn-db         pool, tenant transaction helper, migrations
  platyn-jobs       the Postgres job queue
  platyn-xml        vendor XML/SOAP parsing
  platyn-vendors    vendor client implementations
  platyn-catalog    the global catalog domain
  platyn-identity   tenants, users, sessions, roles
  platyn-shop       companies, customers, deals
  platyn-design     artwork, proofs, approvals
  platyn-storage    S3 / MinIO
  platyn-api        Axum router, middleware, handlers
  platyn-testkit    fixtures and test harness

The catalog sync is a separate deployable (platyn-worker) sharing the workspace but not the process. It runs under its own database role, platyn_catalog, which holds zero grants on app.*.

Consequences

• Transactions stay local. A deal update is one BEGIN/COMMIT, not a saga.
• Crate boundaries are compiler-enforced. A layering violation is a build error, not a review comment someone might miss.
• The blast radius of a sync bug is bounded by grants, not by discipline. The sync process cannot write a customer’s order because Postgres will not let it.
• The sync deploys and scales independently of the request path. A catalog crawl chewing memory cannot page the API’s tail latency.
• The monolith will eventually need splitting. The crate boundaries are where the seams already are, so that split is a build-target change rather than a rewrite.
• You can still write a cross-domain query. Nothing physically stops platyn-shop from reaching into platyn-catalog’s tables. Boundaries are enforced at the crate level, not the schema level. Review has to hold that line.

---

ADR-03 — Multi-tenancy via tenant_id + Postgres RLS

Context

Every tenant table needs a tenant_id, and every query needs to filter on it. The question is what enforces the filter.

Application-level filtering — WHERE tenant_id = $1 in each query — fails on the query someone forgets. That query does not error; it returns another shop’s customer list. The bug is invisible in code review, invisible in tests written against a single-tenant fixture, and catastrophic in production.

Database-per-tenant makes migrations, connection pooling, and cross-tenant analytics into a permanent tax, for a product whose tenants are small shops.

Decision

Shared schema, tenant_id UUID on every table in app.*, and Postgres row-level security as the enforcement mechanism. Application code does not filter by tenant; it cannot see rows it should not.

Every tenant table gets the same policy, applied by one function so it cannot drift:

api/migrations/0001_extensions_and_roles.sql

CREATE OR REPLACE FUNCTION app.current_tenant_id() RETURNS uuid
LANGUAGE sql STABLE PARALLEL SAFE AS $$
  SELECT nullif(current_setting('platyn.tenant_id', true), '')::uuid
$$;

CREATE OR REPLACE FUNCTION app.apply_tenant_rls(tbl regclass) RETURNS void
LANGUAGE plpgsql AS $$
BEGIN
  EXECUTE format('ALTER TABLE %s ENABLE ROW LEVEL SECURITY', tbl);
  EXECUTE format('ALTER TABLE %s FORCE  ROW LEVEL SECURITY', tbl);
  -- USING gates reads and which rows UPDATE/DELETE can see.
  -- WITH CHECK stops writing a row stamped with someone else's tenant_id.
  EXECUTE format($p$
    CREATE POLICY tenant_isolation ON %s
      USING      (tenant_id = app.current_tenant_id())
      WITH CHECK (tenant_id = app.current_tenant_id())
  $p$, tbl);
END $$;

Three details are load-bearing.

FORCE ROW LEVEL SECURITY, not just ENABLE. ENABLE exempts the table owner. Connect as the migration user — which is exactly what a hand-run script or a hastily written test does — and isolation silently disappears while every query still returns plausible results. FORCE closes it. The runtime roles are additionally NOBYPASSRLS and are not table owners.

set_config('platyn.tenant_id', $1, true), never SET LOCAL.

// Correct: parameterised, transaction-local.
sqlx::query!("SELECT set_config('platyn.tenant_id', $1, true)", tenant_id.to_string())
    .execute(&mut *tx)
    .await?;

-- Wrong, two ways over.
SET LOCAL platyn.tenant_id = '...';  -- cannot bind a parameter: string interpolation,
                                     -- and therefore an injection surface on the one
                                     -- value that defines the security boundary.
SET       platyn.tenant_id = '...';  -- session-scoped: behind a transaction pooler the
                                     -- setting outlives the transaction and leaks onto
                                     -- the next tenant to borrow that connection.

set_config(..., true) takes bind parameters and is transaction-local. Both problems, one call.

A missing tenant fails closed. Unset, current_setting('platyn.tenant_id', true) returns NULL. tenant_id = NULL is NULL, not true, so every policy admits zero rows. A forgotten begin_tenant() produces an empty list — a bug you notice immediately — never another shop’s data.

Two tables are deliberately outside the pattern:

• app.users — global identity. One human can work for several shops; the user row is not owned by any of them. Access is mediated through app.memberships, which is RLS-scoped.
• app.sessions — session lookup happens before the tenant is known. It is the step that determines the tenant, so it cannot be gated by it.

Consequences

• The dangerous default is safe. The query someone forgets to filter returns nothing.
• All access goes through one helper. platyn-db exposes a transaction constructor that sets the tenant; there is no other way to get a connection in a request handler.
• RLS costs a little planning time per query. Measured in the tens of microseconds at our row counts. Policies are simple equality on an indexed column.
• Migrations must remember SELECT app.apply_tenant_rls('app.new_table') — and forgetting is caught automatically. Migration 0008 adds app.rls_audit(), which returns every table in app.* that carries a tenant_id but lacks ENABLE, FORCE, or a policy, minus the tables declared in app.rls_exemptions. CI asserts it returns zero rows. Two tables are exempted, each with a written reason: app.sessions and app.tokens, both resolved by an unguessable secret before any tenant context exists.
• Partitioned tables need the policy on every partition. RLS on a partitioned parent covers queries routed through the parent; selecting a partition directly bypasses it. app.audit_log is partitioned by month, so 0006 applies the policy to the parent and every existing partition, and app.ensure_audit_partition() applies it to each new month’s partition as it is created. rls_audit() includes relkind = 'p' so a partition created without a policy is caught.
• Tables with legitimate NULL-tenant rows need a hand-written policy. app.jobs (platform jobs) and app.decoration_methods (system defaults) both admit NULL in USING. Only jobs admits it in WITH CHECK; decoration_methods deliberately does not, so no tenant can inject a fake system-wide method. That asymmetry is easy to get wrong and has its own test.
• Testing isolation requires connecting as platyn_app. A test running as the owner without FORCE would pass while proving nothing. See Testing.
• Superuser and pg_dump still see everything. RLS is not encryption. It bounds application bugs, not a compromised database credential.

---

ADR-04 — The vendor catalog is global

Context

The blank goods catalog is the product’s centre of gravity: roughly a million variants across S&S Activewear and SanMar, with prices and inventory that move daily.

The instinct in a multi-tenant system is to make everything tenant-scoped. Applied here it is absurd: PC61 is the same Port & Company Essential Tee for every shop in the country. Copying it per tenant means storing the same million rows N times, syncing the same vendor API N times against a shared rate limit, and getting N slightly different answers to “what colors does this come in”.

What genuinely differs per shop is negotiated cost. Two shops buying the same blank from the same vendor pay different prices based on their account tier.

Decision

catalog.* is global. Synced once, centrally, shared by every tenant. No table in that schema has a tenant_id column, and none ever will — the absence is the enforcement mechanism. A leak would require a migration adding a column, which is not something that happens by accident.

Tenant-specific pricing is an overlay in app.*. app.vendor_accounts holds a shop’s negotiated credentials and account number under RLS; cost adjustments resolve against the global variant at quote time. The platform’s own catalog-read credentials are platform secrets in config, not rows in any tenant’s table.

The catalog is read-only to tenants. platyn_app holds SELECT on catalog.* and nothing more. Writes come only from platyn_catalog.

Consequences

• One sync, one rate-limit budget, one set of vendor credentials for catalog reads.
• Storage and sync cost are constant in tenant count. Onboarding the hundredth shop adds no catalog rows and no vendor API calls.
• Every shop sees the same product data, which is also what shops expect — a discontinued style is discontinued for everyone.
• Caching is trivial. No tenant dimension in the cache key, so one warm cache serves all tenants. Keys embed catalog.version, bumped once per successful sync, so invalidation is key rotation.
• A shop cannot add a private style. Custom or house-brand blanks need a separate tenant-scoped table (app.custom_styles) that unions into search results. That is planned work, not a hole in this decision.
• A bad sync is visible to everyone at once. This is the real cost. It is why runs are gated by sanity checks — a run that would retire an implausible share of a vendor’s styles lands in failed_sanity and changes nothing.
• Vendor-specific spellings must be normalised centrally. catalog.sizes is a canonical registry with catalog.size_aliases mapping every vendor spelling onto it. Unmapped aliases land in catalog.size_alias_review — recorded for a human, never guessed, never silently dropped.

---

ADR-05 — Opaque session tokens, not JWT

Context

The reflexive choice for a modern API is a stateless JWT. Its advantage is that the server does not look anything up.

That advantage is precisely wrong for B2B software. In Platyn, revocation is a product feature:

• An employee is fired at 2 p.m. Their access ends at 2 p.m., not whenever the token expires.
• An owner clicks “sign out all devices” after losing a laptop.
• An admin changes someone’s role from sales to viewer, and the next request must reflect it.
• A tenant is suspended for non-payment and every session under it must stop.

A JWT cannot do any of these without a revocation list — a database lookup on every request, which is the thing statelessness was supposed to avoid, plus the complexity of the JWT.

Decision

Opaque bearer tokens backed by app.sessions.

• 256 bits from a CSPRNG, base64url-encoded. Returned to the client exactly once.
• Stored as token_key = SHA-256(pepper || raw_token), where the pepper is application config, not in the database. A stolen database dump alone cannot be used to forge session lookups.
• Two independent expiries: idle_expires_at slides on use; absolute_expires_at does not move.
• revoked_at is a single UPDATE, effective on the next request with zero propagation delay.
• csrf_secret per session, for the BFF’s double-submit check.
• ip_hash and user_agent for anomaly review, hashed rather than stored raw.

SHA-256 for the token; Argon2id for passwords. These look inconsistent and are not:

	Session token	Password
Entropy	256 bits, uniform	maybe 40 bits, human-chosen
Brute-forcible from a hash?	No — no amount of GPU changes that	Yes, cheaply, with a fast hash
Right primitive	SHA-256 (+ pepper)	Argon2id
Cost per verification	microseconds	~50 ms, on purpose

Argon2 on the session token would add ~50 ms of CPU to every authenticated request and buy nothing: you cannot brute-force a uniform 256-bit secret regardless of how slow the hash is. Using a fast hash on a password would be the mirror-image mistake.

Session state lives in Postgres, not Valkey. A cache that evicts under memory pressure would log everyone out at the worst possible moment.

Consequences

• One indexed lookup per request on sessions.token_key, a unique B-tree hit. Cheaper than the Argon2 verification a JWT-with-denylist scheme would still need.
• Revocation is immediate and simple — the feature that motivated the whole decision.
• The session table needs sweeping. A periodic job deletes rows past absolute_expires_at. sessions_expiry indexes exactly that predicate.
• Sessions are a scaling consideration. At our scale, unremarkable. If it ever matters, a short-TTL read-through cache in Valkey is available — with revocation writing through, and Postgres remaining the source of truth.
• Horizontal scaling needs no sticky sessions. Any replica can resolve any token.
• No third-party can validate a token offline. Correct: nothing third-party should be validating Platyn sessions.

---

ADR-06 — Job queue in Postgres, not Valkey

Context

Platyn has to run work outside the request: catalog syncs, artwork rendering and thumbnailing, proof emails, inventory refreshes, PDF generation.

Redis/Valkey-backed queues are the default reflex. They introduce one specific bug that is very hard to get rid of.

Consider approving a proof, which must (a) write the approval row and (b) enqueue the “notify production” job. With the queue in Valkey these are two different systems:

BEGIN;
  INSERT INTO approvals ...;
  -- enqueue to Valkey here → job runs before COMMIT, reads a row that isn't there
COMMIT;
  -- enqueue to Valkey here → process dies in between, job never runs at all

There is no ordering that is correct. The usual fixes — a transactional outbox, or making every job idempotent and retried forever — are real engineering, spent to work around a database choice.

Decision

The job queue is app.jobs, claimed with SELECT ... FOR UPDATE SKIP LOCKED.

-- Claim a batch. SKIP LOCKED lets N workers poll the same table without
-- blocking each other; each gets a disjoint set of rows.
WITH claimed AS (
    SELECT id
      FROM app.jobs
     WHERE status = 'queued'
       AND run_after <= now()
     ORDER BY priority, run_after, id
     FOR UPDATE SKIP LOCKED
     LIMIT $1
)
UPDATE app.jobs j
   SET status = 'running',
       locked_by = $2,
       locked_until = now() + interval '5 minutes',
       attempts = attempts + 1
  FROM claimed c
 WHERE j.id = c.id
RETURNING j.*;

Three indexes carry the design:

api/migrations/0006_jobs_and_audit.sql

-- Bounded by backlog depth, not table size: still sub-millisecond at year five.
CREATE INDEX jobs_claimable ON app.jobs (queue, priority, run_after, id)
    WHERE status = 'queued';

-- Enqueue-once, enforced by the database rather than by a read-then-check.
CREATE UNIQUE INDEX jobs_idempotent ON app.jobs (kind, idempotency_key)
    WHERE idempotency_key IS NOT NULL AND status IN ('queued','running');

CREATE INDEX jobs_stale ON app.jobs (locked_until) WHERE status = 'running';

jobs_claimable is partial, so the index the claim query walks stays proportional to the backlog rather than to every job ever run. jobs_idempotent is what replaced a 60-line existence check in the old code that guarded against duplicate invoices — the guard is now a unique violation the caller handles.

Platform-level work (catalog sync) has tenant_id IS NULL, so this table needs a hand-written policy rather than the generic helper:

CREATE POLICY tenant_isolation ON app.jobs
    USING      (tenant_id IS NULL OR tenant_id = app.current_tenant_id())
    WITH CHECK (tenant_id IS NULL OR tenant_id = app.current_tenant_id());

Enqueueing is an INSERT in the caller’s transaction:

BEGIN;
  INSERT INTO approvals ...;
  INSERT INTO app.jobs (kind, payload) VALUES ('notify_production', $1);
COMMIT;   -- both, or neither. There is no third outcome.

Valkey is a cache and only a cache. It runs with --save "", --appendonly no, and allkeys-lru — configuration that makes durability impossible rather than merely discouraged, so no one can quietly start depending on it.

The outbox

The same transactional argument applies to effects that leave the system — a webhook, a push to an accounting integration, an email. app.outbox records them in the caller’s transaction:

CREATE TABLE app.outbox (
    id             BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    tenant_id      UUID NOT NULL REFERENCES app.tenants(id) ON DELETE CASCADE,
    aggregate_type TEXT NOT NULL,
    aggregate_id   UUID NOT NULL,
    event_type     TEXT NOT NULL,
    payload        JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at   TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON app.outbox (id) WHERE published_at IS NULL;

A publisher drains rows where published_at IS NULL. Delivery is at-least-once; combined with a per-target idempotency key, the observable result is effectively-once. The partial index means the publisher’s query cost tracks the unpublished backlog, not the full event history.

Consequences

• Transactional enqueue, for free. The dual-write problem does not exist. This alone justifies the decision.
• Jobs are queryable. “What is stuck?” is a SELECT, not a Redis CLI expedition. Failed jobs keep their payload and error, so retrying one is an UPDATE.
• Crash recovery is a lease expiry. A worker that dies mid-job leaves lease_expires_at in the past; a sweep requeues it. No separate reaper process, no boot-time reset that stomps on other replicas’ in-flight work.
• Backups include the queue. Restoring the database restores pending work.
• Throughput ceiling is far lower than Redis — thousands per second rather than hundreds of thousands. Platyn’s real volume is dozens per minute. Three orders of magnitude of headroom.
• Polling produces baseline query load. Mitigated with LISTEN/NOTIFY to wake workers, with a slow poll as the fallback, so an idle system is nearly silent.
• Long-running jobs need stop_grace_period tuning. See ADR-08.
• Dead rows need vacuuming. Completed jobs are archived and deleted on a schedule; the table is high-churn and wants an aggressive autovacuum setting.

---

ADR-07 — No Kubernetes yet

Context

Kubernetes is the default answer to “how do we deploy this”, and it is a real cost: a control plane to run or pay for, manifests, an ingress controller, secret management, cluster upgrades, and a debugging surface where a failing pod might be the app, the scheduler, the CNI, or a PodDisruptionBudget.

That cost is worth paying when you have workloads whose scale is unpredictable and bursty.

Platyn does not. The load pattern is boring by construction:

• Request traffic is a few hundred shop employees on a business-hours curve. Two API replicas and two web replicas cover it, with room to spare.
• Background work is dominated by catalog sync, and catalog sync is rate-limited by the vendors, not by us. S&S and SanMar each permit a fixed request rate. Running ten sync workers against two vendors does not go faster — it goes 429.

That last point is the decisive one. Sync capacity scales with vendor count, not customer count. Adding the hundredth shop adds zero sync work, because the catalog is global. Adding a third vendor adds exactly one worker’s worth. The autoscaling that Kubernetes exists to provide has nothing to autoscale.

Decision

Containers on plain hosts, orchestrated by Podman via podman-compose. Same compose topology locally and in production, differing by env file and image tags.

• web and api — two or more replicas each, behind a load balancer, rolling-restarted.
• worker — one replica per vendor sync lane, plus one for general jobs.
• Postgres — managed (RDS or equivalent) in production. Not a database we run ourselves.
• Valkey — a single container. It is a cache; losing it costs a cold period, not data.

Nothing in the design blocks a later move. Every process is a stateless container reading config from the environment, listening on one port, exposing /health/live and /health/ready, and handling SIGTERM with a graceful drain. That is a Deployment manifest away from running on Kubernetes the day the load pattern justifies it.

Consequences

• Deploys are legible. podman-compose pull && podman-compose up -d. A new engineer can hold the whole deployment in their head on day one.
• Local and production are the same topology, which kills an entire category of “works on my machine”.
• No autoscaling. Capacity is a decision someone makes, not a curve. Given the load profile, that decision changes a few times a year.
• Failover is manual. A host dying means someone acts. Accepted at this stage; the mitigation is a warm standby and a documented runbook, not a control plane.
• Multi-host coordination would be awkward. If it becomes necessary, that is the signal to revisit this ADR — and the trigger is hosts, not customers.
• This ADR has an expiry condition, stated up front: revisit when sustained CPU across API replicas exceeds ~60%, or when more than three hosts are in play, or when a vendor lands whose rate limit is high enough that sync throughput becomes our constraint rather than theirs.

---

ADR-08 — Long syncs are cheap to kill, not hard to kill

Context

A full SanMar catalog sync takes hours. Deploys, host reboots, and OOM kills happen on a schedule that does not consult the sync.

The intuitive response is to protect the sync — long grace periods, “please do not deploy during sync” windows, resume-from-scratch logic. Every one of those makes the sync more precious, and precious background work quietly holds the whole deployment process hostage.

There is also a specific failure to design out. A previous system reset stuck sync tasks on boot: any process starting up marked every running sync as failed. With one replica that was a recovery mechanism. With two, a routine deploy of replica B killed replica A’s healthy in-flight sync — and the more the system scaled, the more reliably it broke itself.

Decision

Make syncs cheap to kill instead of hard to kill.

Per-item checkpoints. A run’s cursor column is a JSONB resume point updated as work completes:

{ "brand_idx": 12, "last_style_id": "PC61", "page": 340 }

Killed at any moment, a run resumes from its last checkpoint. The unit of lost work is one item, not one run.

Leases, not status flags. sync_runs carries lease_owner, lease_expires_at, and heartbeat_at. A worker holds a lease and renews it. A dead worker’s lease simply expires and another picks the run up. A booting replica never touches a run whose lease is live, which removes the reset-on-boot failure entirely.

Cooperative cancellation, 90 second grace. SIGTERM sets a cancellation token. The sync loop checks it at every item boundary, checkpoints, marks the run queued for resume, and exits. The compose service allows 90 seconds — generous for finishing one item, far too short to finish a run, which is the intent.

docker-compose.yml

worker:
  # Long syncs abort at item boundaries; 90s is generous. See ADR-08.
  stop_grace_period: 90s

Database-enforced single-flight. The old read-then-check for “is a sync already running” was a TOCTOU race. A partial unique index makes it an invariant:

api/migrations/0002_catalog.sql

CREATE UNIQUE INDEX sync_runs_one_active
    ON catalog.sync_runs (vendor_id, scope)
    WHERE status IN ('queued','running');

A duplicate enqueue now fails with a unique violation, which the caller handles, instead of producing two workers fighting over the same vendor’s rate limit.

Consequences

• Deploys never wait for syncs. Restart whenever. Worst case, one item is redone.
• The reset-on-boot bug cannot recur. Boot logic keys on lease expiry, and a live lease is untouchable regardless of which replica is starting.
• Every sync step must be idempotent. The item interrupted mid-write will be reprocessed. All catalog writes are upserts keyed on (vendor_id, vendor_style_id) or (vendor_id, vendor_sku), with content_hash short-circuiting unchanged rows.
• Checkpoint writes cost. One small UPDATE per item against a low-row table. Negligible next to the vendor HTTP call it follows.
• Resume needs a stable vendor sort order. If a vendor’s pagination is unstable, the cursor is approximate and the sync may redo or skip a page. Handled by treating the cursor as a hint and relying on idempotent upserts plus the retirement sweep for exactness.
• Observability is required, not optional. rows_seen, rows_changed, rows_retired, error_count, and a sync_events log per run — because “it got killed and resumed” must be distinguishable from “it silently did half the work”.

---

ADR-09 — Envelope encryption for vendor credentials

Context

app.vendor_accounts stores each shop’s own vendor credentials — their S&S account number and API key, their SanMar username and password. These are the shop’s commercial relationship. A leak is not “our users need new passwords”, it is “someone can place orders on our customer’s vendor account”.

pgcrypto is the obvious answer and is wrong here in three ways:

1. The key ends up in the database’s blast radius. Passing it as a query parameter puts it in pg_stat_statements, in query logs, and in the memory of the process you were trying to protect against. Putting it in a Postgres setting puts it in the database.
2. Rotation means decrypting and re-encrypting every row with both keys briefly live.
3. No per-tenant separation. One key encrypts everything, so one disclosure exposes everything.

There is also a subtler requirement: a ciphertext must not be relocatable. Copying the secret_ciphertext bytes from tenant A’s row into tenant B’s row must not yield a credential that decrypts.

Decision

Envelope encryption, with the master key outside the database entirely.

flowchart LR
    KMS["KMS master key<br/>(AWS KMS / Vault)<br/>never leaves the HSM"]
    TK["app.tenant_keys<br/>wrapped_dek · key_version"]
    VA["app.vendor_accounts<br/>secret_ciphertext · nonce · key_id"]

    KMS -->|"unwraps"| TK
    TK -->|"DEK decrypts"| VA

    style KMS fill:#FF5B23,stroke:#D9410F,color:#14131A
    style TK fill:#00A3D9,stroke:#007AA6,color:#14131A
    style VA fill:#E4006C,stroke:#B00054,color:#ffffff

1. Each tenant gets a data encryption key (DEK), generated once, 256-bit.
2. The DEK is wrapped by a KMS master key and the wrapped form is stored in app.tenant_keys. The plaintext DEK is never written anywhere.
3. Credentials are sealed with XChaCha20-Poly1305 under the tenant DEK. Ciphertext, nonce, and secret_key_id go in app.vendor_accounts.
4. At use time the API asks KMS to unwrap the DEK, decrypts, and zeroizes the plaintext (the zeroize crate, plus secrecy to keep it out of Debug output and logs).
5. The AAD binds the ciphertext to its row — tenant_id || vendor_account_id || key_version. Move the bytes to another row and authentication fails. This is what makes relocation impossible.

// The AAD is the whole point of this construction.
let aad = [tenant_id.as_bytes(), account_id.as_bytes(), &key_version.to_le_bytes()].concat();
let sealed = XChaCha20Poly1305::new(dek).encrypt(&nonce, Payload { msg: secret, aad: &aad })?;

Rotation is versioned rather than global. tenant_keys is keyed (tenant_id, key_version); a new version is added, new writes use it, existing rows are re-sealed lazily, and the old version is marked retired_at once nothing references it. secret_rotated_at on each account row makes “which credentials are overdue for rotation” a query.

The platform’s own catalog-read credentials are not in this table. The catalog is global (ADR-04), so those are platform secrets in configuration.

Consequences

• The master key is never in the database, in a query, or in a log. Compromising Postgres — dump, replica, backup — yields wrapped DEKs and ciphertext, not credentials.
• Blast radius is one tenant. A leaked DEK exposes one shop.
• Ciphertexts cannot be relocated between tenants, thanks to the AAD.
• Rotation is incremental, per tenant, without a maintenance window.
• A KMS dependency on a hot-ish path. Mitigated by caching unwrapped DEKs in memory with a short TTL. Cached DEKs are Zeroizing<[u8; 32]> and never leave the process.
• KMS unavailability blocks vendor operations but not the rest of the product. Ordering degrades; quoting, art, and production do not.
• Local development needs a KMS stand-in. A file-backed master key, gated behind PLATYN__ENV=local, refusing to start in any other environment.
• Encrypted columns cannot be searched or indexed. Deliberate. account_number and username are stored in plaintext for display and lookup; only the secret is sealed.