verae-staff-iam/docs/PROD-CA-CUTOVER-RUNBOOK.md

Production CA cutover runbook (READY package)

Status: READY package — operator cutover checklist for live device-cert mint / renew / revoke.
Not shipped in staff-iam code. Default POST /enroll/csr stays 501. Lab CA stays default off and is refused when NODE_ENV=production.

This package tells ops how to cut over to a real production CA without using scripts/lab-test-ca.sh or LAB_CA_ENABLE. It does not implement a CA inside the IAM Node process.

Companion	Role
PKI-MINT-BACKLOG.md	Shipped surface + live-mint product checklist
AUTHZ-ENROLL-DESIGN.md	AuthN/AuthZ gates before dropping default 501
CRL-OCSP-DEPLOY-CHECKLIST.md	Revocation distribution at proxy/CA
PRODUCTION-PKI-REFUSE.md	What stays refused under production defaults
TEST-CA-DESIGN.md	Lab fixture only — never production trust
ENROLL.md	mTLS states, proxy headers, status contract

---

Hard rules (do not violate on cutover)

1. LAB_CA_ENABLE default off — never set in production images, systemd units, or k8s manifests.
2. NODE_ENV=production refuses lab even if someone sets LAB_CA_ENABLE=1 (server gate).
3. Do not trust lab-pki/ roots on staff fleets or production reverse proxies.
4. Do not drop default 501 until a real Signer + AuthZ (see AuthZ package) pass go/no-go.
5. Private keys never enter the IAM user store, audit log, or enroll response as PKCS#12 passwords / leaf keys.
6. Proxy remains TLS SoT for “cert present on this request”; IAM mtlsEnrolled is enrollment policy only.

---

Choose production CA model (product decision — pick one)

Model	When to choose	staff-iam role
A. External corporate PKI (ADCS, Venafi, smallstep, internal Intermediate)	Org already issues workstation client certs	Stay proxy-only; optional later Signer HTTP client
B. EST / SCEP / ACME device profile	Managed fleet enrollment at edge	Edge enrolls; IAM still proxy headers + flag
C. HSM-backed delegated Signer service	Product wants in-app POST /enroll/csr live mint	IAM calls Signer over mTLS; never holds CA key
D. Keep external-only forever	Lowest risk; current default	No code change; this runbook stops at proxy cutover

Recommended first cutover: Model A or B (no IAM crypto). Model C only after AuthZ package + Signer contract land.

Document the choice out-of-band (ticket / security design). Do not invent CA hostnames in git.

---

Pre-flight (ops-owned CA + proxy)

• P-01 Issuing CA / intermediate chosen (model A–C above); HSM or offline root procedure documented offline
• P-02 Leaf profile: clientAuth EKU, short lifetime (hours–days preferred), CN/SAN policy agreed with security
• P-03 Reverse proxy terminates mTLS and verifies client chain against production trust store only
• P-04 Proxy forwards enroll headers per ENROLL.md (ssl-client-verify, DN, optional PEM presence — names as deployed)
• P-05 CRL/OCSP path ready or short-lifetime-only policy accepted — CRL-OCSP-DEPLOY-CHECKLIST.md
• P-06 Mint / renew / revoke operator procedures exist at the CA (GUI, API, or runbook) outside staff-iam
• P-07 Confirm production staff-iam env: NODE_ENV=production, LAB_CA_ENABLE unset, no lab-pki volume mounts
• P-08 Confirm GET /enroll/status → csrPostStatus: 501, labCa.enabled: false on production host

# production host smoke (expect scaffold — not a cert)
curl -sS https://<staff-iam-host>/enroll/status | jq '{csrPostStatus, labCa}'
# → csrPostStatus: 501, labCa.enabled: false (or equivalent probe false)

curl -sS -o /tmp/csr-body.json -w '%{http_code}\n' -X POST https://<staff-iam-host>/enroll/csr \
  -H 'content-type: application/json' \
  -d '{"csr":"-----BEGIN CERTIFICATE REQUEST-----\nMIIB\n-----END CERTIFICATE REQUEST-----"}'
# → 501 (or 400 if PEM shape invalid) — never 201 under production defaults

---

Cutover path A/B — external PKI (no IAM mint)

Use when product keeps mint outside staff-iam (current shipped design).

Mint (issue workstation client cert)

1. Operator or MDM enrolls device at corporate CA / EST / SCEP / ACME profile.
2. Install leaf + chain in OS/browser client-cert store (org procedure).
3. Browser hits staff door through mTLS-terminating proxy.
4. Confirm GET /enroll/status:
   • proxyHeadersSeen / mtlsPresent / proxy.verifyOk as configured
   • state progresses not_enrolled → pending_proxy → ready for admin flag
5. IAM admin: PUT /users/:username { "mtlsEnrolled": true }
6. Only then consider STAFF_MTLS_ENFORCE=1 on the door.

Renew

1. Renew at CA before notAfter (org cadence; prefer automated short-lived renew).
2. Replace client cert in workstation store.
3. No staff-iam API required unless metadata binding ships later (backlog).
4. Spot-check /enroll/status still shows proxy verify after renew.

Revoke / offboard

1. CA revoke serial (reason keyCompromise or cessationOfOperation).
2. IAM: mtlsEnrolled: false (+ kill sessions if policy requires).
3. Wait for proxy CRL/OCSP cache TTL or flush — CRL-OCSP-DEPLOY-CHECKLIST.md.
4. Verify: client no longer yields proxy.verifyOk; enforce path denies.

Do not call POST /enroll/csr expecting a production cert — it returns 501.

---

Cutover path C — delegated Signer (future live mint)

Blocked until product implements real Signer + AuthZ. This section is the go/no-go gate, not permission to code.

Go criteria (all required)

• C-01 Signer service deployed (HSM or equivalent); mTLS client identity for staff-iam only
• C-02 AUTHZ-ENROLL-DESIGN.md gates implemented and tested (session + role + rate limit + PoP)
• C-03 Full CSR ASN.1 validation (not header-shape only); CN/SAN bind to IAM user — never trust client CN alone
• C-04 Issue returns cert chain PEM only (no private key / PKCS#12 password in JSON)
• C-05 Persist metadata only: serial, fingerprint, notAfter (no leaf private key)
• C-06 Renew + revoke call Signer; offboard runbook dual-writes CA revoke + mtlsEnrolled: false
• C-07 Default 501 removed only when Signer healthy; failure modes stay fail-closed (502/503), never lab CA
• C-08 Tests: production env still refuses LAB_CA_ENABLE; no demo PEMs; audit never stores CSR PEM body
• C-09 ENROLL.md / README / this runbook updated for the live path in the same release
• C-10 Security sign-off on CA model + key custody

Signer contract sketch (names only)

Signer.issue({ csrPem, subjectUserId, deviceId? }) → { certificatePem, chainPem?, serial, notAfter }
Signer.renew({ serial | csrPem, subjectUserId }) → same as issue
Signer.revoke({ serial, reason }) → { revokedAt }

• Transport: mTLS or SPIFFE between staff-iam and Signer — not lab shell-out.
• staff-iam maps HTTP:
  • POST /enroll/csr → Signer.issue (AuthZ’d)
  • future renew/revoke routes → Signer.renew / Signer.revoke
• Lab path (lab-test-ca.sh) remains CI/dev fixture only; never selected when NODE_ENV=production.

Rollback

1. Feature-flag or deploy previous staff-iam build that restores default 501.
2. Leave external proxy mTLS + mtlsEnrolled path working (path A/B still valid).
3. Revoke any leaves issued during failed cutover at the CA.
4. Do not “fix” with LAB_CA_ENABLE=1 in production.

---

Production env attestation (required every deploy)

Check	Expected
NODE_ENV	production
LAB_CA_ENABLE	unset / empty
LAB_PKI_DIR	unset; no lab-pki volume
POST /enroll/csr without live Signer	501 not_implemented
GET /enroll/status .labCa	enabled/ready false
Trust store on proxy	production intermediate/root only
Image contents	no committed lab-pki/ keys (gitignored)

---

Do not

• Do not enable LAB_CA_ENABLE on production, staging-shared, or customer-adjacent hosts
• Do not point production proxies at lab-pki/lab-root.crt.pem
• Do not drop 501 without C-01…C-10
• Do not return PKCS#12 or private keys from staff-iam
• Do not treat mtlsEnrolled: false alone as cryptographic revoke (always CA revoke too)
• Do not invent CA URLs, RA credentials, or HSM slot IDs in this repository

---

Still left (ops / product — outside this READY package)

• Deploy real CA or Signer (hostnames, credentials, HSM)
• Execute mint/renew/revoke on live serials
• Wire production proxy trust + headers
• Implement AuthZ + Signer client code (separate engineering change gated by C-01…C-10)
• Production CRL/OCSP endpoints — CRL-OCSP-DEPLOY-CHECKLIST.md

Related

• PRODUCTION-PKI-REFUSE.md
• PKI-MINT-BACKLOG.md
• AUTHZ-ENROLL-DESIGN.md
• CRL-OCSP-DEPLOY-CHECKLIST.md
• TEST-CA-DESIGN.md
• ENROLL.md
• scripts/lab-test-ca.sh — LAB ONLY