George Lambert
7e8fc7c1f9
Convert OPEN-ISSUES #4 external items to READY ops packages without shipping production mint. LAB_CA_ENABLE stays default off; NODE_ENV= production still refuses lab; default POST /enroll/csr remains 501. - docs/PROD-CA-CUTOVER-RUNBOOK.md (mint/renew/revoke cutover) - docs/AUTHZ-ENROLL-DESIGN.md (gates before dropping 501) - docs/CRL-OCSP-DEPLOY-CHECKLIST.md (proxy/CA revoke deploy) - Cross-links from README, ENROLL, PKI-MINT-BACKLOG, refuse/design docs npm test: 19/19 PASS
183 lines
9.4 KiB
Markdown
183 lines
9.4 KiB
Markdown
# Production CA cutover runbook (READY package)
|
||
|
||
**Status: READY package** — operator cutover checklist for live device-cert **mint / renew / revoke**.
|
||
**Not shipped in staff-iam code.** Default `POST /enroll/csr` stays **501**. Lab CA stays **default off** and is **refused** when `NODE_ENV=production`.
|
||
|
||
This package tells ops how to cut over to a **real** production CA without using `scripts/lab-test-ca.sh` or `LAB_CA_ENABLE`. It does **not** implement a CA inside the IAM Node process.
|
||
|
||
| Companion | Role |
|
||
|-----------|------|
|
||
| [PKI-MINT-BACKLOG.md](./PKI-MINT-BACKLOG.md) | Shipped surface + live-mint product checklist |
|
||
| [AUTHZ-ENROLL-DESIGN.md](./AUTHZ-ENROLL-DESIGN.md) | AuthN/AuthZ gates before dropping default 501 |
|
||
| [CRL-OCSP-DEPLOY-CHECKLIST.md](./CRL-OCSP-DEPLOY-CHECKLIST.md) | Revocation distribution at proxy/CA |
|
||
| [PRODUCTION-PKI-REFUSE.md](./PRODUCTION-PKI-REFUSE.md) | What stays refused under production defaults |
|
||
| [TEST-CA-DESIGN.md](./TEST-CA-DESIGN.md) | Lab fixture only — never production trust |
|
||
| [ENROLL.md](../ENROLL.md) | mTLS states, proxy headers, status contract |
|
||
|
||
---
|
||
|
||
## Hard rules (do not violate on cutover)
|
||
|
||
1. **LAB_CA_ENABLE default off** — never set in production images, systemd units, or k8s manifests.
|
||
2. **`NODE_ENV=production` refuses lab** even if someone sets `LAB_CA_ENABLE=1` (server gate).
|
||
3. **Do not trust `lab-pki/` roots** on staff fleets or production reverse proxies.
|
||
4. **Do not drop default 501** until a real `Signer` + AuthZ (see AuthZ package) pass go/no-go.
|
||
5. **Private keys never** enter the IAM user store, audit log, or enroll response as PKCS#12 passwords / leaf keys.
|
||
6. **Proxy remains TLS SoT** for “cert present on this request”; IAM `mtlsEnrolled` is enrollment policy only.
|
||
|
||
---
|
||
|
||
## Choose production CA model (product decision — pick one)
|
||
|
||
| Model | When to choose | staff-iam role |
|
||
|-------|----------------|----------------|
|
||
| **A. External corporate PKI** (ADCS, Venafi, smallstep, internal Intermediate) | Org already issues workstation client certs | Stay proxy-only; optional later `Signer` HTTP client |
|
||
| **B. EST / SCEP / ACME device profile** | Managed fleet enrollment at edge | Edge enrolls; IAM still proxy headers + flag |
|
||
| **C. HSM-backed delegated Signer service** | Product wants in-app `POST /enroll/csr` live mint | IAM calls Signer over mTLS; **never** holds CA key |
|
||
| **D. Keep external-only forever** | Lowest risk; current default | No code change; this runbook stops at proxy cutover |
|
||
|
||
**Recommended first cutover:** Model **A** or **B** (no IAM crypto). Model **C** only after AuthZ package + Signer contract land.
|
||
|
||
Document the choice out-of-band (ticket / security design). Do not invent CA hostnames in git.
|
||
|
||
---
|
||
|
||
## Pre-flight (ops-owned CA + proxy)
|
||
|
||
- [ ] **P-01** Issuing CA / intermediate chosen (model A–C above); HSM or offline root procedure documented offline
|
||
- [ ] **P-02** Leaf profile: clientAuth EKU, short lifetime (hours–days preferred), CN/SAN policy agreed with security
|
||
- [ ] **P-03** Reverse proxy terminates mTLS and verifies client chain against **production** trust store only
|
||
- [ ] **P-04** Proxy forwards enroll headers per [ENROLL.md](../ENROLL.md) (`ssl-client-verify`, DN, optional PEM presence — names as deployed)
|
||
- [ ] **P-05** CRL/OCSP path ready or short-lifetime-only policy accepted — [CRL-OCSP-DEPLOY-CHECKLIST.md](./CRL-OCSP-DEPLOY-CHECKLIST.md)
|
||
- [ ] **P-06** Mint / renew / revoke operator procedures exist at the CA (GUI, API, or runbook) **outside** staff-iam
|
||
- [ ] **P-07** Confirm production staff-iam env: `NODE_ENV=production`, **`LAB_CA_ENABLE` unset**, no `lab-pki` volume mounts
|
||
- [ ] **P-08** Confirm `GET /enroll/status` → `csrPostStatus: 501`, `labCa.enabled: false` on production host
|
||
|
||
```sh
|
||
# production host smoke (expect scaffold — not a cert)
|
||
curl -sS https://<staff-iam-host>/enroll/status | jq '{csrPostStatus, labCa}'
|
||
# → csrPostStatus: 501, labCa.enabled: false (or equivalent probe false)
|
||
|
||
curl -sS -o /tmp/csr-body.json -w '%{http_code}\n' -X POST https://<staff-iam-host>/enroll/csr \
|
||
-H 'content-type: application/json' \
|
||
-d '{"csr":"-----BEGIN CERTIFICATE REQUEST-----\nMIIB\n-----END CERTIFICATE REQUEST-----"}'
|
||
# → 501 (or 400 if PEM shape invalid) — never 201 under production defaults
|
||
```
|
||
|
||
---
|
||
|
||
## Cutover path A/B — external PKI (no IAM mint)
|
||
|
||
Use when product keeps mint outside staff-iam (current shipped design).
|
||
|
||
### Mint (issue workstation client cert)
|
||
|
||
1. Operator or MDM enrolls device at corporate CA / EST / SCEP / ACME profile.
|
||
2. Install leaf + chain in OS/browser client-cert store (org procedure).
|
||
3. Browser hits staff door through mTLS-terminating proxy.
|
||
4. Confirm `GET /enroll/status`:
|
||
- `proxyHeadersSeen` / `mtlsPresent` / `proxy.verifyOk` as configured
|
||
- `state` progresses `not_enrolled` → `pending_proxy` → ready for admin flag
|
||
5. IAM admin: `PUT /users/:username` `{ "mtlsEnrolled": true }`
|
||
6. Only then consider `STAFF_MTLS_ENFORCE=1` on the door.
|
||
|
||
### Renew
|
||
|
||
1. Renew at CA **before** `notAfter` (org cadence; prefer automated short-lived renew).
|
||
2. Replace client cert in workstation store.
|
||
3. No staff-iam API required unless metadata binding ships later (backlog).
|
||
4. Spot-check `/enroll/status` still shows proxy verify after renew.
|
||
|
||
### Revoke / offboard
|
||
|
||
1. CA revoke serial (reason `keyCompromise` or `cessationOfOperation`).
|
||
2. IAM: `mtlsEnrolled: false` (+ kill sessions if policy requires).
|
||
3. Wait for proxy CRL/OCSP cache TTL or flush — [CRL-OCSP-DEPLOY-CHECKLIST.md](./CRL-OCSP-DEPLOY-CHECKLIST.md).
|
||
4. Verify: client no longer yields `proxy.verifyOk`; enforce path denies.
|
||
|
||
**Do not** call `POST /enroll/csr` expecting a production cert — it returns **501**.
|
||
|
||
---
|
||
|
||
## Cutover path C — delegated Signer (future live mint)
|
||
|
||
Blocked until product implements real `Signer` + AuthZ. This section is the **go/no-go gate**, not permission to code.
|
||
|
||
### Go criteria (all required)
|
||
|
||
- [ ] **C-01** Signer service deployed (HSM or equivalent); mTLS client identity for staff-iam only
|
||
- [ ] **C-02** [AUTHZ-ENROLL-DESIGN.md](./AUTHZ-ENROLL-DESIGN.md) gates implemented and tested (session + role + rate limit + PoP)
|
||
- [ ] **C-03** Full CSR ASN.1 validation (not header-shape only); CN/SAN bind to IAM user — never trust client CN alone
|
||
- [ ] **C-04** Issue returns cert **chain PEM only** (no private key / PKCS#12 password in JSON)
|
||
- [ ] **C-05** Persist metadata only: serial, fingerprint, notAfter (no leaf private key)
|
||
- [ ] **C-06** Renew + revoke call Signer; offboard runbook dual-writes CA revoke + `mtlsEnrolled: false`
|
||
- [ ] **C-07** Default 501 removed **only** when Signer healthy; failure modes stay fail-closed (502/503), never lab CA
|
||
- [ ] **C-08** Tests: production env still refuses `LAB_CA_ENABLE`; no demo PEMs; audit never stores CSR PEM body
|
||
- [ ] **C-09** ENROLL.md / README / this runbook updated for the live path in the same release
|
||
- [ ] **C-10** Security sign-off on CA model + key custody
|
||
|
||
### Signer contract sketch (names only)
|
||
|
||
```
|
||
Signer.issue({ csrPem, subjectUserId, deviceId? }) → { certificatePem, chainPem?, serial, notAfter }
|
||
Signer.renew({ serial | csrPem, subjectUserId }) → same as issue
|
||
Signer.revoke({ serial, reason }) → { revokedAt }
|
||
```
|
||
|
||
- Transport: mTLS or SPIFFE between staff-iam and Signer — **not** lab shell-out.
|
||
- staff-iam maps HTTP:
|
||
- `POST /enroll/csr` → `Signer.issue` (AuthZ’d)
|
||
- future renew/revoke routes → `Signer.renew` / `Signer.revoke`
|
||
- Lab path (`lab-test-ca.sh`) remains CI/dev fixture only; never selected when `NODE_ENV=production`.
|
||
|
||
### Rollback
|
||
|
||
1. Feature-flag or deploy previous staff-iam build that restores default **501**.
|
||
2. Leave external proxy mTLS + `mtlsEnrolled` path working (path A/B still valid).
|
||
3. Revoke any leaves issued during failed cutover at the CA.
|
||
4. **Do not** “fix” with `LAB_CA_ENABLE=1` in production.
|
||
|
||
---
|
||
|
||
## Production env attestation (required every deploy)
|
||
|
||
| Check | Expected |
|
||
|-------|----------|
|
||
| `NODE_ENV` | `production` |
|
||
| `LAB_CA_ENABLE` | **unset** / empty |
|
||
| `LAB_PKI_DIR` | unset; no lab-pki volume |
|
||
| `POST /enroll/csr` without live Signer | **501** `not_implemented` |
|
||
| `GET /enroll/status` `.labCa` | enabled/ready **false** |
|
||
| Trust store on proxy | production intermediate/root only |
|
||
| Image contents | no committed `lab-pki/` keys (gitignored) |
|
||
|
||
---
|
||
|
||
## Do not
|
||
|
||
- [ ] Do **not** enable `LAB_CA_ENABLE` on production, staging-shared, or customer-adjacent hosts
|
||
- [ ] Do **not** point production proxies at `lab-pki/lab-root.crt.pem`
|
||
- [ ] Do **not** drop 501 without C-01…C-10
|
||
- [ ] Do **not** return PKCS#12 or private keys from staff-iam
|
||
- [ ] Do **not** treat `mtlsEnrolled: false` alone as cryptographic revoke (always CA revoke too)
|
||
- [ ] Do **not** invent CA URLs, RA credentials, or HSM slot IDs in this repository
|
||
|
||
---
|
||
|
||
## Still left (ops / product — outside this READY package)
|
||
|
||
- Deploy real CA or Signer (hostnames, credentials, HSM)
|
||
- Execute mint/renew/revoke on live serials
|
||
- Wire production proxy trust + headers
|
||
- Implement AuthZ + Signer client code (separate engineering change gated by C-01…C-10)
|
||
- Production CRL/OCSP endpoints — [CRL-OCSP-DEPLOY-CHECKLIST.md](./CRL-OCSP-DEPLOY-CHECKLIST.md)
|
||
|
||
## Related
|
||
|
||
- [PRODUCTION-PKI-REFUSE.md](./PRODUCTION-PKI-REFUSE.md)
|
||
- [PKI-MINT-BACKLOG.md](./PKI-MINT-BACKLOG.md)
|
||
- [AUTHZ-ENROLL-DESIGN.md](./AUTHZ-ENROLL-DESIGN.md)
|
||
- [CRL-OCSP-DEPLOY-CHECKLIST.md](./CRL-OCSP-DEPLOY-CHECKLIST.md)
|
||
- [TEST-CA-DESIGN.md](./TEST-CA-DESIGN.md)
|
||
- [ENROLL.md](../ENROLL.md)
|
||
- [`scripts/lab-test-ca.sh`](../scripts/lab-test-ca.sh) — LAB ONLY
|