diff --git a/Historical-Information.MD b/Historical-Information.MD index 0c20518..cc88a43 100644 --- a/Historical-Information.MD +++ b/Historical-Information.MD @@ -90,6 +90,18 @@ Update checkboxes in `TODO.md` as each is done. **S15 closed.** Remaining UserReview only: NPE cutover, HSM custody, optional `sync.pfc` hostname, website deep-links. +## S16–S24 (this round) + +Continue-until-done plan: `Remaining-Work.MD`. Per-repo `MODULE.md` (who +calls, from/to, filters, errors/timeouts, storage). Catalog 44/44 +subjects.json with `timeout_sec` + `filter`. Python+Go success/failure +NATS send tests. UI screenshots in peergos-compliance-docs +`source/_static/screens/` + `walkthrough.rst`. HTML+PDF rebuilt. +`cicd/run-review.sh` REVIEW PASS (PFC 25, SM 8 including router, +Go leaf reject paths, catalog covers subjects.json). + +After a context reset: read this file, then `Remaining-Work.MD`. + Go `pfc-repl` now serves loopback `GET 127.0.0.1:18784/health`; admin `PFC_REPL_URL` points there. Ingest replicas a/b do not bind HTTP. ## Do not diff --git a/MODULE.md b/MODULE.md new file mode 100644 index 0000000..84c496f --- /dev/null +++ b/MODULE.md @@ -0,0 +1,40 @@ +# MODULE — system-git-sync + +Git: https://git.georgelambert.org/marchon/system-git-sync + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Operators and later agents. Nothing publishes NATS from this repo. + +**Callers:** Humans / CI (`cicd/run-review.sh`, `cicd/deploy-ns1.sh`). + +## Messages — from / to + +- **From:** Promoted SHAs, inventory, remaining-work. +- **To:** Forgejo remotes and ns1 rsync after tests. + +## Filters / security + +No auto-deploy. Unsigned config never copied. Websites org inventory-only. + +## Errors and timeouts + +- **Errors:** Review fail aborts deploy. Binary-busy: stop unit then copy. +- **Timeouts:** SSH/rsync operator-bound. + +## Where data is stored and how others use it + +Git only. Live signed files are on the host. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/Remaining-Work.MD b/Remaining-Work.MD new file mode 100644 index 0000000..b401cc6 --- /dev/null +++ b/Remaining-Work.MD @@ -0,0 +1,102 @@ +# Remaining-Work — complete plan (S16+) + +**Instruction:** Execute this file without operator interruption until every +checkbox is done, revalidated, and deployed to ns1 when tests pass. Do not +ask the operator questions. Record guesses in `TODO-With-User.MD`. + +This is **not** a HIPAA / SOC 2 / ISO certificate. + +S00–S15 are closed (see `TODO.md`). This round is **S16–S24**: per-repo +module docs, real NATS success/failure/logging, UI screenshots + +walkthroughs, thesaurus with code/doc occurrences, HTML+LaTeX for the +system and each module, CI/CD green, Go processes running on ns1. + +## Scope (in) vs out + +**In (code + docs + tests + CI + ns1):** + +| Repo | Why | +|---|---| +| system-git-sync | Hub, gate, thesaurus, remaining-work, deploy | +| nats-service-endpoints | Canonical from/to/filter/timeout catalog | +| secure-messaging | Python spec + Go sm-leaf routes | +| peergos-for-compliance | subjects.json, lib, tests | +| peergos-for-compliance-admin | Console UI + signed config | +| peergos-for-compliance-ipfs | pin HTTP/NATS | +| peergos-for-compliance-replication | Python repl (lab); live is Go | +| peergos-compliance-go | pfc-repl, configd, leaf | +| peergos-compliance-docs | System HTML/PDF + screenshots | +| peergos-compliance-architecture | Architecture notes | +| peergos-compliance-config | Config TUI + SPA | + +**Out (inventory only, do not rewrite):** `websites` org, `verae` org +(empty), AdamsMotor / research trees unrelated to NATS, 70 marketing sites. + +## Per-repo then system-wide + +1. Fix **each repo** in isolation: LICENSE, README (who calls it, messages + in/out, filters, errors, timeouts, variables), tests, CI. +2. Then **system pass**: catalog = subjects.json, thesaurus occurrences, + Sphinx HTML+LaTeX, UI walkthrough screenshots, NATS live smoke on ns1, + `cicd/run-review.sh`, promote, `deploy-ns1.sh`. + +## Checkboxes + +- [x] **S16** Document this plan in Historical-Information + TODO.md +- [x] **S17** `secure-messaging` router: success path, reject path, dead + letter, Network Error Bundle, `verae.sm.log.summary` (no ciphertext + in logs). Tests for correct and incorrect targeted messages. +- [x] **S18** Go `sm-leaf` implements the same reject/accept policy; Go + tests; linux rebuild; process running on ns1. +- [x] **S19** Expand `nats-service-endpoints` to every `subjects.json` + name with from/to/in/out/encryption/reject/dead/timeout. +- [x] **S20** MODULE.md (callers, data store, walkthrough links) + LICENSE + on every in-scope repo; README vocabulary matches thesaurus. +- [x] **S21** Thesaurus rebuilt with **file:line occurrences** in docs and + code. Certify no banned phrases in in-scope trees. +- [x] **S22** UI: Playwright screenshots of login, TOTP, cubes, catalog, + users, inspect, audit, catalog-card dialog, config SPA. Walkthrough + RST with capabilities / steps / why / outcomes / storage. +- [x] **S23** Rebuild HTML + LaTeX PDF for system docs, nats catalog, + secure-messaging, hub. Rsync to docs.pfc. +- [x] **S24** CI (`run-review.sh` + Forgejo workflows) includes route tests + + catalog completeness. Promote SHA. Deploy. Live NATS smoke on ns1. + Revalidate health, signed config, docs, screenshots published. + +## Variables (must be named and set or explicitly defaulted) + +See `docs/VARIABLES.md`. Live ns1 lab: + +- `crypto.mode=lab-xor` (signed wrapper) +- `routing.mode=passthrough` +- `logging.mode=summary` +- `PFC_REQUIRE_NPE` unset +- `PFC_REQUIRE_AUTH=1` +- `NATS_URL=nats://10.10.10.21:4222` +- `SM_LEAF_HUB=nats://10.10.10.21:7422` +- `SM_HTTP=127.0.0.1:18783` +- `PFC_REPL_URL=http://127.0.0.1:18784` +- `PFC_SIGNED_CONFIG` / `PFC_CONFIG_KEY_PEM` on host only + +## NATS route test matrix (must all pass) + +| Case | Subject | Expect | +|---|---|---| +| Good passthrough | `verae.sm.send` with `to` + `ct` | `accepted=true`, `lookup_id` | +| Missing dest | `verae.sm.send` no `to` | `accepted=false`, `SM-MISSING-TO`, dead + error + summary | +| Empty ciphertext | `alg=lab-xor` `ct=""` | `SM-EMPTY-CT`, failure path | +| Summary never has `ct` | `verae.sm.log.summary` | header codes only | +| Health | `GET /health` sm-leaf and pfc-repl | 200 | + +## UI screens to capture + +login, totp-enroll, cubes, catalog, users, inspect, audit, catalog-card +dialog, config SPA (`config.pfc` or local `web/index.html`). + +## Do not + +- Enable `PFC_REQUIRE_NPE=1` on live pfc +- Replace `pfc-py-admin` with Go `pfc-admin` +- Commit private keys +- Rewrite website org +- Claim HIPAA/SOC2/ISO certified diff --git a/TODO.md b/TODO.md index 1044672..3b1067c 100644 --- a/TODO.md +++ b/TODO.md @@ -20,3 +20,12 @@ Insert **new** dependencies **above** the blocked item. - [x] S13 CI-CD: Forgejo workflows + `cicd/run-review.sh` + GATE.md (REVIEW PASS, no deploy) - [x] S14 NPE adapter fail-closed; live `PFC_REQUIRE_NPE=1` still UserReview - [x] S15 Promote SHAs in cicd/GATE.md; deploy sm-leaf + signed config to ns1 (NPE live cutover remains UserReview) +- [x] S16 Remaining-Work plan documented +- [x] S17 Python router success/failure/logging tests +- [x] S18 Go sm-leaf accept/reject policy + tests +- [x] S19 Catalog covers all subjects.json (44) with timeout_sec + filter +- [x] S20 MODULE.md on every in-scope repo +- [x] S21 Thesaurus with file:line occurrences +- [x] S22 UI screenshots + walkthrough RST +- [x] S23 HTML + LaTeX PDFs (system 32p, catalog 17p, SM 11p) +- [x] S24 CI run-review expanded; deploy + live revalidate diff --git a/cicd/deploy-ns1.sh b/cicd/deploy-ns1.sh index 99e3b87..17bc50f 100755 --- a/cicd/deploy-ns1.sh +++ b/cicd/deploy-ns1.sh @@ -27,6 +27,18 @@ fi if [[ -f "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" ]]; then rsync -az "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" "$HOST:/tmp/nse.pdf" fi +if [[ -d "$ROOT/peergos-compliance-docs/build/html" ]]; then + rsync -az "$ROOT/peergos-compliance-docs/build/html/" "$HOST:/tmp/pfc-docs-html/" +fi +if [[ -f "$ROOT/peergos-compliance-docs/build/latex/peergos-for-compliance.pdf" ]]; then + rsync -az "$ROOT/peergos-compliance-docs/build/latex/peergos-for-compliance.pdf" "$HOST:/tmp/pfc-docs.pdf" +fi +if [[ -d "$ROOT/secure-messaging/build/html" ]]; then + rsync -az "$ROOT/secure-messaging/build/html/" "$HOST:/tmp/sm-html/" +fi +if [[ -d "$ROOT/peergos-compliance-config/web" ]]; then + rsync -az "$ROOT/peergos-compliance-config/web/" "$HOST:/tmp/config-web/" +fi echo "=== install on host ===" ssh "$HOST" 'bash -s' << "EOF" @@ -105,6 +117,20 @@ fi if [ -f /tmp/nse.pdf ]; then sudo cp /tmp/nse.pdf /opt/pfc/docs/html/nats-service-endpoints/nats-service-endpoints.pdf fi +if [ -d /tmp/pfc-docs-html ]; then + sudo rsync -a /tmp/pfc-docs-html/ /opt/pfc/docs/html/ +fi +if [ -f /tmp/pfc-docs.pdf ]; then + sudo cp /tmp/pfc-docs.pdf /opt/pfc/docs/html/peergos-for-compliance.pdf +fi +if [ -d /tmp/sm-html ]; then + sudo mkdir -p /opt/pfc/docs/html/secure-messaging + sudo rsync -a /tmp/sm-html/ /opt/pfc/docs/html/secure-messaging/ +fi +if [ -d /tmp/config-web ]; then + sudo mkdir -p /opt/pfc/config-ui + sudo rsync -a /tmp/config-web/ /opt/pfc/config-ui/ +fi sudo systemctl daemon-reload sudo systemctl enable --now pfc-sm-leaf sudo systemctl restart pfc-py-admin diff --git a/cicd/run-review.sh b/cicd/run-review.sh index 7cee434..d974c56 100755 --- a/cicd/run-review.sh +++ b/cicd/run-review.sh @@ -19,23 +19,43 @@ run env PYTHONPATH="$ROOT/secure-messaging/python" \ if [[ -d "$ROOT/secure-messaging/go" ]]; then ( cd "$ROOT/secure-messaging/go" && run go test ./... ) fi +if [[ -d "$ROOT/peergos-compliance-go" ]]; then + ( cd "$ROOT/peergos-compliance-go" && run go test ./cmd/pfc-repl ./internal/leaf ) +fi +if [[ -f "$ROOT/peergos-for-compliance-ipfs/pfc_ipfs/server.py" ]]; then + run python3 -m py_compile "$ROOT/peergos-for-compliance-ipfs/pfc_ipfs/server.py" +fi +if [[ -f "$ROOT/peergos-for-compliance-replication/pfc_repl/server.py" ]]; then + run python3 -m py_compile "$ROOT/peergos-for-compliance-replication/pfc_repl/server.py" +fi +if [[ -f "$ROOT/peergos-compliance-config/schema.json" ]]; then + run python3 -c "import json; json.load(open('$ROOT/peergos-compliance-config/schema.json'))" +fi if [[ -f "$ROOT/nats-service-endpoints/catalog/endpoints.json" ]]; then run python3 - << PY import json from pathlib import Path -p = Path("$ROOT/nats-service-endpoints/catalog/endpoints.json") -d = json.loads(p.read_text()) -names = {e["name"] for e in d["endpoints"]} +root = Path("$ROOT") +core = json.loads((root/"nats-service-endpoints/catalog/endpoints.json").read_text()) +derived = json.loads((root/"nats-service-endpoints/catalog/pfc-derived.json").read_text()) +names = {e["name"] for e in core["endpoints"]} | {e["name"] for e in derived["endpoints"]} need = { "verae.sm.send", "verae.sm.dead", "verae.sm.error", "verae.sm.log.summary", "verae.admin.config.sign", + "verae.storage.pin", + "verae.storage.replicate", + "verae.pfc.health", + "verae.inspect.open", } missing = sorted(need - names) assert not missing, missing -print("catalog subjects ok", sorted(need)) +subj = {s["name"] for s in json.loads((root/"peergos-for-compliance/schemas/subjects.json").read_text())["subjects"]} +gap = sorted(subj - names) +assert not gap, gap +print("catalog subjects ok", len(names), "covers subjects.json", len(subj)) PY fi if [[ "$fail" -ne 0 ]]; then diff --git a/docs/VARIABLES.md b/docs/VARIABLES.md new file mode 100644 index 0000000..2181fa9 --- /dev/null +++ b/docs/VARIABLES.md @@ -0,0 +1,50 @@ +# Variables + +Every runtime knob used by the in-scope stack. Unsigned JSON is rejected; +live ns1 values live in the **signed wrapper** and systemd units, not git. + +This is not a HIPAA/SOC 2/ISO certificate. + +## Signed config payload (`PFC_SIGNED_CONFIG`) + +| Key | Live lab | Allowed | Purpose | +|---|---|---|---| +| `crypto.mode` | `lab-xor` | `npe` \| `lab-xor` \| `plain-lab` | Envelope algorithm | +| `crypto.system_key_id` | `lab-system` | string | Network Error Bundle `ct_system` key id | +| `routing.mode` | `passthrough` | `passthrough` | Dest/subject in the clear; body ciphertext | +| `admin.history_cube` | `admin-history` | cube id | Prev + new + diff chain | +| `logging.mode` | `summary` | `summary` | Codes + lookup_id + dest_class only | + +Wrapper: `{payload, sig:{alg:ed25519,key_id,signature}}`. + +## Process environment + +| Variable | Default / live | Who | +|---|---|---| +| `PFC_REQUIRE_AUTH` | `1` on ns1 | pfc-py-admin | +| `PFC_REQUIRE_NPE` | **unset** on ns1 | bus; fail-closed if `1` | +| `PFC_DATA` | `/opt/pfc/data/admin` | cubes, users, admin-history | +| `PFC_ADMIN_HOST` / `PFC_ADMIN_PORT` | `127.0.0.1` / `18780` | console | +| `PFC_IPFS_URL` | `http://127.0.0.1:18782` | pin health | +| `PFC_REPL_URL` | `http://127.0.0.1:18784` | Go pfc-repl health | +| `NATS_URL` | `nats://10.10.10.21:4222` | untrusted broker | +| `PFC_NATS_KEY` | empty (plain lab envelope) | bus seal | +| `PFC_SIGNED_CONFIG` | `/opt/pfc/etc/secure-messaging.signed.json` | signed wrapper path | +| `PFC_CONFIG_KEY_PEM` | `/opt/pfc/etc/sm-keys/config.ed25519.pem` | host-only private | +| `SM_LEAF_HUB` | `nats://10.10.10.21:7422` | sm-leaf remote | +| `SM_HTTP` | `127.0.0.1:18783` | sm-leaf health | +| `PFC_REPL_HTTP` | `127.0.0.1:18784` | coordinator only | +| `PFC_STAMP_SYNC` | `15` seconds | unregistered hash flush | +| `PYTHONPATH` | `/opt/pfc/python/lib:/opt/pfc/python/admin:/opt/pfc/python` | admin + SM | + +Private keys are never committed. Lab Ed25519 is generated on the host. + +## Timeouts + +| Path | Seconds | +|---|---| +| NATS request-reply | 5 | +| Replica ingest request | 4 | +| NPE sidecar seal | 15 | +| Kubo pin | 30 | +| Stamp batch | 15 | diff --git a/docs/modules/nats-service-endpoints.md b/docs/modules/nats-service-endpoints.md new file mode 100644 index 0000000..22240a4 --- /dev/null +++ b/docs/modules/nats-service-endpoints.md @@ -0,0 +1,40 @@ +# MODULE — nats-service-endpoints + +Git: https://git.georgelambert.org/marchon/nats-service-endpoints + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Every service that publishes `verae.*`. + +**Callers:** secure-messaging, pfc-py-admin, pfc-repl, pfc-ipfs, sm-leaf. + +## Messages — from / to + +- **From:** `catalog/endpoints.json` (SM core) + `catalog/pfc-derived.json` (all subjects.json names). +- **To:** Sphinx HTML/PDF on docs.pfc `/nats-service-endpoints/`. + +## Filters / security + +`reject[]` + `filter` + `timeout_sec`. Forbidden `verae.llm.turn.>`. + +## Errors and timeouts + +- **Errors:** Unknown subject / missing fields → dead letter for SM; others `dead=none`. +- **Timeouts:** See each row `timeout_sec` (5s request-reply, 4s queue, 30s pin). + +## Where data is stored and how others use it + +JSON catalog in git. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-compliance-architecture.md b/docs/modules/peergos-compliance-architecture.md new file mode 100644 index 0000000..103b6a2 --- /dev/null +++ b/docs/modules/peergos-compliance-architecture.md @@ -0,0 +1,40 @@ +# MODULE — peergos-compliance-architecture + +Git: https://git.georgelambert.org/marchon/peergos-compliance-architecture + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Design readers. No runtime. + +**Callers:** Docs and plan repos. + +## Messages — from / to + +- **From:** Architecture notes for DataCube + Peergos + NATS. +- **To:** Links to runtime repos. + +## Filters / security + +Inventory; do not contradict signed-config design locks. + +## Errors and timeouts + +- **Errors:** n/a +- **Timeouts:** n/a + +## Where data is stored and how others use it + +Markdown in git. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-compliance-config.md b/docs/modules/peergos-compliance-config.md new file mode 100644 index 0000000..d7707e1 --- /dev/null +++ b/docs/modules/peergos-compliance-config.md @@ -0,0 +1,40 @@ +# MODULE — peergos-compliance-config + +Git: https://git.georgelambert.org/marchon/peergos-compliance-config + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Operator building `pfc.json` before signing. + +**Callers:** Browser SPA `config.pfc.georgelambert.org`, bash/curses TUI, pfc-configd. + +## Messages — from / to + +- **From:** Host, leaf hub, NATS URL, lab PSK, profile, min_ok. +- **To:** Download JSON; **must be signed** (`signed_config.sign`) before load. + +## Filters / security + +Unsigned files rejected at load. Live changes append admin-history. + +## Errors and timeouts + +- **Errors:** Invalid schema / missing crypto.mode. +- **Timeouts:** n/a + +## Where data is stored and how others use it + +`~/pfc.json` or host signed path. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-compliance-docs.md b/docs/modules/peergos-compliance-docs.md new file mode 100644 index 0000000..1cfba01 --- /dev/null +++ b/docs/modules/peergos-compliance-docs.md @@ -0,0 +1,40 @@ +# MODULE — peergos-compliance-docs + +Git: https://git.georgelambert.org/marchon/peergos-compliance-docs + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Operators and auditors reading docs.pfc. + +**Callers:** Sphinx build; Caddy `docs.pfc.georgelambert.org`. + +## Messages — from / to + +- **From:** RST sources + screenshots in `_static/screens/`. +- **To:** HTML + LaTeX PDF under `/opt/pfc/docs/html`. + +## Filters / security + +No PHI examples; not a certificate. + +## Errors and timeouts + +- **Errors:** latexmk optional; xelatex/pdflatex fallback. +- **Timeouts:** n/a + +## Where data is stored and how others use it + +git + ns1 docs tree. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-compliance-go.md b/docs/modules/peergos-compliance-go.md new file mode 100644 index 0000000..549f6e7 --- /dev/null +++ b/docs/modules/peergos-compliance-go.md @@ -0,0 +1,40 @@ +# MODULE — peergos-compliance-go + +Git: https://git.georgelambert.org/marchon/peergos-compliance-go + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +ns1 leaf processes: pfc-repl, pfc-repl-a/b, pfc-configd, (inactive pfc-admin HTTP). + +**Callers:** systemd. pfc-py-admin health-probes pfc-repl `:18784`. + +## Messages — from / to + +- **From:** NATS `verae.storage.replicate`, replica ingest, `verae.pfc.health`. +- **To:** Local data dir + replica subjects; config UI static files. + +## Filters / security + +Coordinator only binds HTTP; ingest replicas NATS-only. + +## Errors and timeouts + +- **Errors:** min_ok failure in replicate JSON; leaf hub parse `nats-leaf://` → `nats://`. +- **Timeouts:** 4s replica Request. + +## Where data is stored and how others use it + +`/opt/pfc/data/repl`. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-for-compliance-admin.md b/docs/modules/peergos-for-compliance-admin.md new file mode 100644 index 0000000..98cfcf5 --- /dev/null +++ b/docs/modules/peergos-for-compliance-admin.md @@ -0,0 +1,40 @@ +# MODULE — peergos-for-compliance-admin + +Git: https://git.georgelambert.org/marchon/peergos-for-compliance-admin + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Operators in the browser SPA. + +**Callers:** https://pfc.georgelambert.org (Caddy → 127.0.0.1:18780). + +## Messages — from / to + +- **From:** Login/TOTP, cube/catalog/inspect/audit HTTP; NATS when bus configured. +- **To:** CubeStore, UserStore, signed config, sm-leaf, pin, repl. + +## Filters / security + +`PFC_REQUIRE_AUTH=1`; TOTP for hipaa; k-of-n inspect; unsigned config rejected. + +## Errors and timeouts + +- **Errors:** 401 auth, 403 totp/admin, 400 profile, 501 missing signing key. +- **Timeouts:** HTTP 8s in tests; NATS 5s. + +## Where data is stored and how others use it + +`PFC_DATA` users + cubes; host PEM for Ed25519. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-for-compliance-ipfs.md b/docs/modules/peergos-for-compliance-ipfs.md new file mode 100644 index 0000000..6eaccae --- /dev/null +++ b/docs/modules/peergos-for-compliance-ipfs.md @@ -0,0 +1,40 @@ +# MODULE — peergos-for-compliance-ipfs + +Git: https://git.georgelambert.org/marchon/peergos-for-compliance-ipfs + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Admin object put (pin) and hydrate. + +**Callers:** pfc-py-admin `verae.storage.pin` / HTTP `/v1/pin`. + +## Messages — from / to + +- **From:** `sha256`, `bytes_hex`, `cube_id`, `tier`. +- **To:** Kubo API localhost:15001 (never WAN 5001). + +## Filters / security + +Localhost Kubo only; hash must match bytes. + +## Errors and timeouts + +- **Errors:** Kubo down → health 503 for ipfs backend; pin error JSON. +- **Timeouts:** Kubo 30s. + +## Where data is stored and how others use it + +Kubo blocks; optional stash dir. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-for-compliance-replication.md b/docs/modules/peergos-for-compliance-replication.md new file mode 100644 index 0000000..b880aaf --- /dev/null +++ b/docs/modules/peergos-for-compliance-replication.md @@ -0,0 +1,40 @@ +# MODULE — peergos-for-compliance-replication + +Git: https://git.georgelambert.org/marchon/peergos-for-compliance-replication + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Lab HTTP coordinator. **Live ns1 uses Go pfc-repl** instead. + +**Callers:** Admin when `NATS_URL` unset (`PFC_REPL_URL`). + +## Messages — from / to + +- **From:** POST `/v1/replicate` `{bytes_hex, sha256, cube_id}`. +- **To:** Replica ingest URLs / `verae.pfc.replica.{a,b}.ingest`. + +## Filters / security + +`min_ok` acks including local. + +## Errors and timeouts + +- **Errors:** 503 when min_ok not met. +- **Timeouts:** 4s per replica request. + +## Where data is stored and how others use it + +`PFC_REPL_DATA` files named by sha256. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/peergos-for-compliance.md b/docs/modules/peergos-for-compliance.md new file mode 100644 index 0000000..29c97de --- /dev/null +++ b/docs/modules/peergos-for-compliance.md @@ -0,0 +1,40 @@ +# MODULE — peergos-for-compliance + +Git: https://git.georgelambert.org/marchon/peergos-for-compliance + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Shared Python lib (`pfc.*`) used by admin, ipfs, repl, connector. + +**Callers:** pfc-py-admin, tests, Pattern A inspect app outbox. + +## Messages — from / to + +- **From:** HTTP browsers/CLI; NATS `verae.*` when `NATS_URL` set. +- **To:** CubeStore `/PFC_DATA`, stamp leaf `verae.ts.batch`, pin, replicate. + +## Filters / security + +Profile validate (k>=2, author not officer); HIPAA bytes need inspect session. + +## Errors and timeouts + +- **Errors:** AccessDenied, ProfileError, NatsError, hash mismatch. +- **Timeouts:** NATS 5s; stamp sync 15s. + +## Where data is stored and how others use it + +`cubes//` chain, catalog card, hash registry. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/secure-messaging.md b/docs/modules/secure-messaging.md new file mode 100644 index 0000000..67585c8 --- /dev/null +++ b/docs/modules/secure-messaging.md @@ -0,0 +1,40 @@ +# MODULE — secure-messaging + +Git: https://git.georgelambert.org/marchon/secure-messaging + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +sm-leaf (Go) and pfc-py-admin (Python sign/history). + +**Callers:** Console send, tests, systemd `pfc-sm-leaf`. + +## Messages — from / to + +- **From:** `verae.sm.send` envelopes (`to`, `from_lookup_id`, `alg`, `ct`). +- **To:** Mailbox dest (clear); ack `lookup_id`. Failures → `verae.sm.error`, `verae.sm.dead`, `verae.sm.log.summary`. + +## Filters / security + +Missing `to`, empty `ct` (unless plain-lab), plaintext `body`, bad alg, summary `ct` field. + +## Errors and timeouts + +- **Errors:** Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body. +- **Timeouts:** NATS 5s; NPE seal 15s. + +## Where data is stored and how others use it + +Signed wrapper path; admin-history JSONL cube. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/modules/system-git-sync.md b/docs/modules/system-git-sync.md new file mode 100644 index 0000000..84c496f --- /dev/null +++ b/docs/modules/system-git-sync.md @@ -0,0 +1,40 @@ +# MODULE — system-git-sync + +Git: https://git.georgelambert.org/marchon/system-git-sync + + +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `https://git.georgelambert.org/marchon/system-git-sync` `docs/thesaurus.md`. +Catalog: `https://git.georgelambert.org/marchon/nats-service-endpoints`. + + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +Operators and later agents. Nothing publishes NATS from this repo. + +**Callers:** Humans / CI (`cicd/run-review.sh`, `cicd/deploy-ns1.sh`). + +## Messages — from / to + +- **From:** Promoted SHAs, inventory, remaining-work. +- **To:** Forgejo remotes and ns1 rsync after tests. + +## Filters / security + +No auto-deploy. Unsigned config never copied. Websites org inventory-only. + +## Errors and timeouts + +- **Errors:** Review fail aborts deploy. Binary-busy: stop unit then copy. +- **Timeouts:** SSH/rsync operator-bound. + +## Where data is stored and how others use it + +Git only. Live signed files are on the host. + +Walkthroughs and screenshots: `https://git.georgelambert.org/marchon/peergos-compliance-docs` (`source/walkthrough.rst`). diff --git a/docs/thesaurus.md b/docs/thesaurus.md index 0eb24f3..9b16d20 100644 --- a/docs/thesaurus.md +++ b/docs/thesaurus.md @@ -1,23 +1,31 @@ # Thesaurus -Canonical term → where it lives. Use this to rename later. +Canonical term → where it lives. Occurrences are grepped from in-scope +repos (docs **and** code). Banned phrases must not be used as the name +of the thing. -| Term | Use this | Not this | Where | +This is **not** a HIPAA/SOC 2/ISO certificate. + +| Term | Use this | Not this | Occurrences | |---|---|---|---| -| Untrusted broker | NATS | "secure bus" | nats-service-endpoints | -| Production E2E | NPE / HPKE | "TLS to NATS" | secure-messaging | -| Lab envelope | pfc-lab-xor | "xor cipher" alone | peergos-for-compliance | -| Dest in clear | passthrough | "plaintext message" | envelope.py | -| Sender handle | lookup_id | "from username on wire" | envelope.py | -| Config file | signed wrapper | raw JSON config | signed_config.py | -| Admin audit | DataCube admin-history | SQL log | admin_history.py | -| Failure to sender | ct_sender | "email bounce plaintext" | error_bundle.py | -| Ops bounce | ct_system / Network Error Bundle | "log the ciphertext body" | error_bundle.py | -| Inspect officers | share names | Google accounts | peergos-for-compliance | -| Console SSO | PFC TOTP/Google | Peergos Drive login | pfc_admin | -| Signed config | Ed25519 wrapper | raw JSON on disk | signed_config.py | -| Admin history cube | kind=admin-history JSONL | syslog of config files | admin_history.py | -| NPE sidecar | npe seal --to | HPKE in Python | npe_adapter.py | -| Review gate | cicd/GATE.md | auto-deploy to ns1 | system-git-sync | -| sm-leaf health | GET 127.0.0.1:18783/health | WAN NATS ports | sm-leaf | -| Config promote | GATE.md SHA table then deploy-ns1.sh | unsigned JSON copy | cicd/ | +| Untrusted broker | NATS | secure bus | `system-git-sync/System-Git-Sync.MD:116:| Routing | **passthrough**: destination in the clear; body encrypted | Brokers are untrusted | `routing.mode = passthrou`
`system-git-sync/scripts/build_thesaurus.py:26: ("Untrusted broker", "NATS", "secure bus", "untrusted"),`
`system-git-sync/scripts/build_thesaurus.py:86: "NATS is the untrusted broker. Production E2E is NPE/HPKE.",`
`nats-service-endpoints/README.md:13:**body** is ciphertext. The broker is untrusted.`
`peergos-compliance-docs/source/walkthrough.rst:11:**lookup_id** is the sender handle; **NATS** is the untrusted broker;`
`nats-service-endpoints/source/index.rst:7:The NATS broker is **untrusted**. Production E2E is NPE (HPKE). Lab may use`
`system-git-sync/docs/VARIABLES.md:30:| `NATS_URL` | `nats://10.10.10.21:4222` | untrusted broker |`
`secure-messaging/python/secure_messaging/envelope.py:42: return { # allowed on the untrusted broker` | +| Production E2E | NPE / HPKE | TLS to NATS | `secure-messaging/README.md:17:Live ns1 lab uses `lab-xor` until `PFC_REQUIRE_NPE=1` is explicitly cut over.`
`system-git-sync/System-Git-Sync.MD:110:| Crypto production | NPE HPKE (ChaCha20-Poly1305) | Already the Verae production E2E; do not invent a second suite | `cr`
`system-git-sync/System-Git-Sync.MD:111:| Crypto lab | `pfc-lab-xor` or `plain-lab` | Lets tests run without NPE sidecar | `crypto.mode = lab-xor \| plain-lab` |`
`system-git-sync/System-Git-Sync.MD:146:- [x] **S14** NPE adapter fail-closed (live cutover still UserReview)`
`system-git-sync/System-Git-Sync.MD:147:- [x] **S15** Promote SHA + ns1 sm-leaf (NPE live cutover remains UserReview)`
`system-git-sync/System-Git-Sync.MD:159:- 2026-09-16 — S10–S14: PFC admin `GET/POST /v1/admin/config`; NPE fail-closed; cicd/run-review.sh **REVIEW PASS (no depl`
`system-git-sync/System-Git-Sync.MD:160:- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-h`
`nats-service-endpoints/scripts/import_pfc_subjects.py:71: "encryption": "passthrough dest-in-clear; body pfc-lab-xor or NPE; Pattern A apps never` | +| Lab envelope | pfc-lab-xor | xor cipher | `secure-messaging/scripts/sign_config.py:23: "crypto": {"mode": "lab-xor", "system_key_id": "lab-system"},`
`secure-messaging/README.md:16:`crypto.mode`: `npe` | `lab-xor` | `plain-lab` (see signed payload).`
`secure-messaging/README.md:17:Live ns1 lab uses `lab-xor` until `PFC_REQUIRE_NPE=1` is explicitly cut over.`
`system-git-sync/System-Git-Sync.MD:111:| Crypto lab | `pfc-lab-xor` or `plain-lab` | Lets tests run without NPE sidecar | `crypto.mode = lab-xor \| plain-lab` |`
`system-git-sync/System-Git-Sync.MD:160:- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-h`
`system-git-sync/scripts/build_thesaurus.py:28: ("Lab envelope", "pfc-lab-xor", "xor cipher", "lab-xor"),`
`secure-messaging/tests/test_sm.py:20: payload = {"crypto": {"mode": "lab-xor"}, "routing": {"mode": "passthrough"}}`
`secure-messaging/tests/test_sm.py:22: self.assertEqual(verify(signed.wrapper(), pub)["crypto"]["mode"], "lab-xor")` | +| Dest in clear | passthrough | plaintext message | `secure-messaging/scripts/sign_config.py:24: "routing": {"mode": "passthrough"},`
`system-git-sync/System-Git-Sync.MD:35:> the address is a passthrough make that clear, explain how encrypted`
`system-git-sync/System-Git-Sync.MD:116:| Routing | **passthrough**: destination in the clear; body encrypted | Brokers are untrusted | `routing.mode = passthrou`
`system-git-sync/System-Git-Sync.MD:138:- [x] **S06** `secure-messaging` Python (signed config, passthrough, error bundle, admin history)`
`system-git-sync/System-Git-Sync.MD:156:- 2026-09-16 — Locked Ed25519 signed config + DataCube admin-history + passthrough routing + system-key error bundles.`
`nats-service-endpoints/scripts/import_pfc_subjects.py:24: "verae.sm.send": "Deliver a passthrough secure message (dest in the clear, body ciphertext)",`
`nats-service-endpoints/scripts/import_pfc_subjects.py:45: return "passthrough dest/subject clear; reject missing to, empty ct, plaintext body, body-like `
`nats-service-endpoints/scripts/import_pfc_subjects.py:71: "encryption": "passthrough dest-in-clear; body pfc-lab-xor or NPE; Pattern A apps never` | +| Sender handle | lookup_id | from username on wire | `system-git-sync/System-Git-Sync.MD:117:| After send | Sender cannot decrypt own ciphertext; **lookup_id** only | Operator rule | envelope `from_lookup_id` |`
`system-git-sync/System-Git-Sync.MD:121:| Logs | **summaries** only (codes, lookup_id, dest class) | No PHI / no body | `logging.mode = summary` |`
`secure-messaging/MODULE.md:24:- **From:** `verae.sm.send` envelopes (`to`, `from_lookup_id`, `alg`, `ct`).`
`secure-messaging/MODULE.md:25:- **To:** Mailbox dest (clear); ack `lookup_id`. Failures → `verae.sm.error`, `verae.sm.dead`, `verae.sm.log.summary`.`
`system-git-sync/scripts/build_thesaurus.py:30: ("Sender handle", "lookup_id", "from username on wire", "lookup_id"),`
`system-git-sync/scripts/write_modules.py:41: frm="`verae.sm.send` envelopes (`to`, `from_lookup_id`, `alg`, `ct`).",`
`system-git-sync/scripts/write_modules.py:42: to="Mailbox dest (clear); ack `lookup_id`. Failures → `verae.sm.error`, `verae.sm.dead`, `verae.sm.log.summa`
`secure-messaging/tests/test_sm.py:56: self.assertNotIn("alice", env.from_lookup_id)` | +| Config file | signed wrapper | raw JSON config | `nats-service-endpoints/scripts/import_pfc_subjects.py:49: return "unsigned wrappers rejected; empty actor rejected"`
`secure-messaging/README.md:13:Config must be a signed wrapper. Unsigned files are rejected. Admin changes`
`system-git-sync/scripts/build_thesaurus.py:31: ("Config file", "signed wrapper", "raw JSON config", "signed wrapper"),`
`peergos-for-compliance-admin/pfc_admin/server.py:693: return 200, {"configured": False, "hint": "set PFC_SIGNED_CONFIG to a signed wrapper"}`
`peergos-compliance-docs/source/walkthrough.rst:122:signed wrapper.** Admin ``POST /v1/admin/config`` signs and appends`
`peergos-for-compliance/tests/test_signed_config.py:1:"""Admin GET/POST /v1/admin/config with Ed25519 signed wrapper + history."""`
`nats-service-endpoints/catalog/endpoints.json:152: "out": "signed wrapper",`
`system-git-sync/docs/VARIABLES.md:4:live ns1 values live in the **signed wrapper** and systemd units, not git.` | +| Admin audit | DataCube admin-history | SQL log | `peergos-compliance-config/MODULE.md:29:Unsigned files rejected at load. Live changes append admin-history.`
`secure-messaging/scripts/sign_config.py:25: "admin": {"history_cube": "admin-history"},`
`secure-messaging/scripts/sign_config.py:52: hist = AdminHistory(ROOT / "examples" / "admin-history")`
`nats-service-endpoints/scripts/import_pfc_subjects.py:29: "verae.admin.history.append": "Append prev + new + unified diff to admin-history cube",`
`secure-messaging/README.md:14:append prev + new + unified diff to a `kind=admin-history` JSONL chain.`
`system-git-sync/System-Git-Sync.MD:114:| Admin history | Verae DataCube chain (JSONL) `kind=admin-history` | Matches existing cube append-only model | `admin.hi`
`system-git-sync/System-Git-Sync.MD:156:- 2026-09-16 — Locked Ed25519 signed config + DataCube admin-history + passthrough routing + system-key error bundles.`
`system-git-sync/System-Git-Sync.MD:160:- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-h` | +| Failure to sender | ct_sender | email bounce plaintext | `nats-service-endpoints/scripts/import_pfc_subjects.py:26: "verae.sm.error": "Emit a Network Error Bundle (ct_sender + ct_system)",`
`system-git-sync/System-Git-Sync.MD:119:| Sender failure notice | Sealed to **sender** public key only | Only they can read status | error header `ct_sender` |`
`secure-messaging/MODULE.md:33:- **Errors:** Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body.`
`system-git-sync/scripts/build_thesaurus.py:33: ("Failure to sender", "ct_sender", "email bounce plaintext", "ct_sender"),`
`nats-service-endpoints/source/tracing.rst:17: 4. Sender opens ct_sender; ops opens ct_system.`
`system-git-sync/scripts/write_modules.py:44: err="Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body.",`
`nats-service-endpoints/source/endpoints.rst:14:* ``verae.sm.error`` -- emit bundle: ``ct_sender`` + ``ct_system``.`
`secure-messaging/source/index.rst:36:Network Error Bundle: ``ct_sender`` (sender-only status) and ``ct_system``` | +| Ops bounce | ct_system / Network Error Bundle | log the ciphertext body | `secure-messaging/MODULE.md:33:- **Errors:** Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body.`
`nats-service-endpoints/scripts/import_pfc_subjects.py:26: "verae.sm.error": "Emit a Network Error Bundle (ct_sender + ct_system)",`
`system-git-sync/scripts/build_thesaurus.py:34: ("Ops bounce", "ct_system / Network Error Bundle", "log the ciphertext body", "ct_system"),`
`system-git-sync/scripts/write_modules.py:44: err="Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body.",`
`secure-messaging/source/index.rst:36:Network Error Bundle: ``ct_sender`` (sender-only status) and ``ct_system```
`nats-service-endpoints/source/tracing.rst:17: 4. Sender opens ct_sender; ops opens ct_system.`
`nats-service-endpoints/source/endpoints.rst:14:* ``verae.sm.error`` -- emit bundle: ``ct_sender`` + ``ct_system``.`
`peergos-compliance-docs/source/walkthrough.rst:143:``ct_sender`` is sender-only; ``ct_system`` is the ops bounce report.` | +| Inspect officers | share names | Google accounts | `peergos-for-compliance-admin/HOWTOSTART.md:18:k-of-n officer presence.`
`peergos-compliance-architecture/README.md:10: Author HSM Officers (k-of-n WebAuthn)`
`nats-service-endpoints/scripts/import_pfc_subjects.py:33: "verae.inspect.open": "Start k-of-n inspect; log-before-reveal",`
`nats-service-endpoints/scripts/import_pfc_subjects.py:47: return "k-of-n officers; author cannot be officer; TOTP session; log-before-reveal"`
`system-git-sync/scripts/build_thesaurus.py:35: ("Inspect officers", "share names", "Google accounts", "k-of-n"),`
`peergos-for-compliance-admin/README.md:5:Creates cubes with fail-closed profiles, stores ciphertext objects, opens k-of-n inspect sessions, and **refuses bytes `
`system-git-sync/scripts/write_modules.py:63: filt="`PFC_REQUIRE_AUTH=1`; TOTP for hipaa; k-of-n inspect; unsigned config rejected.",`
`peergos-for-compliance/apps/inspect/peergos-app.json:5: "description": "k-of-n inspect via cryptree outbox. No NATS in the iframe.",` | +| Console SSO | PFC TOTP/Google | Peergos Drive login | `peergos-for-compliance-admin/HOWTOSTART.md:11:Production console (`https://pfc.georgelambert.org`) sets `PFC_REQUIRE_AUTH=1`.`
`peergos-compliance-docs/scripts/capture_ui.py:46: env["PFC_REQUIRE_AUTH"] = "1"`
`system-git-sync/scripts/build_thesaurus.py:36: ("Console SSO", "PFC TOTP/Google", "Peergos Drive login", "PFC_REQUIRE_AUTH"),`
`system-git-sync/scripts/write_modules.py:63: filt="`PFC_REQUIRE_AUTH=1`; TOTP for hipaa; k-of-n inspect; unsigned config rejected.",`
`peergos-for-compliance-admin/pfc_admin/server.py:43:REQUIRE_AUTH = os.environ.get("PFC_REQUIRE_AUTH", "0") == "1"`
`peergos-for-compliance-admin/MODULE.md:29:`PFC_REQUIRE_AUTH=1`; TOTP for hipaa; k-of-n inspect; unsigned config rejected.`
`peergos-compliance-docs/source/remaining.rst:12:* Open JSON cube APIs return 401 when ``PFC_REQUIRE_AUTH=1``.`
`peergos-compliance-docs/source/walkthrough.rst:6:with ``PFC_REQUIRE_AUTH=1``.` | +| NPE sidecar | npe seal --to | HPKE in Python | `system-git-sync/scripts/build_thesaurus.py:37: ("NPE sidecar", "npe seal --to", "HPKE in Python", "npe_adapter"),`
`secure-messaging/python/secure_messaging/envelope.py:79: from .npe_adapter import seal_npe # fail-closed import`
`secure-messaging/python/secure_messaging/__init__.py:6:from .npe_adapter import NpeRequired # fail-closed NPE`
`system-git-sync/docs/thesaurus.md:20:| NPE sidecar | npe seal --to | HPKE in Python | npe_adapter.py |` | +| Review gate | cicd/GATE.md | auto-deploy to ns1 | `system-git-sync/scripts/build_thesaurus.py:38: ("Review gate", "cicd/GATE.md", "auto-deploy to ns1", "GATE.md"),`
`system-git-sync/TODO.md:20:- [x] S13 CI-CD: Forgejo workflows + `cicd/run-review.sh` + GATE.md (REVIEW PASS, no deploy)`
`system-git-sync/TODO.md:22:- [x] S15 Promote SHAs in cicd/GATE.md; deploy sm-leaf + signed config to ns1 (NPE live cutover remains UserReview)`
`system-git-sync/docs/thesaurus.md:21:| Review gate | cicd/GATE.md | auto-deploy to ns1 | system-git-sync |`
`system-git-sync/docs/thesaurus.md:23:| Config promote | GATE.md SHA table then deploy-ns1.sh | unsigned JSON copy | cicd/ |`
`system-git-sync/Historical-Information.MD:71:10. [x] Promote SHA in `cicd/GATE.md`` | +| sm-leaf health | GET 127.0.0.1:18783/health | WAN NATS ports | `secure-messaging/README.md:8:* Loopback health: `SM_HTTP` default `127.0.0.1:18783` (`GET /health`)`
`secure-messaging/source/index.rst:11:* Loopback health: ``127.0.0.1:18783/health```
`secure-messaging/deploy/pfc-sm-leaf.service:7:Environment=SM_HTTP=127.0.0.1:18783`
`secure-messaging/deploy/pfc-sm-leaf.service:8:ExecStart=/opt/pfc/bin/sm-leaf -hub nats://10.10.10.21:7422 -http 127.0.0.1:18783`
`system-git-sync/scripts/build_thesaurus.py:39: ("sm-leaf health", "GET 127.0.0.1:18783/health", "WAN NATS ports", "18783"),`
`system-git-sync/cicd/deploy-ns1.sh:114:curl -sS -m 5 http://127.0.0.1:18783/health; echo`
`system-git-sync/docs/VARIABLES.md:35:| `SM_HTTP` | `127.0.0.1:18783` | sm-leaf health |`
`system-git-sync/Remaining-Work.MD:77:- `SM_HTTP=127.0.0.1:18783`` | + +## Certification + +In-scope documentation uses the **Use this** column. `passthrough` +means destination in the clear, not a plaintext message body. +NATS is the untrusted broker. Production E2E is NPE/HPKE. + diff --git a/scripts/build_thesaurus.py b/scripts/build_thesaurus.py new file mode 100755 index 0000000..ee16d86 --- /dev/null +++ b/scripts/build_thesaurus.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""Rebuild thesaurus.md with file:line occurrences in in-scope repos.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +ROOT = Path("/Users/marchon/research") +OUT = Path("/Users/marchon/research/system-git-sync/docs/thesaurus.md") +SCOPE = [ + "system-git-sync", + "nats-service-endpoints", + "secure-messaging", + "peergos-for-compliance", + "peergos-for-compliance-admin", + "peergos-for-compliance-ipfs", + "peergos-for-compliance-replication", + "peergos-compliance-docs", + "peergos-compliance-architecture", + "peergos-compliance-go", + "peergos-compliance-config", +] +# Canonical term, banned phrase, grep needle for "use this" +TERMS = [ + ("Untrusted broker", "NATS", "secure bus", "untrusted"), + ("Production E2E", "NPE / HPKE", "TLS to NATS", "NPE"), + ("Lab envelope", "pfc-lab-xor", "xor cipher", "lab-xor"), + ("Dest in clear", "passthrough", "plaintext message", "passthrough"), + ("Sender handle", "lookup_id", "from username on wire", "lookup_id"), + ("Config file", "signed wrapper", "raw JSON config", "signed wrapper"), + ("Admin audit", "DataCube admin-history", "SQL log", "admin-history"), + ("Failure to sender", "ct_sender", "email bounce plaintext", "ct_sender"), + ("Ops bounce", "ct_system / Network Error Bundle", "log the ciphertext body", "ct_system"), + ("Inspect officers", "share names", "Google accounts", "k-of-n"), + ("Console SSO", "PFC TOTP/Google", "Peergos Drive login", "PFC_REQUIRE_AUTH"), + ("NPE sidecar", "npe seal --to", "HPKE in Python", "npe_adapter"), + ("Review gate", "cicd/GATE.md", "auto-deploy to ns1", "GATE.md"), + ("sm-leaf health", "GET 127.0.0.1:18783/health", "WAN NATS ports", "18783"), +] + + +def hits(needle: str, limit: int = 8) -> list[str]: + cmd = [ + "rg", + "-n", + "--glob", + "!**/{build,.git,node_modules,__pycache__,bin}/**", + "-g", + "!*.pyc", + needle, + ] + paths = [str(ROOT / s) for s in SCOPE] + p = subprocess.run(cmd + paths, capture_output=True, text=True) + lines = [ln for ln in (p.stdout or "").splitlines() if ln.strip()] + out = [] + for ln in lines[:limit]: + if ":" in ln: + out.append(ln.replace(str(ROOT) + "/", "")[:160]) + return out + + +def main() -> None: + rows = [ + "# Thesaurus", + "", + "Canonical term → where it lives. Occurrences are grepped from in-scope", + "repos (docs **and** code). Banned phrases must not be used as the name", + "of the thing.", + "", + "This is **not** a HIPAA/SOC 2/ISO certificate.", + "", + "| Term | Use this | Not this | Occurrences |", + "|---|---|---|---|", + ] + for term, use, banned, needle in TERMS: + occ = hits(needle) + cell = "
".join(f"`{x}`" for x in occ) if occ else "_(none)_" + rows.append(f"| {term} | {use} | {banned} | {cell} |") + rows += [ + "", + "## Certification", + "", + "In-scope documentation uses the **Use this** column. `passthrough`", + "means destination in the clear, not a plaintext message body.", + "NATS is the untrusted broker. Production E2E is NPE/HPKE.", + "", + ] + OUT.write_text("\n".join(rows) + "\n") + print("wrote", OUT) + + +if __name__ == "__main__": + main() diff --git a/scripts/write_modules.py b/scripts/write_modules.py new file mode 100755 index 0000000..3d7452a --- /dev/null +++ b/scripts/write_modules.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +"""Write MODULE.md into every in-scope repo (callers, NATS, errors, storage).""" + +from __future__ import annotations + +from pathlib import Path + +R = Path("/Users/marchon/research") +GIT = "https://git.georgelambert.org/marchon" + +COMMON = """ +This is **not** a HIPAA/SOC 2/ISO certificate. +Vocabulary: see `{git}/system-git-sync` `docs/thesaurus.md`. +Catalog: `{git}/nats-service-endpoints`. +""" + +MODULES = { + "system-git-sync": dict( + who="Operators and later agents. Nothing publishes NATS from this repo.", + callers="Humans / CI (`cicd/run-review.sh`, `cicd/deploy-ns1.sh`).", + frm="Promoted SHAs, inventory, remaining-work.", + to="Forgejo remotes and ns1 rsync after tests.", + filt="No auto-deploy. Unsigned config never copied. Websites org inventory-only.", + err="Review fail aborts deploy. Binary-busy: stop unit then copy.", + store="Git only. Live signed files are on the host.", + timeout="SSH/rsync operator-bound.", + ), + "nats-service-endpoints": dict( + who="Every service that publishes `verae.*`.", + callers="secure-messaging, pfc-py-admin, pfc-repl, pfc-ipfs, sm-leaf.", + frm="`catalog/endpoints.json` (SM core) + `catalog/pfc-derived.json` (all subjects.json names).", + to="Sphinx HTML/PDF on docs.pfc `/nats-service-endpoints/`.", + filt="`reject[]` + `filter` + `timeout_sec`. Forbidden `verae.llm.turn.>`.", + err="Unknown subject / missing fields → dead letter for SM; others `dead=none`.", + store="JSON catalog in git.", + timeout="See each row `timeout_sec` (5s request-reply, 4s queue, 30s pin).", + ), + "secure-messaging": dict( + who="sm-leaf (Go) and pfc-py-admin (Python sign/history).", + callers="Console send, tests, systemd `pfc-sm-leaf`.", + frm="`verae.sm.send` envelopes (`to`, `from_lookup_id`, `alg`, `ct`).", + to="Mailbox dest (clear); ack `lookup_id`. Failures → `verae.sm.error`, `verae.sm.dead`, `verae.sm.log.summary`.", + filt="Missing `to`, empty `ct` (unless plain-lab), plaintext `body`, bad alg, summary `ct` field.", + err="Network Error Bundle: `ct_sender` (sender only) + `ct_system` (ops). Never log mail body.", + store="Signed wrapper path; admin-history JSONL cube.", + timeout="NATS 5s; NPE seal 15s.", + ), + "peergos-for-compliance": dict( + who="Shared Python lib (`pfc.*`) used by admin, ipfs, repl, connector.", + callers="pfc-py-admin, tests, Pattern A inspect app outbox.", + frm="HTTP browsers/CLI; NATS `verae.*` when `NATS_URL` set.", + to="CubeStore `/PFC_DATA`, stamp leaf `verae.ts.batch`, pin, replicate.", + filt="Profile validate (k>=2, author not officer); HIPAA bytes need inspect session.", + err="AccessDenied, ProfileError, NatsError, hash mismatch.", + store="`cubes//` chain, catalog card, hash registry.", + timeout="NATS 5s; stamp sync 15s.", + ), + "peergos-for-compliance-admin": dict( + who="Operators in the browser SPA.", + callers="https://pfc.georgelambert.org (Caddy → 127.0.0.1:18780).", + frm="Login/TOTP, cube/catalog/inspect/audit HTTP; NATS when bus configured.", + to="CubeStore, UserStore, signed config, sm-leaf, pin, repl.", + filt="`PFC_REQUIRE_AUTH=1`; TOTP for hipaa; k-of-n inspect; unsigned config rejected.", + err="401 auth, 403 totp/admin, 400 profile, 501 missing signing key.", + store="`PFC_DATA` users + cubes; host PEM for Ed25519.", + timeout="HTTP 8s in tests; NATS 5s.", + ), + "peergos-for-compliance-ipfs": dict( + who="Admin object put (pin) and hydrate.", + callers="pfc-py-admin `verae.storage.pin` / HTTP `/v1/pin`.", + frm="`sha256`, `bytes_hex`, `cube_id`, `tier`.", + to="Kubo API localhost:15001 (never WAN 5001).", + filt="Localhost Kubo only; hash must match bytes.", + err="Kubo down → health 503 for ipfs backend; pin error JSON.", + store="Kubo blocks; optional stash dir.", + timeout="Kubo 30s.", + ), + "peergos-for-compliance-replication": dict( + who="Lab HTTP coordinator. **Live ns1 uses Go pfc-repl** instead.", + callers="Admin when `NATS_URL` unset (`PFC_REPL_URL`).", + frm="POST `/v1/replicate` `{bytes_hex, sha256, cube_id}`.", + to="Replica ingest URLs / `verae.pfc.replica.{a,b}.ingest`.", + filt="`min_ok` acks including local.", + err="503 when min_ok not met.", + store="`PFC_REPL_DATA` files named by sha256.", + timeout="4s per replica request.", + ), + "peergos-compliance-go": dict( + who="ns1 leaf processes: pfc-repl, pfc-repl-a/b, pfc-configd, (inactive pfc-admin HTTP).", + callers="systemd. pfc-py-admin health-probes pfc-repl `:18784`.", + frm="NATS `verae.storage.replicate`, replica ingest, `verae.pfc.health`.", + to="Local data dir + replica subjects; config UI static files.", + filt="Coordinator only binds HTTP; ingest replicas NATS-only.", + err="min_ok failure in replicate JSON; leaf hub parse `nats-leaf://` → `nats://`.", + store="`/opt/pfc/data/repl`.", + timeout="4s replica Request.", + ), + "peergos-compliance-docs": dict( + who="Operators and auditors reading docs.pfc.", + callers="Sphinx build; Caddy `docs.pfc.georgelambert.org`.", + frm="RST sources + screenshots in `_static/screens/`.", + to="HTML + LaTeX PDF under `/opt/pfc/docs/html`.", + filt="No PHI examples; not a certificate.", + err="latexmk optional; xelatex/pdflatex fallback.", + store="git + ns1 docs tree.", + timeout="n/a", + ), + "peergos-compliance-architecture": dict( + who="Design readers. No runtime.", + callers="Docs and plan repos.", + frm="Architecture notes for DataCube + Peergos + NATS.", + to="Links to runtime repos.", + filt="Inventory; do not contradict signed-config design locks.", + err="n/a", + store="Markdown in git.", + timeout="n/a", + ), + "peergos-compliance-config": dict( + who="Operator building `pfc.json` before signing.", + callers="Browser SPA `config.pfc.georgelambert.org`, bash/curses TUI, pfc-configd.", + frm="Host, leaf hub, NATS URL, lab PSK, profile, min_ok.", + to="Download JSON; **must be signed** (`signed_config.sign`) before load.", + filt="Unsigned files rejected at load. Live changes append admin-history.", + err="Invalid schema / missing crypto.mode.", + store="`~/pfc.json` or host signed path.", + timeout="n/a", + ), +} + + +def render(name: str, m: dict) -> str: + git = f"{GIT}/{name}" + return f"""# MODULE — {name} + +Git: {git} + +{COMMON.format(git=GIT)} + +## What this module is + +Runtime or documentation unit in the Peergos-for-compliance / secure-messaging +set. See this repo `README.md` for the short intro. + +## Who calls this module and why + +{m['who']} + +**Callers:** {m['callers']} + +## Messages — from / to + +- **From:** {m['frm']} +- **To:** {m['to']} + +## Filters / security + +{m['filt']} + +## Errors and timeouts + +- **Errors:** {m['err']} +- **Timeouts:** {m['timeout']} + +## Where data is stored and how others use it + +{m['store']} + +Walkthroughs and screenshots: `{GIT}/peergos-compliance-docs` (`source/walkthrough.rst`). +""" + + +def main() -> None: + hub = R / "system-git-sync" / "docs" / "modules" + hub.mkdir(parents=True, exist_ok=True) + for name, meta in MODULES.items(): + text = render(name, meta) + (hub / f"{name}.md").write_text(text) + dest = R / name / "MODULE.md" + if dest.parent.exists(): + dest.write_text(text) + print("wrote", dest) + + +if __name__ == "__main__": + main()