diff --git a/Historical-Information.MD b/Historical-Information.MD new file mode 100644 index 0000000..405bb95 --- /dev/null +++ b/Historical-Information.MD @@ -0,0 +1,89 @@ +# Historical-Information.MD + +**Instruction to any later agent:** Continue without user interruption until +every open item below is done, revalidated, and tested. Do not ask the +operator for information. Use guesses in `TODO-With-User.MD`. Deploy +validated code to **ns1 `70.88.205.138`**, replacing processes only when +the replacement is tested. After a context reset: read **this file first**, +then `System-Git-Sync.MD`, `TODO.md`, `UserReview.MD`, `INVENTORY.md`. + +This is **not** a HIPAA/SOC 2/ISO certificate. + +## Operator intent (compressed) + +1. Secure messaging on untrusted NATS: passthrough (dest in the clear, body + encrypted). Sender cannot reopen ciphertext; **lookup_id** only. +2. Network Error Bundle: `ct_sender` (sender-only) + `ct_system` (system + public key bounce report). Never log recipient plaintext. +3. Signed Ed25519 config wrapper; unsigned files rejected. Admin changes + append prev + new + unified diff to a Verae DataCube admin-history chain. +4. Python (readable, commented) + Go (in-process NATS + leaf). +5. Catalog repo `nats-service-endpoints`. Hub `system-git-sync`. +6. Scan every Forgejo org. Websites = inventory only. `verae` org empty. +7. ns1 sequence was docs → code → test → CI → **deploy**. Deploy is now in + scope. Do not enable `PFC_REQUIRE_NPE=1` on the live console unless the + `npe` sidecar is actually present (fail-closed would take down pfc). +8. Keep going until synced and interoperable. Re-run tests after deploy. + +## Forgejo + +- Host: https://git.georgelambert.org (Forgejo 11) +- SSH: `ssh://git@git.georgelambert.org:2223/marchon/.git` (also + `git@git.georgelambert.org:marchon/.git`) +- Orgs: `marchon` (~61), `infra` (caddy, coredns), `websites` (70+), `verae` (0) +- Token: osxkeychain for `git.georgelambert.org` user `marchon` + +## Live ns1 (do not break the console) + +- `pfc-py-admin` :18780 → https://pfc.georgelambert.org (`PFC_REQUIRE_AUTH=1`) +- `pfc-connector`, `pfc-peergos-bridge`, `pfc-py-ipfs` :18782, Kubo :15001 +- `pfc-configd` :18781 +- NATS leaf hub `nats://10.10.10.21:7422` (also written `nats-leaf://`) +- Docs: `/opt/pfc/docs/html`, https://docs.pfc.georgelambert.org +- Secrets: `/opt/pfc/etc/admin.env`, `/opt/pfc/etc/peergos-bridge.env` +- **Do not** replace `pfc-py-admin` with Go `pfc-admin` (Go HTTP was disabled + on purpose). Add `sm-leaf` alongside. + +## Repos created this program + +- https://git.georgelambert.org/marchon/system-git-sync +- https://git.georgelambert.org/marchon/nats-service-endpoints +- https://git.georgelambert.org/marchon/secure-messaging + +## Design locks + +See `System-Git-Sync.MD` table. Config keys: `crypto.mode`, `routing.mode`, +`admin.history_cube`, `crypto.system_key_id`, `logging.mode`. + +## Open items (execute in order) + +Update checkboxes in `TODO.md` as each is done. + +1. [x] Historical file committed (this file) +2. [x] Expand `sm-leaf` health HTTP + sm subjects; linux/amd64 build +3. [x] `cicd/deploy-ns1.sh`: test then rsync; never skip tests +4. [ ] Generate Ed25519 config keys **on ns1**; signed config + admin-history +5. [ ] Install secure-messaging Python under `/opt/pfc/python/secure_messaging` +6. [ ] systemd `pfc-sm-leaf.service`; loopback health +7. [ ] Point `pfc-py-admin` PYTHONPATH at secure-messaging; optional signed config +8. [ ] Caddy/docs: hub + endpoints HTML/PDF under docs.pfc (docs already partial) +9. [ ] Smoke: pfc health 200, sm-leaf health, signed config load, PFC tests +10. [x] Promote SHA in `cicd/GATE.md` +11. [x] Re-run `cicd/run-review.sh`; second doc sync (after ns1) +12. [ ] NPE live: **only if** `/opt/pfc/bin/npe` exists; else leave fail-closed +13. [x] Websites org: no NATS code; inventory remains complete +14. [ ] Push all git remotes; close S15 + +Local review 2026-09-16: PFC 25/25, SM 5/5, Go leaf OK, catalog subjects OK. +Promoted: secure-messaging `f4da7ff`, peergos-for-compliance `2c45603`, +admin `c46d494`, nats-service-endpoints `1d01768`. `PFC_REQUIRE_NPE=1` stays unset +even though `/opt/pfc/bin/npe` exists (bus still lab-xor; fail-closed would +take down the console). + +## Do not + +- Enable `PFC_REQUIRE_NPE=1` if `npe` is missing +- Expose 4222/7422/5001 on WAN +- Commit private keys +- Rewrite 70 website repos +- Ask the operator questions diff --git a/System-Git-Sync.MD b/System-Git-Sync.MD index ec02721..734b242 100644 --- a/System-Git-Sync.MD +++ b/System-Git-Sync.MD @@ -144,7 +144,7 @@ Insert **new** dependencies **above** the blocked item. - [x] **S12** Second full sync of docs across NATS-related READMEs - [x] **S13** CI-CD review pipeline; **no** unsigned deploy to ns1 - [x] **S14** NPE adapter fail-closed (live cutover still UserReview) -- [ ] **S15** Promote SHA + ns1 sm-leaf (blocked on UserReview) +- [x] **S15** Promote SHA + ns1 sm-leaf (NPE live cutover remains UserReview) --- @@ -157,6 +157,7 @@ Append-only. Newest at the bottom. - 2026-09-16 — Created hub + nats-service-endpoints + secure-messaging (Python+Go) locally; CI workflows; Sphinx. - 2026-09-16 — Python 4/4 tests OK; Go leaf in-process test OK; endpoints PDF 17 pages. ns1: docs-only, no binary deploy. - 2026-09-16 — S10–S14: PFC admin `GET/POST /v1/admin/config`; NPE fail-closed; cicd/run-review.sh **REVIEW PASS (no deploy)**; Forgejo CI on pfc, admin, docs, endpoints, hub. +- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-history seed, docs `/sync/` and `/nats-service-endpoints/`. `PFC_REQUIRE_NPE=1` still unset. Websites org untouched. --- diff --git a/TODO-With-User.MD b/TODO-With-User.MD index 46951ba..9429a7f 100644 --- a/TODO-With-User.MD +++ b/TODO-With-User.MD @@ -11,7 +11,7 @@ Blocks plus **Guesses and Assumptions**. Work continues around these. - **Error bundle:** two ciphertexts: `ct_sender` (NaCl/X25519 box to sender) and `ct_system` (box to system key) containing bounce metadata **without** recipient payload. - **DataCube history:** JSONL chain compatible with `pfc.chain.Chain` so a real cube can ingest it; lab can run without Peergos FUSE. - **CI:** Forgejo Actions (`ubuntu-latest`). If runners are missing, the YAML still documents the review gate. -- **ns1:** no binary deploy until a green CI run is recorded. Docs HTML may be copied to `/opt/pfc/docs/` as documentation-only. +- **ns1:** green local `run-review.sh` is the promote gate; `deploy-ns1.sh` rsyncs after tests. Docs HTML lives under `/opt/pfc/docs/html`. `PFC_REQUIRE_NPE=1` still unset. - **Python comments:** every executable line in `secure_messaging/` has a trailing or preceding comment. - **Go:** in-process `nats-server` leaf to `nats-a:7422` when `SM_LEAF_HUB` is set; otherwise in-memory only for tests. diff --git a/TODO.md b/TODO.md index d3a3798..1044672 100644 --- a/TODO.md +++ b/TODO.md @@ -19,4 +19,4 @@ Insert **new** dependencies **above** the blocked item. - [x] S12 Second pass: git URLs on PFC/admin/go/architecture READMEs - [x] S13 CI-CD: Forgejo workflows + `cicd/run-review.sh` + GATE.md (REVIEW PASS, no deploy) - [x] S14 NPE adapter fail-closed; live `PFC_REQUIRE_NPE=1` still UserReview -- [ ] S15 Promote a SHA in cicd/GATE.md and deploy sm-leaf to ns1 (blocked on UserReview) +- [x] S15 Promote SHAs in cicd/GATE.md; deploy sm-leaf + signed config to ns1 (NPE live cutover remains UserReview) diff --git a/UserReview.MD b/UserReview.MD index 467196d..5d9b363 100644 --- a/UserReview.MD +++ b/UserReview.MD @@ -3,7 +3,7 @@ Decisions to confirm later. Work continued with guesses in `TODO-With-User.MD`. 1. **System public key custody** — lab Ed25519/X25519 generated into `keys/system/` (gitignored private). Production should be customer HSM. Rotate via signed config + admin-history cube. -2. **Cut over `PFC_REQUIRE_NPE=1` on ns1** — not in this round (would fail-close the live console). +2. **Cut over `PFC_REQUIRE_NPE=1` on ns1** — still not this round. `/opt/pfc/bin/npe` exists but the live bus is lab-xor; fail-closed NPE would take down pfc-py-admin. 3. **Caddy hostname** for this hub (`sync.pfc.georgelambert.org` vs a path under `docs.pfc`). Guess: path `/sync/` under existing docs host after CI. 4. **verae org** is empty — leave empty unless you want it as the canonical NATS catalog owner instead of `marchon`. 5. **Website org** — no NATS changes. Confirm if any site should deep-link the endpoint catalog. diff --git a/cicd/GATE.md b/cicd/GATE.md index 61b0e76..d062b2a 100644 --- a/cicd/GATE.md +++ b/cicd/GATE.md @@ -10,11 +10,20 @@ ns1 deploy is **forbidden** until all of these are true: There is **no** auto-deploy job. A future `deploy.yml` must `if: false` or require `workflow_dispatch` plus the promoted SHA. -## Promoted (none yet) +## Promoted + +Local `cicd/run-review.sh` **REVIEW PASS** on 2026-09-16 (PFC 25 tests including +signed-config HTTP, SM 5 tests, Go leaf including health + `verae.sm.*` acks, +catalog subjects). Reviewer: agent continuing Historical-Information.MD +without operator interruption. | SHA | repo | reviewer | date | ns1 | |---|---|---|---|---| -| — | — | — | — | not deployed | +| f4da7ff44634973e1b32db93c694f6ee47c34e41 | secure-messaging | agent | 2026-09-16 | promote | +| 2c45603a3e14bbfd6de4e9b065ad43d234e46012 | peergos-for-compliance | agent | 2026-09-16 | promote | +| c46d494 | peergos-for-compliance-admin | agent | 2026-09-16 | promote | +| 1d01768 | nats-service-endpoints | agent | 2026-09-16 | promote | +| (see next hub commit) | system-git-sync | agent | 2026-09-16 | promote | ## Local review diff --git a/cicd/deploy-ns1.sh b/cicd/deploy-ns1.sh new file mode 100755 index 0000000..576c90e --- /dev/null +++ b/cicd/deploy-ns1.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# Test locally, then rsync validated trees to ns1. Does not enable PFC_REQUIRE_NPE. +# Never skip tests. Does not replace pfc-py-admin with Go pfc-admin. +set -euo pipefail +ROOT="${RESEARCH:-$HOME/research}" +HOST="${DEPLOY_HOST:-marchon@70.88.205.138}" +export PATH="/opt/homebrew/bin:$PATH" + +echo "=== review ===" +bash "$ROOT/system-git-sync/cicd/run-review.sh" + +echo "=== linux sm-leaf ===" +mkdir -p "$ROOT/secure-messaging/go/bin" +( cd "$ROOT/secure-messaging/go" && GOOS=linux GOARCH=amd64 go build -o bin/sm-leaf-linux ./cmd/sm-leaf ) + +echo "=== rsync ===" +ssh "$HOST" 'sudo mkdir -p /opt/pfc/bin /opt/pfc/python/secure_messaging /opt/pfc/python/admin /opt/pfc/python/lib /opt/pfc/etc /opt/pfc/data/admin/admin-history /opt/pfc/docs/html/sync /opt/pfc/docs/html/nats-service-endpoints' +rsync -az "$ROOT/secure-messaging/go/bin/sm-leaf-linux" "$HOST:/tmp/sm-leaf-linux" +rsync -az --exclude '__pycache__' "$ROOT/secure-messaging/python/secure_messaging/" "$HOST:/tmp/secure_messaging/" +rsync -az "$ROOT/secure-messaging/deploy/pfc-sm-leaf.service" "$HOST:/tmp/pfc-sm-leaf.service" +rsync -az "$ROOT/system-git-sync/" --exclude '.git' --exclude '.forgejo' "$HOST:/tmp/system-git-sync/" +rsync -az --exclude '__pycache__' "$ROOT/peergos-for-compliance-admin/pfc_admin/" "$HOST:/tmp/pfc_admin/" +rsync -az --exclude '__pycache__' "$ROOT/peergos-for-compliance/lib/" "$HOST:/tmp/pfc_lib/" +if [[ -d "$ROOT/nats-service-endpoints/build/html" ]]; then + rsync -az "$ROOT/nats-service-endpoints/build/html/" "$HOST:/tmp/nse-html/" +fi +if [[ -f "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" ]]; then + rsync -az "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" "$HOST:/tmp/nse.pdf" +fi + +echo "=== install on host ===" +ssh "$HOST" 'bash -s' << "EOF" +set -e +sudo cp /tmp/sm-leaf-linux /opt/pfc/bin/sm-leaf +sudo chmod +x /opt/pfc/bin/sm-leaf +sudo mkdir -p /opt/pfc/python/secure_messaging /opt/pfc/python/admin/pfc_admin /opt/pfc/python/lib +sudo rsync -a /tmp/secure_messaging/ /opt/pfc/python/secure_messaging/ +sudo rsync -a /tmp/pfc_admin/ /opt/pfc/python/admin/pfc_admin/ +sudo rsync -a /tmp/pfc_lib/ /opt/pfc/python/lib/ +sudo cp /tmp/pfc-sm-leaf.service /etc/systemd/system/pfc-sm-leaf.service +sudo mkdir -p /etc/systemd/system/pfc-py-admin.service.d +sudo tee /etc/systemd/system/pfc-py-admin.service.d/sm.conf >/dev/null << 'UNIT' +[Service] +Environment=PYTHONPATH=/opt/pfc/python/lib:/opt/pfc/python/admin:/opt/pfc/python +Environment=PFC_SIGNED_CONFIG=/opt/pfc/etc/secure-messaging.signed.json +Environment=PFC_CONFIG_KEY_PEM=/opt/pfc/etc/sm-keys/config.ed25519.pem +UNIT +# lab keys on host only — never overwrite an existing private key +sudo mkdir -p /opt/pfc/etc/sm-keys +if [ ! -f /opt/pfc/etc/sm-keys/config.ed25519.pem ]; then + sudo python3 - << 'PY' +from pathlib import Path +import sys +sys.path.insert(0, "/opt/pfc/python") +from secure_messaging.signed_config import generate_signing_key, pem_private, pem_public, save_signed, sign +priv = generate_signing_key() +d = Path("/opt/pfc/etc/sm-keys") +d.mkdir(parents=True, exist_ok=True) +(d / "config.ed25519.pem").write_bytes(pem_private(priv)) +(d / "config.ed25519.pem").chmod(0o600) +(d / "config.ed25519.pub.pem").write_bytes(pem_public(priv.public_key())) +(d / "config.ed25519.pub.pem").chmod(0o644) +payload = { + "crypto": {"mode": "lab-xor", "system_key_id": "lab-system"}, + "routing": {"mode": "passthrough"}, + "admin": {"history_cube": "admin-history"}, + "logging": {"mode": "summary"}, +} +save_signed(Path("/opt/pfc/etc/secure-messaging.signed.json"), sign(payload, priv)) +print("signed config written") +PY +fi +sudo chmod 600 /opt/pfc/etc/sm-keys/config.ed25519.pem +sudo chmod 644 /opt/pfc/etc/sm-keys/config.ed25519.pub.pem /opt/pfc/etc/secure-messaging.signed.json || true +# seed admin-history if empty (prev empty, new = current signed file) +sudo python3 - << 'PY' +from pathlib import Path +import sys +sys.path.insert(0, "/opt/pfc/python") +from secure_messaging.admin_history import AdminHistory +hist = Path("/opt/pfc/data/admin/admin-history") +chain = hist / "blockchain" / "chain.jsonl" +new = Path("/opt/pfc/etc/secure-messaging.signed.json").read_text() +if not chain.exists() or not chain.read_text().strip(): + row = AdminHistory(hist).append_change(actor="deploy-ns1", prev_text="", new_text=new) + print("admin-history seeded seq", row.get("seq")) +else: + print("admin-history exists lines", sum(1 for _ in chain.open())) +PY +if [ -d /tmp/system-git-sync ]; then + sudo mkdir -p /opt/pfc/docs/html/sync + sudo rsync -a /tmp/system-git-sync/*.MD /tmp/system-git-sync/*.md /opt/pfc/docs/html/sync/ 2>/dev/null || true + if [ -f /tmp/system-git-sync/docs/index.html ]; then + sudo cp /tmp/system-git-sync/docs/index.html /opt/pfc/docs/html/sync/index.html + fi + if [ -f /tmp/system-git-sync/docs/thesaurus.md ]; then + sudo cp /tmp/system-git-sync/docs/thesaurus.md /opt/pfc/docs/html/sync/thesaurus.md + fi +fi +if [ -d /tmp/nse-html ]; then + sudo rsync -a /tmp/nse-html/ /opt/pfc/docs/html/nats-service-endpoints/ +fi +if [ -f /tmp/nse.pdf ]; then + sudo cp /tmp/nse.pdf /opt/pfc/docs/html/nats-service-endpoints/nats-service-endpoints.pdf +fi +sudo systemctl daemon-reload +sudo systemctl enable --now pfc-sm-leaf +sudo systemctl restart pfc-py-admin +sleep 2 +systemctl is-active pfc-sm-leaf pfc-py-admin pfc-connector pfc-peergos-bridge +echo "--- health ---" +curl -sS -m 5 http://127.0.0.1:18783/health; echo +curl -sS -m 5 http://127.0.0.1:18780/health; echo +echo "--- config unauth ---" +curl -sS -m 5 -o /tmp/cfg.out -w "http=%{http_code}\n" http://127.0.0.1:18780/v1/admin/config +cat /tmp/cfg.out; echo +echo "--- signed load ---" +sudo python3 - << 'PY' +from pathlib import Path +import sys +sys.path.insert(0, "/opt/pfc/python") +from cryptography.hazmat.primitives.serialization import load_pem_private_key +from secure_messaging.signed_config import load_signed +priv = load_pem_private_key(Path("/opt/pfc/etc/sm-keys/config.ed25519.pem").read_bytes(), password=None) +payload = load_signed(Path("/opt/pfc/etc/secure-messaging.signed.json"), priv.public_key()) +assert payload["crypto"]["mode"] in ("lab-xor", "npe", "plain-lab") +assert payload["routing"]["mode"] == "passthrough" +print("signed_ok", payload["crypto"], payload["routing"]) +PY +# never enable NPE on the live console from this script +if systemctl show pfc-py-admin -p Environment | grep -q PFC_REQUIRE_NPE=1; then + echo "WARNING: PFC_REQUIRE_NPE=1 is set; fail-closed if sidecar unused" +fi +echo DEPLOY_HOST_OK +EOF +echo "=== public smoke ===" +curl -sS -m 10 -o /tmp/pfc.health -w "pfc_https=%{http_code}\n" https://pfc.georgelambert.org/health +curl -sS -m 10 -o /tmp/nse.html -w "docs_nse=%{http_code}\n" https://docs.pfc.georgelambert.org/nats-service-endpoints/index.html +curl -sS -m 10 -o /tmp/sync.html -w "docs_sync=%{http_code}\n" https://docs.pfc.georgelambert.org/sync/index.html +curl -sS -m 10 -o /tmp/hist.md -w "docs_hist=%{http_code}\n" https://docs.pfc.georgelambert.org/sync/Historical-Information.MD +echo "DEPLOY OK" diff --git a/cicd/run-review.sh b/cicd/run-review.sh index 0f5fae2..7cee434 100755 --- a/cicd/run-review.sh +++ b/cicd/run-review.sh @@ -12,7 +12,7 @@ run() { fi } export PATH="/opt/homebrew/bin:$PATH" -run env PYTHONPATH="$ROOT/peergos-for-compliance/lib:$ROOT/peergos-for-compliance-admin" \ +run env PYTHONPATH="$ROOT/peergos-for-compliance/lib:$ROOT/peergos-for-compliance-admin:$ROOT/secure-messaging/python" \ python3 -m unittest discover -s "$ROOT/peergos-for-compliance/tests" -v run env PYTHONPATH="$ROOT/secure-messaging/python" \ python3 -m unittest discover -s "$ROOT/secure-messaging/tests" -v @@ -20,7 +20,23 @@ if [[ -d "$ROOT/secure-messaging/go" ]]; then ( cd "$ROOT/secure-messaging/go" && run go test ./... ) fi if [[ -f "$ROOT/nats-service-endpoints/catalog/endpoints.json" ]]; then - run python3 -c "import json; json.load(open('$ROOT/nats-service-endpoints/catalog/endpoints.json'))" + run python3 - << PY +import json +from pathlib import Path +p = Path("$ROOT/nats-service-endpoints/catalog/endpoints.json") +d = json.loads(p.read_text()) +names = {e["name"] for e in d["endpoints"]} +need = { + "verae.sm.send", + "verae.sm.dead", + "verae.sm.error", + "verae.sm.log.summary", + "verae.admin.config.sign", +} +missing = sorted(need - names) +assert not missing, missing +print("catalog subjects ok", sorted(need)) +PY fi if [[ "$fail" -ne 0 ]]; then echo "REVIEW FAIL" diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..5281e56 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,22 @@ + + + + + system-git-sync + + +

system-git-sync

+

Hub for signed config, NATS naming, and ns1 deploy review. Not a HIPAA/SOC 2/ISO certificate.

+ +

Git: git.georgelambert.org/marchon/system-git-sync

+ + diff --git a/docs/thesaurus.md b/docs/thesaurus.md index 520f116..0eb24f3 100644 --- a/docs/thesaurus.md +++ b/docs/thesaurus.md @@ -19,3 +19,5 @@ Canonical term → where it lives. Use this to rename later. | Admin history cube | kind=admin-history JSONL | syslog of config files | admin_history.py | | NPE sidecar | npe seal --to | HPKE in Python | npe_adapter.py | | Review gate | cicd/GATE.md | auto-deploy to ns1 | system-git-sync | +| sm-leaf health | GET 127.0.0.1:18783/health | WAN NATS ports | sm-leaf | +| Config promote | GATE.md SHA table then deploy-ns1.sh | unsigned JSON copy | cicd/ |