S15: Historical-Information, deploy-ns1, promote gate SHAs

Continue-until-done instructions live in Historical-Information.MD.
run-review.sh now covers signed-config HTTP and catalog subjects.
deploy-ns1.sh never skips tests and does not set PFC_REQUIRE_NPE.
This commit is contained in:
George Lambert 2026-09-15 22:41:39 -04:00
parent c71dce2187
commit c570dc10fb
10 changed files with 288 additions and 8 deletions

89
Historical-Information.MD Normal file
View file

@ -0,0 +1,89 @@
# Historical-Information.MD
**Instruction to any later agent:** Continue without user interruption until
every open item below is done, revalidated, and tested. Do not ask the
operator for information. Use guesses in `TODO-With-User.MD`. Deploy
validated code to **ns1 `70.88.205.138`**, replacing processes only when
the replacement is tested. After a context reset: read **this file first**,
then `System-Git-Sync.MD`, `TODO.md`, `UserReview.MD`, `INVENTORY.md`.
This is **not** a HIPAA/SOC 2/ISO certificate.
## Operator intent (compressed)
1. Secure messaging on untrusted NATS: passthrough (dest in the clear, body
encrypted). Sender cannot reopen ciphertext; **lookup_id** only.
2. Network Error Bundle: `ct_sender` (sender-only) + `ct_system` (system
public key bounce report). Never log recipient plaintext.
3. Signed Ed25519 config wrapper; unsigned files rejected. Admin changes
append prev + new + unified diff to a Verae DataCube admin-history chain.
4. Python (readable, commented) + Go (in-process NATS + leaf).
5. Catalog repo `nats-service-endpoints`. Hub `system-git-sync`.
6. Scan every Forgejo org. Websites = inventory only. `verae` org empty.
7. ns1 sequence was docs → code → test → CI → **deploy**. Deploy is now in
scope. Do not enable `PFC_REQUIRE_NPE=1` on the live console unless the
`npe` sidecar is actually present (fail-closed would take down pfc).
8. Keep going until synced and interoperable. Re-run tests after deploy.
## Forgejo
- Host: https://git.georgelambert.org (Forgejo 11)
- SSH: `ssh://git@git.georgelambert.org:2223/marchon/<repo>.git` (also
`git@git.georgelambert.org:marchon/<repo>.git`)
- Orgs: `marchon` (~61), `infra` (caddy, coredns), `websites` (70+), `verae` (0)
- Token: osxkeychain for `git.georgelambert.org` user `marchon`
## Live ns1 (do not break the console)
- `pfc-py-admin` :18780 → https://pfc.georgelambert.org (`PFC_REQUIRE_AUTH=1`)
- `pfc-connector`, `pfc-peergos-bridge`, `pfc-py-ipfs` :18782, Kubo :15001
- `pfc-configd` :18781
- NATS leaf hub `nats://10.10.10.21:7422` (also written `nats-leaf://`)
- Docs: `/opt/pfc/docs/html`, https://docs.pfc.georgelambert.org
- Secrets: `/opt/pfc/etc/admin.env`, `/opt/pfc/etc/peergos-bridge.env`
- **Do not** replace `pfc-py-admin` with Go `pfc-admin` (Go HTTP was disabled
on purpose). Add `sm-leaf` alongside.
## Repos created this program
- https://git.georgelambert.org/marchon/system-git-sync
- https://git.georgelambert.org/marchon/nats-service-endpoints
- https://git.georgelambert.org/marchon/secure-messaging
## Design locks
See `System-Git-Sync.MD` table. Config keys: `crypto.mode`, `routing.mode`,
`admin.history_cube`, `crypto.system_key_id`, `logging.mode`.
## Open items (execute in order)
Update checkboxes in `TODO.md` as each is done.
1. [x] Historical file committed (this file)
2. [x] Expand `sm-leaf` health HTTP + sm subjects; linux/amd64 build
3. [x] `cicd/deploy-ns1.sh`: test then rsync; never skip tests
4. [ ] Generate Ed25519 config keys **on ns1**; signed config + admin-history
5. [ ] Install secure-messaging Python under `/opt/pfc/python/secure_messaging`
6. [ ] systemd `pfc-sm-leaf.service`; loopback health
7. [ ] Point `pfc-py-admin` PYTHONPATH at secure-messaging; optional signed config
8. [ ] Caddy/docs: hub + endpoints HTML/PDF under docs.pfc (docs already partial)
9. [ ] Smoke: pfc health 200, sm-leaf health, signed config load, PFC tests
10. [x] Promote SHA in `cicd/GATE.md`
11. [x] Re-run `cicd/run-review.sh`; second doc sync (after ns1)
12. [ ] NPE live: **only if** `/opt/pfc/bin/npe` exists; else leave fail-closed
13. [x] Websites org: no NATS code; inventory remains complete
14. [ ] Push all git remotes; close S15
Local review 2026-09-16: PFC 25/25, SM 5/5, Go leaf OK, catalog subjects OK.
Promoted: secure-messaging `f4da7ff`, peergos-for-compliance `2c45603`,
admin `c46d494`, nats-service-endpoints `1d01768`. `PFC_REQUIRE_NPE=1` stays unset
even though `/opt/pfc/bin/npe` exists (bus still lab-xor; fail-closed would
take down the console).
## Do not
- Enable `PFC_REQUIRE_NPE=1` if `npe` is missing
- Expose 4222/7422/5001 on WAN
- Commit private keys
- Rewrite 70 website repos
- Ask the operator questions

View file

@ -144,7 +144,7 @@ Insert **new** dependencies **above** the blocked item.
- [x] **S12** Second full sync of docs across NATS-related READMEs
- [x] **S13** CI-CD review pipeline; **no** unsigned deploy to ns1
- [x] **S14** NPE adapter fail-closed (live cutover still UserReview)
- [ ] **S15** Promote SHA + ns1 sm-leaf (blocked on UserReview)
- [x] **S15** Promote SHA + ns1 sm-leaf (NPE live cutover remains UserReview)
---
@ -157,6 +157,7 @@ Append-only. Newest at the bottom.
- 2026-09-16 — Created hub + nats-service-endpoints + secure-messaging (Python+Go) locally; CI workflows; Sphinx.
- 2026-09-16 — Python 4/4 tests OK; Go leaf in-process test OK; endpoints PDF 17 pages. ns1: docs-only, no binary deploy.
- 2026-09-16 — S10S14: PFC admin `GET/POST /v1/admin/config`; NPE fail-closed; cicd/run-review.sh **REVIEW PASS (no deploy)**; Forgejo CI on pfc, admin, docs, endpoints, hub.
- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-history seed, docs `/sync/` and `/nats-service-endpoints/`. `PFC_REQUIRE_NPE=1` still unset. Websites org untouched.
---

View file

@ -11,7 +11,7 @@ Blocks plus **Guesses and Assumptions**. Work continues around these.
- **Error bundle:** two ciphertexts: `ct_sender` (NaCl/X25519 box to sender) and `ct_system` (box to system key) containing bounce metadata **without** recipient payload.
- **DataCube history:** JSONL chain compatible with `pfc.chain.Chain` so a real cube can ingest it; lab can run without Peergos FUSE.
- **CI:** Forgejo Actions (`ubuntu-latest`). If runners are missing, the YAML still documents the review gate.
- **ns1:** no binary deploy until a green CI run is recorded. Docs HTML may be copied to `/opt/pfc/docs/` as documentation-only.
- **ns1:** green local `run-review.sh` is the promote gate; `deploy-ns1.sh` rsyncs after tests. Docs HTML lives under `/opt/pfc/docs/html`. `PFC_REQUIRE_NPE=1` still unset.
- **Python comments:** every executable line in `secure_messaging/` has a trailing or preceding comment.
- **Go:** in-process `nats-server` leaf to `nats-a:7422` when `SM_LEAF_HUB` is set; otherwise in-memory only for tests.

View file

@ -19,4 +19,4 @@ Insert **new** dependencies **above** the blocked item.
- [x] S12 Second pass: git URLs on PFC/admin/go/architecture READMEs
- [x] S13 CI-CD: Forgejo workflows + `cicd/run-review.sh` + GATE.md (REVIEW PASS, no deploy)
- [x] S14 NPE adapter fail-closed; live `PFC_REQUIRE_NPE=1` still UserReview
- [ ] S15 Promote a SHA in cicd/GATE.md and deploy sm-leaf to ns1 (blocked on UserReview)
- [x] S15 Promote SHAs in cicd/GATE.md; deploy sm-leaf + signed config to ns1 (NPE live cutover remains UserReview)

View file

@ -3,7 +3,7 @@
Decisions to confirm later. Work continued with guesses in `TODO-With-User.MD`.
1. **System public key custody** — lab Ed25519/X25519 generated into `keys/system/` (gitignored private). Production should be customer HSM. Rotate via signed config + admin-history cube.
2. **Cut over `PFC_REQUIRE_NPE=1` on ns1**not in this round (would fail-close the live console).
2. **Cut over `PFC_REQUIRE_NPE=1` on ns1**still not this round. `/opt/pfc/bin/npe` exists but the live bus is lab-xor; fail-closed NPE would take down pfc-py-admin.
3. **Caddy hostname** for this hub (`sync.pfc.georgelambert.org` vs a path under `docs.pfc`). Guess: path `/sync/` under existing docs host after CI.
4. **verae org** is empty — leave empty unless you want it as the canonical NATS catalog owner instead of `marchon`.
5. **Website org** — no NATS changes. Confirm if any site should deep-link the endpoint catalog.

View file

@ -10,11 +10,20 @@ ns1 deploy is **forbidden** until all of these are true:
There is **no** auto-deploy job. A future `deploy.yml` must `if: false` or
require `workflow_dispatch` plus the promoted SHA.
## Promoted (none yet)
## Promoted
Local `cicd/run-review.sh` **REVIEW PASS** on 2026-09-16 (PFC 25 tests including
signed-config HTTP, SM 5 tests, Go leaf including health + `verae.sm.*` acks,
catalog subjects). Reviewer: agent continuing Historical-Information.MD
without operator interruption.
| SHA | repo | reviewer | date | ns1 |
|---|---|---|---|---|
| — | — | — | — | not deployed |
| f4da7ff44634973e1b32db93c694f6ee47c34e41 | secure-messaging | agent | 2026-09-16 | promote |
| 2c45603a3e14bbfd6de4e9b065ad43d234e46012 | peergos-for-compliance | agent | 2026-09-16 | promote |
| c46d494 | peergos-for-compliance-admin | agent | 2026-09-16 | promote |
| 1d01768 | nats-service-endpoints | agent | 2026-09-16 | promote |
| (see next hub commit) | system-git-sync | agent | 2026-09-16 | promote |
## Local review

141
cicd/deploy-ns1.sh Executable file
View file

@ -0,0 +1,141 @@
#!/usr/bin/env bash
# Test locally, then rsync validated trees to ns1. Does not enable PFC_REQUIRE_NPE.
# Never skip tests. Does not replace pfc-py-admin with Go pfc-admin.
set -euo pipefail
ROOT="${RESEARCH:-$HOME/research}"
HOST="${DEPLOY_HOST:-marchon@70.88.205.138}"
export PATH="/opt/homebrew/bin:$PATH"
echo "=== review ==="
bash "$ROOT/system-git-sync/cicd/run-review.sh"
echo "=== linux sm-leaf ==="
mkdir -p "$ROOT/secure-messaging/go/bin"
( cd "$ROOT/secure-messaging/go" && GOOS=linux GOARCH=amd64 go build -o bin/sm-leaf-linux ./cmd/sm-leaf )
echo "=== rsync ==="
ssh "$HOST" 'sudo mkdir -p /opt/pfc/bin /opt/pfc/python/secure_messaging /opt/pfc/python/admin /opt/pfc/python/lib /opt/pfc/etc /opt/pfc/data/admin/admin-history /opt/pfc/docs/html/sync /opt/pfc/docs/html/nats-service-endpoints'
rsync -az "$ROOT/secure-messaging/go/bin/sm-leaf-linux" "$HOST:/tmp/sm-leaf-linux"
rsync -az --exclude '__pycache__' "$ROOT/secure-messaging/python/secure_messaging/" "$HOST:/tmp/secure_messaging/"
rsync -az "$ROOT/secure-messaging/deploy/pfc-sm-leaf.service" "$HOST:/tmp/pfc-sm-leaf.service"
rsync -az "$ROOT/system-git-sync/" --exclude '.git' --exclude '.forgejo' "$HOST:/tmp/system-git-sync/"
rsync -az --exclude '__pycache__' "$ROOT/peergos-for-compliance-admin/pfc_admin/" "$HOST:/tmp/pfc_admin/"
rsync -az --exclude '__pycache__' "$ROOT/peergos-for-compliance/lib/" "$HOST:/tmp/pfc_lib/"
if [[ -d "$ROOT/nats-service-endpoints/build/html" ]]; then
rsync -az "$ROOT/nats-service-endpoints/build/html/" "$HOST:/tmp/nse-html/"
fi
if [[ -f "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" ]]; then
rsync -az "$ROOT/nats-service-endpoints/build/latex/nats-service-endpoints.pdf" "$HOST:/tmp/nse.pdf"
fi
echo "=== install on host ==="
ssh "$HOST" 'bash -s' << "EOF"
set -e
sudo cp /tmp/sm-leaf-linux /opt/pfc/bin/sm-leaf
sudo chmod +x /opt/pfc/bin/sm-leaf
sudo mkdir -p /opt/pfc/python/secure_messaging /opt/pfc/python/admin/pfc_admin /opt/pfc/python/lib
sudo rsync -a /tmp/secure_messaging/ /opt/pfc/python/secure_messaging/
sudo rsync -a /tmp/pfc_admin/ /opt/pfc/python/admin/pfc_admin/
sudo rsync -a /tmp/pfc_lib/ /opt/pfc/python/lib/
sudo cp /tmp/pfc-sm-leaf.service /etc/systemd/system/pfc-sm-leaf.service
sudo mkdir -p /etc/systemd/system/pfc-py-admin.service.d
sudo tee /etc/systemd/system/pfc-py-admin.service.d/sm.conf >/dev/null << 'UNIT'
[Service]
Environment=PYTHONPATH=/opt/pfc/python/lib:/opt/pfc/python/admin:/opt/pfc/python
Environment=PFC_SIGNED_CONFIG=/opt/pfc/etc/secure-messaging.signed.json
Environment=PFC_CONFIG_KEY_PEM=/opt/pfc/etc/sm-keys/config.ed25519.pem
UNIT
# lab keys on host only — never overwrite an existing private key
sudo mkdir -p /opt/pfc/etc/sm-keys
if [ ! -f /opt/pfc/etc/sm-keys/config.ed25519.pem ]; then
sudo python3 - << 'PY'
from pathlib import Path
import sys
sys.path.insert(0, "/opt/pfc/python")
from secure_messaging.signed_config import generate_signing_key, pem_private, pem_public, save_signed, sign
priv = generate_signing_key()
d = Path("/opt/pfc/etc/sm-keys")
d.mkdir(parents=True, exist_ok=True)
(d / "config.ed25519.pem").write_bytes(pem_private(priv))
(d / "config.ed25519.pem").chmod(0o600)
(d / "config.ed25519.pub.pem").write_bytes(pem_public(priv.public_key()))
(d / "config.ed25519.pub.pem").chmod(0o644)
payload = {
"crypto": {"mode": "lab-xor", "system_key_id": "lab-system"},
"routing": {"mode": "passthrough"},
"admin": {"history_cube": "admin-history"},
"logging": {"mode": "summary"},
}
save_signed(Path("/opt/pfc/etc/secure-messaging.signed.json"), sign(payload, priv))
print("signed config written")
PY
fi
sudo chmod 600 /opt/pfc/etc/sm-keys/config.ed25519.pem
sudo chmod 644 /opt/pfc/etc/sm-keys/config.ed25519.pub.pem /opt/pfc/etc/secure-messaging.signed.json || true
# seed admin-history if empty (prev empty, new = current signed file)
sudo python3 - << 'PY'
from pathlib import Path
import sys
sys.path.insert(0, "/opt/pfc/python")
from secure_messaging.admin_history import AdminHistory
hist = Path("/opt/pfc/data/admin/admin-history")
chain = hist / "blockchain" / "chain.jsonl"
new = Path("/opt/pfc/etc/secure-messaging.signed.json").read_text()
if not chain.exists() or not chain.read_text().strip():
row = AdminHistory(hist).append_change(actor="deploy-ns1", prev_text="", new_text=new)
print("admin-history seeded seq", row.get("seq"))
else:
print("admin-history exists lines", sum(1 for _ in chain.open()))
PY
if [ -d /tmp/system-git-sync ]; then
sudo mkdir -p /opt/pfc/docs/html/sync
sudo rsync -a /tmp/system-git-sync/*.MD /tmp/system-git-sync/*.md /opt/pfc/docs/html/sync/ 2>/dev/null || true
if [ -f /tmp/system-git-sync/docs/index.html ]; then
sudo cp /tmp/system-git-sync/docs/index.html /opt/pfc/docs/html/sync/index.html
fi
if [ -f /tmp/system-git-sync/docs/thesaurus.md ]; then
sudo cp /tmp/system-git-sync/docs/thesaurus.md /opt/pfc/docs/html/sync/thesaurus.md
fi
fi
if [ -d /tmp/nse-html ]; then
sudo rsync -a /tmp/nse-html/ /opt/pfc/docs/html/nats-service-endpoints/
fi
if [ -f /tmp/nse.pdf ]; then
sudo cp /tmp/nse.pdf /opt/pfc/docs/html/nats-service-endpoints/nats-service-endpoints.pdf
fi
sudo systemctl daemon-reload
sudo systemctl enable --now pfc-sm-leaf
sudo systemctl restart pfc-py-admin
sleep 2
systemctl is-active pfc-sm-leaf pfc-py-admin pfc-connector pfc-peergos-bridge
echo "--- health ---"
curl -sS -m 5 http://127.0.0.1:18783/health; echo
curl -sS -m 5 http://127.0.0.1:18780/health; echo
echo "--- config unauth ---"
curl -sS -m 5 -o /tmp/cfg.out -w "http=%{http_code}\n" http://127.0.0.1:18780/v1/admin/config
cat /tmp/cfg.out; echo
echo "--- signed load ---"
sudo python3 - << 'PY'
from pathlib import Path
import sys
sys.path.insert(0, "/opt/pfc/python")
from cryptography.hazmat.primitives.serialization import load_pem_private_key
from secure_messaging.signed_config import load_signed
priv = load_pem_private_key(Path("/opt/pfc/etc/sm-keys/config.ed25519.pem").read_bytes(), password=None)
payload = load_signed(Path("/opt/pfc/etc/secure-messaging.signed.json"), priv.public_key())
assert payload["crypto"]["mode"] in ("lab-xor", "npe", "plain-lab")
assert payload["routing"]["mode"] == "passthrough"
print("signed_ok", payload["crypto"], payload["routing"])
PY
# never enable NPE on the live console from this script
if systemctl show pfc-py-admin -p Environment | grep -q PFC_REQUIRE_NPE=1; then
echo "WARNING: PFC_REQUIRE_NPE=1 is set; fail-closed if sidecar unused"
fi
echo DEPLOY_HOST_OK
EOF
echo "=== public smoke ==="
curl -sS -m 10 -o /tmp/pfc.health -w "pfc_https=%{http_code}\n" https://pfc.georgelambert.org/health
curl -sS -m 10 -o /tmp/nse.html -w "docs_nse=%{http_code}\n" https://docs.pfc.georgelambert.org/nats-service-endpoints/index.html
curl -sS -m 10 -o /tmp/sync.html -w "docs_sync=%{http_code}\n" https://docs.pfc.georgelambert.org/sync/index.html
curl -sS -m 10 -o /tmp/hist.md -w "docs_hist=%{http_code}\n" https://docs.pfc.georgelambert.org/sync/Historical-Information.MD
echo "DEPLOY OK"

View file

@ -12,7 +12,7 @@ run() {
fi
}
export PATH="/opt/homebrew/bin:$PATH"
run env PYTHONPATH="$ROOT/peergos-for-compliance/lib:$ROOT/peergos-for-compliance-admin" \
run env PYTHONPATH="$ROOT/peergos-for-compliance/lib:$ROOT/peergos-for-compliance-admin:$ROOT/secure-messaging/python" \
python3 -m unittest discover -s "$ROOT/peergos-for-compliance/tests" -v
run env PYTHONPATH="$ROOT/secure-messaging/python" \
python3 -m unittest discover -s "$ROOT/secure-messaging/tests" -v
@ -20,7 +20,23 @@ if [[ -d "$ROOT/secure-messaging/go" ]]; then
( cd "$ROOT/secure-messaging/go" && run go test ./... )
fi
if [[ -f "$ROOT/nats-service-endpoints/catalog/endpoints.json" ]]; then
run python3 -c "import json; json.load(open('$ROOT/nats-service-endpoints/catalog/endpoints.json'))"
run python3 - << PY
import json
from pathlib import Path
p = Path("$ROOT/nats-service-endpoints/catalog/endpoints.json")
d = json.loads(p.read_text())
names = {e["name"] for e in d["endpoints"]}
need = {
"verae.sm.send",
"verae.sm.dead",
"verae.sm.error",
"verae.sm.log.summary",
"verae.admin.config.sign",
}
missing = sorted(need - names)
assert not missing, missing
print("catalog subjects ok", sorted(need))
PY
fi
if [[ "$fail" -ne 0 ]]; then
echo "REVIEW FAIL"

22
docs/index.html Normal file
View file

@ -0,0 +1,22 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>system-git-sync</title>
</head>
<body>
<h1>system-git-sync</h1>
<p>Hub for signed config, NATS naming, and ns1 deploy review. Not a HIPAA/SOC 2/ISO certificate.</p>
<ul>
<li><a href="Historical-Information.MD">Historical-Information.MD</a> (read this after a context reset)</li>
<li><a href="System-Git-Sync.MD">System-Git-Sync.MD</a></li>
<li><a href="TODO.md">TODO.md</a></li>
<li><a href="TODO-With-User.MD">TODO-With-User.MD</a></li>
<li><a href="UserReview.MD">UserReview.MD</a></li>
<li><a href="INVENTORY.md">INVENTORY.md</a></li>
<li><a href="thesaurus.md">thesaurus.md</a></li>
<li><a href="/nats-service-endpoints/index.html">NATS-Service-Endpoints catalog</a></li>
</ul>
<p>Git: <a href="https://git.georgelambert.org/marchon/system-git-sync">git.georgelambert.org/marchon/system-git-sync</a></p>
</body>
</html>

View file

@ -19,3 +19,5 @@ Canonical term → where it lives. Use this to rename later.
| Admin history cube | kind=admin-history JSONL | syslog of config files | admin_history.py |
| NPE sidecar | npe seal --to | HPKE in Python | npe_adapter.py |
| Review gate | cicd/GATE.md | auto-deploy to ns1 | system-git-sync |
| sm-leaf health | GET 127.0.0.1:18783/health | WAN NATS ports | sm-leaf |
| Config promote | GATE.md SHA table then deploy-ns1.sh | unsigned JSON copy | cicd/ |