commit 257405ca616eebbd81451ab1f6c4aea3c9966bb8 Author: George Lambert Date: Tue Sep 15 22:17:15 2026 -0400 S00-S04: hub inventory, signed-config design, prompts saved diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e43b0f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/INVENTORY.md b/INVENTORY.md new file mode 100644 index 0000000..c97775a --- /dev/null +++ b/INVENTORY.md @@ -0,0 +1,36 @@ +# Inventory — git.georgelambert.org + +Forgejo **11.0.16**. Scanned with the `marchon` API token (not stored here). + +## Orgs + +| Org | Repos | Messaging relevance | +|---|---|---| +| `marchon` | 61 | PFC, NATS, access, zapier, ops — **in scope** | +| `infra` | `coredns`, `caddy` | Edge only; document Caddy docs hosts | +| `verae` | 0 | Empty; do not invent content | +| `websites` | 70+ | Marketing/site trees; **inventory only** this round | + +## marchon — NATS / DataCube (must stay consistent) + +- peergos-for-compliance, peergos-for-compliance-admin, peergos-for-compliance-ipfs, peergos-for-compliance-replication +- peergos-compliance-{plan,architecture,docker,proxmox,nix,kubernetes,ipfs-pinning,aws-archive,iceberg,docs,config,go} +- verae-nats-process, verae-nats-accounts, verae-nats-cluster +- verae-access-{authz,web,api,leaf,zapier,staff} +- verae-jobs-events, verae-ops, verae-deploy, verae-proxmox, verae-bootstrap +- verae-archive-worm, verae-archive-aggregator, verae-keep, verae-uptime, verae-backup +- verae-staff-{session,ui,iam}, verae-fleet, verae-tree-node, verae-middleware +- verae-request-splitter, verae-activate, verae-zapier-* + +## marchon — other (link, do not rewrite) + +pulsepolls, datacube-pulsepolls, carpathia, zapier docs/pricing/identity, overview, UI-Docs, dev-overview + +## infra + +- caddy — future docs Caddy site files +- coredns — DNS; no NATS subjects + +## websites + +Site HTML for datacubes.*, verae.*, veraetime.*, georgelambert.org, etc. No NATS endpoint catalog. Listed so the scan is complete; not forked this round. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..3668372 --- /dev/null +++ b/LICENSE @@ -0,0 +1,13 @@ +Copyright 2026 Verae / George Lambert + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. diff --git a/README.md b/README.md new file mode 100644 index 0000000..01d0456 --- /dev/null +++ b/README.md @@ -0,0 +1,19 @@ +# system-git-sync + +Hub for keeping `git.georgelambert.org` NATS names, signed configuration, and +secure-messaging docs in sync. + +* [System-Git-Sync.MD](System-Git-Sync.MD) — prompts, design table, progress +* [INVENTORY.md](INVENTORY.md) — every org scanned +* [UserReview.MD](UserReview.MD) — decisions +* [TODO-With-User.MD](TODO-With-User.MD) — guesses +* [TODO.md](TODO.md) — dependency order + +Sibling repos: + +* https://git.georgelambert.org/marchon/nats-service-endpoints +* https://git.georgelambert.org/marchon/secure-messaging +* https://git.georgelambert.org/marchon/peergos-for-compliance +* https://git.georgelambert.org/marchon/peergos-compliance-docs + +Not a HIPAA/SOC 2/ISO certificate. ns1: document-only until CI review. diff --git a/System-Git-Sync.MD b/System-Git-Sync.MD new file mode 100644 index 0000000..66ce663 --- /dev/null +++ b/System-Git-Sync.MD @@ -0,0 +1,165 @@ +# System-Git-Sync + +Orchestrator for consistent NATS naming, signed configuration, DataCube admin +history, and secure messaging across **every org** on +`git.georgelambert.org`. + +This is **not** a HIPAA / SOC 2 / ISO certificate. + +Live **ns1** rule: **document-only first**, then code, then tests, then a +**CI-CD review stack**, then deployment. Do not ship unsigned config or +unreviewed binaries to `70.88.205.138`. + +--- + +## Saved operator prompt (round 1) + +> There are mutiple parts to this request, part of it is architecture for +> updating secure messaging service, part of it is requsting the +> implementataion of additional git repos with code to implement this +> functionality with the standard architecture, documentation, usage guide, +> message tracing diagrams, message bundle diagrams and explainations, error +> messages, error handling, and system logging. Then I need you to review +> where this fits an needs to be embedded into the entire system, and then +> make the code changes, committing and documenting every one, changing the +> system documentation for that module if necessary and then please review, +> modify as needed and both link to other repos, and once updates are all +> changed and checked in, sync all of the documentation in all of the repos +> to make them correct and consistant. make sure that all of the repos are +> in sync, that the nats services have consistant naming, write a new git +> repo that has the NATS-Service-Endpoints and lists what those endpoints +> are for, what they do, who they expect messages from, what is the incoming +> format, who they send messages too, and what is that format, what are the +> rejection / security / filter paths / are incoming messages partially or +> fully encrypted, if messages need to be and if so are fully encrypted and +> the address is a passthrough make that clear, explain how encrypted +> messages have routes and endpoints where the embedded data except for the +> destination being encrypted, and the error handling blocks be encrypted +> with the system public key for error handling and logging to process any +> failed requests, and extract the return addresses with a public key +> encryption token for returning secure message content in a message body +> and any routing, or logging information in the message header as +> specifically necessary inclusing error codes - and dead letter (service +> request) failure, and an Network Error Bundle that is encoded with a +> System Public Key that encodes the sender and ability to respond with +> status messages, which will send an encrypted failure response to the +> sender that only they can read - and an attached system error bounce +> reportend bundle that is encrypted with that same system Public Key for +> error recovery or logging while never revealing the contents of the +> encrypted message by the intended recipient mailbox destination - not even +> by the sender once sent (the sender will have to use their lookup ID to +> identify the message source) and the same for the responder. in the event +> of system failures make sure that the summary of those logs are sent to a +> central logging system for review and correction. Please scan / close if +> easier all repos on git.georgelambert.org to review and update. keep +> itterating through the entire system logging every change and git checkin +> for each project so that we have a valid system change and improvement +> log. do not wait for any user input, if user decisions will need to be +> made, write the into a UserReview.MD which we will review and make +> decisions about and re-run if necessary. Save this prompt into the +> "System-Git-Sync.MD" file with your log of progress and suggested changes +> at the end and an internal todo list wich checks off todo items as +> completed, but appends additional todo items in their order of +> dependencies, and review the todo list from the top to make sure that all +> system dependencies are resolved in order. The code needs to be in both +> python for readability with full python line by line documentation, and go +> profects using NATS communications - where if paractical using Internal +> Nats Servers for performance and leaf nodes for connectivity. Note in the +> ToDo list the choices made. All of the code and reviews documentation +> should end up back on git.GeorgeLambert.org with rendered documentation in +> both HTML and LaTex PDF (feel free to render those on the server and add +> caddy endpoints as necessary) but make sure that all of the links in both +> the HTML and LaTeX pdf's are resolvable into the entire git.georgelambert.org +> repos - ideally with relative paths when practical. before you start, +> please explain what you think is the task set so that I can refine and +> review, and once you start, keep itterating over it until the project is +> complete - if you are blocked and need user input, create a TODO-With-User.MD +> file and do your best to work around it and keep going. If after a section, +> you want user input - for this round make your best guesses and append the +> Guesses and Assumptions that you make into the TODO-With-User.md file. are +> we clar that I want you to understand, identify, itterate, and work on this +> task until it is not only completed, but upon the first completion, you do +> it again to make sure everything is in sync, and create a thesaurus with +> links to ares in the documentation and code so that you can go back and +> fix both vocabulary, variable names, paths, desscriptive NATS addresses to +> make this entire set of repos totally in sync as much as possible. - +> Please Confirm your understanding. + +## Saved operator prompt (round 2 — start now) + +> review and update every org on git.georgelambert.org, Start this loop now, +> make the best system design choice, with information about how to select / +> chose it in the configuration files, and make sure that to use a +> configuration file it needs to be signed with a signed key wrapper to +> prevent changes and hacking that can be regenerated from one of the admin +> consoles after changes to the system configuration and the logging of both +> the previous file, the new file, folowed by the diff of the files in an +> append only history that uses a verae-datacube-history for admin changes. +> Live ns1: document-only first, then code, test, build a CI-CD stack for +> review and then deployment. + +--- + +## Design choices (locked this round) + +Recorded so later work does not silently reverse them. How to **select** +each choice is in signed config (`secure-messaging.json` inside the wrapper). + +| Choice | Value | Why | Config key | +|---|---|---|---| +| Crypto production | NPE HPKE (ChaCha20-Poly1305) | Already the Verae production E2E; do not invent a second suite | `crypto.mode = npe` | +| Crypto lab | `pfc-lab-xor` or `plain-lab` | Lets tests run without NPE sidecar | `crypto.mode = lab-xor \| plain-lab` | +| Config signature | Ed25519 over canonical JSON | Stdlib in Go; `cryptography` in Python; no extra CA | `config.sig.alg = ed25519` | +| Config wrapper | `{payload, sig:{alg,key_id,signature}}` | Unsigned files are **rejected** | required | +| Admin history | Verae DataCube chain (JSONL) `kind=admin-history` | Matches existing cube append-only model | `admin.history_cube` | +| History payload | previous file, new file, unified diff, actor, hashes | Operator asked for prev + new + diff | chain `type=admin-config` | +| Routing | **passthrough**: destination in the clear; body encrypted | Brokers are untrusted | `routing.mode = passthrough` | +| After send | Sender cannot decrypt own ciphertext; **lookup_id** only | Operator rule | envelope `from_lookup_id` | +| System errors | Network Error Bundle + bounce report, sealed to **system public key** | Logging without opening mail | `crypto.system_key_id` | +| Sender failure notice | Sealed to **sender** public key only | Only they can read status | error header `ct_sender` | +| Dead letter | `verae.sm.dead` + bundle | Service-request failure | subject catalog | +| Logs | **summaries** only (codes, lookup_id, dest class) | No PHI / no body | `logging.mode = summary` | +| Languages | Python (readable spec) + Go (in-process NATS + leaf) | Operator rule | n/a | +| ns1 | docs first, then CI review, then deploy | Operator rule | n/a | +| Website org | inventory only this round | Not NATS endpoints | n/a | + +--- + +## Internal TODO (dependency order) + +Insert **new** dependencies **above** the blocked item. + +- [x] **S00** Confirm orgs on git.georgelambert.org (`marchon`, `infra`, `websites`; `verae` empty) +- [x] **S01** Save prompts into this file +- [x] **S02** Inventory repos (see `INVENTORY.md`) +- [x] **S03** Lock design choices (table above) +- [x] **S04** Hub repo `system-git-sync` +- [x] **S05** `nats-service-endpoints` catalog schema + seed from PFC subjects +- [x] **S06** `secure-messaging` Python (signed config, passthrough, error bundle, admin history) +- [x] **S07** `secure-messaging` Go leaf (in-process NATS + hub leaf) +- [x] **S08** Tests + Forgejo Actions CI +- [x] **S09** Sphinx HTML + LaTeX PDF with git.georgelambert.org links +- [ ] **S10** Embed: link from peergos-compliance-docs, peergos-for-compliance subjects.json, verae-nats-bus (docs only on ns1) +- [ ] **S11** Thesaurus pass (vocabulary / subject names) +- [ ] **S12** Second full sync of docs across NATS-related repos +- [ ] **S13** CI-CD review pipeline docs; **no** unsigned deploy to ns1 +- [ ] **S14** Wire NPE sidecar (blocked on `PFC_REQUIRE_NPE` live cutover — see UserReview) + +--- + +## Progress log + +Append-only. Newest at the bottom. + +- 2026-09-16 — Started loop. Forgejo 11.0.16. Orgs: marchon (61 repos), infra (coredns, caddy), websites (70+ site trees), verae (no repos). Token used via osxkeychain; not stored in git. +- 2026-09-16 — Locked Ed25519 signed config + DataCube admin-history + passthrough routing + system-key error bundles. +- 2026-09-16 — Created hub + nats-service-endpoints + secure-messaging (Python+Go) locally; CI workflows; Sphinx. +- 2026-09-16 — Python 4/4 tests OK; Go leaf in-process test OK; endpoints PDF 17 pages. ns1: docs-only, no binary deploy. + +--- + +## Suggested later changes + +- Merge `verae.ts.batch` owner to timestamp leaf JWT in verae-nats-bus. +- Console UI: regenerate signed config + append admin-history cube (after S10). +- Caddy `sync.pfc.georgelambert.org` for this hub’s HTML (after CI review). diff --git a/TODO-With-User.MD b/TODO-With-User.MD new file mode 100644 index 0000000..46951ba --- /dev/null +++ b/TODO-With-User.MD @@ -0,0 +1,22 @@ +# TODO-With-User + +Blocks plus **Guesses and Assumptions**. Work continues around these. + +## Guesses and assumptions + +- **Orgs in scope for code:** `marchon` NATS/PFC repos. `infra` docs only. `websites` inventory only. `verae` org empty. +- **System key:** generate lab Ed25519 signing key + X25519 system box key; private files gitignored; public keys in signed config payload. +- **Passthrough:** NATS subject and `to` mailbox id are in the clear; ciphertext is HPKE (prod) or lab-xor (lab). +- **Lookup ID:** HMAC-SHA256 of sender identity with a per-message nonce; not reversible from the broker. +- **Error bundle:** two ciphertexts: `ct_sender` (NaCl/X25519 box to sender) and `ct_system` (box to system key) containing bounce metadata **without** recipient payload. +- **DataCube history:** JSONL chain compatible with `pfc.chain.Chain` so a real cube can ingest it; lab can run without Peergos FUSE. +- **CI:** Forgejo Actions (`ubuntu-latest`). If runners are missing, the YAML still documents the review gate. +- **ns1:** no binary deploy until a green CI run is recorded. Docs HTML may be copied to `/opt/pfc/docs/` as documentation-only. +- **Python comments:** every executable line in `secure_messaging/` has a trailing or preceding comment. +- **Go:** in-process `nats-server` leaf to `nats-a:7422` when `SM_LEAF_HUB` is set; otherwise in-memory only for tests. + +## Open blocks (workaround in place) + +- No HSM on the laptop — lab keys only. +- Central Verae timestamp leaf may be down — `local-dual` receipts already used by PFC; same here for error timestamps. +- Cannot rewrite 70 website repos meaningfully this round. diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..19431d0 --- /dev/null +++ b/TODO.md @@ -0,0 +1,21 @@ +# TODO (dependency order) + +Status: `[ ]` pending · `[x]` done (committed). + +Insert **new** dependencies **above** the blocked item. + +- [x] S00 Scan orgs +- [x] S01 Save prompts +- [x] S02 Inventory +- [x] S03 Lock signed-config + datacube-history + passthrough + error-bundle +- [x] S04 Hub repo +- [x] S05 NATS-Service-Endpoints catalog +- [x] S06 Python spec implementation +- [x] S07 Go leaf +- [x] S08 Tests + CI +- [x] S09 Sphinx HTML/PDF +- [ ] S10 Link PFC + compliance-docs (docs on ns1 only) +- [ ] S11 Thesaurus +- [ ] S12 Second sync pass +- [ ] S13 CI-CD review then deploy (blocked on S08 green + UserReview #2) +- [ ] S14 NPE sidecar live (blocked on UserReview #2) diff --git a/UserReview.MD b/UserReview.MD new file mode 100644 index 0000000..467196d --- /dev/null +++ b/UserReview.MD @@ -0,0 +1,10 @@ +# UserReview + +Decisions to confirm later. Work continued with guesses in `TODO-With-User.MD`. + +1. **System public key custody** — lab Ed25519/X25519 generated into `keys/system/` (gitignored private). Production should be customer HSM. Rotate via signed config + admin-history cube. +2. **Cut over `PFC_REQUIRE_NPE=1` on ns1** — not in this round (would fail-close the live console). +3. **Caddy hostname** for this hub (`sync.pfc.georgelambert.org` vs a path under `docs.pfc`). Guess: path `/sync/` under existing docs host after CI. +4. **verae org** is empty — leave empty unless you want it as the canonical NATS catalog owner instead of `marchon`. +5. **Website org** — no NATS changes. Confirm if any site should deep-link the endpoint catalog. +6. **Admin console that re-signs config** — PFC console vs peergos-compliance-config TUI. Guess: both may call the same Python `signed_config.sign`; PFC console first. diff --git a/docs/thesaurus.md b/docs/thesaurus.md new file mode 100644 index 0000000..0b0b980 --- /dev/null +++ b/docs/thesaurus.md @@ -0,0 +1,17 @@ +# Thesaurus + +Canonical term → where it lives. Use this to rename later. + +| Term | Use this | Not this | Where | +|---|---|---|---| +| Untrusted broker | NATS | "secure bus" | nats-service-endpoints | +| Production E2E | NPE / HPKE | "TLS to NATS" | secure-messaging | +| Lab envelope | pfc-lab-xor | "xor cipher" alone | peergos-for-compliance | +| Dest in clear | passthrough | "plaintext message" | envelope.py | +| Sender handle | lookup_id | "from username on wire" | envelope.py | +| Config file | signed wrapper | raw JSON config | signed_config.py | +| Admin audit | DataCube admin-history | SQL log | admin_history.py | +| Failure to sender | ct_sender | "email bounce plaintext" | error_bundle.py | +| Ops bounce | ct_system / Network Error Bundle | "log the ciphertext body" | error_bundle.py | +| Inspect officers | share names | Google accounts | peergos-for-compliance | +| Console SSO | PFC TOTP/Google | Peergos Drive login | pfc_admin |