system-git-sync/System-Git-Sync.MD

# System-Git-Sync

Orchestrator for consistent NATS naming, signed configuration, DataCube admin
history, and secure messaging across **every org** on
`git.georgelambert.org`.

This is **not** a HIPAA / SOC 2 / ISO certificate.

Live **ns1** rule: **document-only first**, then code, then tests, then a
**CI-CD review stack**, then deployment. Do not ship unsigned config or
unreviewed binaries to `70.88.205.138`.

---

## Saved operator prompt (round 1)

> There are mutiple parts to this request, part of it is architecture for
> updating secure messaging service, part of it is requsting the
> implementataion of additional git repos with code to implement this
> functionality with the standard architecture, documentation, usage guide,
> message tracing diagrams, message bundle diagrams and explainations, error
> messages, error handling, and system logging. Then I need you to review
> where this fits an needs to be embedded into the entire system, and then
> make the code changes, committing and documenting every one, changing the
> system documentation for that module if necessary and then please review,
> modify as needed and both link to other repos, and once updates are all
> changed and checked in, sync all of the documentation in all of the repos
> to make them correct and consistant. make sure that all of the repos are
> in sync, that the nats services have consistant naming, write a new git
> repo that has the NATS-Service-Endpoints and lists what those endpoints
> are for, what they do, who they expect messages from, what is the incoming
> format, who they send messages too, and what is that format, what are the
> rejection / security / filter paths / are incoming messages partially or
> fully encrypted, if messages need to be and if so are fully encrypted and
> the address is a passthrough make that clear, explain how encrypted
> messages have routes and endpoints where the embedded data except for the
> destination being encrypted, and the error handling blocks be encrypted
> with the system public key for error handling and logging to process any
> failed requests, and extract the return addresses with a public key
> encryption token for returning secure message content in a message body
> and any routing, or logging information in the message header as
> specifically necessary inclusing error codes - and dead letter (service
> request) failure, and an Network Error Bundle that is encoded with a
> System Public Key that encodes the sender and ability to respond with
> status messages, which will send an encrypted failure response to the
> sender that only they can read - and an attached system error bounce
> reportend bundle that is encrypted with that same system Public Key for
> error recovery or logging while never revealing the contents of the
> encrypted message by the intended recipient mailbox destination - not even
> by the sender once sent (the sender will have to use their lookup ID to
> identify the message source) and the same for the responder. in the event
> of system failures make sure that the summary of those logs are sent to a
> central logging system for review and correction. Please scan / close if
> easier all repos on git.georgelambert.org to review and update. keep
> itterating through the entire system logging every change and git checkin
> for each project so that we have a valid system change and improvement
> log. do not wait for any user input, if user decisions will need to be
> made, write the into a UserReview.MD which we will review and make
> decisions about and re-run if necessary. Save this prompt into the
> "System-Git-Sync.MD" file with your log of progress and suggested changes
> at the end and an internal todo list wich checks off todo items as
> completed, but appends additional todo items in their order of
> dependencies, and review the todo list from the top to make sure that all
> system dependencies are resolved in order. The code needs to be in both
> python for readability with full python line by line documentation, and go
> profects using NATS communications - where if paractical using Internal
> Nats Servers for performance and leaf nodes for connectivity. Note in the
> ToDo list the choices made. All of the code and reviews documentation
> should end up back on git.GeorgeLambert.org with rendered documentation in
> both HTML and LaTex PDF (feel free to render those on the server and add
> caddy endpoints as necessary) but make sure that all of the links in both
> the HTML and LaTeX pdf's are resolvable into the entire git.georgelambert.org
> repos - ideally with relative paths when practical. before you start,
> please explain what you think is the task set so that I can refine and
> review, and once you start, keep itterating over it until the project is
> complete - if you are blocked and need user input, create a TODO-With-User.MD
> file and do your best to work around it and keep going. If after a section,
> you want user input - for this round make your best guesses and append the
> Guesses and Assumptions that you make into the TODO-With-User.md file. are
> we clar that I want you to understand, identify, itterate, and work on this
> task until it is not only completed, but upon the first completion, you do
> it again to make sure everything is in sync, and create a thesaurus with
> links to ares in the documentation and code so that you can go back and
> fix both vocabulary, variable names, paths, desscriptive NATS addresses to
> make this entire set of repos totally in sync as much as possible. -
> Please Confirm your understanding.

## Saved operator prompt (round 2 — start now)

> review and update every org on git.georgelambert.org, Start this loop now,
> make the best system design choice, with information about how to select /
> chose it in the configuration files, and make sure that to use a
> configuration file it needs to be signed with a signed key wrapper to
> prevent changes and hacking that can be regenerated from one of the admin
> consoles after changes to the system configuration and the logging of both
> the previous file, the new file, folowed by the diff of the files in an
> append only history that uses a verae-datacube-history for admin changes.
> Live ns1: document-only first, then code, test, build a CI-CD stack for
> review and then deployment.

---

## Design choices (locked this round)

Recorded so later work does not silently reverse them. How to **select**
each choice is in signed config (`secure-messaging.json` inside the wrapper).

| Choice | Value | Why | Config key |
|---|---|---|---|
| Crypto production | NPE HPKE (ChaCha20-Poly1305) | Already the Verae production E2E; do not invent a second suite | `crypto.mode = npe` |
| Crypto lab | `pfc-lab-xor` or `plain-lab` | Lets tests run without NPE sidecar | `crypto.mode = lab-xor \| plain-lab` |
| Config signature | Ed25519 over canonical JSON | Stdlib in Go; `cryptography` in Python; no extra CA | `config.sig.alg = ed25519` |
| Config wrapper | `{payload, sig:{alg,key_id,signature}}` | Unsigned files are **rejected** | required |
| Admin history | Verae DataCube chain (JSONL) `kind=admin-history` | Matches existing cube append-only model | `admin.history_cube` |
| History payload | previous file, new file, unified diff, actor, hashes | Operator asked for prev + new + diff | chain `type=admin-config` |
| Routing | **passthrough**: destination in the clear; body encrypted | Brokers are untrusted | `routing.mode = passthrough` |
| After send | Sender cannot decrypt own ciphertext; **lookup_id** only | Operator rule | envelope `from_lookup_id` |
| System errors | Network Error Bundle + bounce report, sealed to **system public key** | Logging without opening mail | `crypto.system_key_id` |
| Sender failure notice | Sealed to **sender** public key only | Only they can read status | error header `ct_sender` |
| Dead letter | `verae.sm.dead` + bundle | Service-request failure | subject catalog |
| Logs | **summaries** only (codes, lookup_id, dest class) | No PHI / no body | `logging.mode = summary` |
| Languages | Python (readable spec) + Go (in-process NATS + leaf) | Operator rule | n/a |
| ns1 | docs first, then CI review, then deploy | Operator rule | n/a |
| Website org | inventory only this round | Not NATS endpoints | n/a |

---

## Internal TODO (dependency order)

Insert **new** dependencies **above** the blocked item.

- [x] **S00** Confirm orgs on git.georgelambert.org (`marchon`, `infra`, `websites`; `verae` empty)
- [x] **S01** Save prompts into this file
- [x] **S02** Inventory repos (see `INVENTORY.md`)
- [x] **S03** Lock design choices (table above)
- [x] **S04** Hub repo `system-git-sync`
- [x] **S05** `nats-service-endpoints` catalog schema + seed from PFC subjects
- [x] **S06** `secure-messaging` Python (signed config, passthrough, error bundle, admin history)
- [x] **S07** `secure-messaging` Go leaf (in-process NATS + hub leaf)
- [x] **S08** Tests + Forgejo Actions CI
- [x] **S09** Sphinx HTML + LaTeX PDF with git.georgelambert.org links
- [x] **S10** Embed: link from peergos-compliance-docs, peergos-for-compliance subjects.json
- [x] **S11** Thesaurus pass (vocabulary / subject names)
- [x] **S12** Second full sync of docs across NATS-related READMEs
- [x] **S13** CI-CD review pipeline; **no** unsigned deploy to ns1
- [x] **S14** NPE adapter fail-closed (live cutover still UserReview)
- [x] **S15** Promote SHA + ns1 sm-leaf (NPE live cutover remains UserReview)

---

## Progress log

Append-only. Newest at the bottom.

- 2026-09-16 — Started loop. Forgejo 11.0.16. Orgs: marchon (61 repos), infra (coredns, caddy), websites (70+ site trees), verae (no repos). Token used via osxkeychain; not stored in git.
- 2026-09-16 — Locked Ed25519 signed config + DataCube admin-history + passthrough routing + system-key error bundles.
- 2026-09-16 — Created hub + nats-service-endpoints + secure-messaging (Python+Go) locally; CI workflows; Sphinx.
- 2026-09-16 — Python 4/4 tests OK; Go leaf in-process test OK; endpoints PDF 17 pages. ns1: docs-only, no binary deploy.
- 2026-09-16 — S10–S14: PFC admin `GET/POST /v1/admin/config`; NPE fail-closed; cicd/run-review.sh **REVIEW PASS (no deploy)**; Forgejo CI on pfc, admin, docs, endpoints, hub.
- 2026-09-16 — S15: `cicd/deploy-ns1.sh` (tests required), sm-leaf health + `verae.sm.*`, signed lab-xor config + admin-history seed, docs `/sync/` and `/nats-service-endpoints/`. `PFC_REQUIRE_NPE=1` still unset. Websites org untouched.
- 2026-09-16 — S15 closed on ns1: units active, signed_ok, history seq 0, public pfc/docs 200. Replication 503 remains UserReview.
- 2026-09-16 — pfc-repl loopback HTTP health on :18784; public `/health` backends ipfs+replication both 200.

---

## Suggested later changes

- Merge `verae.ts.batch` owner to timestamp leaf JWT in verae-nats-bus.
- Console UI: regenerate signed config + append admin-history cube (after S10).
- Caddy `sync.pfc.georgelambert.org` for this hub’s HTML (after CI review).