marchon/system-git-sync
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

S00-S04: hub inventory, signed-config design, prompts saved

Browse source
This commit is contained in:
George Lambert 2026-09-15 22:17:15 -04:00
commit 257405ca61
9 changed files with 304 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
.DS_Store

36
INVENTORY.md Normal file
View file

@ -0,0 +1,36 @@
# Inventory — git.georgelambert.org
Forgejo **11.0.16**. Scanned with the `marchon` API token (not stored here).
## Orgs
| Org | Repos | Messaging relevance |
|---|---|---|
| `marchon` | 61 | PFC, NATS, access, zapier, ops — **in scope** |
| `infra` | `coredns`, `caddy` | Edge only; document Caddy docs hosts |
| `verae` | 0 | Empty; do not invent content |
| `websites` | 70+ | Marketing/site trees; **inventory only** this round |
## marchon — NATS / DataCube (must stay consistent)
- peergos-for-compliance, peergos-for-compliance-admin, peergos-for-compliance-ipfs, peergos-for-compliance-replication
- peergos-compliance-{plan,architecture,docker,proxmox,nix,kubernetes,ipfs-pinning,aws-archive,iceberg,docs,config,go}
- verae-nats-process, verae-nats-accounts, verae-nats-cluster
- verae-access-{authz,web,api,leaf,zapier,staff}
- verae-jobs-events, verae-ops, verae-deploy, verae-proxmox, verae-bootstrap
- verae-archive-worm, verae-archive-aggregator, verae-keep, verae-uptime, verae-backup
- verae-staff-{session,ui,iam}, verae-fleet, verae-tree-node, verae-middleware
- verae-request-splitter, verae-activate, verae-zapier-*
## marchon — other (link, do not rewrite)
pulsepolls, datacube-pulsepolls, carpathia, zapier docs/pricing/identity, overview, UI-Docs, dev-overview
## infra
- caddy — future docs Caddy site files
- coredns — DNS; no NATS subjects
## websites
Site HTML for datacubes.*, verae.*, veraetime.*, georgelambert.org, etc. No NATS endpoint catalog. Listed so the scan is complete; not forked this round.

13
LICENSE Normal file
View file

@ -0,0 +1,13 @@
Copyright 2026 Verae / George Lambert
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND.

19
README.md Normal file
View file

@ -0,0 +1,19 @@
# system-git-sync
Hub for keeping `git.georgelambert.org` NATS names, signed configuration, and
secure-messaging docs in sync.
* [System-Git-Sync.MD](System-Git-Sync.MD) — prompts, design table, progress
* [INVENTORY.md](INVENTORY.md) — every org scanned
* [UserReview.MD](UserReview.MD) — decisions
* [TODO-With-User.MD](TODO-With-User.MD) — guesses
* [TODO.md](TODO.md) — dependency order
Sibling repos:
* https://git.georgelambert.org/marchon/nats-service-endpoints
* https://git.georgelambert.org/marchon/secure-messaging
* https://git.georgelambert.org/marchon/peergos-for-compliance
* https://git.georgelambert.org/marchon/peergos-compliance-docs
Not a HIPAA/SOC 2/ISO certificate. ns1: document-only until CI review.

165
System-Git-Sync.MD Normal file
View file

@ -0,0 +1,165 @@
# System-Git-Sync
Orchestrator for consistent NATS naming, signed configuration, DataCube admin
history, and secure messaging across **every org** on
`git.georgelambert.org`.
This is **not** a HIPAA / SOC 2 / ISO certificate.
Live **ns1** rule: **document-only first**, then code, then tests, then a
**CI-CD review stack**, then deployment. Do not ship unsigned config or
unreviewed binaries to `70.88.205.138`.
---
## Saved operator prompt (round 1)
> There are mutiple parts to this request, part of it is architecture for
> updating secure messaging service, part of it is requsting the
> implementataion of additional git repos with code to implement this
> functionality with the standard architecture, documentation, usage guide,
> message tracing diagrams, message bundle diagrams and explainations, error
> messages, error handling, and system logging. Then I need you to review
> where this fits an needs to be embedded into the entire system, and then
> make the code changes, committing and documenting every one, changing the
> system documentation for that module if necessary and then please review,
> modify as needed and both link to other repos, and once updates are all
> changed and checked in, sync all of the documentation in all of the repos
> to make them correct and consistant. make sure that all of the repos are
> in sync, that the nats services have consistant naming, write a new git
> repo that has the NATS-Service-Endpoints and lists what those endpoints
> are for, what they do, who they expect messages from, what is the incoming
> format, who they send messages too, and what is that format, what are the
> rejection / security / filter paths / are incoming messages partially or
> fully encrypted, if messages need to be and if so are fully encrypted and
> the address is a passthrough make that clear, explain how encrypted
> messages have routes and endpoints where the embedded data except for the
> destination being encrypted, and the error handling blocks be encrypted
> with the system public key for error handling and logging to process any
> failed requests, and extract the return addresses with a public key
> encryption token for returning secure message content in a message body
> and any routing, or logging information in the message header as
> specifically necessary inclusing error codes - and dead letter (service
> request) failure, and an Network Error Bundle that is encoded with a
> System Public Key that encodes the sender and ability to respond with
> status messages, which will send an encrypted failure response to the
> sender that only they can read - and an attached system error bounce
> reportend bundle that is encrypted with that same system Public Key for
> error recovery or logging while never revealing the contents of the
> encrypted message by the intended recipient mailbox destination - not even
> by the sender once sent (the sender will have to use their lookup ID to
> identify the message source) and the same for the responder. in the event
> of system failures make sure that the summary of those logs are sent to a
> central logging system for review and correction. Please scan / close if
> easier all repos on git.georgelambert.org to review and update. keep
> itterating through the entire system logging every change and git checkin
> for each project so that we have a valid system change and improvement
> log. do not wait for any user input, if user decisions will need to be
> made, write the into a UserReview.MD which we will review and make
> decisions about and re-run if necessary. Save this prompt into the
> "System-Git-Sync.MD" file with your log of progress and suggested changes
> at the end and an internal todo list wich checks off todo items as
> completed, but appends additional todo items in their order of
> dependencies, and review the todo list from the top to make sure that all
> system dependencies are resolved in order. The code needs to be in both
> python for readability with full python line by line documentation, and go
> profects using NATS communications - where if paractical using Internal
> Nats Servers for performance and leaf nodes for connectivity. Note in the
> ToDo list the choices made. All of the code and reviews documentation
> should end up back on git.GeorgeLambert.org with rendered documentation in
> both HTML and LaTex PDF (feel free to render those on the server and add
> caddy endpoints as necessary) but make sure that all of the links in both
> the HTML and LaTeX pdf's are resolvable into the entire git.georgelambert.org
> repos - ideally with relative paths when practical. before you start,
> please explain what you think is the task set so that I can refine and
> review, and once you start, keep itterating over it until the project is
> complete - if you are blocked and need user input, create a TODO-With-User.MD
> file and do your best to work around it and keep going. If after a section,
> you want user input - for this round make your best guesses and append the
> Guesses and Assumptions that you make into the TODO-With-User.md file. are
> we clar that I want you to understand, identify, itterate, and work on this
> task until it is not only completed, but upon the first completion, you do
> it again to make sure everything is in sync, and create a thesaurus with
> links to ares in the documentation and code so that you can go back and
> fix both vocabulary, variable names, paths, desscriptive NATS addresses to
> make this entire set of repos totally in sync as much as possible. -
> Please Confirm your understanding.
## Saved operator prompt (round 2 — start now)
> review and update every org on git.georgelambert.org, Start this loop now,
> make the best system design choice, with information about how to select /
> chose it in the configuration files, and make sure that to use a
> configuration file it needs to be signed with a signed key wrapper to
> prevent changes and hacking that can be regenerated from one of the admin
> consoles after changes to the system configuration and the logging of both
> the previous file, the new file, folowed by the diff of the files in an
> append only history that uses a verae-datacube-history for admin changes.
> Live ns1: document-only first, then code, test, build a CI-CD stack for
> review and then deployment.
---
## Design choices (locked this round)
Recorded so later work does not silently reverse them. How to **select**
each choice is in signed config (`secure-messaging.json` inside the wrapper).
| Choice | Value | Why | Config key |
|---|---|---|---|
| Crypto production | NPE HPKE (ChaCha20-Poly1305) | Already the Verae production E2E; do not invent a second suite | `crypto.mode = npe` |
| Crypto lab | `pfc-lab-xor` or `plain-lab` | Lets tests run without NPE sidecar | `crypto.mode = lab-xor \| plain-lab` |
| Config signature | Ed25519 over canonical JSON | Stdlib in Go; `cryptography` in Python; no extra CA | `config.sig.alg = ed25519` |
| Config wrapper | `{payload, sig:{alg,key_id,signature}}` | Unsigned files are **rejected** | required |
| Admin history | Verae DataCube chain (JSONL) `kind=admin-history` | Matches existing cube append-only model | `admin.history_cube` |
| History payload | previous file, new file, unified diff, actor, hashes | Operator asked for prev + new + diff | chain `type=admin-config` |
| Routing | **passthrough**: destination in the clear; body encrypted | Brokers are untrusted | `routing.mode = passthrough` |
| After send | Sender cannot decrypt own ciphertext; **lookup_id** only | Operator rule | envelope `from_lookup_id` |
| System errors | Network Error Bundle + bounce report, sealed to **system public key** | Logging without opening mail | `crypto.system_key_id` |
| Sender failure notice | Sealed to **sender** public key only | Only they can read status | error header `ct_sender` |
| Dead letter | `verae.sm.dead` + bundle | Service-request failure | subject catalog |
| Logs | **summaries** only (codes, lookup_id, dest class) | No PHI / no body | `logging.mode = summary` |
| Languages | Python (readable spec) + Go (in-process NATS + leaf) | Operator rule | n/a |
| ns1 | docs first, then CI review, then deploy | Operator rule | n/a |
| Website org | inventory only this round | Not NATS endpoints | n/a |
---
## Internal TODO (dependency order)
Insert **new** dependencies **above** the blocked item.
- [x] **S00** Confirm orgs on git.georgelambert.org (`marchon`, `infra`, `websites`; `verae` empty)
- [x] **S01** Save prompts into this file
- [x] **S02** Inventory repos (see `INVENTORY.md`)
- [x] **S03** Lock design choices (table above)
- [x] **S04** Hub repo `system-git-sync`
- [x] **S05** `nats-service-endpoints` catalog schema + seed from PFC subjects
- [x] **S06** `secure-messaging` Python (signed config, passthrough, error bundle, admin history)
- [x] **S07** `secure-messaging` Go leaf (in-process NATS + hub leaf)
- [x] **S08** Tests + Forgejo Actions CI
- [x] **S09** Sphinx HTML + LaTeX PDF with git.georgelambert.org links
- [ ] **S10** Embed: link from peergos-compliance-docs, peergos-for-compliance subjects.json, verae-nats-bus (docs only on ns1)
- [ ] **S11** Thesaurus pass (vocabulary / subject names)
- [ ] **S12** Second full sync of docs across NATS-related repos
- [ ] **S13** CI-CD review pipeline docs; **no** unsigned deploy to ns1
- [ ] **S14** Wire NPE sidecar (blocked on `PFC_REQUIRE_NPE` live cutover — see UserReview)
---
## Progress log
Append-only. Newest at the bottom.
- 2026-09-16 — Started loop. Forgejo 11.0.16. Orgs: marchon (61 repos), infra (coredns, caddy), websites (70+ site trees), verae (no repos). Token used via osxkeychain; not stored in git.
- 2026-09-16 — Locked Ed25519 signed config + DataCube admin-history + passthrough routing + system-key error bundles.
- 2026-09-16 — Created hub + nats-service-endpoints + secure-messaging (Python+Go) locally; CI workflows; Sphinx.
- 2026-09-16 — Python 4/4 tests OK; Go leaf in-process test OK; endpoints PDF 17 pages. ns1: docs-only, no binary deploy.
---
## Suggested later changes
- Merge `verae.ts.batch` owner to timestamp leaf JWT in verae-nats-bus.
- Console UI: regenerate signed config + append admin-history cube (after S10).
- Caddy `sync.pfc.georgelambert.org` for this hub’s HTML (after CI review).

22
TODO-With-User.MD Normal file
View file

@ -0,0 +1,22 @@
# TODO-With-User
Blocks plus **Guesses and Assumptions**. Work continues around these.
## Guesses and assumptions
- **Orgs in scope for code:** `marchon` NATS/PFC repos. `infra` docs only. `websites` inventory only. `verae` org empty.
- **System key:** generate lab Ed25519 signing key + X25519 system box key; private files gitignored; public keys in signed config payload.
- **Passthrough:** NATS subject and `to` mailbox id are in the clear; ciphertext is HPKE (prod) or lab-xor (lab).
- **Lookup ID:** HMAC-SHA256 of sender identity with a per-message nonce; not reversible from the broker.
- **Error bundle:** two ciphertexts: `ct_sender` (NaCl/X25519 box to sender) and `ct_system` (box to system key) containing bounce metadata **without** recipient payload.
- **DataCube history:** JSONL chain compatible with `pfc.chain.Chain` so a real cube can ingest it; lab can run without Peergos FUSE.
- **CI:** Forgejo Actions (`ubuntu-latest`). If runners are missing, the YAML still documents the review gate.
- **ns1:** no binary deploy until a green CI run is recorded. Docs HTML may be copied to `/opt/pfc/docs/` as documentation-only.
- **Python comments:** every executable line in `secure_messaging/` has a trailing or preceding comment.
- **Go:** in-process `nats-server` leaf to `nats-a:7422` when `SM_LEAF_HUB` is set; otherwise in-memory only for tests.
## Open blocks (workaround in place)
- No HSM on the laptop — lab keys only.
- Central Verae timestamp leaf may be down — `local-dual` receipts already used by PFC; same here for error timestamps.
- Cannot rewrite 70 website repos meaningfully this round.

21
TODO.md Normal file
View file

@ -0,0 +1,21 @@
# TODO (dependency order)
Status: `[ ]` pending · `[x]` done (committed).
Insert **new** dependencies **above** the blocked item.
- [x] S00 Scan orgs
- [x] S01 Save prompts
- [x] S02 Inventory
- [x] S03 Lock signed-config + datacube-history + passthrough + error-bundle
- [x] S04 Hub repo
- [x] S05 NATS-Service-Endpoints catalog
- [x] S06 Python spec implementation
- [x] S07 Go leaf
- [x] S08 Tests + CI
- [x] S09 Sphinx HTML/PDF
- [ ] S10 Link PFC + compliance-docs (docs on ns1 only)
- [ ] S11 Thesaurus
- [ ] S12 Second sync pass
- [ ] S13 CI-CD review then deploy (blocked on S08 green + UserReview #2)
- [ ] S14 NPE sidecar live (blocked on UserReview #2)

10
UserReview.MD Normal file
View file

@ -0,0 +1,10 @@
# UserReview
Decisions to confirm later. Work continued with guesses in `TODO-With-User.MD`.
1. **System public key custody** — lab Ed25519/X25519 generated into `keys/system/` (gitignored private). Production should be customer HSM. Rotate via signed config + admin-history cube.
2. **Cut over `PFC_REQUIRE_NPE=1` on ns1** — not in this round (would fail-close the live console).
3. **Caddy hostname** for this hub (`sync.pfc.georgelambert.org` vs a path under `docs.pfc`). Guess: path `/sync/` under existing docs host after CI.
4. **verae org** is empty — leave empty unless you want it as the canonical NATS catalog owner instead of `marchon`.
5. **Website org** — no NATS changes. Confirm if any site should deep-link the endpoint catalog.
6. **Admin console that re-signs config** — PFC console vs peergos-compliance-config TUI. Guess: both may call the same Python `signed_config.sign`; PFC console first.

17
docs/thesaurus.md Normal file
View file

@ -0,0 +1,17 @@
# Thesaurus
Canonical term → where it lives. Use this to rename later.
| Term | Use this | Not this | Where |
|---|---|---|---|
| Untrusted broker | NATS | "secure bus" | nats-service-endpoints |
| Production E2E | NPE / HPKE | "TLS to NATS" | secure-messaging |
| Lab envelope | pfc-lab-xor | "xor cipher" alone | peergos-for-compliance |
| Dest in clear | passthrough | "plaintext message" | envelope.py |
| Sender handle | lookup_id | "from username on wire" | envelope.py |
| Config file | signed wrapper | raw JSON config | signed_config.py |
| Admin audit | DataCube admin-history | SQL log | admin_history.py |
| Failure to sender | ct_sender | "email bounce plaintext" | error_bundle.py |
| Ops bounce | ct_system / Network Error Bundle | "log the ciphertext body" | error_bundle.py |
| Inspect officers | share names | Google accounts | peergos-for-compliance |
| Console SSO | PFC TOTP/Google | Peergos Drive login | pfc_admin |