George Lambert b4150c8250 Milestone 0: import zappier billing, Verae middleware, and Zapier research

Compose-ready workspace: packages/zappier (rate card, portal, Stripe),
packages/verae-zapier-middleware (timestamp + NATS), packages/verae-zapier
(CLI app), vendor/zapier-platform, and research/zapier vendor corpus.

Gate 0 structure checks pass. Product code and research are not yet wired.

2026-09-09 02:37:36 -04:00

2.9 KiB

Raw Blame History

Zapier vendor dataset on NS1 MongoDB

Canonical store: MongoDB 7 in Docker on 70.88.205.138 (NS1), bound to 127.0.0.1:27017 only.

Database: zapier
Collections: apps (9,986), templates (332,441), help_articles (1,272), platform_reference (1,489 — CLI, z.* functions, schema, official docs, example apps)
Meta: meta.ingest

Connect (always through SSH)

# keep this running
./scripts/mongo-tunnel.sh
# or: ssh -N -L 27017:127.0.0.1:27017 ns1

Then on this Mac:

Tool	Connection
mongosh	`source ~/.mcp-env && mongosh "$ZAPIER_MONGO_URI"`
Compass / Cursor MongoDB plugin	`mongodb://127.0.0.1:27017` after the tunnel is up (auth: user `zapier`, authSource `admin`)
Grok MCP (`mongodb` plugin)	`MDB_MCP_CONNECTION_STRING` in `~/.mcp-env` and `~/.grok/config.toml`
Python	`.venv/bin/python scripts/load-mongodb.py`

Do not point Compass at mongodb://70.88.205.138:27017. Port 27017 is not on the public interface.

Credentials live in ~/.mcp-env (mode 600) and ~/.config/zapier-mongo/root.pass on NS1. Not in git.

Reload after a scrape

./scripts/mongo-tunnel.sh &
source ~/.mcp-env
.venv/bin/python scripts/load-mongodb.py

Installed Grok / editor tools

Grok plugins: mongodb, mongodb-atlas (xAI Official / MongoDB agent-skills)
Grok MCP server: mongodb-mcp-server@3 via npx
Cursor + VS Code: mongodb.mongodb-vscode
Local CLI: mongosh 2.9.2
NS1: Docker container zapier-mongo (mongo:7), restart unless-stopped

Heavy jobs run on NS1

The scrape/load workspace on the server is /home/marchon/zapier-research (home is on SSD3).

ssh ns1
cd /home/marchon/zapier-research
# resume scrapes, then:
export ZAPIER_MONGO_URI="mongodb://zapier:$(cat ~/.config/zapier-mongo/root.pass)@127.0.0.1:27017/zapier?authSource=admin"
.venv/bin/python scripts/load-mongodb.py

Laptop is for Compass/Grok/tunnel only. Do not run the 10k-site scrapes locally.

Collections

Collection	Documents	Contents
`apps`	9,986	Identity, contacts, Zapier controls, commercial flags, featured templates, overview
`templates`	332,441	Full public Zap recipe catalog (title, URL, app pair)
`help_articles`	1,272	Zapier help center (get-started, common problems, platform FAQ)
`platform_reference`	~1,777	Toolkit + functions + schema_json, api_function (55), platform_news (49), template (32). Start: `findOne({kind:"guide", key:"build-new-connector"})`.
`meta`	1	Last ingest timestamp

Reload the developer collection on NS1 after recloning repos:

ssh ns1
cd /home/marchon/zapier-research
./scripts/clone-zapier-repos.sh
export ZAPIER_RESEARCH_ROOT=/home/marchon/zapier-research
export ZAPIER_MONGO_URI="mongodb://zapier:$(cat ~/.config/zapier-mongo/root.pass)@127.0.0.1:27017/zapier?authSource=admin"
.venv/bin/python scripts/ingest-platform-reference.py

2.9 KiB Raw Blame History