George Lambert
b4150c8250
Compose-ready workspace: packages/zappier (rate card, portal, Stripe), packages/verae-zapier-middleware (timestamp + NATS), packages/verae-zapier (CLI app), vendor/zapier-platform, and research/zapier vendor corpus. Gate 0 structure checks pass. Product code and research are not yet wired.
80 lines
2.9 KiB
Markdown
80 lines
2.9 KiB
Markdown
# Zapier vendor dataset on NS1 MongoDB
|
|
|
|
Canonical store: **MongoDB 7 in Docker on `70.88.205.138` (NS1)**, bound to **127.0.0.1:27017 only**.
|
|
|
|
Database: `zapier`
|
|
Collections: `apps` (9,986), `templates` (332,441), `help_articles` (1,272), **`platform_reference` (1,489 — CLI, `z.*` functions, schema, official docs, example apps)**
|
|
Meta: `meta.ingest`
|
|
|
|
## Connect (always through SSH)
|
|
|
|
```bash
|
|
# keep this running
|
|
./scripts/mongo-tunnel.sh
|
|
# or: ssh -N -L 27017:127.0.0.1:27017 ns1
|
|
```
|
|
|
|
Then on this Mac:
|
|
|
|
| Tool | Connection |
|
|
|------|------------|
|
|
| mongosh | `source ~/.mcp-env && mongosh "$ZAPIER_MONGO_URI"` |
|
|
| Compass / Cursor MongoDB plugin | `mongodb://127.0.0.1:27017` after the tunnel is up (auth: user `zapier`, authSource `admin`) |
|
|
| Grok MCP (`mongodb` plugin) | `MDB_MCP_CONNECTION_STRING` in `~/.mcp-env` and `~/.grok/config.toml` |
|
|
| Python | `.venv/bin/python scripts/load-mongodb.py` |
|
|
|
|
Do **not** point Compass at `mongodb://70.88.205.138:27017`. Port 27017 is not on the public interface.
|
|
|
|
Credentials live in `~/.mcp-env` (mode 600) and `~/.config/zapier-mongo/root.pass` on NS1. Not in git.
|
|
|
|
## Reload after a scrape
|
|
|
|
```bash
|
|
./scripts/mongo-tunnel.sh &
|
|
source ~/.mcp-env
|
|
.venv/bin/python scripts/load-mongodb.py
|
|
```
|
|
|
|
## Installed Grok / editor tools
|
|
|
|
- Grok plugins: **mongodb**, **mongodb-atlas** (xAI Official / MongoDB agent-skills)
|
|
- Grok MCP server: `mongodb-mcp-server@3` via npx
|
|
- Cursor + VS Code: **mongodb.mongodb-vscode**
|
|
- Local CLI: **mongosh** 2.9.2
|
|
- NS1: Docker container `zapier-mongo` (`mongo:7`), restart unless-stopped
|
|
|
|
## Heavy jobs run on NS1
|
|
|
|
The scrape/load workspace on the server is `/home/marchon/zapier-research` (home is on SSD3).
|
|
|
|
```bash
|
|
ssh ns1
|
|
cd /home/marchon/zapier-research
|
|
# resume scrapes, then:
|
|
export ZAPIER_MONGO_URI="mongodb://zapier:$(cat ~/.config/zapier-mongo/root.pass)@127.0.0.1:27017/zapier?authSource=admin"
|
|
.venv/bin/python scripts/load-mongodb.py
|
|
```
|
|
|
|
Laptop is for Compass/Grok/tunnel only. Do not run the 10k-site scrapes locally.
|
|
|
|
|
|
## Collections
|
|
|
|
| Collection | Documents | Contents |
|
|
|------------|----------:|----------|
|
|
| `apps` | 9,986 | Identity, contacts, Zapier controls, commercial flags, featured templates, overview |
|
|
| `templates` | 332,441 | Full public Zap recipe catalog (title, URL, app pair) |
|
|
| `help_articles` | 1,272 | Zapier help center (get-started, common problems, platform FAQ) |
|
|
| `platform_reference` | ~1,777 | Toolkit + functions + **schema_json**, **api_function** (55), **platform_news** (49), **template** (32). Start: `findOne({kind:"guide", key:"build-new-connector"})`. |
|
|
| `meta` | 1 | Last ingest timestamp |
|
|
|
|
Reload the developer collection on NS1 after recloning repos:
|
|
|
|
```bash
|
|
ssh ns1
|
|
cd /home/marchon/zapier-research
|
|
./scripts/clone-zapier-repos.sh
|
|
export ZAPIER_RESEARCH_ROOT=/home/marchon/zapier-research
|
|
export ZAPIER_MONGO_URI="mongodb://zapier:$(cat ~/.config/zapier-mongo/root.pass)@127.0.0.1:27017/zapier?authSource=admin"
|
|
.venv/bin/python scripts/ingest-platform-reference.py
|
|
```
|