Milestone 0: import zappier billing, Verae middleware, and Zapier research

Compose-ready workspace: packages/zappier (rate card, portal, Stripe),
packages/verae-zapier-middleware (timestamp + NATS), packages/verae-zapier
(CLI app), vendor/zapier-platform, and research/zapier vendor corpus.

Gate 0 structure checks pass. Product code and research are not yet wired.
This commit is contained in:
George Lambert 2026-09-09 02:37:36 -04:00
commit b4150c8250
1364 changed files with 6814366 additions and 0 deletions

View file

@ -0,0 +1,8 @@
node_modules
*.zip
*.log
.zapier-*
.DS_Store
build-boilerplate
*.tgz
.vscode

View file

@ -0,0 +1 @@
v22

View file

@ -0,0 +1,43 @@
# `zapier-platform-core` Architecture
## Purpose
- This package, released as `zapier-platform-core` publicly on [npm](https://www.npmjs.com/package/zapier-platform-core), ships code that partners (literally) depend on.
- Once declared as a dependency and installed, developers can import it to help with testing & TypeScript types.
- It's not actually `require`d directly in app code, but it runs an app's exposed functions at runtime.
- Its main responsibilities are:
- organizing data passed from the Zapier monolith into the developer code (see [A bridge between...](#a-bridge-between-monolith-and-developer) below)
- maintaining the `z` object, on which developers rely (see [Z Object](#z-object) below)
- The `core`, `schema`, and `cli` packages are always released together under matching version numbers.
## Technical Organization
> `core/...`, when used in a path to a file, is shorthand for `zapier-platform/packages/core/...`
### A Bridge Between Monolith and Developer
- The monolith has a method called `call_into_js` which builds an `event` dict and sends it directly to [AWS Lambda](https://aws.amazon.com/lambda/)
- The event is the first argument in the function returned from `createLambdaHandler` (in `core/src/tools/create-lambda-handler.js`).
- That function creates a Node.js `Domain`, which captures all errors. Inside said `domain`, it runs the dev's code with the `event` (which holds, among other things, the `bundle`). Note that the `Domain` class has been deprecated, but is still available for use until it's actually removed.
- There's some app-level middleware that runs (see `src/create-app.js`), then the function itself is run. Output (or errors) are returned from lambda to the monolith.
### Command Handling
- Compiled apps (aka "fully resolved") can have commands run on them. Most commonly, `execute` and `validate`.
- That's set up in `core/src/tools/create-lambda-handler`.
- This is the interface by which tools can perform app-specific tasks.
### Z Object
- At runtime, a developer's `perform` function is invoked with the following signature: `perform(z, bundle)`.
- The `z` object is a collection of functions used commonly by developers and is defined in `core/src/tools/create-lambda-handler.js`
- Its functions are either Zapier-specific (such as `z.cursor` and `z.dehydrate`) or wrappers around common JS functionality (such as `z.JSON.parse` and `z.console.log`) with better error handling or extra logging
- Each of those functions has its own file in `core/src/tools`
- The most important method is probably `z.request` (from `core/src/tools/create-app-request-client.js`), which devs use to make external requests. This is different from normal http requests in a number of ways:
- there are `beforeRequest` and `afterResponse` functions that run before and after the request. They can be declared by the developer or inserted automatically based on the authentication type.
- all requests are logged to our internal logging service, even if a developer uses their own request library (see `core/src/tools/create-http-patch.js`)
### Tests
- Most files in `core/src` have a matching file in `core/test`
- there are test apps in `core/test/*app` directories

View file

@ -0,0 +1 @@
See [CHANGELOG.md](https://github.com/zapier/zapier-platform/blob/main/CHANGELOG.md).

View file

@ -0,0 +1,3 @@
Copyright (c) Zapier, Inc.
This repository is part of Zapier Platform. By downloading, installing, accessing, or using any part of the Zapier Platform, including this repository, you agree to the Zapier Platform Agreement, which can be found at: https://zapier.com/platform/tos. If you do not agree to the Zapier Platform Agreement, you may not download, install, access, or use any part of the Zapier Platform, including this repository.

View file

@ -0,0 +1,27 @@
# Zapier Platform Core
This is the SDK used in Zapier integrations.
## Development
See [CONTRIBUTING.md](https://github.com/zapier/zapier-platform/blob/main/CONTRIBUTING.md) and [ARCHITECTURE.md](https://github.com/zapier/zapier-platform/blob/main/packages/core/ARCHITECTURE.md) of this package in particular.
Useful commands:
* `npm install` for getting started
* `npm test` for running unit tests
* `npm run local-integration-test` for running integration tests locally
* `npm run build-boilerplate -- --debug` for building a `build-boilerplate/*.zip` (if you want to test buildless locally)
### Integration Test on AWS Lambda
Make sure your AWS access key have permission to update and run Lambda functions, and then you can use these commands to run tests on AWS Lambda:
* `npm run deploy-integration-test` builds and deploys a zip to a function named `integration-test-cli` on Lambda
* `npm run lambda-integration-test` runs the integration test using the live Lambda function `integration-test-cli`
## Publishing
Only do this after merging your PR to `main`.
* `npm version [patch|minor|major]` will pull, test, update schema version in dependencies for this package, update docs, increment version in package.json, and push tags, which then will tell Travis to publish to npm

View file

@ -0,0 +1,28 @@
# Usage like so:
# ./build-ec2.sh ec2-user@ip-10-5-40-104.ec2.internal ami.bundle.zip
echo "###### Uploading..."
echo ""
ssh $1 "mkdir -p /home/ec2-user/zapier-core-platform"
scp -rp package.json index.js src include bin $1:/home/ec2-user/zapier-core-platform
echo ""
echo ""
echo "###### Building..."
echo ""
ssh $1 "cd /home/ec2-user/zapier-core-platform && rm -rf node_modules"
ssh $1 "cd /home/ec2-user/zapier-core-platform && npm install --production"
echo ""
echo ""
echo "###### Zipping..."
echo ""
ssh $1 "cd /home/ec2-user/zapier-core-platform && bin/build.sh $2"
echo ""
echo ""
echo "###### Downloading..."
echo ""
scp -rp $1:/home/ec2-user/zapier-core-platform/$2 .
echo ""
echo ""
echo "Done buiding! :-)"
node ./upload-lambda.js

View file

@ -0,0 +1,63 @@
#!/usr/bin/env bash
# Usage like so:
# ./build.sh local.bundle.zip
CORE_REPO_DIR=$(pwd)
BUILD_DIR="./build"
if [ $# -eq 0 ]
then
echo "No arguments supplied."
exit 1
fi
if [ -f $1 ]
then
rm -rf $1
fi
if [ -d $BUILD_DIR ]
then
rm -rf $BUILD_DIR
fi
mkdir $BUILD_DIR
PACK_FILENAME=$(npm pack)
tar xzf $PACK_FILENAME -C $BUILD_DIR
cp -R test $BUILD_DIR/package/
cd $BUILD_DIR/package
npm install --production
echo "Top 10 biggest dependent Node packages, FYI:"
du -s node_modules/* | sort -n -r | head -n 10
find . -print | \
# removing test and example is bold!
grep -v "\.git" | \
grep -v "DS_Store" | \
grep -v "/LICENSE" | \
grep -v "\.min\.js" | \
grep -v "/min/" | \
grep -v "\.html" | \
grep -v "\.css" | \
grep -v "\.png" | \
grep -v "\.gif" | \
grep -v "\.jpg" | \
grep -v "\.md" | \
grep -v "\.sh" | \
grep -v "\.zip" | \
grep -v "tags" | \
zip $1 -@ > /dev/null
cp ./local.bundle.zip $CORE_REPO_DIR/
cd $CORE_REPO_DIR
rm $PACK_FILENAME
rm -rf $BUILD_DIR

View file

@ -0,0 +1,9 @@
#!/usr/bin/env node
const fs = require('fs');
const packageJson = require('../package.json');
packageJson.dependencies['zapier-platform-schema'] = packageJson.version;
fs.writeFile('./package.json', JSON.stringify(packageJson, null, ' ') + '\n');

View file

@ -0,0 +1,40 @@
#!/usr/bin/env node
var fs = require('fs');
var AWS = require('aws-sdk');
console.log('Uploading zip to test Lambda.');
// run it in real lambda...
var lambda = new AWS.Lambda({ apiVersion: '2015-03-31', region: 'us-east-1' });
var fileName = process.argv.slice(2)[0];
var zipFileLambda = fs.readFileSync(fileName);
console.log(zipFileLambda.length, 'bytes of code');
var params = {
Code: {
ZipFile: zipFileLambda,
},
FunctionName: 'integration-test-cli',
Handler: 'index.integrationTestHandler',
Role: 'arn:aws:iam::996097627176:role/allow_nothing_role',
Runtime: 'nodejs8.10',
Description: 'Via node ./lambda-upload.js for dev-platform-cli.',
MemorySize: 512,
Timeout: 30,
};
// lambda.createFunction(
// params,
lambda.updateFunctionCode(
{ FunctionName: params.FunctionName, ZipFile: params.Code.ZipFile },
(err, data) => {
console.log('update code:');
if (err) {
console.log(err, err.stack); // an error occurred
} else {
console.log(data); // successful response
console.log('Now you can try `npm run lambda-integration-test`.');
}
},
);

View file

@ -0,0 +1,6 @@
// not intended to be loaded via require() - copied during build step
const path = require('path');
const zapier = require('zapier-platform-core');
const appPath = path.resolve(__dirname, 'index.js');
// don't `require(appPath)` out here
module.exports = { handler: zapier.createAppHandler(appPath) };

View file

@ -0,0 +1,26 @@
// not intended to be loaded via require() or import() - copied during build step
import zapier from 'zapier-platform-core';
let _appRaw;
try {
_appRaw = await import('{REPLACE_ME_PACKAGE_NAME}');
} catch (err) {
if (
err.code === 'ERR_MODULE_NOT_FOUND' &&
err.message?.includes('{REPLACE_ME_PACKAGE_NAME}')
) {
err.message =
'It seems you are using ESM because your package.json has `"type": "module"`. ' +
'For ESM to work, make sure you specify a valid entry point using `exports` (instead of `main`) in package.json.\n\n' +
err.message;
}
throw err;
}
// Allows a developer to use named exports or default export in entry point
if (_appRaw && _appRaw.default) {
_appRaw = _appRaw.default;
}
export const appRaw = _appRaw;
export const handler = zapier.createAppHandler(_appRaw);

View file

@ -0,0 +1,6 @@
const zapier = require('./src');
zapier.version = require('./package.json').version;
zapier.tools = require('./src/tools/exported');
zapier.errors = require('./src/errors');
zapier.console = require('./src/tools/console-singleton').consoleProxy;
module.exports = zapier;

View file

@ -0,0 +1,27 @@
import zapier from './src/index.js';
import packageJson from './package.json' with { type: 'json' };
import _tools from './src/tools/exported.js';
import _errors from './src/errors.js';
import { consoleProxy } from './src/tools/console-singleton.js';
zapier.version = packageJson.version;
zapier.tools = _tools;
zapier.errors = _errors;
zapier.console = consoleProxy;
// Allows `import { ... } from 'zapier-platform-core'`
export const {
createAppHandler,
createAppTester,
defineApp,
defineCreate,
defineInputField,
defineInputFields,
defineSearch,
defineTrigger,
integrationTestHandler,
console,
tools,
version,
errors,
} = zapier;
// Allows `import zapier from 'zapier-platform-core'`
export default zapier;

View file

@ -0,0 +1,849 @@
'use strict';
const fs = require('fs');
const os = require('os');
const path = require('path');
const { promisify } = require('util');
const AWS = require('aws-sdk');
const crypto = require('crypto');
const nock = require('nock');
const should = require('should');
const createLambdaHandler = require('../src/tools/create-lambda-handler');
const mocky = require('../test/tools/mocky');
const { HTTPBIN_URL } = require('../test/constants');
const sleep = promisify(setTimeout);
const lambda = new AWS.Lambda({
apiVersion: '2015-03-31',
region: 'us-east-1',
});
const runLambda = (event) => {
return new Promise((resolve, reject) => {
const params = {
FunctionName: 'integration-test-cli',
Payload: JSON.stringify(event),
LogType: 'Tail',
};
lambda.invoke(params, (err, data) => {
if (err) {
console.log(err);
return reject(err);
}
const logs = Buffer.from(data.LogResult, 'base64').toString();
console.log('\n=== LOGS ===\n', logs, '\n===\n');
const response = data.Payload ? JSON.parse(data.Payload) : data;
if (response.errorMessage) {
return reject(new Error(response.errorMessage));
}
return resolve(response);
});
});
};
runLambda.testName = 'runLambda';
const runLocally = (event) => {
return new Promise((resolve, reject) => {
const handler = createLambdaHandler(
path.resolve(__dirname, '../test/userapp/'),
);
try {
resolve(handler(event));
} catch (err) {
reject(err);
}
});
};
runLocally.testName = 'runLocally';
const doTest = (runner) => {
describe(`${runner.testName} integration tests`, () => {
afterEach(() => {
// Clear cache files
const tmpdir = os.tmpdir();
const cacheFilenames = ['cli-override.json', 'cli-hash.txt'];
cacheFilenames.forEach((filename) => {
const filepath = path.join(tmpdir, filename);
if (fs.existsSync(filepath)) {
fs.unlinkSync(filepath);
}
});
// Remove all the mocked requests
nock.cleanAll();
});
it('should return data from app function call', () => {
const event = {
command: 'execute',
method: 'resources.list.list.operation.perform',
bundle: {
'param a': 'say, can u see me?',
'param b': 'oh, can u see me too?',
},
};
return runner(event).then((response) => {
should.exist(response.results);
response.results.should.eql([{ id: 1234 }, { id: 5678 }]);
});
});
it('should validate an app', () => {
const event = {
command: 'validate',
};
return runner(event).then((response) => {
should.exist(response.results);
});
});
it('should provide the definition for an app', () => {
const event = {
command: 'definition',
};
return runner(event).then((response) => {
should.exist(response.results);
});
});
it('should do a logging function', () => {
const event = {
command: 'execute',
method: 'resources.loggingfunc.list.operation.perform',
logExtra: {
app_cli_id: 666,
},
};
return runner(event).then((response) => {
should.exist(response.results);
});
});
it('should handle appRawOverride', () => {
const event = {
command: 'execute',
method: 'triggers.fooList.operation.perform',
appRawOverride: {
resources: {
foo: {
key: 'foo',
noun: 'Foo',
list: {
display: {},
operation: {
perform: { source: 'return [{id: 45678}]' },
},
},
},
},
},
};
return runner(event).then((response) => {
response.results.should.deepEqual([{ id: 45678 }]);
});
});
it('should handle appRawOverride as hash', () => {
const definition = {
resources: {
foo: {
key: 'foo',
noun: 'Foo',
list: {
display: {},
operation: {
perform: { source: 'return [{id: 45678}]' },
},
},
},
},
};
mocky.mockRpcCall(definition);
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'triggers.fooList.operation.perform',
appRawOverride: definitionHash,
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.length.should.eql(1);
response.results[0].id.should.eql(45678);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and override inputFields', () => {
const definition = {
creates: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
inputFields: [
{ key: 'name', type: 'string' },
{ key: 'testing', type: 'string' },
],
sample: {
id: 123,
},
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
creates: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
inputFields: [{ key: 'message', type: 'string' }],
sample: {
name: 'sample',
},
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'creates.foo.operation.inputFields',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([{ key: 'message', type: 'string' }]);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and override specific inputField', () => {
const definition = {
creates: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
inputFields: [{ key: 'message', type: 'integer' }],
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
creates: {
foo: {
noun: 'Foobar',
operation: {
inputFields: [{ key: 'message', type: 'string' }],
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'creates.foo.operation.inputFields',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([{ key: 'message', type: 'string' }]);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and overrides with an operation create key', () => {
const definition = {
creates: {
operation: {
key: 'operation',
operation: {
perform: { source: 'return [{id: 12345}]' },
inputFields: [
{ key: 'name', type: 'string' },
{ key: 'testing', type: 'string' },
],
sample: {
id: 123,
},
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
creates: {
operation: {
operation: {
perform: { source: 'return [{id: 123}]' },
inputFields: [{ key: 'message', type: 'string' }],
sample: {
name: 'sample',
},
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'creates.operation.operation.inputFields',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([{ key: 'message', type: 'string' }]);
});
});
it('should handle array of [null, appRawExtension] without resource key collision', async () => {
const definitionExtension = {
// Example app test/userapp/index.js has a 'contact' resource with a
// 'create' operation ('contactCreate' after compiled). So this should
// merge with that and not collide.
creates: {
contactCreate: {
operation: {
perform: {
source: 'return { id: 8, name: "Yoshi" }',
},
},
},
},
};
const event = {
comamnd: 'execute',
method: 'creates.contactCreate.operation.perform',
appRawOverride: [null, definitionExtension],
};
const response = await runner(event);
response.results.should.deepEqual({ id: 8, name: 'Yoshi' });
});
it('should handle array of [appRawOverrideHash, appRawExtension] and override perform with request', () => {
const definition = {
triggers: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: {
url: `${HTTPBIN_URL}/get`,
params: {
id: 54321,
},
},
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
triggers: {
foo: {
noun: 'Foobar',
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'triggers.foo.operation.perform',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([{ id: 12345 }]);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and override perform with source', async () => {
const definition = {
creates: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
creates: {
foo: {
noun: 'Foobar',
operation: {
perform: {
method: 'POST',
url: `${HTTPBIN_URL}/post`,
params: {
id: 54321,
},
},
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'creates.foo.operation.perform',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
const response = await runner(event);
response.results.should.containEql({
args: {
id: ['54321'],
},
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and override perform with function', () => {
const definition = {
searches: {
foo: {
key: 'foo',
noun: 'Foo',
operation: {
perform: '$func$2$f$',
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
searches: {
foo: {
noun: 'Foobar',
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'searches.foo.operation.perform',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([
{
id: 12345,
},
]);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and add perform', () => {
const definition = {
triggers: {
test: {
key: 'test',
noun: 'Foo',
operation: {
type: 'hook',
performList: { source: 'return [{id: 54321}]' },
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
triggers: {
test: {
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'triggers.test.operation.perform',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([
{
id: 12345,
},
]);
});
});
it('should handle array of [appRawOverrideHash, appRawExtension] and add new trigger', () => {
const definition = {
triggers: {
test: {
key: 'test',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 54321}]' },
},
},
},
};
mocky.mockRpcCall(definition);
const definitionExtension = {
triggers: {
perform: {
key: 'perform',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
const definitionHash = crypto
.createHash('md5')
.update(JSON.stringify(definition))
.digest('hex');
const event = {
command: 'execute',
method: 'triggers.perform.operation.perform',
appRawOverride: [definitionHash, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([
{
id: 12345,
},
]);
});
});
it('should handle array of non-existent appRawOverride (CLI app) with an appRawExtension, and add new trigger', () => {
let definition; // CLI apps have no definition override
const definitionExtension = {
triggers: {
perform: {
key: 'perform',
noun: 'Foo',
operation: {
perform: { source: 'return [{id: 12345}]' },
},
},
},
};
const event = {
command: 'execute',
method: 'triggers.perform.operation.perform',
appRawOverride: [definition, definitionExtension],
rpc_base: 'https://mock.zapier.com/platform/rpc/cli',
token: 'fake',
};
return runner(event).then((response) => {
response.results.should.eql([
{
id: 12345,
},
]);
// CLI code should be merged in with extension
event.appRawOverride[0].should.not.eql(undefined);
});
});
it('should handle function source in beforeRequest', async () => {
const definition = {
beforeRequest: [
{
source: "request.headers['X-Foo'] = 'it worked!'; return request",
args: ['request', 'z', 'bundle'],
},
],
creates: {
foo: {
operation: {
perform: {
method: 'POST',
url: `${HTTPBIN_URL}/post`,
},
},
},
},
};
const event = {
command: 'execute',
method: 'creates.foo.operation.perform',
appRawOverride: definition,
};
const response = await runner(event);
response.results.headers['X-Foo'].should.deepEqual(['it worked!']);
});
it('should log requests', () => {
const event = {
command: 'execute',
method: 'resources.requestsugar.list.operation.perform',
logExtra: {
app_cli_id: 666,
},
};
return runner(event).then((response) => {
should.exist(response.results);
});
});
it('should not leave leftover env vars', () => {
const event = {
environment: {
_ZAPIER_ONE_TIME_SECRET: 'foo',
},
method: 'resources.env.list.operation.perform',
};
return runner(event)
.then((response) => {
response.results.length.should.eql(1);
response.results[0].key.should.eql('_ZAPIER_ONE_TIME_SECRET');
response.results[0].value.should.eql('foo');
delete event.environment;
return runner(event);
})
.then((response) => {
response.results.length.should.eql(0);
});
});
describe('hang detection', () => {
let originalEnv;
before(() => {
originalEnv = process.env;
process.env = {
...originalEnv,
LOGGING_ENDPOINT: `${mocky.FAKE_LOG_URL}/input`,
LOGGING_TOKEN: 'fake-token',
};
});
after(() => {
process.env = originalEnv;
});
beforeEach(() => {
mocky.clearLogs();
});
afterEach(() => {
mocky.clearLogs();
});
it('should end logger even a callback runs after lambda handler returns', async () => {
mocky.mockLogServer();
const event = {
method: 'resources.bad_callback.create.operation.perform',
};
const response = await runner(event);
response.results.message.should.eql('ok');
should.not.exist(process.teapot);
// Hopefully long enough for the callback to complete
await sleep(2000);
process.teapot.should.eql("I'm a teapot!");
const logs = mocky.getLogs();
logs.length.should.eql(1);
const log = logs[0];
log.message.should.eql(`418 GET ${HTTPBIN_URL}/status/418`);
});
});
describe('error handling', () => {
let originalEnv;
before(() => {
originalEnv = process.env;
process.env = {
...originalEnv,
LOGGING_ENDPOINT: `${mocky.FAKE_LOG_URL}/input`,
LOGGING_TOKEN: 'fake-token',
};
});
after(() => {
process.env = originalEnv;
});
beforeEach(() => {
mocky.clearLogs();
mocky.mockLogServer();
});
afterEach(() => {
mocky.clearLogs();
});
const testError = (method, errorMessage, logMessage) => {
it(`should catch errors from ${method}`, async () => {
const event = {
command: 'execute',
method,
};
try {
await runner(event);
should(true).eql(false, 'Expected an error!');
} catch (err) {
should.exist(err);
err.message.should.startWith(errorMessage);
}
if (logMessage) {
const logs = mocky.getLogs();
const log = logs[0];
log.message.should.startWith(logMessage);
}
});
};
testError(
'resources.failerfunc404.list.operation.perform',
'{"status":404,', // ResponseError JSON-encodes the response body in its .message
`404 GET ${HTTPBIN_URL}/status/404`,
);
testError(
'resources.failerfuncasync.list.operation.perform',
'Uncaught failure on async function!',
'Uncaught error: Error: Uncaught failure on async function!',
);
testError(
'resources.failerfunc.list.operation.perform',
'Failer on sync function!',
'Unhandled error: Error: Failer on sync function!',
);
testError(
'resources.failerfuncpromise.list.operation.perform',
'Failer on promise function!',
);
});
});
};
if (process.argv.indexOf('integration-test') > 0) {
if (process.argv.indexOf('--lambda') > 0) {
doTest(runLambda);
} else if (process.argv.indexOf('--local') > 0) {
doTest(runLocally);
} else {
doTest(runLambda);
doTest(runLocally);
}
}
module.exports = {
runLambda,
runLocally,
doTest,
};

View file

@ -0,0 +1,90 @@
{
"name": "zapier-platform-core",
"version": "19.1.0",
"description": "The core SDK for CLI apps in the Zapier Developer Platform.",
"repository": "zapier/zapier-platform",
"homepage": "https://platform.zapier.com/",
"author": "Zapier Engineering <contact@zapier.com>",
"license": "SEE LICENSE IN LICENSE",
"types": "types/index.d.ts",
"exports": {
".": {
"types": "./types/index.d.ts",
"require": "./index.js",
"import": "./index.mjs"
},
"./src/*": {
"require": "./src/*.js"
}
},
"files": [
"/include/",
"/index.js",
"/index.mjs",
"/src/",
"/types/"
],
"scripts": {
"preversion": "git pull && pnpm test",
"version": "node bin/bump-dependencies.js && pnpm install && git add package.json pnpm-lock.yaml",
"postversion": "git push && git push --tags",
"main-tests": "mocha -t 20s --recursive test --exit",
"type-tests": "tsd --files types/**/*.test-d.ts",
"solo-test": "test $(OPT_OUT_PATCH_TEST_ONLY=yes mocha --recursive test -g 'should be able to opt out of patch' -R json | jq '.stats.passes') -eq 1 && echo 'Ran 1 test and it passed!'",
"test": "pnpm main-tests && pnpm solo-test && pnpm type-tests",
"test:debug": "mocha inspect -t 10s --recursive test",
"debug": "mocha -t 10s --inspect-brk --recursive test",
"test:w": "mocha -t 10s --recursive test --watch",
"integration-test": "mocha -t 20s integration-test",
"local-integration-test": "mocha -t 10s integration-test --local",
"lambda-integration-test": "mocha -t 10s integration-test --lambda",
"smoke-test": "mocha -t 2m smoke-test",
"lint": "eslint src test",
"lint:fix": "eslint --fix src test",
"build-integration-test": "bin/build.sh local.bundle.zip",
"upload-integration-test": "bin/upload-lambda.js local.bundle.zip",
"deploy-integration-test": "pnpm build-integration-test && pnpm upload-integration-test",
"validate": "pnpm test && pnpm smoke-test && pnpm lint"
},
"engines": {
"node": ">=16",
"npm": ">=5.6.0"
},
"engineStrict": true,
"dependencies": {
"@zapier/secret-scrubber": "^1.1.2",
"content-disposition": "0.5.4",
"dotenv": "17.2.1",
"fernet": "^0.3.3",
"form-data": "4.0.5",
"json-schema-to-ts": "3.1.1",
"lodash": "4.18.1",
"mime-types": "3.0.1",
"node-abort-controller": "3.1.1",
"node-fetch": "2.7.0",
"oauth-sign": "0.9.0",
"semver": "7.7.2",
"zapier-platform-schema": "workspace:*"
},
"devDependencies": {
"@types/node-fetch": "^2.6.11",
"adm-zip": "0.5.16",
"aws-sdk": "^2.1397.0",
"dicer": "^0.3.1",
"fs-extra": "^11.3.0",
"mock-fs": "^5.5.0",
"nock": "^13.5.4",
"tsd": "^0.31.1"
},
"peerDependencies": {
"zapier-platform-legacy-scripting-runner": ">=3"
},
"peerDependenciesMeta": {
"zapier-platform-legacy-scripting-runner": {
"optional": true
}
},
"optionalDependencies": {
"@types/node": "^20.3.1"
}
}

View file

@ -0,0 +1,289 @@
// Any important changes here need to be made to src/smoke-tests/ in the cli repo!
const { spawnSync } = require('child_process');
const crypto = require('crypto');
const fs = require('fs-extra');
const os = require('os');
const path = require('path');
require('should');
const fetch = require('node-fetch');
const CORE_PACKAGE_NAME = 'zapier-platform-core';
const TEST_APPS = [
'basic-auth',
'create',
// 'custom-auth',
// 'oauth2',
// 'resource',
// 'search',
// 'session-auth',
'trigger',
];
const setupZapierRC = () => {
let hasRC = false;
const rcPath = path.join(os.homedir(), '.zapierrc');
if (fs.existsSync(rcPath)) {
hasRC = true;
} else if (process.env.DEPLOY_KEY) {
fs.writeFileSync(
rcPath,
JSON.stringify({ deployKey: process.env.DEPLOY_KEY }),
);
hasRC = true;
}
return hasRC;
};
const setupZapierAppRC = (workdir) => {
let hasAppRC = false;
if (process.env.TEST_APP_ID && process.env.TEST_APP_KEY) {
const rcPath = path.join(workdir, '.zapierapprc');
if (!fs.existsSync(rcPath)) {
fs.writeFileSync(
rcPath,
JSON.stringify({
id: parseInt(process.env.TEST_APP_ID),
key: process.env.TEST_APP_KEY,
}),
);
hasAppRC = true;
}
}
return hasAppRC;
};
const getPackageDir = (dirname) =>
path.resolve(path.dirname(process.cwd()), dirname);
const npmPack = (workingDir) => {
const proc = spawnSync('npm', ['pack'], {
encoding: 'utf8',
cwd: workingDir,
});
const lines = proc.stdout.split('\n');
let filename;
for (let i = lines.length - 1; i >= 0; i--) {
const line = lines[i].trim();
if (line) {
filename = line;
break;
}
}
return filename;
};
const npmPackCore = (schemaPackagePath) => {
// Patch core's package.json to use schema from local
const packageJsonPath = path.join(process.cwd(), 'package.json');
const originalPackageJsonText = fs.readFileSync(packageJsonPath, {
encoding: 'utf8',
});
const packageJson = JSON.parse(originalPackageJsonText);
packageJson.dependencies['zapier-platform-schema'] =
`file:${schemaPackagePath}`;
fs.writeFileSync(packageJsonPath, JSON.stringify(packageJson));
let filename;
try {
const proc = spawnSync('npm', ['pack'], { encoding: 'utf8' });
const lines = proc.stdout.split('\n');
for (let i = lines.length - 1; i >= 0; i--) {
const line = lines[i].trim();
if (line) {
filename = line;
break;
}
}
} finally {
fs.writeFile(packageJsonPath, originalPackageJsonText);
}
return filename;
};
const setupTempWorkingDir = () => {
let workdir;
const tmpBaseDir = os.tmpdir();
while (!workdir || fs.existsSync(workdir)) {
workdir = path.join(
tmpBaseDir,
'zapier-' + crypto.randomBytes(20).toString('hex'),
);
}
fs.mkdirSync(workdir);
return workdir;
};
const copyTestApps = (workdir) => {
const repoRoot = path.dirname(path.dirname(path.dirname(__dirname)));
TEST_APPS.forEach((appName) => {
const srcAppDir = path.join(repoRoot, 'example-apps', appName);
const destAppDir = path.join(workdir, appName);
fs.copySync(srcAppDir, destAppDir, {
filter: (src, dest) => !src.includes('node_modules/'),
});
});
};
const npmInstalls = (coreZipPath, cliZipPath, workdir) => {
// When releasing a new core version, example apps would have bumped their
// core before the new core version is published to npm. So we need to patch
// example app's package.json to use local core temporarily to avoid `npm
// install` error.
const packageJsonPath = path.join(workdir, 'package.json');
const origPackageJsonText = fs.readFileSync(packageJsonPath);
const packageJson = JSON.parse(origPackageJsonText);
packageJson.dependencies[CORE_PACKAGE_NAME] = `file:${coreZipPath}`;
fs.writeFileSync(packageJsonPath, JSON.stringify(packageJson));
try {
spawnSync('npm', ['install'], {
encoding: 'utf8',
cwd: workdir,
});
spawnSync('npm', ['install', '--no-save', cliZipPath], {
encoding: 'utf8',
cwd: workdir,
});
} finally {
fs.writeFileSync(packageJsonPath, origPackageJsonText);
}
};
describe('smoke tests - setup will take some time', () => {
const context = {
// Global context that will be available for all test cases in this test suite
corePackage: {
filename: null,
path: null,
},
schemaPackage: {
filename: null,
path: null,
},
cliPackage: {
filename: null,
path: null,
},
workRepoDir: null,
workAppDir: null,
cliBin: null,
hasRC: false,
hasAppRC: false,
};
before(async () => {
context.hasRC = setupZapierRC();
const cliDir = getPackageDir('cli');
context.cliPackage.filename = npmPack(cliDir);
context.cliPackage.path = path.join(cliDir, context.cliPackage.filename);
const schemaDir = getPackageDir('schema');
context.schemaPackage.filename = npmPack(schemaDir);
context.schemaPackage.path = path.join(
schemaDir,
context.schemaPackage.filename,
);
context.corePackage.filename = npmPackCore(context.schemaPackage.path);
context.corePackage.path = path.join(
process.cwd(),
context.corePackage.filename,
);
context.workRepoDir = setupTempWorkingDir();
copyTestApps(context.workRepoDir);
});
after(() => {
fs.unlinkSync(context.cliPackage.path);
fs.unlinkSync(context.corePackage.path);
fs.unlinkSync(context.schemaPackage.path);
fs.removeSync(context.workRepoDir);
});
it('package size should not change much', async function () {
const baseUrl = 'https://registry.npmjs.org/zapier-platform-core';
let res = await fetch(baseUrl);
const packageInfo = await res.json();
const latestVersion = packageInfo['dist-tags'].latest;
res = await fetch(`${baseUrl}/-/zapier-platform-core-${latestVersion}.tgz`);
const baselineSize = res.headers.get('content-length');
const newSize = fs.statSync(context.corePackage.path).size;
newSize.should.be.within(baselineSize * 0.7, baselineSize * 1.3);
this.test.title += ` (${baselineSize} -> ${newSize} bytes)`;
});
TEST_APPS.forEach((appName) => {
describe(appName, () => {
before(async () => {
context.workAppDir = path.join(context.workRepoDir, appName);
npmInstalls(
context.corePackage.path,
context.cliPackage.path,
context.workAppDir,
);
context.hasAppRC = setupZapierAppRC(context.workAppDir);
context.cliBin = path.join(
context.workAppDir,
'node_modules',
'.bin',
'zapier-platform',
);
});
it('zapier-platform test', () => {
const proc = spawnSync(context.cliBin, ['test'], {
encoding: 'utf8',
cwd: context.workAppDir,
env: {
PATH: process.env.PATH,
DISABLE_ZAPIER_ANALYTICS: 1,
},
});
if (proc.status !== 0) {
console.log(proc.stdout);
console.log(proc.stderr);
}
proc.status.should.eql(0);
});
it('zapier-platform build', function () {
if (!context.hasAppRC) {
this.skip();
return;
}
const proc = spawnSync(
context.cliBin,
['build', '--skip-npm-install'],
{
encoding: 'utf8',
cwd: context.workAppDir,
env: {
PATH: process.env.PATH,
DISABLE_ZAPIER_ANALYTICS: 1,
},
},
);
if (proc.status !== 0) {
console.log(proc.stdout);
console.log(proc.stderr);
}
proc.status.should.eql(0);
});
});
});
});

View file

@ -0,0 +1,20 @@
'use strict';
const STATUSES = require('../../constants').STATUSES;
const _ = require('lodash');
/*
this method creates the correct envelope responses if the app has used a callback url in their code
by signalling to Zapier that this app/method is returning a callback status the task will be placed
in a waiting state until the callback is called.
*/
const callbackStatusCatcher = (output) => {
const input = output.input || {};
const callbackUsed = _.get(input, '_zapier.event.callbackUsed');
if (callbackUsed) {
// output is an envelope so we can set status here
output.status = STATUSES.CALLBACK;
}
return output;
};
module.exports = callbackStatusCatcher;

View file

@ -0,0 +1,57 @@
'use strict';
const _ = require('lodash');
const constants = require('../../constants');
const errors = require('../../errors');
const checks = _.values(require('../../checks'));
/*
Take a look at our output results, run some checks on it, and depending on if we
are running "locally or live" we will "raise or log" the errors, respectively.
*/
const checkOutput = (output) => {
const input = output.input || {};
const _zapier = input._zapier || {};
const event = _zapier.event || {};
const compiledApp = _zapier.app || {};
const runChecks =
event.method && event.command === 'execute' && !_zapier.skipChecks;
const bundleSkipChecks = event.bundle.skipChecks || [];
if (runChecks) {
const rawResults = checks
.filter((check) => {
return (
!bundleSkipChecks.includes(check.name) &&
check.shouldRun(event.method, event.bundle, compiledApp)
);
})
.map((check) => {
return check
.run(event.method, output.results, compiledApp, event.bundle)
.map((err) => ({ name: check.name, error: err }));
});
const checkResults = _.flatten(rawResults);
if (checkResults.length > 0) {
if (constants.IS_TESTING || event.isDeveloper || event.calledFromCli) {
const shortMsgs = checkResults.map((info) => ` - ${info.error}`);
throw new errors.CheckError(
'Invalid API Response:\n' + shortMsgs.join('\n'),
);
} else {
const longMsgs = checkResults.map(
(info) => `Zapier check "${info.name}" failed: ${info.error}`,
);
longMsgs.forEach((err) => input.z.console.error(err));
}
}
}
return output;
};
module.exports = checkOutput;

View file

@ -0,0 +1,39 @@
'use strict';
const constants = require('../../constants');
const cleaner = require('../../tools/cleaner');
const responseStasher = require('../../tools/create-response-stasher');
const largeResponseCachePointer = async (output) => {
const response = cleaner.maskOutput(output);
if (!response.results) {
return output;
}
const autostashLimit = output.input._zapier.event.autostashPayloadOutputLimit;
const payload = JSON.stringify(response.results);
const size = payload.length;
// If autostash limit is defined, and is within the range, stash the response
// If it is -1, stash the response regardless of size
if (
(autostashLimit &&
size >= constants.RESPONSE_SIZE_LIMIT &&
size <= autostashLimit) ||
autostashLimit === -1
) {
const url = await responseStasher(output.input, payload);
output.resultsUrl = url;
output.results = Array.isArray(output.results) ? [] : {};
} else if (autostashLimit && size > autostashLimit) {
// If the limit is defined and is out of range, throw a descriptive error
// indicating the size of the response and the autostash limit
throw new Error(
`Response size of ${size} bytes exceeds maximum allowed size: ${autostashLimit}`,
);
}
return output;
};
module.exports = largeResponseCachePointer;

View file

@ -0,0 +1,12 @@
'use strict';
/*
After app middlewares that waits for all pending promises to resolve.
*/
const waitForPromises = (output) => {
return Promise.all(output.input._zapier.promises || [])
.catch(() => {}) // drop any errors in waiting promises
.then(() => output);
};
module.exports = waitForPromises;

View file

@ -0,0 +1,25 @@
'use strict';
const _ = require('lodash');
// remove sensitive data from bundle before logging
const logSafeBundle = (bundle) => {
return _.omit(bundle, ['authData', 'platformData']);
};
/*
Before middleware that adds a bit of human stack frame context
*/
const addAppContext = (input) => {
const methodName = _.get(input, '_zapier.event.method');
input._zapier.whatHappened.push(`Executing ${methodName} with bundle`);
const bundle = _.get(input, '_zapier.event.bundle', {});
if (Object.keys(bundle).length > 0) {
input._zapier.whatHappened.push(JSON.stringify(logSafeBundle(bundle)));
}
return input;
};
module.exports = addAppContext;

View file

@ -0,0 +1,43 @@
'use strict';
const _ = require('lodash');
const fetch = require('../../tools/fetch');
const { StashedBundleError } = require('../../errors');
const { withRetry } = require('../../tools/retry-utils');
const { decryptBundleWithSecret } = require('../../tools/bundle-encryption');
const fetchStashedBundle = async (input) => {
const stashedBundleKey = _.get(
input,
'_zapier.event.stashedBundleKey',
undefined,
);
const rpc = _.get(input, '_zapier.rpc');
const secret = process.env._ZAPIER_ONE_TIME_SECRET;
if (stashedBundleKey && secret) {
// Use the RPC to get a presigned URL for downloading the data
const rpcResponse = await rpc(
'get_presigned_download_url',
stashedBundleKey,
);
const response = await withRetry(() => fetch(rpcResponse.url));
if (!response.ok) {
const errorMessage = `Failed to read stashed bundle. Status: ${response.status} ${response.statusText}`;
throw new StashedBundleError(errorMessage);
}
try {
const responseText = await response.text();
// Decrypt the bundle
const stashedBundle = decryptBundleWithSecret(responseText, secret);
// Set the bundle to the stashedBundle value
_.set(input, '_zapier.event.bundle', stashedBundle);
} catch (error) {
throw new StashedBundleError(error.message);
}
}
return input;
};
module.exports = fetchStashedBundle;

View file

@ -0,0 +1,56 @@
'use strict';
const _ = require('lodash');
const createAppRequestClient = require('../../tools/create-app-request-client');
const createCache = require('../../tools/create-cache');
const createDehydrator = require('../../tools/create-dehydrator');
const createFileStasher = require('../../tools/create-file-stasher');
const createJSONtool = require('../../tools/create-json-tool');
const createStoreKeyTool = require('../../tools/create-storekey-tool');
const createCallbackHigherOrderFunction = require('../../tools/create-callback-wrapper');
const createLegacyScriptingRunner = require('../../tools/create-legacy-scripting-runner');
const { initialize } = require('../../tools/console-singleton');
const errors = require('../../errors');
const hashing = require('../../tools/hashing');
/*
Before middleware that injects z object.
*/
const injectZObject = (input) => {
const bundle = _.get(input, '_zapier.event.bundle', {});
const zRoot = {
cache: createCache(input),
console: initialize(input),
cursor: createStoreKeyTool(input),
dehydrate: createDehydrator(input, 'method'),
dehydrateFile: createDehydrator(input, 'file'),
errors,
generateCallbackUrl: createCallbackHigherOrderFunction(input),
hash: hashing.hashify,
JSON: createJSONtool(),
require: (moduleName) =>
require(
require.resolve(moduleName, {
paths: module.paths.concat([process.cwd()]),
}),
),
stashFile: createFileStasher(input),
};
const zSkinny = _.extend({}, zRoot);
const z = _.extend({}, zSkinny, {
request: createAppRequestClient(input, { extraArgs: [zSkinny, bundle] }),
});
const runner = createLegacyScriptingRunner(z, input);
if (runner) {
z.legacyScripting = zSkinny.legacyScripting = runner;
}
return _.extend({}, input, { z });
};
module.exports = injectZObject;

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,141 @@
'use strict';
const http = require('http');
const https = require('https');
const { Readable } = require('stream');
const { EventEmitter } = require('events');
// Patches http/https/fetch so outbound requests are intercepted instead of
// hitting the network. `onRequest({ url, headers, method })` is invoked for
// every intercepted call. The original globals are restored on completion
// (success or error). Used by both getAuthTemplate (placeholder authData)
// and renderAuthTemplate (real authData) when invoking authentication.test
// as a function.
//
// Concurrent withHttpCapture calls share a single set of patches so the
// inner call doesn't capture the outer's patched function as its
// "original" — the depth counter only restores on the outermost finally.
let captureDepth = 0;
let savedHttpRequest = null;
let savedHttpsRequest = null;
let savedHttpGet = null;
let savedHttpsGet = null;
let savedFetch = null;
const onRequestStack = [];
const withHttpCapture = async (onRequest, fn) => {
onRequestStack.push(onRequest);
if (captureDepth === 0) {
savedHttpRequest = http.request;
savedHttpsRequest = https.request;
savedHttpGet = http.get;
savedHttpsGet = https.get;
savedFetch = globalThis.fetch;
installPatches();
}
captureDepth++;
try {
return await fn();
} finally {
captureDepth--;
onRequestStack.pop();
if (captureDepth === 0) {
http.request = savedHttpRequest;
https.request = savedHttpsRequest;
http.get = savedHttpGet;
https.get = savedHttpsGet;
globalThis.fetch = savedFetch;
savedHttpRequest =
savedHttpsRequest =
savedHttpGet =
savedHttpsGet =
savedFetch =
null;
}
}
};
const notifyAll = (info) => {
// Notify every nested capture in stack order so each caller's onRequest
// sees the same intercepted request.
for (const cb of onRequestStack) {
cb(info);
}
};
const installPatches = () => {
const patchedRequest = (origFn, protocol) =>
function patchedReq(...args) {
// args can be (url, options, cb), (options, cb), or (url, cb)
let options = {};
const cb =
typeof args[args.length - 1] === 'function'
? args[args.length - 1]
: null;
if (typeof args[0] === 'string' || args[0] instanceof URL) {
const parsed = typeof args[0] === 'string' ? new URL(args[0]) : args[0];
options =
typeof args[1] === 'object' && args[1] !== null
? { url: parsed.href, ...args[1] }
: { url: parsed.href };
} else {
options = args[0] || {};
}
notifyAll({
url:
options.url ||
`${protocol}://${options.host || options.hostname || 'localhost'}${options.path || '/'}`,
headers: options.headers || {},
method: options.method || 'GET',
});
// Return a no-op request that doesn't actually connect.
// Use a real Readable stream so libraries that call .pipe() or
// .setEncoding() (e.g. xmlrpc's SAX deserializer) work correctly.
const fakeReq = new EventEmitter();
fakeReq.write = () => {};
fakeReq.end = () => {
const body =
'<?xml version="1.0"?><methodResponse><params>' +
'<param><value><string>ok</string></value></param>' +
'</params></methodResponse>';
const fakeRes = new Readable({
read() {
this.push(body);
this.push(null);
},
});
fakeRes.statusCode = 200;
fakeRes.headers = { 'content-type': 'text/xml' };
if (cb) {
cb(fakeRes);
}
fakeReq.emit('response', fakeRes);
};
fakeReq.setTimeout = () => fakeReq;
fakeReq.destroy = () => {};
return fakeReq;
};
http.request = patchedRequest(savedHttpRequest, 'http');
https.request = patchedRequest(savedHttpsRequest, 'https');
http.get = patchedRequest(savedHttpGet, 'http');
https.get = patchedRequest(savedHttpsGet, 'https');
globalThis.fetch = async (input, init) => {
const url = typeof input === 'string' ? input : input?.url || '';
const headers = init?.headers || input?.headers || {};
notifyAll({
url,
headers:
headers instanceof Headers
? Object.fromEntries(headers.entries())
: headers,
method: init?.method || input?.method || 'GET',
});
return new Response('{}', { status: 200, headers: {} });
};
};
module.exports = { withHttpCapture };

View file

@ -0,0 +1,219 @@
'use strict';
const vm = require('vm');
const lodash = require('lodash');
// --- Legacy scripting auth support ---
// Minimal reimplementation of the legacy scripting runner's beforeRequest
// middleware, just enough to inject auth fields into headers/params.
// Adapted from zapier-platform-legacy-scripting-runner/middleware-factory.js.
const renderLegacyTemplate = (templateString, context) => {
if (typeof templateString !== 'string') {
return templateString;
}
return templateString.replace(/\{\{([^}]+)\}\}/g, (_, key) => {
const trimmed = key.trim();
return trimmed in context ? context[trimmed] : '';
});
};
const renderAuthMapping = (authMapping, authData) => {
if (!authMapping || Object.keys(authMapping).length === 0) {
return authData;
}
const result = {};
for (const [k, v] of Object.entries(authMapping)) {
result[k] = renderLegacyTemplate(v, authData);
}
return result;
};
const createLegacyBeforeRequest = (app) => {
const authType = app.authentication && app.authentication.type;
const legacy = app.legacy || {};
const authMapping =
(legacy.authentication && legacy.authentication.mapping) || {};
const placement =
(legacy.authentication && legacy.authentication.placement) || 'header';
return (req, z, bundle) => {
const authData = bundle.authData || {};
if (!authData || Object.keys(authData).length === 0) {
return req;
}
if (authType === 'oauth2') {
if (authData.access_token) {
if (placement === 'header' || placement === 'both') {
req.headers.Authorization =
req.headers.Authorization || `Bearer ${authData.access_token}`;
}
if (placement === 'querystring' || placement === 'both') {
req.params = req.params || {};
req.params.access_token =
req.params.access_token || authData.access_token;
}
}
} else if (authType === 'session' || authType === 'custom') {
const rendered = renderAuthMapping(authMapping, authData);
if (placement === 'header' || placement === 'both') {
const lowerHeaders = {};
for (const [k, v] of Object.entries(req.headers)) {
lowerHeaders[k.toLowerCase()] = v;
}
for (const [k, v] of Object.entries(rendered)) {
if (!lowerHeaders[k.toLowerCase()]) {
req.headers[k] = v;
}
}
}
if (placement === 'querystring' || placement === 'both') {
req.params = req.params || {};
for (const [k, v] of Object.entries(rendered)) {
req.params[k] = req.params[k] || v;
}
}
} else if (authType === 'basic' || authType === 'digest') {
// Only override username/password when the legacy authMapping
// explicitly defines them. Otherwise preserve the values the user
// already provided so addBasicAuthHeader (later in the pipeline)
// can use them.
if (authMapping.username) {
bundle.authData.username = renderLegacyTemplate(
authMapping.username,
authData,
);
}
if (authMapping.password) {
bundle.authData.password = renderLegacyTemplate(
authMapping.password,
authData,
);
}
}
return req;
};
};
// Load the Zap object from legacy scriptingSource.
const loadLegacyZap = (compiledApp) => {
const src = compiledApp.legacy && compiledApp.legacy.scriptingSource;
if (!src) {
return null;
}
const sandbox = { Zap: {}, _: lodash, z: { JSON }, $: {} };
try {
vm.runInNewContext(src, sandbox);
} catch {
return null;
}
return sandbox.Zap;
};
// Map typeOf + key to the pre-method name on the Zap object.
const getLegacyPreMethodName = (typeOf, key) => {
if (!key) {
return null;
}
switch (typeOf) {
case 'trigger':
return `${key}_pre_poll`;
case 'create':
return `${key}_pre_write`;
case 'search':
return `${key}_pre_search`;
default:
return null;
}
};
// Get the operation URL from the legacy app config.
const getLegacyOperationUrl = (compiledApp, typeOf, key) => {
const pluralType =
typeOf === 'trigger'
? 'triggers'
: typeOf === 'create'
? 'creates'
: typeOf === 'search'
? 'searches'
: null;
if (!pluralType || !key) {
return '';
}
const legacy = compiledApp.legacy || {};
return (
(legacy[pluralType] &&
legacy[pluralType][key] &&
legacy[pluralType][key].operation &&
legacy[pluralType][key].operation.url) ||
''
);
};
// Build a `legacyScripting` object suitable for stubZ that mirrors what
// production's legacy-scripting-runner provides for `z.legacyScripting`:
// - `beforeRequest(req, z, bundle)` applies the legacy auth mapping.
// - `afterResponse` is a no-op.
// - `run(bundle, typeOf, key)` builds the operation URL, applies legacy
// auth, optionally runs the pre method (e.g., `<key>_pre_poll`), and
// delegates to `requestFn` to actually capture/send the request.
//
// `requestFn` is called as `requestFn(request)` and should return a
// response-shaped object.
const buildLegacyScripting = (compiledApp, requestFn, cachedZap) => {
const Zap = cachedZap !== undefined ? cachedZap : loadLegacyZap(compiledApp);
const legacyBeforeRequest = createLegacyBeforeRequest(compiledApp);
return {
beforeRequest: legacyBeforeRequest,
afterResponse: (response) => response,
run: async (bundle, typeOf, key) => {
let request = {
url: getLegacyOperationUrl(compiledApp, typeOf, key),
headers: {
Accept: 'application/json',
'Content-Type': 'application/json; charset=utf-8',
},
params: {},
body: {},
};
// Apply legacy auth middleware (adds Bearer token, etc.)
request = legacyBeforeRequest(request, null, bundle);
bundle.request = request;
// Run the pre method (e.g., Zap.newForm_pre_poll) if it exists.
// Wrap in try/catch — pre methods may crash on placeholder data
// (e.g., accessing env vars or bundle fields that don't exist).
if (Zap && key) {
const preMethodName = getLegacyPreMethodName(typeOf, key);
const preMethod = preMethodName ? Zap[preMethodName] : null;
if (preMethod) {
try {
const legacyBundle = {
...bundle,
auth_fields: bundle.authData || {},
request: { ...request },
};
const modified = await preMethod(legacyBundle);
if (modified) {
request = { ...request, ...modified };
}
} catch {
// Pre method failed — continue with base request
}
}
}
return requestFn(request);
},
};
};
module.exports = {
buildLegacyScripting,
createLegacyBeforeRequest,
loadLegacyZap,
};

View file

@ -0,0 +1,354 @@
'use strict';
const applyMiddleware = require('../middleware');
const ensureArray = require('../tools/ensure-array');
const { createRequestOptions } = require('../tools/request-sugar');
// before middlewares
const addBasicAuthHeader = require('../http-middlewares/before/add-basic-auth-header');
const addQueryParams = require('../http-middlewares/before/add-query-params');
const createInjectInputMiddleware = require('../http-middlewares/before/inject-input');
const prepareRequest = require('../http-middlewares/before/prepare-request');
const oauth1SignRequest = require('../http-middlewares/before/oauth1-sign-request');
const sanitizeHeaders = require('../http-middlewares/before/sanatize-headers');
const { REPLACE_CURLIES } = require('../constants');
const { withHttpCapture } = require('./http-capture');
const { buildLegacyScripting } = require('./legacy-scripting');
/**
* Extracts auth-contributed fields from the captured request by diffing
* against the defaults that requestMerge adds.
*/
const extractTemplate = (capturedReq) => {
const template = {};
if (capturedReq.headers) {
const headers = { ...capturedReq.headers };
delete headers['user-agent'];
if (Object.keys(headers).length > 0) {
template.headers = headers;
}
}
if (capturedReq.params && Object.keys(capturedReq.params).length > 0) {
template.params = capturedReq.params;
}
if (capturedReq.body) {
template.body = capturedReq.body;
}
return template;
};
/**
* Render auth fields from authentication.test by replacing
* {{bundle.authData.*}} placeholders with real credential values.
*/
const renderFromTest = (test, authData) => {
const resolve = (value) => {
if (typeof value !== 'string') {
return value;
}
return value.replace(/\{\{bundle\.authData\.(\w+)\}\}/g, (match, key) =>
authData[key] !== undefined ? authData[key] : match,
);
};
const template = {};
if (test.headers && typeof test.headers === 'object') {
const headers = {};
for (const [key, value] of Object.entries(test.headers)) {
const resolved = resolve(value);
if (resolved !== value) {
headers[key] = resolved;
}
}
if (Object.keys(headers).length > 0) {
template.headers = headers;
}
}
if (test.params && typeof test.params === 'object') {
const params = {};
for (const [key, value] of Object.entries(test.params)) {
const resolved = resolve(value);
if (resolved !== value) {
params[key] = resolved;
}
}
if (Object.keys(params).length > 0) {
template.params = params;
}
}
return template;
};
/**
* renderAuthTemplate command handler.
*
* Runs the real HTTP middleware pipeline with actual credentials from
* bundle.authData to produce a fully-rendered request.
*/
const renderAuthTemplate = async (compiledApp, input) => {
const auth = compiledApp.authentication;
if (!auth) {
return { authType: null, template: {} };
}
const authType = auth.type;
const eventBundle = input._zapier.event.bundle || {};
// Ensure all bundle fields are populated so middleware doesn't crash
// accessing e.g. bundle.inputData.someField or bundle.meta.isLoadingSample
//
// targetRequest carries the partner request being proxied. Signature-based
// auth (AWS SigV4, OAuth1, HMAC) needs the real url/method/body to compute
// a valid signature; absent it, the middleware runs against a stub and
// produces a signature for the wrong request.
//
// customRequestProperties carries integration-internal middleware flags
// (e.g. `withUserToken: true`) that control which credential the
// integration's beforeRequest injects. They are spread onto the synthetic
// request below — kept separate from targetRequest because they are not
// HTTP fields and must not participate in signature computation.
const bundle = {
authData: eventBundle.authData || {},
inputData: eventBundle.inputData || {},
meta: eventBundle.meta || {},
subscribeData: eventBundle.subscribeData || {},
targetUrl: eventBundle.targetUrl || '',
targetRequest: eventBundle.targetRequest || null,
customRequestProperties: eventBundle.customRequestProperties || {},
};
// Rebuild input with the fully-populated bundle so prepareRequest
// (which reads from input._zapier.event.bundle) sees the defaults
const safeInput = {
_zapier: {
...input._zapier,
event: {
...input._zapier.event,
bundle,
},
},
};
// beforeRequest can be either a single function or an array; normalize
// once and reuse below. (Function.length is the arity, not "is defined".)
const beforeRequest = ensureArray(compiledApp.beforeRequest);
// Build the same before-middleware chain as createAppRequestClient
const httpBefores = [
createInjectInputMiddleware(safeInput),
prepareRequest,
...beforeRequest,
];
if (authType === 'basic') {
httpBefores.push(addBasicAuthHeader);
} else if (authType === 'digest') {
// Digest auth requires a challenge-response — can't capture statically,
// but we still run the middleware for renderAuthTemplate.
// For now, skip digest middleware since it makes a real HTTP request.
} else if (authType === 'oauth1') {
httpBefores.push(oauth1SignRequest);
}
httpBefores.push(sanitizeHeaders);
httpBefores.push(addQueryParams);
let capturedReq = null;
const captureFunction = (preparedReq) => {
capturedReq = preparedReq;
return Promise.resolve({
status: 200,
headers: {},
getHeader: () => undefined,
content: '{}',
data: {},
request: preparedReq,
});
};
// Real z.request that hits the network. We have real authData here, so
// beforeRequest that sends a request (e.g., refresh-token) and uses the
// response in subsequent requests can produce the correct rendered template.
// Goes directly to fetch instead of recursing through our middleware
// pipeline — that would re-invoke beforeRequest for every sub-request and
// likely loop. Real refresh-token-style middlewares call separate endpoints
// that don't need the same auth applied.
const realRequest = async (reqOrUrl, options) => {
const req = { ...createRequestOptions(reqOrUrl, options) };
const response = await fetch(req.url, {
method: req.method || 'GET',
headers: req.headers,
body: req.body,
});
const content = await response.text();
let data = {};
try {
data = JSON.parse(content);
} catch {
// response is not JSON — leave data empty
}
return {
status: response.status,
headers: Object.fromEntries(response.headers.entries()),
getHeader: (name) => response.headers.get(name),
content,
data,
throwForStatus: () => {
if (response.status >= 400) {
throw new Error(`Got ${response.status} ${response.statusText}`);
}
},
};
};
const stubZ = {
console: { log: () => {}, error: () => {}, warn: () => {} },
errors: require('../errors'),
JSON: { parse: JSON.parse, stringify: JSON.stringify },
request: realRequest,
};
// Apps whose `beforeRequest` calls `z.legacyScripting.beforeRequest(...)`
// need this stub. It applies the same auth-mapping logic getAuthTemplate
// uses; for legacy basic-auth apps the Authorization header has already
// been added by addBasicAuthHeader, so this is effectively a no-op there.
stubZ.legacyScripting = buildLegacyScripting(compiledApp, realRequest);
const client = applyMiddleware(httpBefores, [], captureFunction, {
skipEnvelope: true,
extraArgs: [stubZ, bundle],
});
const target = bundle.targetRequest || {};
try {
// customRequestProperties is spread first so the explicit HTTP fields
// below always win — registry-supplied flags must never clobber the
// request being signed.
await client({
...bundle.customRequestProperties,
url: target.url || 'https://example.com',
method: target.method || 'GET',
headers: target.headers || {},
params: target.params || {},
body: target.body,
merge: true,
[REPLACE_CURLIES]: true,
});
} catch (err) {
return {
authType,
error: err.message,
template: {},
};
}
if (!capturedReq) {
return {
authType,
template: {},
};
}
const template = extractTemplate(capturedReq);
// Inline auth fallback: if pipeline produced an empty template and
// there's no middleware/requestTemplate, render from authentication.test
const hasBeforeRequest = beforeRequest.length > 0;
const hasRequestTemplate =
compiledApp.requestTemplate &&
Object.keys(compiledApp.requestTemplate).length > 0;
if (
!hasBeforeRequest &&
!hasRequestTemplate &&
Object.keys(template).length === 0 &&
auth.test &&
typeof auth.test !== 'function'
) {
const rendered = renderFromTest(auth.test, bundle.authData);
if (Object.keys(rendered).length > 0) {
return { authType, template: rendered };
}
}
// Inline auth + auth.test is a function: the pipeline can't see the
// auth pattern because it lives inside operation.perform / auth.test
// request configs. Invoke the test function with real authData,
// capturing the prepared request via z.request and falling back to
// raw http/fetch interception.
if (
!hasRequestTemplate &&
Object.keys(template).length === 0 &&
typeof auth.test === 'function'
) {
let testCapturedReq = null;
const testCapture = (preparedReq) => {
if (!testCapturedReq) {
testCapturedReq = preparedReq;
}
return Promise.resolve({
status: 200,
headers: {},
getHeader: () => undefined,
content: '{}',
data: {},
request: preparedReq,
});
};
const testClient = applyMiddleware(httpBefores, [], testCapture, {
skipEnvelope: true,
extraArgs: [stubZ, bundle],
});
const testStubZ = {
...stubZ,
request: async (reqOrUrl, options) => {
const req = { ...createRequestOptions(reqOrUrl, options) };
const response = await testClient({
...req,
method: req.method || 'GET',
headers: req.headers || {},
params: req.params || {},
merge: true,
[REPLACE_CURLIES]: true,
});
return { ...response, throwForStatus: () => {}, json: {} };
},
};
const onRawHttp = (httpReq) => {
if (!testCapturedReq) {
testCapturedReq = httpReq;
}
};
try {
await withHttpCapture(onRawHttp, () => auth.test(testStubZ, bundle));
} catch (_err) {
// testFn may crash parsing the stub response — continue with
// whatever was captured before the crash.
}
if (testCapturedReq) {
const inlineTemplate = extractTemplate(testCapturedReq);
if (Object.keys(inlineTemplate).length > 0) {
return { authType, template: inlineTemplate };
}
}
}
return {
authType,
template,
};
};
module.exports = renderAuthTemplate;

View file

@ -0,0 +1,27 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isCreate = require('./is-create');
/*
Makes sure the results are all objects.
*/
const createIsObject = {
name: 'createIsObject',
shouldRun: isCreate,
run: (method, results) => {
if (!_.isPlainObject(results)) {
const repr = simpleTruncate(JSON.stringify(results), 50);
return [
`Got a non-object result, expected an object from create (${repr})`,
];
}
// assumes a single object not array is legit
return [];
},
};
module.exports = createIsObject;

View file

@ -0,0 +1,30 @@
'use strict';
const isInputOrOutputFields = (method) =>
method.endsWith('.operation.inputFields') ||
method.endsWith('.operation.outputFields');
const dynamicFieldsHaveKeys = {
name: 'dynamicFieldsHaveKeys',
shouldRun: isInputOrOutputFields,
run: (method, results) => {
const lastMethodPart = method.split('.').pop();
if (!Array.isArray(results)) {
const type = typeof results;
return [`${lastMethodPart} must be an array, got ${type}`];
}
const errors = [];
for (let i = 0; i < results.length; i++) {
const field = results[i];
if (!field || !field.key) {
errors.push(`${lastMethodPart}[${i}] is missing a key`);
}
}
return errors;
},
};
module.exports = dynamicFieldsHaveKeys;

View file

@ -0,0 +1,23 @@
'use strict';
/*
An example checker, we still have lots TODO:
* compare trigger, search, action result to resource.sample if available
* validate returned inputFields to schema
* etc...
*/
const exampleChecker = {
name: 'exampleChecker',
shouldRun: (method /*, bundle */) => {
return method && true;
},
run: (method, results) => {
if (results) {
// could return ['Bad thing!'];
return [];
}
return [];
},
};
module.exports = exampleChecker;

View file

@ -0,0 +1,23 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isFirehoseWebhook = require('./is-firehose-webhook');
/*
The firehoseWebhook performSubscriptionKeyList function should always return an array of objects.
*/
const firehoseSubscriptionIsArray = {
name: 'firehoseSubscriptionIsArray',
shouldRun: isFirehoseWebhook,
run: (method, results) => {
if (!_.isArray(results)) {
const repr = simpleTruncate(JSON.stringify(results), 50);
return [`Results must be an array, got: ${typeof results}, (${repr})`];
}
return [];
},
};
module.exports = firehoseSubscriptionIsArray;

View file

@ -0,0 +1,33 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isFirehoseWebhook = require('./is-firehose-webhook');
/*
Makes sure the results are all strings.
*/
const firehoseSubscriptionKeyIsString = {
name: 'firehoseSubscriptionKeyIsString',
shouldRun: isFirehoseWebhook,
run: (method, results) => {
if (!_.isArray(results)) {
return []; // firehose-is-array check will catch if not array
}
const nonStringResult = _.find(results, (result) => {
return !_.isString(result);
});
if (nonStringResult !== undefined) {
const repr = simpleTruncate(JSON.stringify(nonStringResult), 50);
return [
`Got a non-string result in the array, expected only strings (${repr})`,
];
}
return [];
},
};
module.exports = firehoseSubscriptionKeyIsString;

View file

@ -0,0 +1,17 @@
module.exports = {
createIsObject: require('./create-is-object'),
searchIsArrayOrEnvelope: require('./search-is-array-or-envelope'),
triggerIsArray: require('./trigger-is-array'),
triggerIsObject: require('./trigger-is-object'),
triggerHasUniquePrimary: require('./trigger-has-unique-primary'),
triggerHasId: require('./trigger-has-id'),
firehoseSubscriptionIsArray: require('./firehose_is_array'),
firehoseSubscriptionKeyIsString: require('./firehose_is_string'),
performBufferReturnType: require('./perform-buffer-return-type'),
dynamicFieldsHaveKeys: require('./dynamic-fields-have-keys'),
};

View file

@ -0,0 +1,11 @@
module.exports = (method) => {
// `method` will never start with "resources." in production.
// Seems only for testing.
return (
(method.startsWith('creates.') &&
(method.endsWith('.operation.perform') ||
method.endsWith('.operation.performBuffer'))) ||
(method.startsWith('resources.') &&
method.endsWith('.create.operation.perform'))
);
};

View file

@ -0,0 +1,3 @@
module.exports = (method) => {
return method === 'firehoseWebhooks.performSubscriptionKeyList';
};

View file

@ -0,0 +1,7 @@
module.exports = (method) => {
return (
(method.startsWith('searches.') && method.endsWith('.operation.perform')) ||
(method.startsWith('resources.') &&
method.endsWith('.search.operation.perform'))
);
};

View file

@ -0,0 +1,9 @@
module.exports = (method) => {
return (
// `method` will never start with "resources." in production.
// Seems only for testing.
(method.startsWith('triggers.') && method.endsWith('.operation.perform')) ||
(method.startsWith('resources.') &&
method.endsWith('.list.operation.perform'))
);
};

View file

@ -0,0 +1,65 @@
const _ = require('lodash');
const performBufferEchoesIds = {
name: 'performBufferReturnType',
shouldRun: (method, bundle) => {
return (
Array.isArray(bundle.buffer) &&
method.endsWith('.operation.performBuffer') &&
method.startsWith('creates.')
);
},
run: (method, results, compiledApp, bundle) => {
if (!_.isPlainObject(results)) {
// create-is-object should have caught this
return [];
}
const inputIds = bundle.buffer
.map((b) => {
return b && b.meta ? b.meta.id : null;
})
.filter((id) => id);
const outputIds = Object.keys(results);
const missingIds = inputIds.filter((id) => !outputIds.includes(id));
if (missingIds.length > 0) {
const LIMIT = 3;
let missingIdsStr = missingIds.slice(0, LIMIT).join(', ');
const remainingCount = missingIds.length - LIMIT;
if (remainingCount > 0) {
// Don't want to flood the user with too many IDs
missingIdsStr += `, and ${remainingCount} more`;
}
return [`Result object is missing these IDs as keys: ${missingIdsStr}`];
}
const errors = [];
for (const id of inputIds) {
const item = results[id];
if (!_.isPlainObject(item)) {
errors.push(`Result object member with ID '${id}' must be an object`);
} else if (
!_.isPlainObject(item.outputData) &&
typeof item.error !== 'string'
) {
errors.push(
`Result object member with ID '${id}' must have 'outputData' object or 'error' string`,
);
}
if (errors.length >= 4) {
// No need to flood the user with too many errors
break;
}
}
return errors;
},
};
module.exports = performBufferEchoesIds;

View file

@ -0,0 +1,61 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isSearch = require('./is-search');
const hasCanPaginate = (searchKey, compiledApp) => {
const canPaginate =
compiledApp?.searches?.[searchKey]?.operation?.canPaginate;
return canPaginate;
};
/*
Searches should return an array of objects,
or a response envelope like { results: [...], paging_token: '...' }
when canPaginate is true.
*/
const searchIsArrayOrEnvelope = {
name: 'searchIsArrayOrEnvelope',
shouldRun: isSearch,
run: (method, results, compiledApp) => {
const searchKey = method.split('.', 2)[1];
const truncatedResults = simpleTruncate(JSON.stringify(results), 50);
if (hasCanPaginate(searchKey, compiledApp)) {
// if paging is supported and results is an object (indicating pagination), it must have results and paging_token
if (_.isPlainObject(results)) {
if (!_.has(results, 'results') || !_.has(results, 'paging_token')) {
return [
`Paginated search results must be an object containing results and paging_token, got: ${truncatedResults}`,
];
}
if (
!_.isString(results.paging_token) &&
!_.isNull(results.paging_token) &&
!_.isUndefined(results.paging_token)
) {
return [
`"paging_token" must be a string or null or undefined, got: ${typeof results.paging_token}`,
];
}
// pass to array check below
results = results.results;
} else {
return [
`Paginated search results must be an object, got: ${typeof results}, (${truncatedResults})`,
];
}
}
if (!_.isArray(results)) {
return [
`Search results must be an array, got: ${typeof results}, (${truncatedResults})`,
];
}
return [];
},
};
module.exports = searchIsArrayOrEnvelope;

View file

@ -0,0 +1,66 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isTrigger = require('./is-trigger');
/*
Makes sure the results all have an ID in them.
*/
const triggerHasId = {
name: 'triggerHasId',
shouldRun: (method, bundle, compiledApp) => {
// Hooks will have a bundle.cleanedRequest and we don't need to check they've got an id
if (!isTrigger(method) || bundle.cleanedRequest) {
return false;
}
const triggerKey = method.split('.', 2)[1];
if (!triggerKey) {
// Unreachable, but just in case
return false;
}
const outputFields = _.get(compiledApp, [
'triggers',
triggerKey,
'operation',
'outputFields',
]);
if (!outputFields || !Array.isArray(outputFields)) {
return true;
}
// This check is only necessary if either:
// - field.primary not set for all fields
// - field.primary is set for `id` field
let hasPrimary = false;
for (const field of outputFields) {
if (!field) {
continue; // just in case
}
if (field.primary) {
if (field.key === 'id') {
return true;
} else {
hasPrimary = true;
}
}
}
return !hasPrimary;
},
run: (method, results) => {
const missingIdResult = _.find(results, (result) => {
return !result || _.isUndefined(result.id) || _.isNull(result.id);
});
if (missingIdResult) {
const repr = simpleTruncate(JSON.stringify(missingIdResult), 250);
return [`Got a result missing the "id" property (${repr})`];
}
return [];
},
};
module.exports = triggerHasId;

View file

@ -0,0 +1,119 @@
'use strict';
const _ = require('lodash');
const isTrigger = require('./is-trigger');
const getPreferredPrimaryKeys = (compiledApp, triggerKey) => {
const defaultPrimaryKeys = ['id'];
if (!triggerKey) {
return defaultPrimaryKeys;
}
const outputFields = _.get(compiledApp, [
'triggers',
triggerKey,
'operation',
'outputFields',
]);
if (!outputFields || !Array.isArray(outputFields)) {
return defaultPrimaryKeys;
}
const primaryKeys = outputFields
.filter((f) => f && f.primary && f.key)
.map((f) => f.key);
return primaryKeys.length > 0 ? primaryKeys : defaultPrimaryKeys;
};
const isPrimitive = (v) => {
return (
v === null ||
v === undefined ||
typeof v === 'string' ||
typeof v === 'number' ||
typeof v === 'boolean'
);
};
// Gets array v where v[i] === result[primaryKeys[i]] and stringifies v into a string.
// Throws TypeError if any of the values are not primitive.
const stringifyValuesFromPrimaryKeys = (result, primaryKeys) => {
const values = primaryKeys
.map((k, i) => {
let v = result[k];
if (v === undefined) {
// undefined is not a valid JSON value. Here we convert it to string so we will
// have `{"id":"undefined"}` in the error message rather than `{}`,
// which is confusing.
v = 'undefined';
}
if (!isPrimitive(v)) {
throw new TypeError(
`As part of primary key, field "${k}" must be a primitive (non-object like number or string)`,
);
}
return [k, v];
})
.reduce((acc, [k, v]) => {
acc[k] = v;
return acc;
}, {});
return JSON.stringify(values);
};
/*
Makes sure the primary keys are unique among the results
*/
const triggerHasUniquePrimary = {
name: 'triggerHasUniquePrimary',
shouldRun: isTrigger,
run: (method, results, compiledApp) => {
const triggerKey = method.split('.', 2)[1];
const primaryKeys = getPreferredPrimaryKeys(compiledApp, triggerKey);
const usingDefaultPrimary =
primaryKeys.length === 1 && primaryKeys[0] === 'id';
const idCount = {};
if (!Array.isArray(results)) {
// One item can't have duplicates
return [];
}
for (const result of results) {
if (!result) {
// this'll get caught elsewhere, but we don't want to blow up this check
continue;
}
let uniqueKey;
try {
uniqueKey = stringifyValuesFromPrimaryKeys(result, primaryKeys);
} catch (e) {
return [e.message];
}
const count = (idCount[uniqueKey] = (idCount[uniqueKey] || 0) + 1);
if (count > 1) {
if (usingDefaultPrimary && uniqueKey === '{"id":"undefined"}') {
// This is for backward compatibility. By default, `id` is used as
// primary key. But if `results` have no `id` field, let it pass here
// and the other check trigger-has-id will catch it.
continue;
}
return [
`Got two or more results with primary key of \`${uniqueKey}\`, primary key should be unique`,
];
}
}
return [];
},
};
module.exports = triggerHasUniquePrimary;

View file

@ -0,0 +1,23 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isTrigger = require('./is-trigger');
/*
Triggers should always return an array of objects.
*/
const triggerIsArray = {
name: 'triggerIsArray',
shouldRun: isTrigger,
run: (method, results) => {
if (!_.isArray(results)) {
const repr = simpleTruncate(JSON.stringify(results), 50);
return [`Results must be an array, got: ${typeof results}, (${repr})`];
}
return [];
},
};
module.exports = triggerIsArray;

View file

@ -0,0 +1,33 @@
'use strict';
const _ = require('lodash');
const { simpleTruncate } = require('../tools/data');
const isTrigger = require('./is-trigger');
/*
Makes sure the results are all objects.
*/
const triggerIsObject = {
name: 'triggerIsObject',
shouldRun: isTrigger,
run: (method, results) => {
if (!_.isArray(results)) {
return []; // trigger-is-array check will catch if not array
}
const nonObjectResult = _.find(results, (result) => {
return !_.isPlainObject(result);
});
if (nonObjectResult !== undefined) {
const repr = simpleTruncate(JSON.stringify(nonObjectResult), 50);
return [
`Got a non-object result in the array, expected only objects (${repr})`,
];
}
return [];
},
};
module.exports = triggerIsObject;

View file

@ -0,0 +1,82 @@
'use strict';
const _processArgs = process.argv.join(' ');
const IS_TESTING =
_processArgs.indexOf('mocha') > 0 ||
_processArgs.indexOf('jest') > 0 ||
(process.env.NODE_ENV || '').startsWith('test');
const KILL_MIN_LIMIT = 250;
const KILL_MAX_LIMIT = 450 * 1000 * 1000;
const RESPONSE_SIZE_LIMIT = 6291456;
const UPLOAD_MAX_SIZE = 1000 * 1000 * 1000 * 1; // 1GB, in zapier backend too
const NON_STREAM_UPLOAD_MAX_SIZE = 1000 * 1000 * 150;
const ENCODED_FILENAME_MAX_LENGTH = 1000; // 1KB - S3 Metadata max is 2048
const HYDRATE_DIRECTIVE_HOIST = '$HOIST$';
const RENDER_ONLY_METHODS = [
'authentication.oauth2Config.authorizeUrl',
'authentication.oauth1Config.authorizeUrl',
];
const REPLACE_CURLIES = Symbol('replaceCurlies');
const REQUEST_OBJECT_SHORTHAND_OPTIONS = {
[REPLACE_CURLIES]: true,
};
const DEFAULT_LOGGING_HTTP_ENDPOINT = 'https://httplogger.zapier.com/input';
const DEFAULT_LOGGING_HTTP_API_KEY = 'R24hzu86v3jntwtX2DtYECeWAB'; // It's ok, this isn't PROD
const SAFE_LOG_KEYS = [
'account_id',
'api_title',
'app_cli_id',
'app_cli_title',
'app_cli_version',
'app_cli_version_id',
'customuser_id',
'facility',
'object_action',
'object_id',
'object_root_id',
'object_type',
'request_method',
'request_type',
'response_status_code',
'selected_api',
'timestamp',
'trigger_subscription_id',
];
const STATUSES = {
CALLBACK: 'CALLBACK',
SUCCESS: 'SUCCESS',
};
const packageJson = require('../package.json');
const PACKAGE_NAME = packageJson.name;
const PACKAGE_VERSION = packageJson.version;
module.exports = {
DEFAULT_LOGGING_HTTP_API_KEY,
DEFAULT_LOGGING_HTTP_ENDPOINT,
ENCODED_FILENAME_MAX_LENGTH,
HYDRATE_DIRECTIVE_HOIST,
IS_TESTING,
KILL_MAX_LIMIT,
KILL_MIN_LIMIT,
NON_STREAM_UPLOAD_MAX_SIZE,
PACKAGE_NAME,
PACKAGE_VERSION,
RENDER_ONLY_METHODS,
REPLACE_CURLIES,
REQUEST_OBJECT_SHORTHAND_OPTIONS,
RESPONSE_SIZE_LIMIT,
SAFE_LOG_KEYS,
STATUSES,
UPLOAD_MAX_SIZE,
};

View file

@ -0,0 +1,50 @@
'use strict';
const applyMiddleware = require('./middleware');
const ensureArray = require('./tools/ensure-array');
const schemaTools = require('./tools/schema');
// before middles
const injectZObject = require('./app-middlewares/before/z-object');
const addAppContext = require('./app-middlewares/before/add-app-context');
const fetchStashedBundle = require('./app-middlewares/before/fetch-stashed-bundle');
// after middles
const checkOutput = require('./app-middlewares/after/checks');
const largeResponseCachePointer = require('./app-middlewares/after/large-response-cacher');
const callbackStatusCatcher = require('./app-middlewares/after/callback-status-catcher');
const waitForPromises = require('./app-middlewares/after/wait-for-promises');
const createCommandHandler = require('./create-command-handler');
/*
Create a z-app from an app definition.
Applies standard middlewares that we want on every z-app, but
caller can supply custom before and after middlewares.
*/
const createApp = (appRaw) => {
const frozenCompiledApp = schemaTools.prepareApp(appRaw);
// standard before middlewares
const befores = [
fetchStashedBundle,
addAppContext,
injectZObject,
...ensureArray(frozenCompiledApp.beforeApp),
];
// standard after middlewares
const afters = [
checkOutput,
largeResponseCachePointer,
waitForPromises,
callbackStatusCatcher,
...ensureArray(frozenCompiledApp.afterApp),
];
const app = createCommandHandler(frozenCompiledApp);
return applyMiddleware(befores, afters, app);
};
module.exports = createApp;

View file

@ -0,0 +1,40 @@
'use strict';
const schemaTools = require('./tools/schema');
const execute = require('./execute');
const executeRequest = require('./execute-request');
const getAuthTemplate = require('./auth-template/get-auth-template');
const renderAuthTemplate = require('./auth-template/render-auth-template');
const { handleError } = require('./errors');
const commandHandlers = {
execute,
validate: schemaTools.validateApp,
definition: schemaTools.serializeApp,
request: (app, input) => executeRequest(input),
getAuthTemplate,
renderAuthTemplate,
};
/*
Creates middleware app that can process z-app app definitions, handling
commands like 'execute', 'validate', 'definition', 'request'.
*/
const createCommandHandler = (compiledApp) => {
return async (input) => {
const command = input._zapier.event.command || 'execute'; // validate || definition || request
const handler = commandHandlers[command];
if (!handler) {
throw new Error(`Unexpected command ${command}`);
}
try {
return await handler(compiledApp, input);
} catch (err) {
return handleError(err);
}
};
};
module.exports = createCommandHandler;

View file

@ -0,0 +1,117 @@
'use strict';
const util = require('util');
const _ = require('lodash');
class AppError extends Error {
constructor(message, code, status) {
super(
JSON.stringify({
message,
code,
status,
}),
);
this.name = 'AppError';
this.doNotContextify = true;
}
}
class ResponseError extends Error {
constructor(response) {
let content;
try {
content = response.content;
} catch (err) {
// Stream request (z.request({raw: true})) doesn't have response.content
content = null;
}
super(
JSON.stringify({
status: response.status,
headers: {
'content-type': response.headers.get('content-type'),
'retry-after': response.headers.get('retry-after'),
},
content,
request: {
url: response.request.url,
},
}),
);
this.name = 'ResponseError';
this.doNotContextify = true;
}
}
class ThrottledError extends Error {
constructor(message, delay) {
super(
JSON.stringify({
message,
delay,
}),
);
this.name = 'ThrottledError';
this.doNotContextify = true;
}
}
// Make some of the errors we'll use!
const createError = (name) => {
const NewError = function (message = '') {
this.name = name;
this.message = message;
Error.call(this);
Error.captureStackTrace(this, this.constructor);
};
util.inherits(NewError, Error);
return NewError;
};
const names = [
'CheckError',
'DehydrateError',
'ExpiredAuthError',
'HaltedError',
'MethodDoesNotExist',
'NotImplementedError',
'RefreshAuthError',
'RequireModuleError',
'StashedBundleError',
'StopRequestError',
];
const exceptions = _.reduce(
names,
(col, name) => {
col[name] = createError(name);
return col;
},
{
Error: AppError,
ResponseError,
ThrottledError,
},
);
const isRequireError = ({ name, message }) =>
name === 'ReferenceError' && message === 'require is not defined';
const handleError = (...args) => {
const [error] = args;
const { RequireModuleError } = exceptions;
if (isRequireError(error)) {
throw new RequireModuleError(
'For technical reasons, use z.require() instead of require().',
);
}
throw error;
};
module.exports = {
...exceptions,
handleError,
};

View file

@ -0,0 +1,16 @@
'use strict';
const _ = require('lodash');
const responseCleaner = require('./tools/response-cleaner');
const executeRequest = (input) => {
const bundle = input._zapier.event.bundle || {};
const options = _.extend({}, bundle.request || {});
if (!options.url) {
throw new Error('Missing url for request');
}
return input.z.request(options).then(responseCleaner);
};
module.exports = executeRequest;

View file

@ -0,0 +1,98 @@
'use strict';
const _ = require('lodash');
const addQueryParams = require('./http-middlewares/before/add-query-params');
const ensureArray = require('./tools/ensure-array');
const injectInput = require('./http-middlewares/before/inject-input');
const prepareRequest = require('./http-middlewares/before/prepare-request');
const constants = require('./constants');
const executeHttpRequest = (input, options) => {
options = {
// shorthand requests should always throw _unless_ the object specifically opts out
// this covers godzilla devs who use shorthand requests (most of them) that rely on the throwing behavior
// when we set the app-wide skip for everyone, we don't want their behavior to change
// so, this line takes precedence over the global setting, but not the local one (`options`)
skipThrowForStatus: false,
...options,
...constants.REQUEST_OBJECT_SHORTHAND_OPTIONS,
};
return input.z.request(options).then((response) => {
if (response.data === undefined) {
throw new Error(
'Response needs to be JSON, form-urlencoded or parsed in middleware.',
);
}
return response.data;
});
};
const executeInputOutputFields = (inputOutputFields, input) => {
inputOutputFields = ensureArray(inputOutputFields);
return Promise.all(
inputOutputFields.map((field) =>
_.isFunction(field) ? field(input.z, input.bundle) : field,
),
).then((fields) => _.flatten(fields));
};
const executeCallbackMethod = (z, bundle, method) => {
return new Promise((resolve, reject) => {
const callback = (err, output) => {
if (err) {
reject(err);
} else {
resolve(output);
}
};
method(z, bundle, callback);
});
};
const isInputOutputFields = (methodName) =>
methodName.match(/\.(inputFields|outputFields)$/);
const isRenderOnly = (methodName) =>
_.indexOf(constants.RENDER_ONLY_METHODS, methodName) >= 0;
const execute = (app, input) => {
const z = input.z;
const methodName = input._zapier.event.method;
const method = _.get(app, methodName);
const bundle = input._zapier.event.bundle || {};
if (isInputOutputFields(methodName)) {
return executeInputOutputFields(method, input);
} else if (_.isFunction(method)) {
// TODO: would be nice to be a bit smarter about this
// either by only setting props we know are used or by
// moving this safing code into before middleware
bundle.authData = bundle.authData || {};
bundle.inputData = bundle.inputData || {};
if (method.length >= 3) {
return executeCallbackMethod(input.z, bundle, method);
}
return method(z, bundle);
} else if (_.isObject(method) && method.url) {
const options = method;
if (isRenderOnly(methodName)) {
const requestWithInput = {
...injectInput(input)(options),
...constants.REQUEST_OBJECT_SHORTHAND_OPTIONS,
};
const preparedRequest = addQueryParams(prepareRequest(requestWithInput));
return preparedRequest.url;
}
return executeHttpRequest(input, options);
} else {
throw new Error(
`Error: Could not find the method to call: ${input._zapier.event.method}`,
);
}
};
module.exports = execute;

View file

@ -0,0 +1,80 @@
'use strict';
const stream = require('stream');
// Prepare a request/reponse to be logged to the backend.
// Generally respects the "Zapier" request and resp object format.
const prepareRequestLog = (req, resp) => {
req = req || {};
resp = resp || {};
let responseBody;
if (!req.raw) {
if (typeof resp.content !== 'string') {
responseBody = JSON.stringify(resp.content);
} else {
responseBody = resp.content;
}
} else {
responseBody = '<probably streaming data>';
}
let requestBody = req.body;
if (requestBody instanceof stream) {
// Avoid JSON.stringify form data or any streams
requestBody = '<streaming data>';
}
const data = {
log_type: 'http',
request_type: 'devplatform-outbound',
request_url: req.url,
request_method: req.method || 'GET',
request_headers: req.headers,
request_data: requestBody,
request_via_client: true,
response_status_code: resp.status,
response_headers: resp.headers,
response_content: responseBody,
};
if (req._requestStart) {
data.request_duration_ms = new Date() - req._requestStart;
}
if (req.url && req.url.indexOf('?') !== -1) {
data.request_url = req.url.split('?')[0];
data.request_params = req.url.split('?').slice(1).join('?');
}
return {
message: `${data.response_status_code} ${data.request_method} ${data.request_url}`,
data,
};
};
/*
Log a response and it's original request to our logger.
*/
const logResponse = (resp) => {
const logger = resp.request.input._zapier.logger;
const logs = prepareRequestLog(resp.request, resp);
let infoMsg = `Received ${resp.status} code from ${resp.request.url}`;
if (logs.data.request_duration_ms) {
infoMsg += ` after ${logs.data.request_duration_ms}ms`;
}
const whatHappened = resp.request.input._zapier.whatHappened;
whatHappened.push(infoMsg);
whatHappened.push(
`Received content "${String(logs.data.response_content).substr(0, 100)}"`,
);
// steamroll any results/errors with org response!
return logger(logs.message, logs.data)
.then(() => resp)
.catch(() => resp);
};
module.exports = { logResponse, prepareRequestLog };

View file

@ -0,0 +1,27 @@
// prepare headers object - plain object for serialization later
const plainHeaders = (headers) => {
const _headers = {};
headers.forEach((value, name) => {
_headers[name] = value;
});
return _headers;
};
// Return the normal resp.headers, but with more goodies (toJSON support).
const replaceHeaders = (resp) => {
const getHeader = (name) => resp.headers.get(name);
Object.defineProperty(resp.headers, 'toJSON', {
enumerable: false,
value: () => plainHeaders(resp.headers),
});
return {
headers: resp.headers,
getHeader,
};
};
module.exports = {
replaceHeaders,
};

View file

@ -0,0 +1,113 @@
'use strict';
const _ = require('lodash');
const querystring = require('querystring');
const { scrub, findSensitiveValues } = require('@zapier/secret-scrubber');
const { replaceHeaders } = require('./middleware-utils');
const { FORM_TYPE } = require('../../tools/http');
const errors = require('../../errors');
const {
findSensitiveValuesFromAuthData,
} = require('../../tools/secret-scrubber');
const buildSensitiveValues = (bundle) => {
const authData = bundle?.authData || {};
const result = [
...findSensitiveValuesFromAuthData(authData),
...findSensitiveValues(process.env),
];
return [...new Set(result)];
};
const _throwForStatus = (response, bundle) => {
// calling this always throws, regardless of the skipThrowForStatus value
// eslint-disable-next-line yoda
if (400 <= response.status && response.status < 600) {
// Create a cleaned version of the response to avoid sensitive data leaks
try {
// Find sensitive values from environment variables and bundle authData
const sensitiveValues = buildSensitiveValues(bundle);
if (sensitiveValues.length > 0 && response?.request?.url) {
response.request.url = scrub(response.request.url, sensitiveValues);
}
} catch (err) {
// don't fail the whole request if we can't scrub for some reason
}
throw new errors.ResponseError(response);
}
};
const prepareRawResponse = (resp, request, bundle) => {
// TODO: if !2xx should we go ahead and get response.content for them?
// retain the response signature for raw control
const extendedResp = {
request,
skipThrowForStatus: request.skipThrowForStatus,
};
const outResp = _.extend(resp, extendedResp, replaceHeaders(resp));
outResp.throwForStatus = () => {
_throwForStatus(outResp, bundle);
};
Object.defineProperty(outResp, 'content', {
get: function () {
throw new Error(
'You passed {raw: true} in request() - the response.content property is not ' +
'available! Try response.body.pipe() for streaming, response.buffer() for a ' +
'buffer, or response.text() for string.',
);
},
});
return outResp;
};
const prepareContentResponse = async (resp, request, bundle) => {
// TODO: does it make sense to not trim the signature? more equivalence to raw...
const content = await resp.text();
// trim down the response signature a ton for simplicity
const preppedResp = {
url: resp.url,
status: resp.status,
redirected: resp.redirected,
json: undefined,
data: undefined,
content,
request,
// only controls if _we_ call throwForStatus automatically
skipThrowForStatus: request.skipThrowForStatus,
};
const outResp = _.extend(preppedResp, replaceHeaders(resp));
try {
if (outResp.headers.get('content-type') === FORM_TYPE) {
outResp.data = querystring.parse(content);
} else {
outResp.data = JSON.parse(content);
outResp.json = JSON.parse(content); // DEPRECATED (not using reference to isolate)
}
} catch (_e) {}
outResp.throwForStatus = () => {
_throwForStatus(outResp, bundle);
};
return outResp;
};
// Provide a standardized plain JS responseObj for common consumption, or raw response for streaming.
const prepareResponse = (resp, z, bundle) => {
const request = resp.input;
delete resp.input;
const responseFunc = request.raw
? prepareRawResponse
: prepareContentResponse;
return responseFunc(resp, request, bundle);
};
module.exports = prepareResponse;

View file

@ -0,0 +1,36 @@
'use strict';
const { Error } = require('../../errors');
const disallowedRedirectHosts = [
// Loopback addresses (IPv4)
'localhost',
'127.0.0.1',
// Loopback addresses (IPv6)
'::1',
'[::1]',
];
function isDisallowedAfterRedirect(url) {
try {
const { hostname } = new URL(url);
return disallowedRedirectHosts.includes(hostname);
} catch (e) {
// If URL parsing fails, consider it allowed
// (being permissive just in case it affects backwards compatibility)
return false;
}
}
const throwForDisallowedHostnameAfterRedirect = (resp) => {
// Looking at the response URL instead of the request URL
// because the response URL can change after a redirect
if (resp.redirected && isDisallowedAfterRedirect(resp.url)) {
throw new Error('Redirecting to disallowed hostname');
}
return resp;
};
module.exports = throwForDisallowedHostnameAfterRedirect;

View file

@ -0,0 +1,16 @@
'use strict';
const { RefreshAuthError } = require('../../errors');
/**
* Raise a RefreshAuthError _before_ any other error handling happens. Behaves more closely to the 9.x behavior rather than 10.x
*/
const throwForStaleAuth = (resp) => {
if (resp.status === 401) {
throw new RefreshAuthError();
}
return resp;
};
module.exports = throwForStaleAuth;

View file

@ -0,0 +1,10 @@
'use strict';
const throwForStatusMiddleware = (response) => {
if (!response.skipThrowForStatus) {
response.throwForStatus();
}
return response;
};
module.exports = throwForStatusMiddleware;

View file

@ -0,0 +1,33 @@
'use strict';
const { ThrottledError } = require('../../errors');
/**
* Raise a ThrottledError for 429 responses _before_ dev's afterResponse middleware,
* unless throwForThrottlingEarly is set to true on the request.
* Behaves similarly to throwForStaleAuth but for throttling.
*/
const throwForThrottling = (resp) => {
// throwForThrottlingEarly has to be explicitly set to false to disable this
// middleware. By default, when it's undefined or null, we want this
// middleware to run.
if (resp.request?.throwForThrottlingEarly === false) {
return resp;
}
if (resp.status === 429) {
const retryAfter = resp.headers.get('retry-after');
let delay = retryAfter ? parseInt(retryAfter, 10) : null;
if (Number.isNaN(delay)) {
delay = null;
}
throw new ThrottledError(
'The server returned 429 (Too Many Requests)',
delay,
);
}
return resp;
};
module.exports = throwForThrottling;

View file

@ -0,0 +1,26 @@
'use strict';
// Computes the basic auth header for the request
const addBasicAuthHeader = (req, z, bundle) => {
if (
bundle.authData &&
(bundle.authData.username || bundle.authData.password)
) {
const username = bundle.authData.username || '';
const password = bundle.authData.password || '';
const buff = Buffer.from(`${username}:${password}`, 'utf8');
const header = 'Basic ' + buff.toString('base64');
if (req.headers) {
req.headers.Authorization = header;
} else {
req.headers = {
Authorization: header,
};
}
}
return req;
};
module.exports = addBasicAuthHeader;

View file

@ -0,0 +1,87 @@
'use strict';
const fetch = require('node-fetch');
const { NotImplementedError } = require('../../errors');
const { md5 } = require('../../tools/hashing');
const { parseDictHeader } = require('../../tools/http');
const buildDigestHeader = (username, password, url, method, creds) => {
if (creds.algorithm && creds.algorithm.toUpperCase() !== 'MD5') {
throw new NotImplementedError(
"algorithm 'MD5-SESS' and 'SHA' are not implemented yet",
);
}
const path = new URL(url).pathname;
const HA1 = md5(`${username}:${creds.realm}:${password}`);
const HA2 = md5(`${method}:${path}`);
let response, cnonce;
if (!creds.qop) {
response = md5(`${HA1}:${creds.nonce}:${HA2}`);
} else if (
creds.qop === 'auth' ||
creds.qop.split(',').indexOf('auth') >= 0
) {
cnonce = md5(Date.now().toString()).substr(0, 16);
response = md5(`${HA1}:${creds.nonce}:00000001:${cnonce}:auth:${HA2}`);
} else {
throw new NotImplementedError(
"qop other than 'auth' is not implemented yet",
);
}
let base =
`username="${username}", realm="${creds.realm}", nonce="${creds.nonce}", ` +
`uri="${path}", response="${response}"`;
if (creds.opaque) {
base += `, opaque="${creds.opaque}"`;
}
if (creds.algorithm) {
base += `, algorithm="${creds.algorithm}"`;
}
if (creds.qop) {
base += `, qop="${creds.qop}"`;
}
if (cnonce) {
base += `, nc=00000001, cnonce="${cnonce}"`;
}
return `Digest ${base}`;
};
const addDigestAuthHeader = async (request, z, bundle) => {
// Send a request without any auth header. Expect 401 and get nonce and other
// necessary info from WWW-Authenticate header to do digest auth.
// TODO: May reuse nonce to save a request
const method = request.method || 'GET';
const res = await fetch(request.url, { method });
if (res.status === 401) {
let credstr = res.headers.get('www-authenticate');
if (credstr && credstr.startsWith('Digest ')) {
credstr = credstr.substr(7);
const creds = parseDictHeader(credstr);
request.headers = request.headers || {};
request.headers.Authorization = buildDigestHeader(
bundle.authData.username,
bundle.authData.password,
request.url,
method,
creds,
);
const cookie = res.headers.get('set-cookie');
if (cookie) {
request.headers.Cookie = res.headers.get('set-cookie');
}
}
}
return request;
};
module.exports = addDigestAuthHeader;

View file

@ -0,0 +1,47 @@
'use strict';
const querystring = require('querystring');
const { normalizeEmptyParamFields } = require('../../tools/cleaner');
const hasQueryParams = ({ params = {} }) => Object.keys(params).length;
// Take params off of req.params and append to url - "?a=1&b=2"".
// This middleware should run *after* custom middlewares, because
// custom middlewares might add params.
const addQueryParams = (req) => {
if (hasQueryParams(req)) {
const splitter = req.url.includes('?') ? '&' : '?';
normalizeEmptyParamFields(req);
let stringifiedParams = querystring.stringify(req.params, '&', '=', {
encodeURIComponent: req.encodeURIComponent,
});
// it goes against spec, but for compatibility, some APIs want certain
// characters (mostly $) unencoded
if (req.skipEncodingChars) {
for (let i = 0; i < req.skipEncodingChars.length; i++) {
const char = req.skipEncodingChars.charAt(i);
const valToReplace = querystring.escape(char);
if (valToReplace === char) {
continue;
}
// no replaceAll in JS yet, coming in a node version soon!
stringifiedParams = stringifiedParams.replace(
new RegExp(valToReplace, 'g'),
char,
);
}
}
if (stringifiedParams) {
req.url += `${splitter}${stringifiedParams}`;
}
}
delete req.params;
return req;
};
module.exports = addQueryParams;

View file

@ -0,0 +1,22 @@
'use strict';
const http = require('http');
const https = require('https');
const httpAgent = new http.Agent({ rejectUnauthorized: false });
const httpsAgent = new https.Agent({ rejectUnauthorized: false });
const disableSSLCertCheck = (req) => {
if (req.agent) {
req.agent.options.rejectUnauthorized = false;
} else if (req.url.startsWith('https://')) {
// Need to dynamically choose a different agent because redirection can be
// across HTTPS and HTTP.
// See https://github.com/node-fetch/node-fetch/tree/6ee9d318#custom-agent
req.agent = (parsedURL) =>
parsedURL.protocol === 'http:' ? httpAgent : httpsAgent;
}
return req;
};
module.exports = disableSSLCertCheck;

View file

@ -0,0 +1,15 @@
'use strict';
/*
Creates HTTP before middleware that adds some app context
to HTTP request options, including the app and event.
Useful for HTTP middlewares that need stuff from the app or event.
*/
const injectInput = (input) => {
return (req) => {
return { ...req, input };
};
};
module.exports = injectInput;

View file

@ -0,0 +1,97 @@
'use strict';
const crypto = require('crypto');
const urllib = require('url');
const querystring = require('querystring');
const _ = require('lodash');
const oauth = require('oauth-sign');
const { getContentType, FORM_TYPE } = require('../../tools/http');
const stripQueryFromUrl = (url) => {
const u = new urllib.URL(url);
return `${u.protocol}//${u.host}${u.pathname}`;
};
const collectAuthParams = (req) => {
const params = _.omit(req.auth, [
'oauth_consumer_secret',
'oauth_token_secret',
]);
_.defaults(params, {
oauth_version: '1.0A',
oauth_signature_method: 'HMAC-SHA1',
oauth_nonce: crypto.randomBytes(20).toString('hex'),
oauth_timestamp: Math.floor(Date.now() / 1000),
});
return params;
};
// Implements https://tools.ietf.org/html/rfc5849#section-3.4.1.3.1
const collectParamsForBaseString = (req, authParams) => {
const params = _.clone(authParams);
const makeArrayOnDupeKey = (objValue, srcValue) => {
if (Array.isArray(objValue)) {
if (Array.isArray(srcValue)) {
return objValue.concat(srcValue);
}
objValue.push(srcValue);
return objValue;
}
if (Array.isArray(srcValue)) {
return [objValue].concat(srcValue);
}
if (objValue === undefined) {
return srcValue;
}
return [objValue, srcValue];
};
_.extendWith(params, req.params, makeArrayOnDupeKey);
_.extendWith(
params,
querystring.parse(new urllib.URL(req.url).search.substr(1)),
makeArrayOnDupeKey,
);
if (req.body && getContentType(req.headers) === FORM_TYPE) {
_.extendWith(params, querystring.parse(req.body), makeArrayOnDupeKey);
}
return params;
};
const buildAuthorizationHeader = (params) => {
const paramList = _.map(
params,
(v, k) => `${oauth.rfc3986(k)}="${oauth.rfc3986(v)}"`,
);
return `OAuth ${paramList.join(',')}`;
};
const oauth1SignRequest = (req) => {
if (!_.isEmpty(req.auth)) {
const signMethod = req.auth.oauth_signature_method || 'HMAC-SHA1';
const authParams = collectAuthParams(req);
const paramsForBaseString = collectParamsForBaseString(req, authParams);
authParams.oauth_signature = oauth.sign(
signMethod,
req.method,
stripQueryFromUrl(req.url),
paramsForBaseString,
req.auth.oauth_consumer_secret,
req.auth.oauth_token_secret,
);
// Implements https://tools.ietf.org/html/rfc5849#section-3.5.1
req.headers.Authorization = buildAuthorizationHeader(authParams);
// TODO: Form-encoded body (section 3.5.2) and querystring (3.5.3)?
}
return req;
};
module.exports = oauth1SignRequest;

View file

@ -0,0 +1,209 @@
'use strict';
const _ = require('lodash');
const stream = require('stream');
const querystring = require('querystring');
const {
createBundleBank,
normalizeEmptyBodyFields,
recurseReplaceBank,
} = require('../../tools/cleaner');
const requestMerge = require('../../tools/request-merge');
const {
getContentType,
FORM_TYPE,
JSON_TYPE_UTF8,
} = require('../../tools/http');
const { REPLACE_CURLIES } = require('../../constants');
const isStream = (obj) => obj instanceof stream.Stream;
const isPromise = (obj) => obj && typeof obj.then === 'function';
const sugarBody = (req) => {
// move into the body as raw, set headers for coerce, merge to work
req.headers = req.headers || {};
if (!req.body && req.form) {
req.body = req.form;
req.headers['content-type'] = FORM_TYPE;
delete req.form;
}
if (!req.body && req.json) {
req.body = req.json;
req.headers['content-type'] = JSON_TYPE_UTF8;
delete req.json;
}
return req;
};
// Be careful not to JSONify a stream or buffer, stuff like that
const coerceBody = (req) => {
const contentType = getContentType(req.headers || {});
// No need for body on get
if (req.method === 'GET' && (!req.allowGetBody || _.isEmpty(req.body))) {
delete req.body;
}
// auto coerce form if header says so
if (contentType === FORM_TYPE && req.body && !_.isString(req.body)) {
normalizeEmptyBodyFields(req);
req.body = querystring.stringify(req.body).replace(/%20/g, '+');
}
if (isStream(req.body)) {
// leave a stream/pipe alone!
} else if (isPromise(req.body)) {
// leave a promise alone!
} else if (Buffer.isBuffer(req.body)) {
// leave a buffer alone!
} else if (req.body && !_.isString(req.body)) {
normalizeEmptyBodyFields(req);
// this is a general - popular fallback
req.body = JSON.stringify(req.body);
if (!contentType) {
req.headers['content-type'] = JSON_TYPE_UTF8;
}
}
return req;
};
// Wrap up the request in a promise - if needed.
const finalRequest = (req) => {
if (isPromise(req.body)) {
return req.body.then((reqBodyRes) => {
if (
reqBodyRes &&
reqBodyRes.body &&
typeof reqBodyRes.body.pipe === 'function'
) {
req.body = reqBodyRes.body;
} else if (
reqBodyRes &&
reqBodyRes.content &&
typeof reqBodyRes.content === 'string'
) {
req.body = reqBodyRes.content;
} else {
req.body = reqBodyRes;
// we could inspect response headers from reqBodyRes
// and apply content type to req - but maybe later
req = coerceBody(req);
}
return req;
});
} else {
return req;
}
};
const throwForCurlies = (value, path) => {
path = path || [];
if (typeof value === 'string') {
if (/{{\s*(bundle|process)\.[^}]*}}/.test(value)) {
throw new Error(
'z.request() no longer supports {{bundle.*}} or {{process.*}} as of v17 ' +
"unless it's used in a shorthand request defined by the integration. " +
'Zapier Customers: Remove "{{curly braces}}" from your request. ' +
'Developers: Use JavaScript template literals instead. ' +
`Value in violation: "${value}" in attribute "${path.join('.')}".`,
);
}
} else if (Array.isArray(value)) {
for (let i = 0; i < value.length; i++) {
const item = value[i];
throwForCurlies(item, [...path, String(i)]);
}
} else if (_.isPlainObject(value)) {
for (const [k, v] of Object.entries(value)) {
throwForCurlies(v, [...path, k]);
}
}
};
const prepareRequest = function (req) {
const input = req.input || {};
// We will want to use _.defaultsDeep if one of these nested values ever defaults to true.
req = _.defaults(req, {
merge: true,
removeMissingValuesFrom: {
params: false,
body: false,
},
// read default from app flags, but always defer to the request object if the value was set
skipThrowForStatus: _.get(
input,
['_zapier', 'app', 'flags', 'skipThrowForStatus'],
false,
),
throwForThrottlingEarly: _.get(
input,
['_zapier', 'app', 'flags', 'throwForThrottlingEarly'],
true,
),
});
req = sugarBody(req);
if (req[REPLACE_CURLIES] || req.merge) {
const bank = createBundleBank(
input?._zapier?.event || {},
req.serializeValueForCurlies,
);
const requestReplaceable = {
url: req.url,
headers: req.headers,
params: req.params,
body: req.body,
};
if (req[REPLACE_CURLIES]) {
// replace {{curlies}} in the request
req = {
...req,
...recurseReplaceBank(requestReplaceable, bank),
};
} else {
// throw if there's {{curlies}} in the request
throwForCurlies(requestReplaceable);
}
if (req.merge) {
// Always replace {{curlies}} in reqeustTemplate regardless of
// req[REPLACE_CURLIES]
const requestTemplate = input._zapier?.app?.requestTemplate || {};
const templateReplaceable = {
url: requestTemplate.url,
headers: requestTemplate.headers,
params: requestTemplate.params,
body: requestTemplate.body,
};
const renderedTemplate = recurseReplaceBank(templateReplaceable, bank);
// Apply app.requestTemplate to request
req = requestMerge(renderedTemplate, req);
}
}
req = coerceBody(req);
req._requestStart = new Date();
const whatHappened = req.input._zapier.whatHappened;
if (whatHappened) {
whatHappened.push(`Starting ${req.method} request to ${req.url}`);
}
return finalRequest(req);
};
module.exports = prepareRequest;

View file

@ -0,0 +1,14 @@
'use strict';
// Middleware to trim whitespace from header keys and values
const sanitizeHeaders = (req) => {
req.headers = Object.fromEntries(
Object.entries(req.headers || {}).map(([key, value]) => [
key.trim(),
typeof value === 'string' ? value.trim() : value,
]),
);
return req;
};
module.exports = sanitizeHeaders;

View file

@ -0,0 +1,21 @@
'use strict';
const createLambdaHandler = require('./tools/create-lambda-handler');
const createAppTester = require('./tools/create-app-tester');
const { consoleProxy } = require('./tools/console-singleton');
let _integrationTestHandler;
const integrationTestHandler = (event, context, callback) => {
const testAppPath = require.resolve('../test/userapp');
_integrationTestHandler =
_integrationTestHandler || createLambdaHandler(testAppPath);
return _integrationTestHandler(event, context, callback);
};
module.exports = {
createAppHandler: createLambdaHandler,
createAppTester,
integrationTestHandler,
console: consoleProxy,
...require('./typeHelpers'),
};

View file

@ -0,0 +1,111 @@
'use strict';
const _ = require('lodash');
const envelope = require('./tools/envelope');
/**
Applies before and after middleware functions, returning
a function that takes arguments and returns a promise that
returns a result.
A before middleware function looks like this:
(input) => { Promise.resolve(input); }
It takes a input object, and returns a promise that returns
the new input object, to pass down to the next middleware in the
chain. For an app middleware the input object would include
the input event, and any other meta information or utility objects.
For example here is a before middleware that adds a 'z'
property to the input:
(input) => {
input.z = {};
return Promise.resolve(input);
};
Before middleware can modify or clone the input input, and
subsqeuent middlewares will receive the new input.
After middleware takes a output object, which includes the results
from previous after middlewares. The output object also include the
input object returned by the before middlewares:
(output) => Promise.resolve(output)
options.skipEnvelope parameter controls whether or not the output object
should have an wrapper envelope that includes the input, or just return the raw
output. The default is false.
*/
const enrichErrorMessages = (error, input) => {
if (error.doNotContextify) {
throw error;
}
if (input._zapier && input._zapier.whatHappened) {
const details = input._zapier.whatHappened.map((f) => ` ${f}`).join('\n');
error.message = `${error.message}\nWhat happened:\n${details}\n ${error.message}`;
}
throw error;
};
const applyMiddleware = (befores, afters, app, options) => {
options = _.defaults({}, options, {
skipEnvelope: false,
extraArgs: [],
});
const ensureEnvelope = (maybeEnvelope) => {
if (!options.skipEnvelope) {
// they returned just the results; put them back in the envelope
return envelope.ensureOutputEnvelope(maybeEnvelope);
}
return maybeEnvelope;
};
return (input) => {
const beforeMiddleware = async (beforeInput) => {
let newInput = beforeInput;
for (const func of befores) {
const args = [newInput].concat(options.extraArgs);
const maybePromise = func.apply(undefined, args);
// legacy scripting runner returns a Promise for beforeRequest
if (typeof maybePromise !== 'object') {
throw new Error('Middleware should return an object.');
}
newInput = await Promise.resolve(maybePromise);
}
return newInput;
};
const afterMiddleware = async (output) => {
for (const func of afters) {
const args = [output].concat(options.extraArgs);
const maybePromise = func.apply(undefined, args);
if (typeof maybePromise !== 'object') {
throw new Error('Middleware should return an object.');
}
output = await Promise.resolve(maybePromise);
output = ensureEnvelope(output);
}
return output;
};
const promise = async (input) => {
const newInput = await beforeMiddleware(input);
try {
let output = await app(newInput);
output = await ensureEnvelope(output);
output.input = newInput;
return afterMiddleware(output);
} catch (error) {
return enrichErrorMessages(error, newInput);
}
};
return promise(input);
};
};
module.exports = applyMiddleware;

View file

@ -0,0 +1,88 @@
'use strict';
const crypto = require('crypto');
const fernet = require('fernet');
const zlib = require('zlib');
/**
* Decrypt a bundle using secret key
*
* This matches the backend:
* 1. Hash the secret with SHA256 to get 32 bytes
* 2. Base64url encode those bytes to make Fernet-compatible key
* 3. Use Fernet library to decrypt (handles all token parsing internally)
* 4. Base64 decode the decrypted string to get compressed binary data
* 5. Decompress the data using gzip
*
* @param {string} bundle - The bundle represented as an encrypted token
* @param {string} secret - The secret key for decryption
* @returns {Object} The decrypted and decompressed bundle object
*/
const decryptBundleWithSecret = (bundle, secret) => {
try {
// Validate input
if (!bundle || typeof bundle !== 'string') {
throw new Error('Invalid object from s3 - must be a non-empty string');
}
if (!secret || typeof secret !== 'string') {
throw new Error('Invalid secret - must be a non-empty string');
}
// Step 1: Create the same key as backend
// Hash the secret and take first 32 bytes, then base64url encode for Fernet
const keyHash = crypto.createHash('sha256').update(secret).digest();
const keyBytes = keyHash.subarray(0, 32); // Take first 32 bytes
const fernetKey = keyBytes.toString('base64url'); // Use built-in base64url encoding
// Use Fernet library to decrypt (handles all the token parsing)
const secretObj = new fernet.Secret(fernetKey);
const token = new fernet.Token({
secret: secretObj,
token: bundle,
ttl: 0,
});
// Step 2: Decrypt the token - this should now be a valid UTF-8 string (base64 encoded)
let decryptedString;
try {
decryptedString = token.decode();
} catch (fernetError) {
throw new Error(`Fernet decryption failed: ${fernetError.message}`);
}
// Step 3: The decrypted data should be a base64 encoded string
// Base64 decode it to get the compressed binary data
let compressedBytes;
try {
compressedBytes = Buffer.from(decryptedString, 'base64');
} catch (base64Error) {
throw new Error(`Base64 decoding failed: ${base64Error.message}`);
}
// Step 4: The data is compressed, so we need to decompress it
let decompressed;
try {
// Try to decompress first (for new format with compression)
decompressed = zlib.gunzipSync(compressedBytes).toString('utf8');
} catch (decompressionError) {
// If decompression fails, assume it's the old format without compression
// This provides backward compatibility
console.warn(
'Bundle decompression failed, falling back to uncompressed format:',
decompressionError.message,
);
decompressed = decryptedString;
}
// Step 5: Parse JSON
try {
return JSON.parse(decompressed);
} catch (error) {
throw new Error('Invalid JSON in decrypted bundle');
}
} catch (error) {
throw new Error(`Bundle decryption failed: ${error.message}`);
}
};
module.exports = {
decryptBundleWithSecret,
};

View file

@ -0,0 +1,217 @@
'use strict';
const _ = require('lodash');
const { defaults, pick, pipe } = require('lodash/fp');
const {
flattenPaths,
getObjectType,
isPlainObj,
recurseReplace,
} = require('./data');
const DEFAULT_BUNDLE = {
authData: {},
inputData: {},
meta: {},
subscribeData: {},
targetUrl: '',
};
const isCurlies = /{{.*?}}/g;
const recurseCleanFuncs = (obj, path) => {
// mainly turn functions into $func${arity}${arguments}$
path = path || [];
if (typeof obj === 'function') {
const usesArguments =
obj.toString().indexOf('arguments') !== -1 ? 't' : 'f';
// TODO: could optimize $func$0$f$ as "pure" and just render them
return `$func$${obj.length}$${usesArguments}$`;
} else if (Array.isArray(obj)) {
return obj.map((value, i) => {
return recurseCleanFuncs(value, path.concat([i]));
});
} else if (isPlainObj(obj)) {
const newObj = {};
Object.keys(obj).forEach((key) => {
const value = obj[key];
newObj[key] = recurseCleanFuncs(value, path.concat([key]));
});
return newObj;
}
return obj;
};
const findNextCurlies = (str) => {
const start = str.indexOf('{{');
if (start < 0) {
return {
start: -1,
end: -1,
};
}
const end = str.indexOf('}}', start + 2);
if (end < 0) {
return {
start: -1,
end: -1,
};
}
return {
start,
end: end + 2,
};
};
// Recurse a nested object replace all instances of keys->vals in the bank.
const recurseReplaceBank = (obj, bank = {}) => {
const replacer = (input) => {
if (typeof input !== 'string') {
return input;
}
const outputItems = [];
let inputString = input;
// 1000 iterations is just a static upper bound to make infinite loops
// impossible. Who would have 1000 {{curlies}} in a string... right?
const MAX_ITERATIONS = 1000;
for (let i = 0; i < MAX_ITERATIONS; i++) {
const { start, end } = findNextCurlies(inputString);
if (start < 0) {
outputItems.push(inputString);
break;
}
const head = inputString.slice(0, start);
const key = inputString.slice(start, end);
const tail = inputString.slice(end);
if (head) {
outputItems.push(head);
}
const replacementValue = bank[key];
if (replacementValue == null) {
// No match in the bank
outputItems.push(key);
} else {
const isPartOfString = head || tail;
if (
isPartOfString &&
(Array.isArray(replacementValue) || _.isPlainObject(replacementValue))
) {
const bareKey = key.slice(2, -2); // '{{key}}' -> 'key'
throw new TypeError(
'Cannot reliably interpolate objects or arrays into a string. ' +
`Variable \`${bareKey}\` is an ${getObjectType(
replacementValue,
)}:\n"${replacementValue}"`,
);
} else {
outputItems.push(replacementValue);
}
}
inputString = tail;
if (!inputString) {
break;
}
if (i === MAX_ITERATIONS - 1) {
// The input string does have more than 1000 {{curlies}}, just return
// the rest of the string without replacing.
outputItems.push(inputString);
}
}
if (outputItems.length === 1 && typeof outputItems[0] !== 'string') {
return outputItems[0];
} else {
return outputItems.join('');
}
};
return recurseReplace(obj, replacer);
};
const finalizeBundle = pipe(
pick(Object.keys(DEFAULT_BUNDLE)),
defaults(DEFAULT_BUNDLE),
);
// Takes a raw app and bundle and composes a bank of {{key}}->val
const createBundleBank = (event = {}, serializeFunc = (x) => x) => {
const bank = {
bundle: finalizeBundle(event.bundle),
process: {
env: _.extend({}, process.env || {}),
},
};
const options = { preserve: { 'bundle.inputData': true } };
const flattenedBank = flattenPaths(bank, options);
return Object.entries(flattenedBank).reduce((coll, [key, value]) => {
coll[`{{${key}}}`] = serializeFunc(value);
return coll;
}, {});
};
const maskOutput = (output) =>
_.pick(output, 'results', 'status', 'resultsUrl');
// These normalize functions are called after the initial before middleware that
// cleans the request. The reason is that we need to know why a value is empty
// later on. If we resolve all templates (even with undefined values) first, we
// don't know _why_ it was an empty string. Was it from a user supplied value in
// an earlier Zap step? Or was it a null value? Each has different results depending
// on how the partner has configued their integration.
const normalizeEmptyRequestFields = (shouldCleanup, field, req) => {
const handleEmpty = (key) => {
const value = req[field][key] || '';
const cleaned = value.replace(isCurlies, '');
if (value !== cleaned) {
req[field][key] = cleaned;
}
if (!cleaned && req.removeMissingValuesFrom[field]) {
delete req[field][key];
}
};
Object.entries(req[field]).forEach(([key, value]) => {
if (shouldCleanup(value)) {
handleEmpty(key);
}
});
};
const isEmptyQueryParam = (value) =>
value === '' ||
value === null ||
value === undefined ||
(typeof value === 'string' && value.search(isCurlies) >= 0);
const normalizeEmptyParamFields = normalizeEmptyRequestFields.bind(
null,
isEmptyQueryParam,
'params',
);
const normalizeEmptyBodyFields = normalizeEmptyRequestFields.bind(
null,
(v) => typeof v === 'string' && v.search(isCurlies) >= 0,
'body',
);
module.exports = {
createBundleBank,
maskOutput,
normalizeEmptyBodyFields,
normalizeEmptyParamFields,
recurseCleanFuncs,
recurseReplaceBank,
};

View file

@ -0,0 +1,49 @@
'use strict';
const createLoggerConsole = require('./create-logger-console');
/**
* Shared console instance that can be used standalone or as z.console.
* Uses createLoggerConsole to create the single instance shared between both.
*/
// Shared console instance - will be initialized by middleware
let loggerConsole = null;
/**
* Initialize the console with the input context from middleware.
* This is called by the z-object middleware and uses createLoggerConsole.
* Creates a new logger console for each Lambda invocation to ensure proper isolation.
* Returns the shared console instance for z.console to use.
*/
const initialize = (input) => {
// Always create a new logger console for each invocation to ensure
// log context isolation between Lambda invocations
loggerConsole = createLoggerConsole(input);
return loggerConsole;
};
/**
* Reset the singleton for testing purposes
*/
const reset = () => {
loggerConsole = null;
};
/**
* Proxy that behaves like a console object.
* Forwards calls to loggerConsole if initialized, otherwise to global console.
*/
const consoleProxy = new Proxy(console, {
get(target, prop, receiver) {
// Use loggerConsole if initialized, otherwise fall back to global console
const actualConsole = loggerConsole || target;
return Reflect.get(actualConsole, prop, actualConsole);
},
});
module.exports = {
consoleProxy,
initialize,
reset,
};

View file

@ -0,0 +1,81 @@
'use strict';
const _ = require('lodash');
const createRequestClient = require('./create-request-client');
const ensureArray = require('./ensure-array');
const ensurePath = require('./ensure-path');
// before middles
const addBasicAuthHeader = require('../http-middlewares/before/add-basic-auth-header');
const addDigestAuthHeader = require('../http-middlewares/before/add-digest-auth-header');
const addQueryParams = require('../http-middlewares/before/add-query-params');
const createInjectInputMiddleware = require('../http-middlewares/before/inject-input');
const disableSSLCertCheck = require('../http-middlewares/before/disable-ssl-cert-check');
const oauth1SignRequest = require('../http-middlewares/before/oauth1-sign-request');
const prepareRequest = require('../http-middlewares/before/prepare-request');
const sanitizeHeaders = require('../http-middlewares/before/sanatize-headers');
// after middles
const { logResponse } = require('../http-middlewares/after/log-response');
const prepareResponse = require('../http-middlewares/after/prepare-response');
const throwForStaleAuth = require('../http-middlewares/after/throw-for-stale-auth');
const throwForThrottling = require('../http-middlewares/after/throw-for-throttling');
const throwForStatusMiddleware = require('../http-middlewares/after/throw-for-status');
const throwForDisallowedHostnameAfterRedirect = require('../http-middlewares/after/throw-for-disallowed-hostname-after-redirect');
const createAppRequestClient = (input, options) => {
input = ensurePath(input, '_zapier.app');
const app = input._zapier.app;
options = _.defaults({}, options, {
skipDefaultMiddle: true,
extraArgs: [],
});
const httpBefores = [
createInjectInputMiddleware(input),
prepareRequest,
].concat(ensureArray(app.beforeRequest));
if (app.authentication) {
if (app.authentication.type === 'basic') {
httpBefores.push(addBasicAuthHeader);
} else if (app.authentication.type === 'digest') {
httpBefores.push(addDigestAuthHeader);
} else if (app.authentication.type === 'oauth1') {
httpBefores.push(oauth1SignRequest);
}
}
httpBefores.push(sanitizeHeaders);
httpBefores.push(addQueryParams);
const verifySSL = _.get(input, '_zapier.event.verifySSL');
if (verifySSL === false) {
httpBefores.push(disableSSLCertCheck);
}
let includeAutoRefresh = false;
if (
app.authentication &&
(app.authentication.type === 'session' ||
(app.authentication.type === 'oauth2' &&
_.get(app, 'authentication.oauth2Config.autoRefresh')))
) {
includeAutoRefresh = true;
}
const httpAfters = [
prepareResponse,
throwForDisallowedHostnameAfterRedirect,
logResponse,
...(includeAutoRefresh ? [throwForStaleAuth] : []),
throwForThrottling,
...ensureArray(app.afterResponse),
throwForStatusMiddleware,
];
return createRequestClient(httpBefores, httpAfters, options);
};
module.exports = createAppRequestClient;

View file

@ -0,0 +1,70 @@
'use strict';
const createLambdaHandler = require('./create-lambda-handler');
const resolveMethodPath = require('./resolve-method-path');
const { isFunction } = require('lodash');
const { genId } = require('./data');
const { shouldPaginate } = require('./should-paginate');
// A shorthand compatible wrapper for testing.
const createAppTester = (appRaw, { customStoreKey } = {}) => {
const handler = createLambdaHandler(appRaw);
const randomSeed = genId();
const appTester = (methodOrFunc, bundle, clearZcacheBeforeUse = false) => {
bundle = bundle || {};
let method = resolveMethodPath(appRaw, methodOrFunc, false);
if (!method) {
if (isFunction(methodOrFunc)) {
// definitely have a function but didn't find it on the app; it's an adhoc
appRaw._testRequest = (z, bundle) => methodOrFunc(z, bundle);
method = resolveMethodPath(appRaw, appRaw._testRequest);
} else {
throw new Error(
`Unable to find the following on your App instance: ${JSON.stringify(
methodOrFunc,
)}`,
);
}
}
const storeKey = shouldPaginate(appRaw, method)
? customStoreKey
? `testKey-${customStoreKey}`
: `testKey-${method}-${randomSeed}`
: null;
if (clearZcacheBeforeUse) {
appTester.zcacheTestObj = {};
}
const event = {
command: 'execute',
method,
bundle,
storeKey,
callback_url: 'https://auth-json-server.zapier-staging.com/echo',
zcacheTestObj: appTester.zcacheTestObj,
};
if (process.env.LOG_TO_STDOUT) {
event.logToStdout = true;
}
if (process.env.DETAILED_LOG_TO_STDOUT) {
event.detailedLogToStdout = true;
}
return handler(event).then((resp) => {
delete appRaw._testRequest; // clear adHocFunc so tests can't affect each other
return resp.results;
});
};
appTester.zcacheTestObj = {};
return appTester;
};
module.exports = createAppTester;

View file

@ -0,0 +1,74 @@
'use strict';
const _ = require('lodash');
const JSON = require('./create-json-tool')();
const ensureJSONEncodable = require('./ensure-json-encodable');
const createCache = (input) => {
const rpc = _.get(input, '_zapier.rpc');
const runValidationChecks = (
rpc,
key,
value = null,
ttl = null,
scope = null,
nx = null,
) => {
if (!rpc) {
throw new Error('rpc is not available');
}
if (!_.isString(key)) {
throw new TypeError('key must be a string');
}
if (ttl != null && !_.isInteger(ttl)) {
throw new TypeError('ttl must be an integer');
}
if (
scope !== null &&
(!Array.isArray(scope) ||
!scope.every((v) => v === 'user' || v === 'auth'))
) {
throw new TypeError(
'scope must be an array of strings with values "user" or "auth"',
);
}
if (nx !== null && !_.isBoolean(nx)) {
throw new TypeError('nx must be a boolean');
}
ensureJSONEncodable(value);
};
return {
get: async (key, scope = null) => {
runValidationChecks(rpc, key, scope);
const result = await rpc('zcache_get', key, scope);
return result ? JSON.parse(result) : null;
},
set: async (key, value, ttl = null, scope = null, nx = null) => {
runValidationChecks(rpc, key, value, ttl, scope, nx);
return await rpc(
'zcache_set',
key,
JSON.stringify(value),
ttl,
scope,
nx,
);
},
delete: async (key, scope = null) => {
runValidationChecks(rpc, key, scope);
return await rpc('zcache_delete', key, scope);
},
};
};
module.exports = createCache;

View file

@ -0,0 +1,20 @@
'use strict';
const _ = require('lodash');
// this returns a higher order function that allows users to generate the callback URL
// generation doesn't actually generate the URL (although we could use RPC if we wanted to)
// instead it relies on the callbackUrl being passed in by the platform
// as such it'll always return the same value.
// calling this inner function will also mark the task as a CALLBACK status on return which will
// effectively pause the Zap
const createCallbackHigherOrderFunction = (input) => {
const callbackUrl = _.get(input, '_zapier.event.callback_url');
return () => {
_.set(input, '_zapier.event.callbackUsed', true);
return callbackUrl;
};
};
module.exports = createCallbackHigherOrderFunction;

View file

@ -0,0 +1,34 @@
'use strict';
const _ = require('lodash');
const { DehydrateError } = require('../errors');
const resolveMethodPath = require('./resolve-method-path');
const wrapHydrate = require('./wrap-hydrate');
const createDehydrator = (input, type = 'method') => {
const app = _.get(input, '_zapier.app');
return (func, inputData, cacheExpiration) => {
inputData = inputData || {};
if (inputData.inputData) {
throw new DehydrateError(
'Oops! You passed a full `bundle` - really you should pass what you want under `inputData`!',
);
}
const payload = {
type,
method: resolveMethodPath(app, func),
// inputData vs. bundle is a legacy oddity
bundle: _.omit(inputData, 'environment'), // don't leak the environment
};
if (cacheExpiration) {
payload.cacheExpiration = cacheExpiration;
}
return wrapHydrate(payload);
};
};
module.exports = createDehydrator;

View file

@ -0,0 +1,284 @@
'use strict';
const fs = require('fs');
const os = require('os');
const path = require('path');
const { pipeline, Readable } = require('stream');
const { promisify } = require('util');
const { randomBytes } = require('crypto');
const _ = require('lodash');
const contentDisposition = require('content-disposition');
const mime = require('mime-types');
const {
ENCODED_FILENAME_MAX_LENGTH,
UPLOAD_MAX_SIZE,
NON_STREAM_UPLOAD_MAX_SIZE,
} = require('../constants');
const uploader = require('./uploader');
const DEFAULT_FILE_NAME = 'unnamedfile';
const DEFAULT_CONTENT_TYPE = 'application/octet-stream';
const streamPipeline = promisify(pipeline);
const filenameFromURL = (url) => {
try {
return decodeURIComponent(path.posix.basename(new URL(url).pathname));
} catch (error) {
return null;
}
};
const filenameFromHeader = (response) => {
const cd = response.headers.get('content-disposition');
let filename;
if (cd) {
try {
filename = contentDisposition.parse(cd).parameters.filename;
} catch (error) {
return null;
}
}
return filename || null;
};
const resolveRemoteStream = async (stream) => {
// Download to a temp file, get the file size, and create a readable stream
// from the temp file.
//
// The streamPipeline usage is taken from
// https://github.com/node-fetch/node-fetch#streams
const tmpFilePath = path.join(
os.tmpdir(),
'stash-' + randomBytes(16).toString('hex'),
);
try {
await streamPipeline(stream, fs.createWriteStream(tmpFilePath));
} catch (error) {
try {
fs.unlinkSync(tmpFilePath);
} catch (e) {
// File doesn't exist? Probably okay
}
throw error;
}
const length = fs.statSync(tmpFilePath).size;
const readStream = fs.createReadStream(tmpFilePath);
readStream.on('end', () => {
// Burn after reading
try {
fs.unlinkSync(tmpFilePath);
} catch (e) {
// TODO: We probably want to log warning here
}
});
return {
streamOrData: readStream,
length,
};
};
const resolveResponseToStream = async (response) => {
// Get filename from content-disposition header or URL
let filename =
filenameFromHeader(response) ||
filenameFromURL(response.url || _.get(response, ['request', 'url'])) ||
DEFAULT_FILE_NAME;
const contentType = response.headers.get('content-type');
if (contentType && !path.extname(filename)) {
const ext = mime.extension(contentType);
if (ext && ext !== 'bin') {
filename += '.' + ext;
}
}
if (response.body && typeof response.body.pipe === 'function') {
// streamable response created by z.request({ raw: true })
return {
...(await resolveRemoteStream(response.body)),
contentType: contentType || DEFAULT_CONTENT_TYPE,
filename,
};
}
// regular response created by z.request({ raw: false })
return {
streamOrData: response.content,
length: Buffer.byteLength(response.content),
contentType: contentType || DEFAULT_CONTENT_TYPE,
filename,
};
};
const resolveStreamWithMeta = async (stream) => {
const isLocalFile = stream.path && fs.existsSync(stream.path);
if (isLocalFile) {
const filename = path.basename(stream.path);
return {
streamOrData: stream,
length: fs.statSync(stream.path).size,
contentType: mime.lookup(filename) || DEFAULT_CONTENT_TYPE,
filename,
};
}
return {
...(await resolveRemoteStream(stream)),
contentType: DEFAULT_CONTENT_TYPE,
filename: DEFAULT_FILE_NAME,
};
};
// Returns an object with fields:
// * streamOrData: a readable stream, a string, or a Buffer
// * length: content length in bytes
// * contentType
// * filename
const resolveToBufferStringStream = async (responseOrData) => {
if (typeof responseOrData === 'string' || responseOrData instanceof String) {
// The .toString() call only makes a difference for the String object case.
// It converts a String object to a regular string.
const str = responseOrData.toString();
return {
streamOrData: str,
length: Buffer.byteLength(str),
contentType: 'text/plain',
filename: `${DEFAULT_FILE_NAME}.txt`,
};
} else if (Buffer.isBuffer(responseOrData)) {
return {
streamOrData: responseOrData,
length: responseOrData.length,
contentType: DEFAULT_CONTENT_TYPE,
filename: DEFAULT_FILE_NAME,
};
} else if (
(responseOrData.body && typeof responseOrData.body.pipe === 'function') ||
typeof responseOrData.content === 'string'
) {
return resolveResponseToStream(responseOrData);
} else if (typeof responseOrData.pipe === 'function') {
return resolveStreamWithMeta(responseOrData);
}
throw new TypeError(
`z.stashFile() cannot stash type '${typeof responseOrData}'. ` +
'Pass it a request, readable stream, string, or Buffer.',
);
};
const ensureUploadMaxSizeNotExceeded = (streamOrData, length) => {
let uploadMaxSize = NON_STREAM_UPLOAD_MAX_SIZE;
let uploadMethod = 'non-streaming';
if (streamOrData instanceof Readable) {
uploadMaxSize = UPLOAD_MAX_SIZE;
uploadMethod = 'streaming';
}
if (length && length > uploadMaxSize) {
throw new Error(
`${length} bytes is too big, ${uploadMaxSize} is the max for ${uploadMethod} data.`,
);
}
};
// S3's max metadata size is 2KB
// If the filename needs to be encoded, both the filename
// and encoded filename are included in the Content-Disposition header
const ensureMetadataMaxSizeNotExceeded = (filename) => {
const filenameMaxSize = ENCODED_FILENAME_MAX_LENGTH;
if (filename) {
const encodedFilename = encodeURIComponent(filename);
if (
encodedFilename !== filename &&
encodedFilename.length > filenameMaxSize
) {
throw new Error(
`URI-Encoded Filename is too long at ${encodedFilename.length}, ${ENCODED_FILENAME_MAX_LENGTH} is the max.`,
);
}
}
};
// Designed to be some user provided function/api.
const createFileStasher = (input) => {
const rpc = _.get(input, '_zapier.rpc');
return async (requestOrData, knownLength, filename, contentType) => {
// TODO: maybe this could be smart?
// if it is already a public url, do we pass through? or upload?
if (!rpc) {
throw new Error('rpc is not available');
}
const isRunningOnHydrator = _.get(
input,
'_zapier.event.method',
'',
).startsWith('hydrators.');
const isRunningOnCreate = _.get(
input,
'_zapier.event.method',
'',
).startsWith('creates.');
if (!isRunningOnHydrator && !isRunningOnCreate) {
throw new Error(
'Files can only be stashed within a create or hydration function/method.',
);
}
// requestOrData can be one of these:
// * string
// * Buffer
// * z.request() - a Promise of a regular response
// * z.request({ raw: true }) - a Promise of a "streamable" response
// * await z.request() - a regular response
// * await z.request({ raw: true }) - a streamable response
//
// After the following, requestOrData is resolved to responseOrData, which
// is either:
// - string
// - Buffer
// - a regular response
// - a streamable response
const [signedPostData, responseOrData] = await Promise.all([
rpc('get_presigned_upload_post_data'),
requestOrData,
]);
if (responseOrData.throwForStatus) {
responseOrData.throwForStatus();
}
const {
streamOrData,
length,
contentType: _contentType,
filename: _filename,
} = await resolveToBufferStringStream(responseOrData);
const finalLength = knownLength || length;
ensureUploadMaxSizeNotExceeded(streamOrData, finalLength);
ensureMetadataMaxSizeNotExceeded(filename || _filename);
return uploader(
signedPostData,
streamOrData,
finalLength,
filename || _filename,
contentType || _contentType,
);
};
};
module.exports = createFileStasher;

View file

@ -0,0 +1,142 @@
const zlib = require('zlib');
const _ = require('lodash');
const { ALLOWED_HTTP_DATA_CONTENT_TYPES, getContentType } = require('./http');
const constants = require('../constants');
const createHttpPatch = (event) => {
const httpPatch = (object, logger) => {
const originalRequest = object.request;
// Important not to reuse logger between calls, because we always destroy
// the logger at the end of a Lambda call.
object.zapierLogger = logger;
// Avoids multiple patching and memory leaks (mostly when running tests locally)
if (object.patchedByZapier) {
return;
}
object.patchedByZapier = true;
// Proxy the request method
object.request = (options, callback) => {
// `options` can be an object or a string. If options is a string, it is
// automatically parsed with url.parse().
// See https://nodejs.org/docs/latest-v14.x/api/http.html#http_http_request_options_callback
let requestUrl;
if (typeof options === 'string') {
requestUrl = options;
} else if (typeof options.url === 'string') {
// XXX: Somehow options.url is available for some requests although
// http.request doesn't really accept it. Without this else-if, many
// HTTP requests don't work. Should take a deeper look at this
// weirdness.
requestUrl = options.url;
} else {
requestUrl =
options.href ||
`${options.protocol || 'https:'}//${options.host}${options.path}`;
}
const loggerUrl =
process.env.LOGGING_ENDPOINT || constants.DEFAULT_LOGGING_HTTP_ENDPOINT;
// Ignore logger requests
if (requestUrl.indexOf(loggerUrl) !== -1) {
return originalRequest(options, callback);
}
// Ignore requests made via the request client (z.request)
if (_.get(options.headers, 'user-agent', []).indexOf('Zapier') !== -1) {
return originalRequest(options, callback);
}
// Proxy the callback to get the response
const newCallback = function (response) {
const chunks = [];
// Only include request or response data for specific content types
// which we are able to read in logs and which are not typically too large
// Limitation: This doesn't capture the request content-type if it's set afterwards, like:
// const req = https.request(options, callback);
// req.setHeader('Content-Type', 'text/plain');
const requestContentType = getContentType(options.headers || {});
const responseContentType = getContentType(response.headers || {});
const shouldIncludeRequestData =
ALLOWED_HTTP_DATA_CONTENT_TYPES.has(requestContentType);
const shouldIncludeResponseData =
ALLOWED_HTTP_DATA_CONTENT_TYPES.has(responseContentType);
const sendToLogger = (responseBody) => {
// Prepare data for GL
const logData = {
log_type: 'http',
// Using a custom request_type to differentiate from z.request() since `request_via_client` is not logged in GL
request_type: 'patched-devplatform-outbound',
request_url: requestUrl,
request_method: options.method || 'GET',
request_headers: options.headers,
request_data: shouldIncludeRequestData
? options.body || ''
: '<unsupported format>',
request_via_client: false,
response_status_code: response.statusCode,
response_headers: response.headers,
response_content: shouldIncludeResponseData
? responseBody
: '<unsupported format>',
};
object.zapierLogger(
`${logData.response_status_code} ${logData.request_method} ${logData.request_url}`,
logData,
);
};
const logResponse = () => {
// Decode gzip if needed
if (response.headers['content-encoding'] === 'gzip') {
const buffer = Buffer.concat(chunks);
zlib.gunzip(buffer, (err, decoded) => {
const responseBody = err
? 'Could not decode response body.'
: decoded.toString();
sendToLogger(responseBody);
});
} else {
const responseBody = _.map(chunks, (chunk) =>
chunk.toString(),
).join('');
sendToLogger(responseBody);
}
};
const originalEmit = response.emit;
response.emit = function (event, ...args) {
if (event === 'data') {
chunks.push(args[0]);
}
return originalEmit.apply(this, [event, ...args]);
};
response.on('end', logResponse);
response.on('error', logResponse);
// If there was a callback, call it now
if (_.isFunction(callback)) {
callback.apply(this, arguments);
}
};
return originalRequest(options, newCallback);
};
};
return httpPatch;
};
module.exports = createHttpPatch;

View file

@ -0,0 +1,44 @@
'use strict';
/*
Creates input object for the middleware chain, from the app definition,
AWS event object, and logger function. The logger is a function that
logs somewhere, and returns a promise.
The returned input object contains all the very basic, core fields that every
middleware, *and* our AWS handler function, can depend on and assume exist.
Subsequent middlewares may stick other things on input (like the z object).
*/
const createInput = (app, event, logger, logBuffer, rpc) => {
return {
// Expose bundle to dev apps
bundle: event.bundle,
// The _zapier namespace is 'private'. It has things
// we need, but that dev apps should not need.
_zapier: {
// Compiled app definition
app,
// The raw AWS event object
event,
// List of promises to wait on. Stick unresolved promises on
// here, and we will wait for them to complete before calling
// the AWS callback.
promises: [],
// Our internal rpc client
rpc,
// Logger function that returns a promise
logger,
logBuffer,
whatHappened: [],
},
};
};
module.exports = createInput;

View file

@ -0,0 +1,32 @@
'use strict';
const _ = require('lodash');
const parse = (str) => {
const newError = (message) => {
const error = new SyntaxError(message);
Error.captureStackTrace(error, parse);
return error;
};
if (!_.isString(str)) {
throw newError(`Error parsing response. "${str}" is not a string.`);
}
try {
return JSON.parse(str);
} catch (err) {
const preview = str.substr(0, 100);
throw newError(`Error parsing response. We got: "${preview}"`);
}
};
// Similar API to JSON built in but catches errors with nicer tracebacks.
const createJSONtool = () => {
return {
parse,
stringify: JSON.stringify,
};
};
module.exports = createJSONtool;

View file

@ -0,0 +1,295 @@
'use strict';
const domain = require('domain'); // eslint-disable-line n/no-deprecated-api
const fs = require('fs');
const os = require('os');
const path = require('path');
const _ = require('lodash');
const checkMemory = require('./memory-checker');
const cleaner = require('./cleaner');
const constants = require('../constants');
const createApp = require('../create-app');
const createHttpPatch = require('./create-http-patch');
const createInput = require('./create-input');
const createLogger = require('./create-logger');
const createRpcClient = require('./create-rpc-client');
const environmentTools = require('./environment');
const schemaTools = require('./schema');
const wrapFetchWithLogger = require('./fetch-logger');
const isDefinedPrimitive = (value) => {
return (
value === null ||
typeof value === 'string' ||
typeof value === 'number' ||
typeof value === 'boolean'
);
};
const shouldFullyReplace = (path) => {
// covers inputFields, outputFields, sample, throttle, etc
const isOperation = path[path.length - 2] === 'operation';
return isOperation;
};
const extendAppRaw = (base, extension, path) => {
if (extension === undefined) {
return base;
} else if (isDefinedPrimitive(extension)) {
return extension;
} else if (Array.isArray(extension)) {
return [...extension];
} else if (_.isPlainObject(extension)) {
path = path || [];
if (shouldFullyReplace(path)) {
return extension;
} else {
const baseObject = _.isPlainObject(base) ? base : {};
const result = { ...baseObject };
for (const [key, value] of Object.entries(extension)) {
const newPath = [...path, key];
result[key] = extendAppRaw(baseObject[key], value, newPath);
}
return result;
}
}
throw new TypeError('Unexpected extension type');
};
const mayMoveCreatesToResourcesInExtension = (base, extension) => {
// The backend sends an extension for creates.KEY.operation.perform as a
// special case for legacy-scripting-runner.
// For details, see the MR description at
// https://gitlab.com/zapier/zapier/-/merge_requests/57964
//
// We need to make sure that 'creates.{key}Create' in extension won't collide
// with 'resources.{key}.create' in base. Otherwise, the checks in
// compileApp() will throw an error. So here we move 'creates.{key}Create' to
// 'resources.{key}.create' in extension if base has 'resources.{key}.create'.
//
// There's a regression test: Search for 'resource key collision' in
// integration-test.js.
if (
!_.isPlainObject(base.resources) ||
!_.isPlainObject(extension.creates) ||
_.isEmpty(base.resources) ||
_.isEmpty(extension.creates)
) {
return extension;
}
const creates = extension.creates;
extension.creates = {};
extension.resources = extension.resources || {};
for (const [key, resource] of Object.entries(base.resources)) {
const standaloneCreate = creates[key + 'Create'];
if (resource.create && standaloneCreate) {
delete standaloneCreate.key;
delete standaloneCreate.noun;
extension.resources[key] = {
...extension.resources[key],
create: standaloneCreate,
};
}
}
return extension;
};
const getAppRawOverride = async (rpc, appRawOverride) => {
let appRawExtension;
if (Array.isArray(appRawOverride) && appRawOverride.length > 1) {
// If appRawOverride is too big, we send an md5 hash instead of JSON, so
// appRawOverride can be:
// - [appDefinition, {'creates': {'foo': {...}}}]
// - ['<hash>', {'creates': {'foo': {...}}}] if appRawOverride is too big
appRawExtension = appRawOverride[1];
appRawOverride = appRawOverride[0];
if (typeof appRawOverride !== 'string') {
appRawExtension = mayMoveCreatesToResourcesInExtension(
appRawOverride,
appRawExtension,
);
appRawOverride = extendAppRaw(appRawOverride, appRawExtension);
return appRawOverride;
}
} else if (typeof appRawOverride === 'object') {
return appRawOverride;
}
// Lambda keeps the container and /tmp directory around for a bit,
// so we can use that to "cache" the hash and override we fetched
// from RPC before.
const tmpdir = os.tmpdir();
const overridePath = path.join(tmpdir, 'cli-override.json');
const hashPath = path.join(tmpdir, 'cli-hash.txt');
// Check if it's "cached", to prevent unnecessary RPC calls
if (
fs.existsSync(hashPath) &&
fs.existsSync(overridePath) &&
fs.readFileSync(hashPath).toString() === appRawOverride
) {
appRawOverride = JSON.parse(fs.readFileSync(overridePath).toString());
appRawOverride = extendAppRaw(appRawOverride, appRawExtension);
return appRawOverride;
}
const fetchedOverride = await rpc('get_definition_override');
// store or "cache" override
fs.writeFileSync(hashPath, appRawOverride);
fs.writeFileSync(overridePath, JSON.stringify(fetchedOverride));
return extendAppRaw(fetchedOverride, appRawExtension);
};
const loadApp = async (event, rpc, appRawOrPath) => {
let appRaw;
if (typeof appRawOrPath === 'string') {
// CommonJS route - most CLI integrations go through here.
const appPath = appRawOrPath;
appRaw = require(appPath);
} else {
// ESM route - CLI integrations that use ESM go through here.
// Some tests and UI "buildless" integrations also use this route.
appRaw = appRawOrPath;
}
if (event && event.appRawOverride) {
if (
Array.isArray(event.appRawOverride) &&
event.appRawOverride.length > 1 &&
!event.appRawOverride[0]
) {
event.appRawOverride[0] = appRaw;
}
return getAppRawOverride(rpc, event.appRawOverride);
}
return appRaw;
};
const createLambdaHandler = (appRawOrPath) => {
const handlerPromise = async (event, context = {}) => {
// If we're running out of memory or file descriptors, force exit the process.
// The backend will try again via @retry(ProcessExitedException).
checkMemory(event);
environmentTools.cleanEnvironment();
// Copy bundle environment into process.env *before* creating the logger and
// loading app code, so that the logger gets the endpoint from process.env,
// and top level app code can get bundle environment vars via process.env.
environmentTools.applyEnvironment(event);
// Create logger outside of domain, so we can use in both error and run callbacks.
const logBuffer = [];
const logger = createLogger(event, { logBuffer });
const logErrorAndReturn = async (logMsg, logData, err) => {
await logger(logMsg, logData);
// Check for `.message` in case someone did `throw "My Error"`
if (!constants.IS_TESTING && err && !err.doNotContextify && err.message) {
err.message += `\n\nConsole logs:\n${logBuffer
.map((s) => ` ${s.message}`)
.join('')}`;
}
return err;
};
const handlerDomain = domain.create();
// TODO ideally we should not use a promise here
// also ideally we could remove domain since it's deprecated...
return new Promise((resolve, reject) => {
handlerDomain.on('error', async (err) => {
// This error handler is only called when someone (we or devs) missed
// catching an error in a callback. Errors thrown by promises should be
// already caught by the try-catch below.
// Notice this one starts with "Uncaught error" while the other one starts
// with "Unhandled error". You can use this to distinguish between them in
// the logs.
// This "Uncaught error" handler doesn't always get called. The behavior
// is a bit unpredictable based on my testing. Say, for example, if the
// Lambda handler returns before the error is thrown, this handler won't
// be executed.
const logMsg = `Uncaught error: ${err}\n${
(err && err.stack) || '<stack>'
}`;
const logData = { err, log_type: 'error' };
const loggedErr = await logErrorAndReturn(logMsg, logData, err);
await logger.end();
reject(loggedErr);
});
handlerDomain.run(async () => {
const rpc = createRpcClient(event);
try {
const appRaw = await loadApp(event, rpc, appRawOrPath);
const app = createApp(appRaw);
const { skipHttpPatch } = appRaw.flags || {};
// Adds logging for _all_ kinds of http(s) requests, no matter the library
if (
!skipHttpPatch &&
(!event.calledFromCli || event.calledFromCliInvoke)
) {
const httpPatch = createHttpPatch(event);
httpPatch(require('http'), logger);
httpPatch(require('https'), logger); // 'https' needs to be patched separately
if (global.fetch) {
global.fetch = wrapFetchWithLogger(global.fetch, logger);
}
}
// TODO: Avoid calling prepareApp(appRaw) repeatedly here as createApp()
// already calls prepareApp() but just doesn't return it.
const compiledApp = schemaTools.prepareApp(appRaw);
const input = createInput(compiledApp, event, logger, logBuffer, rpc);
const output = await app(input);
const result = cleaner.maskOutput(output);
await logger.end();
resolve(result);
} catch (err) {
const logMsg = `Unhandled error: ${err}\n${
(err && err.stack) || '<stack>'
}`;
const logData = { err, log_type: 'error' };
const loggedErr = await logErrorAndReturn(logMsg, logData, err);
await logger.end();
reject(loggedErr);
}
});
});
};
// For backwards compatibility with older versions of Zapier CLI,
// support both callback and Promise styles
const handler = (event, context = {}, callback) => {
// If callback is provided, indicating <v17, call it
if (typeof callback === 'function') {
handlerPromise(event, context)
.then((result) => callback(null, result))
.catch((err) => callback(err));
return;
}
// If no callback is provided, indicating >=v17, return a Promise
return handlerPromise(event, context);
};
return handler;
};
module.exports = createLambdaHandler;

View file

@ -0,0 +1,69 @@
'use strict';
const path = require('path');
const _ = require('lodash');
const semver = require('semver');
const createLegacyScriptingRunner = (z, input) => {
const app = _.get(input, '_zapier.app');
// once we have node 14 everywhere, this can be:
// let source = _.get(app, 'legacy.scriptingSource') ?? app.legacyScriptingSource;
let source = _.get(app, 'legacy.scriptingSource');
source = source === undefined ? app.legacyScriptingSource : source;
if (source === undefined) {
// Don't initialize z.legacyScripting for a pure CLI app
return null;
}
if (!source) {
// Even if the app has no scripting, we still rely on legacy-scripting-runner
// to run some scriptingless operations
source = 'var Zap = {};';
}
// Only UI-built app will have this legacy-scripting-runner dependency, so we
// need to make it an optional dependency
let LegacyScriptingRunner, version;
try {
LegacyScriptingRunner = require('zapier-platform-legacy-scripting-runner');
version =
require('zapier-platform-legacy-scripting-runner/package.json').version;
} catch (e) {
// Find it in cwd, in case we're developing legacy-scripting-runner itself
const cwd = process.cwd();
try {
const pkg = require(path.join(cwd, 'package.json'));
if (pkg.name === 'zapier-platform-legacy-scripting-runner') {
LegacyScriptingRunner = require(cwd);
version = 'dev';
}
} catch (e2) {
// Do nothing
}
if (!LegacyScriptingRunner) {
// Only warn when the package is installed but failed to load
const isNotInstalled =
e.code === 'MODULE_NOT_FOUND' &&
e.message?.includes('zapier-platform-legacy-scripting-runner');
if (!isNotInstalled) {
console.warn(
'Failed to load zapier-platform-legacy-scripting-runner.\nError details:',
e.message,
);
}
return null;
}
}
if (version === 'dev' || semver.gte(version, '3.0.0')) {
return LegacyScriptingRunner(source, z, input);
}
return LegacyScriptingRunner(source, z, app);
};
module.exports = createLegacyScriptingRunner;

View file

@ -0,0 +1,29 @@
'use strict';
const stream = require('stream');
const Console = require('console').Console;
const createLoggerConsole = (input) => {
const doWrite = (data, chunk, encoding, next) => {
const promise = input._zapier.logger(chunk.toString(), data);
// stash the promise in input, so we can wait on it later
input._zapier.promises.push(promise);
next();
(console[data.log_type] || console.log)(chunk.toString());
};
const stdout = new stream.Writable({
write: doWrite.bind(undefined, { log_type: 'console' }),
});
const stderr = new stream.Writable({
write: doWrite.bind(undefined, { log_type: 'error' }),
});
return new Console(stdout, stderr);
};
module.exports = createLoggerConsole;

View file

@ -0,0 +1,370 @@
'use strict';
const { promisify } = require('util');
const { Transform } = require('stream');
const { parse: querystringParse } = require('querystring');
const _ = require('lodash');
const { AbortController } = require('node-abort-controller');
const request = require('./request-client-internal');
const { simpleTruncate, recurseReplace, truncateData } = require('./data');
const {
DEFAULT_LOGGING_HTTP_API_KEY,
DEFAULT_LOGGING_HTTP_ENDPOINT,
SAFE_LOG_KEYS,
} = require('../constants');
const { unheader } = require('./http');
const { scrub, findSensitiveValues } = require('@zapier/secret-scrubber');
const { findSensitiveValuesFromAuthData } = require('./secret-scrubber');
// The payload size per request to stream logs. This should be slighly lower
// than the limit (16 MB) on the server side.
const LOG_STREAM_BYTES_LIMIT = 15 * 1024 * 1024;
const DEFAULT_LOGGER_TIMEOUT = 200;
const sleep = promisify(setTimeout);
const MAX_LENGTH = 3500;
const truncateString = (str) => simpleTruncate(str, MAX_LENGTH, ' [...]');
const formatHeaders = (headers = {}) => {
if (_.isEmpty(headers)) {
return undefined;
}
if (_.isString(headers)) {
// we had a bug where headers coming in as strings weren't getting censored. If something calls this with stringified headers, we'll bow out. Pass the raw object instead.
return 'ERR - refusing to log possibly uncensored headers';
}
return Object.entries(unheader(headers))
.map(([header, value]) => {
return `${header}: ${value}`;
})
.join('\n');
};
const maybeStringify = (d) => {
if (_.isPlainObject(d) || Array.isArray(d)) {
return JSON.stringify(d);
}
return d;
};
// format HTTP request details into string suitable for printing to stdout
const httpDetailsLogMessage = (data) => {
if (data.log_type !== 'http') {
return '';
}
const trimmedData = _.reduce(
data,
(result, value, key) => {
result[key] = value;
if (typeof value === 'string') {
result[key] = truncateString(value);
}
return result;
},
{},
);
if (trimmedData.request_params) {
trimmedData.request_params = '?' + trimmedData.request_params;
}
return `\
${trimmedData.request_method || 'GET'} ${trimmedData.request_url}${
trimmedData.request_params || ''
}
${formatHeaders(trimmedData.request_headers) || ''}
${maybeStringify(trimmedData.request_data) || ''}
${trimmedData.response_status_code || 0}
${formatHeaders(trimmedData.response_headers) || ''}
${maybeStringify(trimmedData.response_content) || ''}
`.trim();
};
const toStdout = (event, msg, data) => {
if (data.log_type === 'http' && event.detailedLogToStdout) {
const extra = httpDetailsLogMessage(data);
if (extra) {
console.log(extra);
}
} else {
console.log(String(msg).replace(/\n$/, ''));
}
};
// try to parse json; if successful, find secrets in it
const attemptFindSecretsInStr = (s, isGettingNewSecret) => {
let parsedRespContent;
try {
parsedRespContent = JSON.parse(s) || {};
} catch {
return [];
}
if (isGettingNewSecret && typeof parsedRespContent === 'string') {
// Likely the response content itself is a secret
return [parsedRespContent];
}
return findSensitiveValues(parsedRespContent);
};
const buildSensitiveValues = (event, data) => {
const bundle = event.bundle || {};
const authData = bundle.authData || {};
const result = [
...findSensitiveValuesFromAuthData(authData),
...findSensitiveValues(process.env),
...findSensitiveValues(data),
];
// for our http logs (genrated by prepareRequestLog), make sure that we try to parse the content to find any new strings
// (such as what comes back in the response during an auth refresh)
const isGettingNewSecret =
event.method &&
(event.method.endsWith('refreshAccessToken') ||
event.method.endsWith('sessionConfig.perform') ||
event.method.endsWith('oauth1Config.getAccessToken'));
for (const prop of ['response_content', 'request_data']) {
if (data[prop]) {
result.push(...attemptFindSecretsInStr(data[prop], isGettingNewSecret));
}
}
if (data.request_params) {
result.push(...findSensitiveValues(querystringParse(data.request_params)));
}
// unique- no point in duplicates
return [...new Set(result)];
};
class LogStream extends Transform {
constructor(options) {
super(options);
this.bytesWritten = 0;
this.controller = new AbortController();
this.request = this._newRequest(options.url, options.token);
}
_newRequest(url, token) {
const httpOptions = {
url,
method: 'POST',
headers: {
'Content-Type': 'application/x-ndjson',
'X-Token': token,
},
body: this,
signal: this.controller.signal,
};
return request(httpOptions).catch((err) => {
if (err.name === 'AbortError') {
return {
status: 200,
content: 'aborted',
};
}
// Swallow logging errors. This will show up in AWS logs at least.
// Don't need to log for AbortError because that happens when we abort
// on purpose.
console.error('Error making log request:', err);
});
}
_transform(chunk, encoding, callback) {
this.push(chunk);
this.bytesWritten += Buffer.byteLength(chunk, encoding);
callback();
}
abort() {
this.controller.abort();
}
}
// Implements singleton for LogStream. The goal is for every sendLog() call we
// reuse the same request until the request body grows too big and exceeds
// LOG_STREAM_BYTES_LIMIT.
class LogStreamFactory {
constructor() {
this._logStream = null;
this.ended = false;
}
getOrCreate(url, token) {
if (this._logStream) {
if (this._logStream.bytesWritten < LOG_STREAM_BYTES_LIMIT) {
// Reuse the same request for efficiency
return this._logStream;
}
// End this one before creating another
this._logStream.end();
}
this._logStream = new LogStream({ url, token });
return this._logStream;
}
// Ends the logger and gets a response from the log server. Optionally takes
// timeoutToAbort to specify how many milliseconds we want to wait before
// force aborting the connection to the log server.
async end(timeoutToAbort = DEFAULT_LOGGER_TIMEOUT) {
// Mark the factory as ended. This suggests that any logStream.write() that
// follows should end() right away.
this.ended = true;
let response;
if (this._logStream) {
this._logStream.end();
const clock =
timeoutToAbort > 0 ? sleep(timeoutToAbort) : Promise.resolve(undefined);
const responsePromise = this._logStream.request;
const result = await Promise.race([clock, responsePromise]);
const isTimeout = !result;
if (isTimeout) {
this._logStream.abort();
// Expect to get a `{content: 'aborted'}` response
response = await responsePromise;
} else {
response = result;
}
this._logStream = null;
}
return response;
}
}
const sendLog = async (logStreamFactory, options, event, message, data) => {
data = _.extend({}, data || {}, event.logExtra || {});
data.log_type = data.log_type || 'console';
data.request_headers = unheader(data.request_headers);
data.response_headers = unheader(data.response_headers);
const sensitiveValues = buildSensitiveValues(event, data);
// data.input and data.output have the ability to grow unbounded; the following caps the size to a reasonable amount
if (data.log_type === 'bundle') {
data.input = truncateData(data.input, MAX_LENGTH);
data.output = truncateData(data.output, MAX_LENGTH);
}
// scrub throws an error if there are no secrets
let safeMessage, safeData;
if (sensitiveValues.length) {
safeMessage = scrub(message, sensitiveValues);
safeData = scrub(data, sensitiveValues);
} else {
safeMessage = message;
safeData = data;
}
let safeKeyData = _.pick(data, SAFE_LOG_KEYS);
if (event.logFieldMaxLength != null && event.logFieldMaxLength >= 0) {
const truncate = (s) =>
simpleTruncate(s, event.logFieldMaxLength, ' [...]');
safeMessage = truncate(safeMessage);
safeData = recurseReplace(safeData, truncate);
safeKeyData = recurseReplace(safeKeyData, truncate);
}
// Keep safe log keys uncensored
Object.entries(safeKeyData).forEach(([key, value]) => {
safeData[key] = value;
});
safeData.request_headers = formatHeaders(safeData.request_headers);
safeData.response_headers = formatHeaders(safeData.response_headers);
if (event.logToStdout) {
toStdout(event, message, safeData);
}
if (event.customLogger && typeof event.customLogger === 'function') {
// For `zapier invoke` command
event.customLogger(safeMessage, safeData);
}
if (options.logBuffer && data.log_type === 'console') {
// Cap size of messages in log buffer, in case devs log humongous things.
options.logBuffer.push({ type: safeData.log_type, message: safeMessage });
}
if (options.token) {
const logStream = logStreamFactory.getOrCreate(
options.endpoint,
options.token,
);
logStream.write(
// JSON Lines format: It's important the serialized JSON object itself has
// no line breaks, and after an object it ends with a line break.
JSON.stringify({ message: safeMessage, data: safeData }) + '\n',
);
if (logStreamFactory.ended) {
// Lambda handler calls logger.end() at the end. But what if there's a
// (bad) callback that is still running after the Lambda handler returns?
// We need to make sure the bad callback ends the logger as well.
// Otherwise, it will hang!
logStreamFactory.end(DEFAULT_LOGGER_TIMEOUT);
}
}
};
/*
Creates low level logging function that POSTs to endpoint (GL by default).
Use internally; do not expose to devs.
Usage:
const logger = createLogger(event, options);
// These will reuse the same request to the log server
logger('log message here', { log_type: 'console' });
logger('another log', { log_type: 'console' });
logger('200 GET https://example.com', { log_type: 'http' });
// After an invocation, the Lambda handler MUST call logger.end() to close
// the log stream. Otherwise, it will hang!
logger.end().finally(() => {
// anything else you want to do to finish an invocation
});
*/
const createLogger = (event, options) => {
options = options || {};
event = event || {};
options = _.defaults(options, {
endpoint: process.env.LOGGING_ENDPOINT || DEFAULT_LOGGING_HTTP_ENDPOINT,
apiKey: process.env.LOGGING_API_KEY || DEFAULT_LOGGING_HTTP_API_KEY,
token: process.env.LOGGING_TOKEN || event.token,
});
const logStreamFactory = new LogStreamFactory();
const logger = sendLog.bind(undefined, logStreamFactory, options, event);
logger.end = async (timeoutToAbort = DEFAULT_LOGGER_TIMEOUT) => {
return logStreamFactory.end(timeoutToAbort);
};
return logger;
};
module.exports = createLogger;

View file

@ -0,0 +1,40 @@
'use strict';
const _ = require('lodash');
const ensureArray = require('./ensure-array');
const request = require('./request-client');
const requestSugar = require('./request-sugar');
const applyMiddleware = require('../middleware');
// before middles
const prepareRequest = require('../http-middlewares/before/prepare-request');
// after middles
const prepareResponse = require('../http-middlewares/after/prepare-response');
const createRequestClient = (befores, afters, options) => {
options = _.defaults({}, options, {
skipDefaultMiddle: false,
skipEnvelope: true,
extraArgs: [],
});
const httpBefores = [];
const httpAfters = [];
if (!options.skipDefaultMiddle) {
httpBefores.push(prepareRequest);
httpAfters.push(prepareResponse);
}
const client = applyMiddleware(
httpBefores.concat(ensureArray(befores)),
httpAfters.concat(ensureArray(afters)),
request,
options,
);
return requestSugar.addUrlOrOptions(client);
};
module.exports = createRequestClient;

View file

@ -0,0 +1,32 @@
'use strict';
const _ = require('lodash');
const uploader = require('./uploader');
const crypto = require('crypto');
const { withRetry } = require('./retry-utils');
// responseStasher uploads the data and returns the URL that points to that data.
const stashResponse = async (input, response) => {
const rpc = _.get(input, '_zapier.rpc');
if (!rpc) {
throw new Error('rpc is not available');
}
const signedPostData = await rpc('get_presigned_upload_post_data');
// Encode the response to base64 to avoid uploading any non-ascii characters
const encodedResponse = Buffer.from(response).toString('base64');
return withRetry(
_.partial(
uploader,
signedPostData,
encodedResponse,
encodedResponse.length,
crypto.randomUUID() + '.txt',
'text/plain',
),
);
};
module.exports = stashResponse;

View file

@ -0,0 +1,155 @@
'use strict';
const _ = require('lodash');
const constants = require('../constants');
const request = require('./request-client-internal');
const { genId } = require('./data');
const RPC_HOSTS = {
'https://zapier.com': 'https://rpc.zapier.com/cli',
'https://zapier-staging.com': 'https://rpc.zapier-staging.com/cli',
};
const FALLBACK_RPC =
RPC_HOSTS[process.env.ZAPIER_BASE_ENDPOINT] ||
(process.env.ZAPIER_BASE_ENDPOINT
? `${process.env.ZAPIER_BASE_ENDPOINT}/platform/rpc/cli`
: RPC_HOSTS['https://zapier.com']);
const rpcCacheMock = (zcacheTestObj, method, key, value = null, ttl = null) => {
if (method === 'zcache_get') {
const result = key in zcacheTestObj ? zcacheTestObj[key] : null;
return result;
}
if (method === 'zcache_set') {
zcacheTestObj[key] = value;
return true;
}
if (method === 'zcache_delete') {
if (key in zcacheTestObj) {
delete zcacheTestObj[key];
return true;
}
return false;
}
throw new Error(`Unexpected method '${method}'`);
};
const rpcCursorMock = (cursorTestObj, method, key, value = null) => {
if (method === 'get_cursor') {
return cursorTestObj[key] || null;
}
if (method === 'set_cursor') {
cursorTestObj[key] = value;
return null;
}
throw new Error(`Unexpected method '${method}'`);
};
const createRpcClient = (event) => {
return async function (method) {
const params = _.toArray(arguments);
params.shift();
const zcacheMethods = ['zcache_get', 'zcache_set', 'zcache_delete'];
if (
zcacheMethods.includes(method) &&
_.isPlainObject(event.zcacheTestObj)
) {
const [key, value = null] = params;
return rpcCacheMock(event.zcacheTestObj, method, key, value);
}
const cursorMethods = ['get_cursor', 'set_cursor'];
if (
cursorMethods.includes(method) &&
_.isPlainObject(event.cursorTestObj)
) {
const [key, value = null] = params;
return rpcCursorMock(event.cursorTestObj, method, key, value);
}
const id = genId();
const body = JSON.stringify({
id,
storeKey: event.storeKey,
method,
params,
});
const req = {
method: 'POST',
url: `${event.rpc_base || FALLBACK_RPC}`,
body,
headers: {},
};
if (event.token) {
req.headers['X-Token'] = event.token;
} else if (process.env.ZAPIER_DEPLOY_KEY) {
req.headers['X-Deploy-Key'] = process.env.ZAPIER_DEPLOY_KEY;
} else {
if (constants.IS_TESTING) {
throw new Error(
'No deploy key found. Make sure you set the `ZAPIER_DEPLOY_KEY` environment variable ' +
'to write tests that rely on the RPC API (i.e. z.stashFile)',
);
} else {
throw new Error('No token found - cannot call RPC');
}
}
// RPC can fail, so let's retry.
// Be careful what we throw here as this will be forwarded to the user.
const maxRetries = 3;
let attempt = 0;
let res;
while (attempt < maxRetries) {
// We will throw here, which will be caught by catch logic to either retry or bubble up.
try {
res = await request(req);
if (res.status >= 500) {
throw new Error('Unable to reach the RPC server');
}
if (res.status === 413) {
throw new Error('The request is too large to be processed');
}
if (res.content) {
// check if the ids match
if (res.content.id !== id) {
throw new Error(
`Got id ${res.content.id} but expected ${id} when calling RPC`,
);
}
if (res.content.error) {
throw new Error(res.content.error);
}
return res.content.result;
} else {
throw new Error(`Got a ${res.status} when calling RPC`);
}
} catch (err) {
attempt++;
if (attempt === maxRetries || (res && res.status < 500)) {
throw new Error(
`RPC request failed after ${attempt} attempts: ${err.message}`,
);
}
// sleep for 100ms before retrying
await new Promise((resolve) => setTimeout(resolve, 100));
}
}
};
};
module.exports = createRpcClient;

View file

@ -0,0 +1,31 @@
'use strict';
const _ = require('lodash');
const createStoreKeyTool = (input) => {
const rpc = _.get(input, '_zapier.rpc');
return {
get: () => {
if (!rpc) {
return Promise.reject(new Error('rpc is not available'));
}
return rpc('get_cursor');
},
set: (cursor) => {
if (!rpc) {
return Promise.reject(new Error('rpc is not available'));
}
if (!_.isString(cursor)) {
return Promise.reject(new TypeError('cursor value must be a string'));
}
return rpc('set_cursor', cursor);
},
};
};
module.exports = createStoreKeyTool;

View file

@ -0,0 +1,346 @@
'use strict';
const _ = require('lodash');
const memoize = require('./memoize');
const plainModule = require('./plain');
const isPlainObj = (o) => {
return (
o &&
typeof o === 'object' &&
(o.constructor === Object || o.constructor === plainModule.constructor)
);
};
const comparison = (obj, needle) => obj === needle;
const getObjectType = (obj) => {
if (_.isPlainObject(obj)) {
return 'Object';
}
if (Array.isArray(obj)) {
return 'Array';
}
return _.capitalize(typeof obj);
};
// Returns a path for the deeply nested haystack where
// you could find the needle. If the needle is a plain
// object we try _.isEqual (which could be slow!).
// TODO: might be nice to WeakMap memoize
const findMapDeep = (haystack, needle, comp) => {
comp = comp || comparison;
const finder = (obj, path) => {
path = path || [];
if (comp(obj, needle)) {
return path;
} else if (Array.isArray(obj)) {
for (let i = obj.length - 1; i >= 0; i--) {
const value = obj[i];
const found = finder(value, path.concat([`[${i}]`]));
if (found !== undefined) {
return found;
}
}
} else if (isPlainObj(obj)) {
const keys = Object.keys(obj);
for (let i = keys.length - 1; i >= 0; i--) {
const key = keys[i];
const value = obj[key];
const found = finder(value, path.concat([`.${key}`]));
if (found !== undefined) {
return found;
}
}
}
return undefined;
};
const path = finder(haystack);
if (path && path.length) {
return _.trim(path.join(''), '.');
} else {
return undefined;
}
};
const memoizedFindMapDeep = memoize(findMapDeep);
const deepCopy = (obj) => {
return _.cloneDeepWith(obj, (value) => {
if (_.isFunction(value)) {
return value;
}
return undefined;
});
};
const jsonCopy = (obj) => {
return JSON.parse(JSON.stringify(obj));
};
const deepFreeze = (obj) => {
Object.freeze(obj);
Object.getOwnPropertyNames(obj).forEach(function (prop) {
if (
Object.prototype.hasOwnProperty.call(obj, prop) && // https://eslint.org/docs/rules/no-prototype-builtins
(typeof obj[prop] === 'object' || typeof obj[prop] === 'function') &&
obj[prop] !== null &&
!Object.isFrozen(obj[prop])
) {
deepFreeze(obj[prop]);
}
});
return obj;
};
// Recurse a nested object replace stuff according to the function.
const recurseReplace = (obj, replacer, options = {}) => {
if (options.all) {
obj = replacer(obj);
}
if (Array.isArray(obj)) {
return obj.map((value) => {
return recurseReplace(value, replacer, options);
});
} else if (isPlainObj(obj)) {
const newObj = {};
Object.keys(obj).forEach((key) => {
const value = obj[key];
newObj[key] = recurseReplace(value, replacer, options);
});
return newObj;
} else {
obj = replacer(obj);
}
return obj;
};
const _IGNORE = {};
// Flatten a nested object.
const flattenPaths = (data, { preserve = {} } = {}) => {
const out = {};
const recurse = (obj, prop = '') => {
if (_.isPlainObject(obj)) {
Object.entries(obj).forEach(([key, value]) => {
const newProp = prop ? `${prop}.${key}` : key;
if (preserve[prop]) {
out[newProp] = value;
}
const subValue = recurse(value, newProp);
if (subValue !== _IGNORE) {
out[newProp] = subValue;
}
});
return _IGNORE;
} else {
return obj;
}
};
recurse(data);
return out;
};
// A simpler, and memory-friendlier version of _.truncate()
const simpleTruncate = (string, length, suffix) => {
if (string == null) {
return string;
}
if (!string || !string.toString) {
return '';
}
const finalString = string.toString();
if (finalString.length === 0) {
return '';
} else if (finalString.length > length) {
const cutoff = suffix ? length - suffix.length : length;
return finalString.substr(0, cutoff) + (suffix || '');
}
return string;
};
/**
* Adds an item to an object or array.
* If the parent is an object, the value will be set at the specified key.
* If the parent is an array, the value will be added to the end of the array (and the key will be ignored).
* Used by truncateData.
*
* @param {object | any[]} parent An object or array.
* @param {string} key The key to set the value at (objects only; ignored for arrays).
* @param {any} value The value to add.
*/
const _addItem = (parent, key, value) => {
if (Array.isArray(parent)) {
parent.push(value);
} else {
parent[key] = value;
}
};
/**
* Examines an object entry or array entry (`item`) and determines its "cost" (how many characters will it take to add it to `parent`).
* If the item is a string, `availableSpace` is used to determine if the string should be truncated.
* If the item is an object or array, its entries / values are added to `queue`.
* Used by truncateData.
*
* @param {any[]} queue
* @param {object | any[]} parent
* @param {string} key
* @param {any} item
* @param {number} availableSpace
* @returns
*/
const _processItem = (queue, parent, key, item, availableSpace) => {
let itemLength = 0;
let itemToAdd = item;
let wasTruncated = false;
itemLength += key.length; // array keys are empty strings, so this is a noop
itemLength += key.length ? 3 : 0; // objects get +2 for "" around the key and +1 for the :
itemLength += 1; // arrays and objects both have +1 for commas between entries
if (parent && typeof parent === 'object' && _.isEmpty(parent)) {
itemLength -= 1; // this is the first entry for an object or array; remove the count for a comma
}
if (typeof item === 'number' || typeof item === 'boolean' || item == null) {
itemLength += String(item).length;
} else if (typeof item === 'string') {
const overhead = itemLength + 2; // the minimum amount of space needed after truncation
if (item.length + overhead > availableSpace) {
// this string is going to push us over the edge; truncate it
itemToAdd = simpleTruncate(item, availableSpace - overhead, ' [...]');
wasTruncated = true;
}
itemLength += itemToAdd.length + 2; // 2 for quotes around the string value
} else if (typeof item === 'object') {
itemLength += 2; // '{}' or '[]'
let entries;
if (Array.isArray(item)) {
const newArr = [];
itemToAdd = newArr;
entries = item.map((subValue) => [newArr, '', subValue]);
} else {
const newObj = {};
itemToAdd = newObj;
entries = Object.entries(item).map(([subKey, subValue]) => [
newObj,
subKey,
subValue,
]);
}
queue.unshift(...entries);
} else {
// JSON.stringify doesn't usually really do anything for any other typeofs
// we're just going to use `undefined` and hope for the best
itemLength += 'undefined'.length;
itemToAdd = undefined;
}
return [itemLength, itemToAdd, wasTruncated];
};
/**
* Takes a given `data` object or array and copies pieces of that data into `output` until its stringified length fits in `maxLength` characters.
*
* In general, output should track with `JSON.stringify(item).substring(0, maxLength)` (i.e. depth-first traversal of arrays and object entries), but in a JSON-aware way.
* If the item's initial stringified length is less than or equal to `maxLength`, the item is returned as-is.
* @param {object | any[]} data The JSON object or array to be truncated.
* @param {number} maxLength The maximum length of JSON.stringify(output). Note that this may not be the exact output length, but it serves as an upper bound. Minimum value is 40.
* @returns {object | any[]} The truncated object or array.
*/
const truncateData = (data, maxLength) => {
if (!data || typeof data !== 'object') {
// the following code is only meant to work on objects and arrays
return data;
}
if (JSON.stringify(data).length <= maxLength) {
// no need to truncate
return data;
}
const root = Array.isArray(data) ? [] : {};
let length = 2; // '{}' or '[]'
let dataWasTruncated = false; // used during iteration to track if a string was truncated
const truncateMessageSize = 39; // the overhead required to add a message about truncating data
if (maxLength < 40) {
// adding the truncate message takes 39 characters, but the minimum output (i.e. just the message wrapped in an object or array)
// is 40 characters due to the overhead of the {} or [] characters (+2) minus the comma (-1)
throw new Error(`maxLength must be at least 40`);
}
const queue = Array.isArray(data)
? data.map((value) => [root, '', value])
: Object.entries(data).map(([key, value]) => [root, key, value]);
// iterate over the queue
while (queue.length > 0) {
const [parent, key, item] = queue.shift();
const [itemLength, processedItem, itemWasTruncated] = _processItem(
queue,
parent,
key,
item,
maxLength - length - truncateMessageSize,
);
if (itemWasTruncated) {
// if a string was truncated, we mark the total data as truncated for messaging purposes
dataWasTruncated = true;
}
if (length + itemLength + truncateMessageSize < maxLength) {
// we're still under the max length, add this and keep going
_addItem(parent, key, processedItem);
length += itemLength;
} else {
if (length + itemLength + truncateMessageSize === maxLength) {
// we can fit this item + the truncate message, so let's add it before we stop
_addItem(parent, key, processedItem);
}
dataWasTruncated = true;
break;
}
}
// we can hit the following even if we got through all the items in the queue in the case that any strings were truncated
if (dataWasTruncated) {
if (Array.isArray(root)) {
root.push('NOTE : This data has been truncated.');
} else {
root.NOTE = 'This data has been truncated.';
}
}
return root;
};
const genId = () => parseInt(Math.random() * 100000000);
module.exports = {
deepCopy,
deepFreeze,
findMapDeep,
flattenPaths,
genId,
getObjectType,
isPlainObj,
jsonCopy,
memoizedFindMapDeep,
recurseReplace,
simpleTruncate,
truncateData,
};

View file

@ -0,0 +1,15 @@
'use strict';
const _ = require('lodash');
const ensureArray = (maybeArray) => {
if (_.isArray(maybeArray)) {
return maybeArray;
}
if (_.isNil(maybeArray)) {
return [];
}
return [maybeArray];
};
module.exports = ensureArray;

View file

@ -0,0 +1,36 @@
'use strict';
const _ = require('lodash');
const ensureJSONEncodable = (obj, path = null, visited = null) => {
if (obj === null || _.isBoolean(obj) || _.isNumber(obj) || _.isString(obj)) {
return;
}
path = path || [];
if (!_.isPlainObject(obj) && !_.isArray(obj)) {
const typeName = typeof obj;
const pathStr = path.join('.');
throw new TypeError(
`Type '${typeName}' is not JSON-encodable (path: '${pathStr}')`,
);
}
visited = visited || new Set();
if (visited.has(obj)) {
const pathStr = path.join('.');
throw new TypeError(
`Circular structure is not JSON-encodable (path: '${pathStr}')`,
);
}
visited.add(obj);
for (const key in obj) {
ensureJSONEncodable(obj[key], path.concat(key), visited);
}
};
module.exports = ensureJSONEncodable;

View file

@ -0,0 +1,12 @@
'use strict';
const ensurePath = (obj, path) => {
obj = obj || {};
path.split('.').reduce((coll, key) => {
coll[key] = coll[key] || {};
return coll[key];
}, obj);
return obj;
};
module.exports = ensurePath;

View file

@ -0,0 +1,23 @@
'use strict';
const _ = require('lodash');
const STATUSES = require('../constants').STATUSES;
/*
An envelope is a dumb "wrapper" for results - allowing us
to do fancy things in the future like add more context or
stash large results.
*/
const OUTPUT_ENVELOPE_TYPE = 'OutputEnvelope';
const isOutputEnvelope = (obj) =>
_.isObject(obj) && obj.__type === OUTPUT_ENVELOPE_TYPE;
const ensureOutputEnvelope = (results) =>
isOutputEnvelope(results)
? results
: { __type: OUTPUT_ENVELOPE_TYPE, results, status: STATUSES.SUCCESS };
module.exports = {
ensureOutputEnvelope,
isOutputEnvelope,
};

View file

@ -0,0 +1,73 @@
'use strict';
const _ = require('lodash');
const path = require('path');
const dotenv = require('dotenv');
const ensurePath = require('./ensure-path');
const { IS_TESTING } = require('../constants');
const ENV_VARS_TO_CLEAN = ['_ZAPIER_ONE_TIME_SECRET'];
// Copy bundle environment into process.env, and vice versa,
// for convenience and compatibility with native environment vars.
const applyEnvironment = (event) => {
event = ensurePath(event, 'bundle');
_.extend(process.env, event.environment || {});
};
// Remove junk from process.env.
const cleanEnvironment = () => {
// not really a security measure - just prevent useless security bounty emails
if (
!process.env.ZAPIER_SUPPRESS_CLEAN_ENVIRONMENT &&
(process.env.AWS_LAMBDA_FUNCTION_VERSION ||
process.env.AWS_LAMBDA_FUNCTION_NAME)
) {
delete process.env.AWS_ACCESS_KEY_ID;
delete process.env.AWS_SECURITY_TOKEN;
delete process.env.AWS_SESSION_TOKEN;
delete process.env.AWS_SECRET_ACCESS_KEY;
}
// Lambda may reuse container, which leaves leftovers in process.env. Let's clean
// up Zapier specific stuff here.
ENV_VARS_TO_CLEAN.forEach((name) => {
delete process.env[name];
});
};
const localFilepath = (filename) => {
return path.join(process.cwd(), filename || '');
};
const injectEnvironmentFile = (filename) => {
if (filename) {
filename = localFilepath(filename);
}
// reads ".env" if filename is falsy, needs full path otherwise
let result = dotenv.config({ path: filename, quiet: true });
if (result.error) {
// backwards compatibility
result = dotenv.config({
path: localFilepath('.environment'),
quiet: true,
});
if (result.parsed && !IS_TESTING) {
console.log(
[
'\nWARNING: `.environment` files will no longer be read by default in the next major version.',
'Either rename your file to `.env` or explicitly call this function with a filename:',
'\n zapier.tools.env.inject(".environment");\n\n',
].join('\n'),
);
}
}
};
module.exports = {
applyEnvironment,
cleanEnvironment,
injectEnvironmentFile,
};

View file

@ -0,0 +1,8 @@
const injectEnvironmentFile = require('./environment').injectEnvironmentFile;
// Intended to be available on zapier.tools - IE: zapier.tools.env.inject();
module.exports = {
env: {
inject: injectEnvironmentFile,
},
};

View file

@ -0,0 +1,119 @@
const _ = require('lodash');
const { ALLOWED_HTTP_DATA_CONTENT_TYPES } = require('./http');
const stringifyRequestData = (data) => {
// Be careful not to consume the data if it's a stream
if (typeof data === 'string') {
return data;
} else if (data instanceof URLSearchParams) {
return data.toString();
} else if (global.FormData && data instanceof global.FormData) {
return '<FormData>';
} else {
// See https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#setting_a_body
// for other possible body types
return '<unsupported format>';
}
};
const normalizeRequestInfo = (input, init) => {
const result = {
method: 'GET',
headers: {},
data: '',
};
if (global.Request && input instanceof global.Request) {
result.url = input.url;
result.method = input.method || result.method;
result.headers =
Object.fromEntries(input.headers.entries()) || result.headers;
if (input.body) {
result.data = stringifyRequestData(input.body);
}
} else if (typeof input.toString === 'function') {
// This condition includes `typeof input === 'string'` as well
result.url = input.toString();
} else {
// Don't log if we don't recognize the input type
return null;
}
if (init) {
result.method = init.method || result.method;
result.headers = init.headers || result.headers;
if (init.body) {
result.data = stringifyRequestData(init.body);
}
}
return result;
};
// Is this request made by z.request()?
const isZapierUserAgent = (headers) =>
_.get(headers, 'user-agent', []).indexOf('Zapier') !== -1;
const shouldIncludeResponseContent = (contentType) => {
if (!contentType) {
return false;
}
for (const ctype of ALLOWED_HTTP_DATA_CONTENT_TYPES) {
if (contentType.includes(ctype)) {
return true;
}
}
return false;
};
const stringifyResponseContent = async (response) => {
// Be careful not to consume the original response body, which is why we clone it
return response.clone().text();
};
// Usage:
// global.fetch = wrapFetchWithLogger(global.fetch, logger);
const wrapFetchWithLogger = (fetchFunc, logger) => {
if (fetchFunc.patchedByZapier) {
// Important not to reuse logger between calls, because we always destroy
// the logger at the end of a Lambda call.
fetchFunc.zapierLogger = logger;
return fetchFunc;
}
const newFetch = async function (input, init) {
const response = await fetchFunc(input, init);
const requestInfo = normalizeRequestInfo(input, init);
if (requestInfo && !isZapierUserAgent(requestInfo.headers)) {
const responseContentType = response.headers.get('content-type');
newFetch.zapierLogger(
`${response.status} ${requestInfo.method} ${requestInfo.url}`,
{
log_type: 'http',
request_type: 'patched-devplatform-outbound',
request_url: requestInfo.url,
request_method: requestInfo.method,
request_headers: requestInfo.headers,
request_data: requestInfo.data,
request_via_client: false,
response_status_code: response.status,
response_headers: Object.fromEntries(response.headers.entries()),
response_content: shouldIncludeResponseContent(responseContentType)
? await stringifyResponseContent(response)
: '<unsupported format>',
},
);
}
return response;
};
newFetch.patchedByZapier = true;
newFetch.zapierLogger = logger;
return newFetch;
};
module.exports = wrapFetchWithLogger;

View file

@ -0,0 +1,82 @@
'use strict';
const { Writable } = require('stream');
const fetch = require('node-fetch');
// XXX: PatchedRequest is to get past node-fetch's check that forbids GET requests
// from having a body here:
// https://github.com/node-fetch/node-fetch/blob/v2.6.0/src/request.js#L75-L78
class PatchedRequest extends fetch.Request {
constructor(url, opts) {
const origMethod = ((opts && opts.method) || 'GET').toUpperCase();
const isGetWithBody =
(origMethod === 'GET' || origMethod === 'HEAD') && opts && opts.body;
let newOpts = opts;
if (isGetWithBody) {
// Temporary remove body to fool fetch.Request constructor
newOpts = { ...opts, body: null };
}
super(url, newOpts);
this._isGetWithBody = isGetWithBody;
if (isGetWithBody) {
// Restore the body. The body is stored internally using a Symbol key. We
// can't just do this[Symbol('Body internals')] as the symbol is internal.
// We need to use Object.getOwnPropertySymbols() to get all the keys and
// find the one that holds the body.
const keys = Object.getOwnPropertySymbols(this);
for (const k of keys) {
if (this[k].body !== undefined) {
this[k].body = Buffer.from(String(opts.body));
break;
}
}
}
}
get body() {
if (this._isGetWithBody && !this._bodyCalled) {
// This assumes node-fetch's check that disallows a GET request to have a
// body happens on the first time it calls this.body. Might not work if
// node-fetch breaks the assumption.
this._bodyCalled = true;
return null;
}
return super.body;
}
}
const newFetch = (url, opts) => {
const request = new PatchedRequest(url, opts);
// fetch actually accepts a Request object as an argument. It'll clone the
// request internally, that's why the PatchedRequest.body hack works.
const responsePromise = fetch(request);
// node-fetch clones request.body and use the cloned body internally. We need
// to make sure to consume the original body stream so its internal buffer is
// not filled up, which causes it to pause.
// See https://github.com/node-fetch/node-fetch/issues/151
//
// Exclude form-data object to be consistent with
// https://github.com/node-fetch/node-fetch/blob/v2.6.6/src/body.js#L403-L412
if (
request.body &&
typeof request.body.pipe === 'function' &&
typeof request.body.getBoundary !== 'function'
) {
const nullStream = new Writable();
nullStream._write = function (chunk, encoding, done) {
done();
};
request.body.pipe(nullStream);
}
return responsePromise;
};
module.exports = newFetch;

View file

@ -0,0 +1,32 @@
'use strict';
const crypto = require('crypto');
// Helpful handler for doing z.hash('sha256', 'my password')
const hashify = (algo, s, encoding, inputEncoding) => {
encoding = encoding || 'hex';
inputEncoding = inputEncoding || 'binary';
const hasher = crypto.createHash(algo);
hasher.update(s, inputEncoding);
return hasher.digest(encoding);
};
// Clean up sensitive values in a hashed manner so they don't get logged.
const snipify = (s) => {
if (!['string', 'number'].includes(typeof s)) {
return null;
}
const str = String(s);
const length = str.length;
const salted = str + (process.env.SECRET_SALT || 'doesntmatterreally');
const hashed = hashify('sha256', salted);
return `:censored:${length}:${hashed.substr(0, 10)}:`;
};
const md5 = (s) => crypto.createHash('md5').update(s).digest('hex');
module.exports = {
hashify,
md5,
snipify,
};

View file

@ -0,0 +1,134 @@
const _ = require('lodash');
const fetch = require('node-fetch');
const FORM_TYPE = 'application/x-www-form-urlencoded';
const JSON_TYPE = 'application/json';
const JSON_TYPE_UTF8 = 'application/json; charset=utf-8';
const BINARY_TYPE = 'application/octet-stream';
const HTML_TYPE = 'text/html';
const TEXT_TYPE = 'text/plain';
const TEXT_TYPE_UTF8 = 'text/plain; charset=utf-8';
const YAML_TYPE = 'application/yaml';
const XML_TEXT_TYPE = 'text/xml';
const XML_APPLICATION_TYPE = 'application/xml';
const JSONAPI_TYPE = 'application/vnd.api+json';
const ALLOWED_HTTP_DATA_CONTENT_TYPES = new Set([
FORM_TYPE,
JSON_TYPE,
JSON_TYPE_UTF8,
HTML_TYPE,
TEXT_TYPE,
TEXT_TYPE_UTF8,
YAML_TYPE,
XML_TEXT_TYPE,
XML_APPLICATION_TYPE,
JSONAPI_TYPE,
]);
const getContentType = (headers) => {
for (const [key, value] of Object.entries(headers)) {
if (key.toLowerCase() === 'content-type') {
if (Array.isArray(value) && value.length > 0) {
return value[0];
} else if (typeof value === 'string') {
return value;
}
return null;
}
}
return null;
};
// This function splits a comma-separated string described by RFC 2068 Section 2.
// Ported from https://github.com/python/cpython/blob/f081fd83/Lib/urllib/request.py#L1399-L1440
const parseHttpList = (s) => {
const res = [];
let part = '';
let escape = false;
let quote = false;
for (let i = 0; i < s.length; i++) {
const cur = s.charAt(i);
if (escape) {
part += cur;
escape = false;
continue;
}
if (quote) {
if (cur === '\\') {
escape = true;
continue;
} else if (cur === '"') {
quote = false;
}
part += cur;
continue;
}
if (cur === ',') {
res.push(part);
part = '';
continue;
}
if (cur === '"') {
quote = true;
}
part += cur;
}
if (part) {
res.push(part);
}
return res.map((x) => x.trim());
};
// Parse lists of key, value pairs as described by RFC 2068 Section 2 and convert them
// into an associative array.
// Ported from https://github.com/requests/requests/blob/d2962f1d/requests/utils.py#L342-L373
const parseDictHeader = (s) => {
const res = {};
const items = parseHttpList(s);
items.forEach((item) => {
if (item.includes('=')) {
const parts = item.split('=');
const name = parts[0];
let value = parts.slice(1).join('=');
if (value.charAt(0) === '"' && value.charAt(value.length - 1) === '"') {
value = value.substring(1, value.length - 1);
}
res[name] = value;
} else {
res[item] = null;
}
});
return res;
};
const unheader = (h) =>
h instanceof fetch.Headers && _.isFunction(h.toJSON) ? h.toJSON() : h;
module.exports = {
FORM_TYPE,
JSON_TYPE,
JSON_TYPE_UTF8,
BINARY_TYPE,
HTML_TYPE,
TEXT_TYPE,
YAML_TYPE,
XML_TEXT_TYPE,
XML_APPLICATION_TYPE,
JSONAPI_TYPE,
ALLOWED_HTTP_DATA_CONTENT_TYPES,
getContentType,
parseDictHeader,
unheader,
};

View file

@ -0,0 +1,218 @@
'use strict';
/* eslint-disable */
// Extracted from assets/app/common/memoize.js. Author @shauser.
var _slicedToArray = (function () {
function sliceIterator(arr, i) {
var _arr = [];
var _n = true;
var _d = false;
var _e = undefined;
try {
for (
var _i = arr[Symbol.iterator](), _s;
!(_n = (_s = _i.next()).done);
_n = true
) {
_arr.push(_s.value);
if (i && _arr.length === i) break;
}
} catch (err) {
_d = true;
_e = err;
} finally {
try {
if (!_n && _i['return']) _i['return']();
} finally {
if (_d) throw _e;
}
}
return _arr;
}
return function (arr, i) {
if (Array.isArray(arr)) {
return arr;
} else if (Symbol.iterator in Object(arr)) {
return sliceIterator(arr, i);
} else {
throw new TypeError(
'Invalid attempt to destructure non-iterable instance',
);
}
};
})();
var _lodash = require('lodash');
var _lodash2 = _interopRequireDefault(_lodash);
function _interopRequireDefault(obj) {
return obj && obj.__esModule ? obj : { default: obj };
}
// Prevents `map` from becoming too large and taking up
// a huge amount of memory. Once `map.size` reaches `max`,
// `step` items are removed from `map`, with the oldest
// items being removed first.
var enforceSize = function enforceSize(max, step, map) {
// If `map` isn't too big yet then we don't need to do anything.
if (map.size < max) {
return;
}
// Otherwise we need to trim it down to `targetSize`
var targetSize = max - step;
// ...in a `for` loop so we can bail once we have
// trimmed it down enough.
var _iteratorNormalCompletion = true;
var _didIteratorError = false;
var _iteratorError = undefined;
try {
for (
var _iterator = map[Symbol.iterator](), _step;
!(_iteratorNormalCompletion = (_step = _iterator.next()).done);
_iteratorNormalCompletion = true
) {
var _step$value = _slicedToArray(_step.value, 1),
key = _step$value[0];
// If we're still too big, slim down...
if (map.size > targetSize) {
map.delete(key);
} else {
// ...otherwise break early since we're slim enough.
return;
}
}
} catch (err) {
_didIteratorError = true;
_iteratorError = err;
} finally {
try {
if (!_iteratorNormalCompletion && _iterator.return) {
_iterator.return();
}
} finally {
if (_didIteratorError) {
throw _iteratorError;
}
}
}
};
// Maps primitive keys to object values so that
// object values can be used in `WeakMap`.
var primitiveMap = new Map();
// Returns the object mapped to `primitive`.
var getObjectForPrimitive = function getObjectForPrimitive(primitive) {
return primitiveMap.get(primitive);
};
// Creates an object for a `primitive` so that `primitive`
// can be "used" as a key within a `WeakMap`.
var setObjectForPrimitive = function setObjectForPrimitive(primitive) {
primitiveMap.set(primitive, {});
// Prevent `primitiveMap` from becoming massive and taking
// up a huge chunk of memory.
enforceSize(1000, 100, primitiveMap);
return getObjectForPrimitive(primitive);
};
// Top level `WeakMap` that holds all of the cached values.
// The trunk/base of the `WeakMap` tree.
var weakCache = new WeakMap();
// Normalizes `arg` to an object if it isn't already an object
// since `WeakMap` keys must be objects. Does _not_ try to
// create an object for `arg` if it's a primitive; it only
// looks up existing objects.
var normalizeArg = function normalizeArg(arg) {
return _lodash2.default.isObject(arg) ? arg : getObjectForPrimitive(arg);
};
// Trees are `WeakMap`s all the way down and the last
// one will have this special `valueKey` in it.
var valueKey = {};
// Looks up the memoized value in the `WeakMap` tree
// based on `args`, returning `undefined` if it's not found.
var getMemoizedValue = function getMemoizedValue(args) {
var valueMap = args.reduce(function (map, arg) {
// `map` should *always* be a `WeakMap` or `undefined` at this point.
// It's `undefined` if the value hasn't been memoized yet.
if (!(map && map.get)) {
return undefined;
}
// Need to ensure `arg` is an object, however since this is
// a lookup function we won't try to create an object for it
// which would be useless.
var argObject = normalizeArg(arg);
// This should return a `WeakMap` if there's a cached value
// for `argObject` or `undefined` if there isn't one.
return map.get(argObject);
}, weakCache);
// If there is a cached value then `valueMap` will be a `WeakMap`.
// If the value hasn't been cached then `valueMap` will be
// `undefined`. We'll explicitly return `undefined` in the ternary.
return valueMap ? valueMap.get(valueKey) : undefined;
};
// Creates a `WeakMap` tree that points from each item
// in the `args` array to `valueToMemoize`.
// Returns `valueToMemoize`.
var memoizeValue = function memoizeValue(args, valueToMemoize) {
return (
args
.reduce(function (map, arg, idx) {
// Get or create an object for `arg` depending
// on if it's a primitive value because `WeakMap`
// necessitates objects for keys.
var argObject = normalizeArg(arg) || setObjectForPrimitive(arg);
// If there's no key in `map` for `argObject` yet,
// we need to add it.
if (!map.has(argObject)) {
var isLast = idx === args.length - 1;
var newMap = new WeakMap();
// If this is the last argument then we need to
// associate the value in the `newMap`.
if (isLast) {
newMap.set(valueKey, valueToMemoize);
}
map.set(argObject, newMap);
}
// Should always return a `WeakMap`.
return map.get(argObject);
// Note that the `|| valueToMemoize` is a workaround for IE11.
// Early versions of it won't `set` frozen objects as keys in
// `WeakMap`s which results in the above `set` potentially failing
// and therefore not returning the value.
}, weakCache)
.get(valueKey) || valueToMemoize
);
};
// Returns a function whose arguments will create a `WeakMap`
// tree that should self-garbage collect. For example if there
// are arguments `a`, `b`, and `c`, `a` will point to a `WeakMap`
// with a key of `b`, which will point to a `WeakMap` with a key
// of `c`, which will point to the result of `fn`.
var memoize = function memoize(fn) {
return function () {
for (
var _len = arguments.length, args = Array(_len), _key = 0;
_key < _len;
_key++
) {
args[_key] = arguments[_key];
}
return (
getMemoizedValue(args) || memoizeValue(args, fn.apply(undefined, args))
);
};
};
module.exports = memoize;

View file

@ -0,0 +1,46 @@
'use strict';
const constants = require('../constants');
const zid = Math.round(Math.random() * Math.pow(10, 15)).toString(16);
let zrun = 0;
const checkMemory = (event) => {
event = event || {};
let memUsage;
try {
memUsage = process.memoryUsage();
} catch (err) {
if (err.code === 'EMFILE') {
console.error(
'Force killing process by Zapier for too many open file descriptors',
);
process.exit(1);
} else {
throw err;
}
}
zrun += 1;
if (!constants.IS_TESTING && !event.calledFromCli) {
console.log('ZID:', zid, 'pid', 'ZRUN:', zrun, 'RSSMEM:', memUsage);
}
if (
zrun > constants.KILL_MIN_LIMIT &&
memUsage.rss > constants.KILL_MAX_LIMIT
) {
// should throw "Process exited before completing request"
// and a @retry in our stack will attempt again - and this
// process will get restarted
console.error('Force killing process by Zapier for memory usage');
/* eslint no-process-exit: 0 */
process.exit(1);
}
};
module.exports = checkMemory;

Some files were not shown because too many files have changed in this diff Show more