Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local testing for functions using hs functions test command #389

Merged
merged 89 commits into from
Jan 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
b14fe87
got module import working
miketalley Nov 17, 2020
1c591d6
got function executing and getting environment variables
miketalley Nov 17, 2020
4850305
passed through sendResponse value to be displayed
miketalley Nov 17, 2020
f86d824
forwarded environment variables to function
miketalley Nov 17, 2020
49e18a3
cleanup and error handling
miketalley Nov 17, 2020
54a38b9
cleanup and adding todos
miketalley Nov 17, 2020
1783174
added non-test file to ignore for jest
miketalley Nov 17, 2020
ec0ecf8
added support for multiple valued of method
miketalley Nov 18, 2020
967d62f
handled functions with no package.json
miketalley Nov 18, 2020
2641989
added error handling for invalid files and directories
miketalley Nov 19, 2020
05674aa
local function environment variables overwrite global environment var…
miketalley Nov 19, 2020
9fb1dcc
made sure to close server on sigint
miketalley Nov 19, 2020
5ae1ea2
added .env file to ignore list for upload
miketalley Nov 19, 2020
5bdbf60
adding some minor fixes
miketalley Nov 20, 2020
f0ecbf1
improved cleanup
miketalley Nov 20, 2020
89082dc
local function tests are done within temporary dir that is cleaned up…
miketalley Nov 20, 2020
6af5891
improved error
miketalley Nov 23, 2020
3ff3ed7
improved cleanup to prevent server from continuing to run when exiting
miketalley Nov 23, 2020
6c4c2ba
formatted data passed to function to mimic shape used in prod
miketalley Nov 23, 2020
25ae9dc
updated environment variables to load at runtime
miketalley Nov 23, 2020
403bf05
updated todo
miketalley Nov 23, 2020
ec6c2cc
added warning for reserved AWS variable use
miketalley Nov 24, 2020
c494675
secrets are now properly loaded based on array values in serverless.json
miketalley Nov 24, 2020
6e3369c
added warning for exceeding secrets limit
miketalley Nov 24, 2020
a15e4d0
added warning for exceeding max dependencies in package.json
miketalley Nov 24, 2020
397b3b1
removed npm install output from logging
miketalley Nov 24, 2020
a65fd42
updated todo
miketalley Nov 24, 2020
e19157b
added function runtime counter and warning for exceeding max runtime
miketalley Nov 24, 2020
23e2103
improved limit warnings with url
miketalley Nov 24, 2020
6107003
updated header values to mimic production
miketalley Nov 24, 2020
4989bd9
added better endpoint output when running server
miketalley Nov 24, 2020
c846343
minor fix and updated todo
miketalley Nov 24, 2020
61ed4e0
updated output of function execution to match hs logs command
miketalley Nov 24, 2020
7641c8f
updated todo
miketalley Nov 24, 2020
d566906
fixed cookie value in headers
miketalley Nov 30, 2020
ba30454
Merge branch 'master' into serverless/local-testing
miketalley Dec 8, 2020
095ec24
merged master and fixed conflict with hs functions package rename to …
miketalley Dec 15, 2020
1142b3a
added cors to allow modules to hit locally running serverless function
miketalley Dec 16, 2020
e3abf4d
added contact data and ability to modify mocked data
miketalley Dec 16, 2020
33a8b45
added ability to modify mocked limits data
miketalley Dec 16, 2020
d2b68f2
removed MAX_DEPS as this is not an actual limit
miketalley Dec 16, 2020
214b0b3
moved local serverless server into separate package
miketalley Dec 17, 2020
a9adcfd
added @hubspot/cms-lib dep
miketalley Dec 17, 2020
8dd4ed1
updated readme
miketalley Dec 17, 2020
206830b
removed comment
miketalley Dec 17, 2020
9bad639
constants are now required in correctly
miketalley Dec 17, 2020
2991275
protected against no logs being passed to logFunctiuonExecution
miketalley Dec 17, 2020
4a23fba
improved port validation
miketalley Dec 17, 2020
d345764
moved outside deps up in order
miketalley Dec 17, 2020
f6c7807
enabled validateInputs
miketalley Dec 17, 2020
e6eaa77
moved logic out of server.js into data and routes and fixed console l…
miketalley Dec 18, 2020
06f4052
fixed issue with data not getting passed to function properly
miketalley Dec 18, 2020
fc001a4
mocked data can now be customized through .env instead of serverless.…
miketalley Dec 18, 2020
a416771
prevented mocked data from polluting secrets
miketalley Dec 18, 2020
e6422d7
Merge pull request #404 from HubSpot/serverless/local-testing-package
miketalley Dec 21, 2020
e4c0075
renamed package to serverless-dev-runtime
miketalley Dec 21, 2020
e07188a
added watch ability to rerun server when changes are detected
miketalley Dec 21, 2020
00cdc08
got watch working and restarting server properly
miketalley Dec 21, 2020
ca683a4
made watch an option
miketalley Dec 21, 2020
d645628
Cleaned up some code and added more error handling
miketalley Dec 22, 2020
dd1550c
updated readme
miketalley Dec 22, 2020
82594b6
methods can now be a string or array
miketalley Dec 22, 2020
d918cf6
fixed logging when required from external project
miketalley Dec 22, 2020
0fd7c13
fixed issue with processLog not being imported with external import o…
miketalley Dec 22, 2020
5149efc
fixed default value for contact when imported externally
miketalley Dec 22, 2020
3cda634
fixed response data shape
miketalley Dec 22, 2020
e2f21ac
allowed passing serverless function folder path with and without .fun…
miketalley Dec 22, 2020
a18df37
preserved statusCode returned by function and made sure errors return…
miketalley Dec 22, 2020
af65471
removed console log
miketalley Dec 22, 2020
cf3e233
fixed visible output when logging execution
miketalley Dec 22, 2020
2c36fdd
merged master and fixed conflict in cms-cli package.json due to updat…
miketalley Jan 4, 2021
7997186
added calculation for memory usage
miketalley Jan 4, 2021
9604d65
fixed loading of new .env config when watched file changes
miketalley Jan 4, 2021
3e05b2b
added warning for unmatched routes
miketalley Jan 6, 2021
449a858
moved call of getFunctionDataContext before load of file to allow pro…
miketalley Jan 6, 2021
7e6ade6
fixed vid value as it is a string not a number
miketalley Jan 6, 2021
312aa35
handled sending error statusCode responses instead of throwing
miketalley Jan 6, 2021
a9a5e13
added json capability to body parser
miketalley Jan 6, 2021
30c4033
cleaned up functionExecutionCallback
miketalley Jan 6, 2021
920ffd1
extended log handlers to allow insertion of more text into header thr…
miketalley Jan 6, 2021
032288d
fixed default value for log
miketalley Jan 6, 2021
e274a5d
changed warning for unmatched routes to get only as it was interferin…
miketalley Jan 6, 2021
3690ae7
changed mock data values
miketalley Jan 13, 2021
7e86955
updated local function path to mimic production path
miketalley Jan 13, 2021
03c5da0
merged master and fixed conflict in packages/cli/package.json using 2…
miketalley Jan 13, 2021
77fa56f
merged master and fixed conflict in lib/logs.js with options passed t…
miketalley Jan 19, 2021
714c7f5
updated package versions
miketalley Jan 19, 2021
59f12f0
updated references of cms-lib to cli-lib
miketalley Jan 19, 2021
1d51169
fixed logging of header insertions due to merge conflict issue
miketalley Jan 19, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions packages/cli-lib/__tests__/__snapshots__/schema.js.snap
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`cms-lib/schema cleanSchema() cleans a full schema 1`] = `
exports[`cli-lib/schema cleanSchema() cleans a full schema 1`] = `
Object {
"associatedObjects": undefined,
"labels": Object {
Expand Down Expand Up @@ -42,7 +42,7 @@ Object {
}
`;

exports[`cms-lib/schema cleanSchema() cleans multiple schema 1`] = `
exports[`cli-lib/schema cleanSchema() cleans multiple schema 1`] = `
Array [
Object {
"associatedObjects": undefined,
Expand Down Expand Up @@ -173,15 +173,15 @@ Array [
]
`;

exports[`cms-lib/schema logSchemas() logs schemas 1`] = `
exports[`cli-lib/schema logSchemas() logs schemas 1`] = `
"╔════════╤════════╤══════════════╗
║ Label │ Name │ objectTypeId ║
║ Schema │ schema │ ║
╚════════╧════════╧══════════════╝
"
`;

exports[`cms-lib/schema writeSchemaToDisk() writes schema to disk 1`] = `
exports[`cli-lib/schema writeSchemaToDisk() writes schema to disk 1`] = `
"{
\\"name\\": \\"schema\\",
\\"labels\\": { \\"singular\\": \\"Schema\\", \\"plural\\": \\"Schemas\\" },
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/errorHandlers.js
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ const fileSystemErrorContext = Object.freeze({
filepath: 'x/y/z.html',
});

describe('cms-lib/errorHandlers', () => {
describe('cli-lib/errorHandlers', () => {
beforeEach(() => {
logger.clear();
});
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/fileMapper.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ function testPathDeterminationFunction(name, pathType, fn, truthy, falsey) {
});
}

describe('cms-lib/fileMapper', () => {
describe('cli-lib/fileMapper', () => {
testPathDeterminationFunction(
'isPathToFile',
'file',
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/hubdb.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const hubdbPublishTableResponse = require('./fixtures/hubdb/hubdbPublishTableRes
jest.mock('../path');
jest.mock('../api/hubdb');

describe('cms-lib/hubdb', () => {
describe('cli-lib/hubdb', () => {
it('downloads hubdb table', async () => {
const accountId = 123;
const tableId = 456;
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/index.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
describe('cms-lib', () => {
describe('cli-lib', () => {
it('works', () => {
expect(true).toBe(true);
});
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/modules.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ const isHubSpot = true;
// ca. 2016 https://github.com/facebook/jest/issues/335#issuecomment-250400941
jest.mock('../lib/walk', () => require('../lib/__mocks__/walk'));

describe('cms-lib/modules', () => {
describe('cli-lib/modules', () => {
describe('isModuleFolder()', () => {
it('should throw on invalid input', () => {
expect(() => isModuleFolder('foo')).toThrow();
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/path.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
const path = require('path');
const { splitHubSpotPath, splitLocalPath } = require('../path');

describe('cms-lib/path', () => {
describe('cli-lib/path', () => {
describe('splitHubSpotPath()', () => {
const testSplit = (filepath, expectedParts, joined) => {
test(filepath, () => {
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/__tests__/schema.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ const basic = require('./fixtures/schema/basic.json');
const full = require('./fixtures/schema/full.json');
const multiple = require('./fixtures/schema/multiple.json');

describe('cms-lib/schema', () => {
describe('cli-lib/schema', () => {
describe('cleanSchema()', () => {
it('cleans a basic schema', () => {
expect(cleanSchema(basic)).toEqual(basic);
Expand Down
2 changes: 1 addition & 1 deletion packages/cli-lib/api/__tests__/fileMapper.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
const fileStreamResponse = require('./fixtures/fileStreamResponse');
const { createFileMapperNodeFromStreamResponse } = require('../fileMapper');

describe('cms-lib/api/fileMapper', () => {
describe('cli-lib/api/fileMapper', () => {
describe('createFileMapperNodeFromStreamResponse()', () => {
const src = '1234/test.html';
it('should return request#tranform to create a FileMapperNode from the octet-stream response', () => {
Expand Down
3 changes: 2 additions & 1 deletion packages/cli-lib/api/fileMapper.js
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ function createFileMapperNodeFromStreamResponse(filePath, response) {
* @returns {Promise}
*/
async function upload(accountId, src, dest, options = {}) {
console.log('upload: ', fs.createReadStream(path.resolve(getCwd(), src)));
return http.post(accountId, {
uri: `${FILE_MAPPER_API_PATH}/upload/${encodeURIComponent(dest)}`,
formData: {
Expand Down Expand Up @@ -160,7 +161,7 @@ async function deleteFile(accountId, filePath, options = {}) {
*/
async function deleteFolder(accountId, folderPath, options = {}) {
logger.warn(
'`cms-lib/api/fileMapper#deleteFolder()` is deprecated. Use `cms-lib/api/fileMapper#deleteFile()` instead.'
'`cli-lib/api/fileMapper#deleteFolder()` is deprecated. Use `cli-lib/api/fileMapper#deleteFile()` instead.'
);
return http.delete(accountId, {
uri: `${FILE_MAPPER_API_PATH}/delete/folder/${folderPath}`,
Expand Down
1 change: 1 addition & 0 deletions packages/cli-lib/ignoreRules.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ const ignoreList = [
'.*', // hidden files/folders
'*.log', // Error log for npm
'*.swp', // Swap file for vim state
'.env', // Dotenv file

// # macOS
'Icon\\r', // Custom Finder icon: http://superuser.com/questions/298785/icon-file-on-os-x-desktop
Expand Down
32 changes: 18 additions & 14 deletions packages/cli-lib/lib/logs.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,31 @@ const formatError = log => {
};

const logHandler = {
UNHANDLED_ERROR: (log, { compact }) => {
return `${formatLogHeader(log)}${compact ? '' : `\n${formatError(log)}`}`;
UNHANDLED_ERROR: (log, options) => {
return `${formatLogHeader(log, options)}${
options.compact ? '' : `\n${formatError(log)}`
}`;
},
HANDLED_ERROR: (log, { compact }) => {
return `${formatLogHeader(log)}${compact ? '' : `\n${formatError(log)}`}`;
HANDLED_ERROR: (log, options) => {
return `${formatLogHeader(log, options)}${
options.compact ? '' : `\n${formatError(log)}`
}`;
},
SUCCESS: (log, { compact }) => {
return `${formatLogHeader(log)}${compact ? '' : `\n${formatLog(log)}`}`;
SUCCESS: (log, options) => {
return `${formatLogHeader(log, options)}${
options.compact ? '' : `\n${log.log}`
}`;
},
};

const formatLogHeader = log => {
const formatLogHeader = (log, options) => {
const color = LOG_STATUS_COLORS[log.status];
const headerInsertion =
options && options.insertions && options.insertions.header;

return `${formatTimestamp(log)}${SEPARATOR}${color(
log.status
)}${SEPARATOR}${formatExecutionTime(log)}`;
};

const formatLog = log => {
return `${log.log}`;
return `${formatTimestamp(log)}${SEPARATOR}${color(log.status)}${
headerInsertion ? `${SEPARATOR}${headerInsertion}` : ''
}${SEPARATOR}${formatExecutionTime(log)}`;
};

const formatStackTrace = log => {
Expand Down
2 changes: 2 additions & 0 deletions packages/cli/commands/functions.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ const {
} = require('../lib/commonOpts');
const list = require('./functions/list');
const deploy = require('./functions/deploy');
const test = require('./functions/test');

exports.command = 'functions';
exports.describe = 'Commands for working with functions';
Expand All @@ -20,6 +21,7 @@ exports.builder = yargs => {
aliases: 'ls',
})
.command(deploy)
.command(test)
.demandCommand(1, '');

return yargs;
Expand Down
86 changes: 86 additions & 0 deletions packages/cli/commands/functions/test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
const {
addAccountOptions,
addConfigOptions,
setLogLevel,
getAccountId,
addUseEnvironmentOptions,
} = require('../../lib/commonOpts');
const { trackCommandUsage } = require('../../lib/usageTracking');
const { logDebugInfo } = require('../../lib/debugInfo');
const {
loadConfig,
validateConfig,
checkAndWarnGitInclusion,
} = require('@hubspot/cli-lib');
const { logger } = require('@hubspot/cli-lib/logger');
const { start: startTestServer } = require('@hubspot/serverless-dev-runtime');
const { validateAccount } = require('../../lib/validation');

const loadAndValidateOptions = async options => {
setLogLevel(options);
logDebugInfo(options);
const { config: configPath } = options;
loadConfig(configPath, options);
checkAndWarnGitInclusion();

if (!(validateConfig() && (await validateAccount(options)))) {
process.exit(1);
}
};

exports.command = 'test <path>';
exports.describe = false;

exports.handler = async options => {
loadAndValidateOptions(options);

const { path: functionPath } = options;
const accountId = getAccountId(options);

trackCommandUsage('functions-test', { functionPath }, accountId);

logger.debug(
`Starting test server for .functions folder with path: ${functionPath}`
);

startTestServer({
accountId,
...options,
});
};

exports.builder = yargs => {
yargs.positional('path', {
describe: 'Path to local .functions folder',
type: 'string',
});
yargs.option('port', {
describe: 'port to run the test server on',
type: 'string',
default: 5432,
});
yargs.option('contact', {
describe: 'pass contact data to the test function',
type: 'boolean',
default: true,
});
yargs.option('watch', {
describe:
'watch the specified .functions folder for changes and restart the server',
type: 'boolean',
default: true,
});

yargs.example([
[
'$0 functions test ./tmp/myFunctionFolder.functions',
'Run a local function test server.',
],
]);

addConfigOptions(yargs, true);
addAccountOptions(yargs, true);
addUseEnvironmentOptions(yargs, true);

return yargs;
};
3 changes: 3 additions & 0 deletions packages/cli/jest.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = {
testPathIgnorePatterns: ['commands/functions/test.js'],
};
1 change: 0 additions & 1 deletion packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
"dependencies": {
"@hubspot/cli-lib": "^2.2.3-beta.3",
"chalk": "^4.1.0",
"express": "^4.17.1",
"inquirer": "^6.3.1",
"open": "^7.0.3",
"ora": "^4.0.3",
Expand Down
67 changes: 67 additions & 0 deletions packages/serverless-dev-runtime/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# @hubspot/serverless-dev-runtime
miketalley marked this conversation as resolved.
Show resolved Hide resolved

A serverless function development runtime that can be used to test CMS serverless functions. This is intended for use with the [CMS CLI](https://developers.hubspot.com/docs/cms/developer-reference/local-development-cms-cli).

## Getting started

For more information on using these tools, see [Local Development Tooling: Getting Started](https://designers.hubspot.com/tutorials/getting-started-with-local-development)

### Installation

#### Using `yarn`

```bash
yarn add @hubspot/cms-cli --dev
```

#### Using `npm`

```bash
npm install @hubspot/cms-cli
```

### Usage

#### CLI Command
To run the CLI command to test a local serverless function run...
```bash
hs functions <localDotFunctionsFolderPath>
```

#### Importing
To start the server, the `start` method can be imported from the `@hubspot/serverless-dev-runtime` package and run with settings like so...

```bash
const { start } = require('@hubspot/serverless-dev-runtime');

start({
accountId: <portalId/accountId>, // default: 123456
contact: <booleanValueToSpecifyIfContactDataShouldBePassedToServerlessFunction>, // default: true
path: <pathToLocalDotFunctionsFolder>, // required
port: <customPortToRunServerOn> // default: 5432
});
```

### Mocked Data
Some of the data that is passed to the serverless function context is mocked. Specifically the `contact` and `limits` properties. It is possible
to modify the mocked data by setting values for specific variables within a `.env` file within the `.functions` folder.

The variables used to modify the data are:

```
HUBSPOT_LIMITS_TIME_REMAINING // default: 600000
HUBSPOT_LIMITS_EXECUTIONS_REMAINING // default: 60
HUBSPOT_CONTACT_VID // default: 123
HUBSPOT_CONTACT_IS_LOGGED_IN // default: false
HUBSPOT_CONTACT_LIST_MEMBERSHIPS // default: []
```

Usage example `.env`:

```
HUBSPOT_LIMITS_TIME_REMAINING=1000
HUBSPOT_LIMITS_EXECUTIONS_REMAINING=2
HUBSPOT_CONTACT_VID=456
HUBSPOT_CONTACT_IS_LOGGED_IN=true
HUBSPOT_CONTACT_LIST_MEMBERSHIPS="some, memberships"
```
5 changes: 5 additions & 0 deletions packages/serverless-dev-runtime/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
const { start } = require('./lib/server');

module.exports = {
start,
};
Loading