Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Source documentation #15

Merged
merged 8 commits into from
Oct 31, 2022
Merged

Add Source documentation #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1b6e173
source documentation, first run
oykmnk Oct 28, 2022
a9b5193
add empty resources.md
oykmnk Oct 28, 2022
13f1232
fmt README
oykmnk Oct 28, 2022
57604f3
fix typo
oykmnk Oct 28, 2022
9bc5452
update configuration options documentation
oykmnk Oct 28, 2022
e8bcfdc
`resource` -> `resources`
oykmnk Oct 28, 2022
a5aef54
format source config table
oykmnk Oct 31, 2022
2f1290b
add resource and source operations they support
oykmnk Oct 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 70 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,73 @@

## General

The [HubSpot](https://hubspot.com/) connector is one of Conduit plugins. It provides both, a source and a destination HubSpot connector.
The [HubSpot](https://hubspot.com/) connector is one of Conduit plugins. It provides both, a source and a destination HubSpot connector.

### Prerequisites

- [Go](https://go.dev/) v1.18
- [HubSpot](https://www.hubspot.com/) account and a [private app](https://developers.hubspot.com/docs/api/private-apps)
- (optional) [golangci-lint](https://github.com/golangci/golangci-lint) v1.49.0

### How to build it

Run `make build`.

### Testing

Run `make test` to run all the unit and integration tests. The integration tests require the environment variable `HUBSPOT_ACCESS_TOKEN` to be set. If the environment variable is empty, the integration tests will be skipped.

## Source

The HubSpot Source Connector uses a private app access token to connect to a HubSpot account and creates records for each resource change detected in a HubSpot account.

### Snapshot capture

When the connector first starts, snapshot mode is enabled. The connector reads items of a resource that you specified. It only reads items that are created before the connector starts to run, batching them by `bufferSize`. Each new batch is processed every `pollingPeriod` duration. Once all items in that initial snapshot are read the connector switches into CDC mode.

### Change Data Capture

When a snapshot is captured the connector starts to listen to data changes. It can track creates, updates, and deletes that occur after the connector is started. But please note that not all resources support all operations. You can check the available resources and operations they support out [here](docs/resources.md).

### Position structure

The connector goes through two modes.

**Snapshot**. The position contains the `initialTimestamp` field that is equal to the timestamp of the first connector run. If a resource supports filtering by id, the position also contains the id of the last processed item in the `itemId` field.

Here's an example of a Snapshot position:

```json
{
"mode": "snapshot",
"itemId": "256",
"initialTimestamp": "2022-10-28T14:58:27Z"
}
```

**CDC**. The position in this mode contains the same fields as in the Snapshot mode plus a `timestamp` that is equal to the `updatedAt` of the last processed item.

Here's an example of a CDC position:

```json
{
"mode": "cdc",
"itemId": "256",
"initialTimestamp": "2022-10-28T14:58:27Z",
"timestamp": "2022-10-28T15:00:50Z"
}
```

### Configuration options

| name | description | required | default |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
| `accessToken` | The private app access token for accessing the HubSpot API. | **true** | |
| `resource` | The HubSpot resource that the connector will work with.<br />You can find a list of the available resources [here](docs/resources.md). | **true** | |
| `maxRetries` | The number of HubSpot API request retries attempts that will be tried before giving up if a request fails. | false | `4` |
| `pollingPeriod` | The duration that defines a period of polling new items. | false | `5s` |
| `bufferSize` | The buffer size for consumed items.<br />It will also be used as a limit when retrieving items from the HubSpot API. | false | `100` |

### Known limitations

- Not all resources support all CDC operations. You can check the available resources and operations they support out [here](docs/resources.md).
24 changes: 24 additions & 0 deletions docs/resources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
Here you can find a list of the available HubSpot resources and operations they support.

| resource | source operations |
| --------------------------------------------------------------------------------------------- | ---------------------------------------- |
| [`cms.blogs.authors`](https://developers.hubspot.com/docs/api/cms/blog-authors) | `snapshot`, `insert`, `update`, `delete` |
| [`cms.blogs.posts`](https://developers.hubspot.com/docs/api/cms/blog-post) | `snapshot`, `insert`, `update`, `delete` |
| [`cms.blogs.tags`](https://developers.hubspot.com/docs/api/cms/blog-tags) | `snapshot`, `insert`, `update`, `delete` |
| [`cms.pages.landing`](https://developers.hubspot.com/docs/api/cms/pages) | `snapshot`, `insert`, `update`, `delete` |
| [`cms.pages.site`](https://developers.hubspot.com/docs/api/cms/pages) | `snapshot`, `insert`, `update`, `delete` |
| [`cms.hubdb.tables`](https://developers.hubspot.com/docs/api/cms/hubdb) | `snapshot`, `insert`, `update` |
| [`cms.urlRedirects`](https://developers.hubspot.com/docs/api/cms/url-redirects) | `snapshot`, `insert`, `update` |
| [`crm.companies`](https://developers.hubspot.com/docs/api/crm/companies) | `snapshot`, `insert`, `update` |
| [`crm.contacts`](https://developers.hubspot.com/docs/api/crm/contacts) | `snapshot`, `insert`, `update` |
| [`crm.deals`](https://developers.hubspot.com/docs/api/crm/deals) | `snapshot`, `insert`, `update` |
| [`crm.feedbackSubmissions`](https://developers.hubspot.com/docs/api/crm/feedback-submissions) | `snapshot`, `insert`, `update` |
| [`crm.lineItems`](https://developers.hubspot.com/docs/api/crm/line-items) | `snapshot`, `insert`, `update` |
| [`crm.products`](https://developers.hubspot.com/docs/api/crm/products) | `snapshot`, `insert`, `update` |
| [`crm.tickets`](https://developers.hubspot.com/docs/api/crm/tickets) | `snapshot`, `insert`, `update` |
| [`crm.quotes`](https://developers.hubspot.com/docs/api/crm/quotes) | `snapshot`, `insert`, `update` |
| [`crm.calls`](https://developers.hubspot.com/docs/api/crm/calls) | `snapshot`, `insert`, `update` |
| [`crm.emails`](https://developers.hubspot.com/docs/api/crm/email) | `snapshot`, `insert`, `update` |
| [`crm.meetings`](https://developers.hubspot.com/docs/api/crm/meetings) | `snapshot`, `insert`, `update` |
| [`crm.notes`](https://developers.hubspot.com/docs/api/crm/notes) | `snapshot`, `insert`, `update` |
| [`crm.tasks`](https://developers.hubspot.com/docs/api/crm/tasks) | `snapshot`, `insert`, `update` |