Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk controller #15

Merged
merged 8 commits into from
Jun 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions doc/2/controllers/bulk/import/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
code: true
type: page
title: import
description: Creates, updates or deletes large amounts of documents as fast as possible.
---

# Impport

Creates, updates or deletes large amounts of documents as fast as possible.

This route is faster than the `document:m*` routes family (e.g. [document:mCreate](/sdk/dart/2/controllers/document/m-create)), but no real-time notifications will be generated, even if some of the documents in the import match subscription filters.

## Arguments

```dart
Future<Map<String, dynamic>> import(
String index, String collection, List<Map<String, dynamic>> documents)
```

| Argument | Type | Description |
|--------------|-------------------|--------------------------------------------------------------|
| `index` | <pre>String</pre> | Index name |
| `collection` | <pre>String</pre> | Collection name |
| `documents` | <pre>List<Map<String, dynamic>></pre> | List of documents to import |

### documents

This API takes a JSON array containing a list of objects working in pairs.
In each pair, the first object specifies the action to perform (the most common is `create`) and the second specifies the document itself, like in the example below:

```json
[
// Action object
{ "create": { "_id": "id" } },
// Document object
{ "myField": "myValue", "myOtherField": "myOtherValue" },

// Another action object
{ "create": { "_id": "another-id" } },
// Another document object
{ "myField": "anotherValue", "myOtherField": "yetAnotherValue" }
];
```

Possible actions are `create`, `index`, `update`, `delete`.

Learn more at [Elasticsearch Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/docs-bulk.html)

## Return

A Map<String, dynamic> containing 2 arrays:

| Property | Type | Description |
| -------- | ------------------- | --------------------------------------------------- |
| `successes` | <pre>List<dynamic></pre> | Array of object containing successful document import |
| `errors` | <pre>List<dynamic></pre> | Array of object containing failed document import |

Each item of the `successes` array is an object containing the action name as key and the corresponding object contain the following properties:

| Property | Type | Description |
| -------- | ------------------- | --------------------------------------------------- |
| `_id` | <pre>String</pre> | Document unique identifier |
| `status` | <pre>int</pre> | HTTP status code for that query |

Each item of the `errors` array is an object containing the action name as key and the corresponding object contain the following properties:

| Property | Type | Description |
| -------- | ------------------- | --------------------------------------------------- |
| `_id` | <pre>String</pre> | Document unique identifier |
| `status` | <pre>int</pre> | HTTP status code for that query |
| `error` | <pre>Map<String, dynamic></pre> | Error object |

Each `error` object contain the following properties:

| Property | Type | Description |
| -------- | ------------------- | --------------------------------------------------- |
| `type` | <pre>String</pre> | Elasticsearch client error type |
| `reason` | <pre>String</pre> | human readable error message |

## Usage

<<< ./snippets/import.dart
13 changes: 13 additions & 0 deletions doc/2/controllers/bulk/import/snippets/import.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
final result = await kuzzle
.bulk
.import(
'nyc-open-data',
'yellow-taxi',
[
{ 'index': { } },
{ 'a': 'document', 'with': 'any', 'number': 'of fields' },
{ 'create': { '_id': 'uniq-id-1' } },
{ 'another': 'document' },
{ 'create': { '_id': 'uniq-id-2' } },
{ 'and': { 'another': 'one' } },
]);
9 changes: 9 additions & 0 deletions doc/2/controllers/bulk/import/snippets/import.test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: Bulk#ImportAsync
description: Creates, updates or deletes large amounts of documents as fast as possible.
hooks:
before: |
curl -X POST kuzzle:7512/nyc-open-data/_create
curl -X PUT kuzzle:7512/nyc-open-data/yellow-taxi
after: curl -X DELETE kuzzle:7512/nyc-open-data
template: default
expected: "Success"
8 changes: 8 additions & 0 deletions doc/2/controllers/bulk/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
code: true
type: branch
title: bulk
description: Bulk Controller
---

# Bulk Controller
63 changes: 63 additions & 0 deletions doc/2/controllers/bulk/mwrite/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
code: true
type: page
title: MWriteAsync
description: Creates or replaces multiple documents directly into the storage engine.
---

# mWrite

This is a low level route intended to bypass Kuzzle actions on document creation, notably:
- check [document validity](/core/2/guides/essentials/data-validation),
- add [kuzzle metadata](/core/2/guides/essentials/document-metadata),
- trigger [realtime notifications](/core/2/guides/essentials/real-time) (unless asked otherwise)

## Arguments

```dart
Future<Map<String, dynamic>> import(
String index,
String collection,
List<Map<String, dynamic>> documents,
{
bool waitForRefresh = false,
})
```

| Argument | Type | Description |
|--------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------|
| `index` | <pre>String</pre> | Index name |
| `collection` | <pre>String</pre> | Collection name |
| `documents` | <pre>List<Map<String, dynamic>></pre> | An array of Map<String, dynamic> representing the documents|
| `waitForRefresh` | <pre>bool</pre><br>(`false`) | If set to true, Kuzzle will not respond until the created/replaced documents are indexed |

### documents

An array of `Map<String, dynamic>`. Each object describes a document to create or replace, by exposing the following properties:
- `_id`: document unique identifier (optional)
- `body`: document content

## Return

Returns a `Map<String, dynamic>` containing 2 arrays: `successes` and `errors`

Each created or replaced document is an object of the `successes` array with the following properties:

| Name | Type | Description |
| --------- | ----------------- | ------------------------------------------------------ |
| `_id` | <pre>String</pre> | Document ID |
| `_version` | <pre>int</pre> | Version of the document in the persistent data storage |
| `_source` | <pre>Map<String, dynamic></pre> | Document content |
| `created` | <pre>bool</pre> | True if the document was created |

Each errored document is an object of the `errors` array with the following properties:

| Name | Type | Description |
| --------- | ----------------- | ------------------------------------------------------ |
| `document` | <pre>Map<String, dynamic></pre> | Document that cause the error |
| `status` | <pre>int</pre> | HTTP error status |
| `reason` | <pre>String</pre> | Human readable reason |

## Usage

<<< ./snippets/mwrite.dart
8 changes: 8 additions & 0 deletions doc/2/controllers/bulk/mwrite/snippets/mwrite.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
final result = await kuzzle
.bulk
.mWrite(
'nyc-open-data',
'yellow-taxi',
[
{'_id': 'foo', 'body': {},},
]);
9 changes: 9 additions & 0 deletions doc/2/controllers/bulk/mwrite/snippets/mwrite.test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: Bulk#MWriteAsync
description: Creates or replaces multiple documents directly into the storage engine.
hooks:
before: |
curl -X POST kuzzle:7512/nyc-open-data/_create
curl -X PUT kuzzle:7512/nyc-open-data/yellow-taxi
after: curl -X DELETE kuzzle:7512/nyc-open-data
template: default
expected: "Success"
44 changes: 44 additions & 0 deletions doc/2/controllers/bulk/write/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
code: true
type: page
title: write
description: Creates or replaces a document directly into the storage engine.
---

# write

Create or replace a document directly into the storage engine.

## Arguments

```dart
Future<Map<String, dynamic>> write(
String index,
String collection,
Map<String, dynamic> document, {
String id,
bool waitForRefresh = false,
})
```

| Argument | Type | Description |
|--------------|--------------------|-----------------------------|
| `index` | <pre>String</pre> | Index name |
| `collection` | <pre>String</pre> | Collection name |
| `content` | <pre>Map<String, dynamic></pre> | Document content to create. |
| `id` | <pre>String</pre><br>(`null`) | Document id
| `waitForRefresh` | <pre>bool</pre><br>(`false`) | If set to `true`, Kuzzle will wait for the persistence layer to finish indexing |

## Return

Returns a `Map<String, dynamic>` with the following properties:

| Property | Type | Description |
|------------|--------------------|-------------------------------------------------|
| `_id` | <pre>String</pre> | Created document unique identifier. |
| `_source` | <pre>Map<String, dynamic></pre> | Document content. |
| `_version` | <pre>int</pre> | Version number of the document |

## Usage

<<< ./snippets/write.dart
13 changes: 13 additions & 0 deletions doc/2/controllers/bulk/write/snippets/write.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
final result = await kuzzle
.bulk
.write(
'nyc-open-data',
'yellow-taxi',
{
'_kuzzle_info': {
'author': '<kuid>',
'createdAt': 1481816934209,
'updatedAt': null,
'updater': null,
}
});
9 changes: 9 additions & 0 deletions doc/2/controllers/bulk/write/snippets/write.test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: Bulk#WriteAsync
description: Creates or replaces a document directly into the storage engine.
hooks:
before: |
curl -X POST kuzzle:7512/nyc-open-data/_create
curl -X PUT kuzzle:7512/nyc-open-data/yellow-taxi
after: curl -X DELETE kuzzle:7512/nyc-open-data
template: default
expected: "Success"
2 changes: 1 addition & 1 deletion doc/2/controllers/document/m-create-or-replace/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Future<Map<String, dynamic>> mCreateOrReplace(
| ------------------ | ------------------------------------------------------- | --------------------------------- |
| `index` | <pre>String</pre> | Index |
| `collection` | <pre>String</pre> | Collection |
| `documents` | <pre>List<Map<String, dynamic>></pre> | ArrayList containing the documents to create |
| `documents` | <pre>List<Map<String, dynamic>></pre> | List containing the documents to create |
| `waitForRefresh` | <pre>bool</pre> | If set to `true`, Kuzzle will wait for the persistence layer to finish indexing |

---
Expand Down
10 changes: 5 additions & 5 deletions doc/2/controllers/document/m-create/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Future<c> mCreate(
| ------------------ | ------------------------------------------------------- | --------------------------------- |
| `index` | <pre>String</pre> | Index |
| `collection` | <pre>String</pre> | Collection |
| `documents` | <pre>List<Map<String, dynamic>></pre> | ArrayList containing the documents to create |
| `documents` | <pre>List<Map<String, dynamic>></pre> | List containing the documents to create |
| `waitForRefresh` | <pre>bool</pre><br>(`false`) | If set to `true`, Kuzzle will wait for the persistence layer to finish indexing |

---
Expand All @@ -35,24 +35,24 @@ Each document has the following properties:
| Arguments | Type | Description |
| ------------------ | -------------------------------------------- | --------------------------------- |
| `_id` | <pre>String</pre> | Optional document ID. Will be auto-generated if not defined. |
| `body` | <pre>List<Map<String, dynamic>></pre> | Document body |
| `body` | <pre>Map<String, dynamic></pre> | Document body |

## Return

A `List<Map<String, dynamic>>` which has a `successes` and `errors`:
A `Map<String, dynamic>` which has a `successes` and `errors`:
Each created document is an object of the `successes` array with the following properties:

| Property | Type | Description |
|------------- |--------------------------------------------- |--------------------------------- |
| `_source` | <pre>List<Map<String, dynamic>></pre> | Created document |
| `_source` | <pre>Map<String, dynamic></pre> | Created document |
| `_id` | <pre>String</pre> | ID of the newly created document |
| `_version` | <pre>int</pre> | Version of the document in the persistent data storage |

Each errored document is an object of the `errors` array with the following properties:

| Property | Type | Description |
|------------- |--------------------------------------------- |--------------------------------- |
| `document` | <pre>List<Map<String, dynamic>></pre> | Document that causes the error |
| `document` | <pre>Map<String, dynamic></pre> | Document that causes the error |
| `status` | <pre>int</pre> | HTTP error status |
| `reason` | <pre>String</pre> | Human readable reason |

Expand Down
4 changes: 2 additions & 2 deletions doc/2/controllers/document/m-get/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ Future<Map<String, dynamic>> mGet(

## Return

A `List<String>` which has a `successes` and `errors`:
A `Map<String, dynamic>` which has a `successes` and `errors`:
Each created document is an object of the `successes` array with the following properties:

| Property | Type | Description |
|------------- |--------------------------------------------- |--------------------------------- |
| `_source` | <pre>List<String></pre> | Document content |
| `_source` | <pre>Map<String, dynamic></pre> | Document content |
| `_id` | <pre>String</pre> | Document ID |
| `_version` | <pre>int</pre> | Version of the document in the persistent data storage |

Expand Down
Loading