feat: improve export #316

sebtiz13 · 2023-08-30T14:25:38Z

What does this PR do ?

Add support for elasticsearch and koncorde language for export
Improve handle error on export, return error message instead white page
Limit export to 10k documents (prevent segfault with more than 25k results)
Prevent error duplicate result on search with sort

prevent segfault with more than 25k results

lib/modules/shared/services/AbstractExporter.ts

rolljee · 2023-08-30T14:52:47Z

lib/modules/shared/services/DigitalTwinExporter.ts

+    const {
+      query,
+      sort,
+      lang = "elasticsearch",


lang was set to koncorde by default before, we should keep the same otherwise it's a breaking change

It's true, but all other requests use lang='elasticsearch' by default, so it's inconsistent.
But need to be confirmed because for measure export some filters are hard-coded in koncorde language.

Doesn't it already default to elasticsearch ? See this

rolljee · 2023-08-30T14:58:53Z

lib/modules/asset/AssetsController.ts

+
+        return new HttpStream(stream);
+      } catch (error) {
+        request.response.configure({


Why adding this instead of throwing a traditional error ?

Because this endpoint is mostly called by browser and traditional error return a JSON that can be hard to understand for user.
Maybe the better solution could be to improve global error handler in the core

Add a comment to explain this in the code :)

# [2.4.0-beta.1](v2.3.2...v2.4.0-beta.1) (2023-08-31) ### Bug Fixes * copy-version script ([72b5e49](72b5e49)) ### Features * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * improve export ([#316](#316)) ([e4e746e](e4e746e)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53))

* feat(export): add support elasticsearch lang * feat: improve handle error * feat(export): limit export to 10k documents prevent segfault with more than 25k results * fix(export): prevent error duplicate result

# [2.4.0-beta.1](v2.3.2...v2.4.0-beta.1) (2023-08-31) ### Bug Fixes * copy-version script ([72b5e49](72b5e49)) ### Features * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * improve export ([#316](#316)) ([51db84a](51db84a)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53))

# [2.4.0-beta.2](v2.4.0-beta.1...v2.4.0-beta.2) (2023-09-05) ### Bug Fixes * copy-version script ([72b5e49](72b5e49)) ### Features * improve export ([#316](#316)) ([51db84a](51db84a))

# [2.4.0-temp-ci-debug.1](v2.3.2...v2.4.0-temp-ci-debug.1) (2024-01-11) ### Bug Fixes * **asset_migration:** fixes multiple assets migration and optimize ([fd546cf](fd546cf)) * **assetsGroups:** disable checkRights on impersonate ([#325](#325)) ([eaa3edc](eaa3edc)) * **assets:** migrate tenant only if user is admin ([c24f942](c24f942)) * **assetsMigrateTenant:** add type for frontend and fix admin only ([e249f13](e249f13)) * **ci:** correct github token in ci ([a5a6b69](a5a6b69)) * copy-version script ([72b5e49](72b5e49)) * correct KuzzleRequest for pipes ([ef932d1](ef932d1)) * **export:** correct scroll in getNamedMeasures ([05fe4fd](05fe4fd)) * improve get internal devices documents ([03ccb29](03ccb29)) * **migrateTenant:** clear groups of newly created assets ([79f6a5a](79f6a5a)) * **npm:** fix repository url ([514dc59](514dc59)) * update semantic-release ([380c8b3](380c8b3)) ### Features * **asset:** add upsert method ([#323](#323)) ([5f87787](5f87787)) * **assets:** assets migrate tenant ([5f05101](5f05101)) * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * **digitalTwin:** allow to modify mapping from application ([#322](#322)) ([89d375a](89d375a)) * **digitalTwin:** implement generic document pipes triggers ([#321](#321)) ([60a21fc](60a21fc)) * improve export ([#316](#316)) ([51db84a](51db84a)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53))

# [2.4.0-elasticsearch-8.1](v2.3.2...v2.4.0-elasticsearch-8.1) (2024-08-06) ### Bug Fixes * **asset_migration:** fixes multiple assets migration and optimize ([fd546cf](fd546cf)) * **assetsGroups:** disable checkRights on impersonate ([#325](#325)) ([eaa3edc](eaa3edc)) * **assets:** migrate tenant only if user is admin ([c24f942](c24f942)) * **assetsMigrateTenant:** add type for frontend and fix admin only ([e249f13](e249f13)) * catch errors when updating the mappings on startup ([#338](#338)) ([e09c5a3](e09c5a3)) * **ci:** correct github token in ci ([a5a6b69](a5a6b69)) * **ci:** fix npm glob not working anymore ([18e79aa](18e79aa)) * copy-version script ([72b5e49](72b5e49)) * correct api types ([#333](#333)) ([05c426f](05c426f)) * correct KuzzleRequest for pipes ([ef932d1](ef932d1)) * correct release script ([f02c285](f02c285)) * **device:** correct device HTTP API ([#354](#354)) ([d9fe9e1](d9fe9e1)) * **engine:** update conflicts ask return values ([0f817c3](0f817c3)) * **export:** correct scroll in getNamedMeasures ([05fe4fd](05fe4fd)) * improve get internal devices documents ([03ccb29](03ccb29)) * improve measure export to be usable with multiple measures in one measure ([#330](#330)) ([dab8d57](dab8d57)) * **migrateTenant:** clear groups of newly created assets ([79f6a5a](79f6a5a)) * **models:** list asset returns commons assets ([#345](#345)) ([2844437](2844437)) * **npm:** fix repository url ([514dc59](514dc59)) * update semantic-release ([380c8b3](380c8b3)) ### Features * add group descriptions to models ([#335](#335)) ([de00fb7](de00fb7)) * add log in decoders ([#339](#339)) ([fe08b7c](fe08b7c)) * add tooltip model by asset model ([#341](#341)) ([865ae6d](865ae6d)) * add UTC export by default for devices and assets ([#342](#342)) ([878ca58](878ca58)) * **asset:** add upsert method ([#323](#323)) ([5f87787](5f87787)) * **assets:** assets migrate tenant ([5f05101](5f05101)) * **assets:** remove assetId in upsert ([#326](#326)) ([2f286de](2f286de)) * **configuration:** allow to specify Elasticsearch index settings ([#337](#337)) ([6d22fa9](6d22fa9)) * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * **digitalTwin:** allow to modify mapping from application ([#322](#322)) ([89d375a](89d375a)) * **digitalTwin:** implement generic document pipes triggers ([#321](#321)) ([60a21fc](60a21fc)) * **engine:** add mappings update conflict ask call ([#340](#340)) ([5cd6aca](5cd6aca)) * implement comprehensible metadata ([#332](#332)) ([e80ef8e](e80ef8e)) * improve export ([#316](#316)) ([51db84a](51db84a)) * **measure:** add device metadata to origin ([#356](#356)) ([1ed9d8b](1ed9d8b)) * **measuremodels:** add optional local names and unit to measure definitions ([#343](#343)) ([50038b2](50038b2)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53)) * **metadata:** add support nested metadata properties ([#348](#348)) ([17eda8b](17eda8b)) * **metadata:** improve asset metadata ([#346](#346)) ([2eb5887](2eb5887)) * patch-in support for ES 8 ([6afbe79](6afbe79)) * upsert devices ([#355](#355)) ([c75abef](c75abef))

# [2.4.0-dev.1](v2.3.2...v2.4.0-dev.1) (2024-09-27) ### Bug Fixes * add specific dev branch in releaserc.js ([e11a823](e11a823)) * **asset_migration:** fixes multiple assets migration and optimize ([fd546cf](fd546cf)) * **assetsGroups:** disable checkRights on impersonate ([#325](#325)) ([eaa3edc](eaa3edc)) * **assets:** migrate tenant only if user is admin ([c24f942](c24f942)) * **assetsMigrateTenant:** add type for frontend and fix admin only ([e249f13](e249f13)) * catch errors when updating the mappings on startup ([#338](#338)) ([e09c5a3](e09c5a3)) * **ci:** correct github token in ci ([a5a6b69](a5a6b69)) * **ci:** fix npm glob not working anymore ([18e79aa](18e79aa)) * copy-version script ([72b5e49](72b5e49)) * correct api types ([#333](#333)) ([05c426f](05c426f)) * correct KuzzleRequest for pipes ([ef932d1](ef932d1)) * correct release script ([f02c285](f02c285)) * **device-metadata:** fetch device's metadata from assigned tenant when available ([#374](#374)) ([acfd8a0](acfd8a0)) * **device:** correct device HTTP API ([#354](#354)) ([d9fe9e1](d9fe9e1)) * **engine:** update conflicts ask return values ([0f817c3](0f817c3)) * **export:** correct scroll in getNamedMeasures ([05fe4fd](05fe4fd)) * improve get internal devices documents ([03ccb29](03ccb29)) * improve measure export to be usable with multiple measures in one measure ([#330](#330)) ([dab8d57](dab8d57)) * **migrateTenant:** clear groups of newly created assets ([79f6a5a](79f6a5a)) * **models:** list asset returns commons assets ([#345](#345)) ([2844437](2844437)) * **npm:** fix repository url ([514dc59](514dc59)) * only list asset models for the requested engine group ([#367](#367)) ([8a14207](8a14207)) * update semantic-release ([380c8b3](380c8b3)) ### Features * add actions to get last measures on digital twins ([#363](#363)) ([b8005a2](b8005a2)) * add dev builds when pushing on 2-dev ([#375](#375)) ([aa11885](aa11885)) * add group descriptions to models ([#335](#335)) ([de00fb7](de00fb7)) * add log in decoders ([#339](#339)) ([fe08b7c](fe08b7c)) * add search actions for the models ([#362](#362)) ([9990e57](9990e57)) * add tooltip model by asset model ([#341](#341)) ([865ae6d](865ae6d)) * add UTC export by default for devices and assets ([#342](#342)) ([878ca58](878ca58)) * **asset:** add upsert method ([#323](#323)) ([5f87787](5f87787)) * **assetrole:** add link / unlink device action to asset admin role ([#357](#357)) ([febf3a6](febf3a6)) * **assets:** assets migrate tenant ([5f05101](5f05101)) * **assets:** remove assetId in upsert ([#326](#326)) ([2f286de](2f286de)) * **configuration:** allow to specify Elasticsearch index settings ([#337](#337)) ([6d22fa9](6d22fa9)) * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * **digitalTwin:** allow to modify mapping from application ([#322](#322)) ([89d375a](89d375a)) * **digitalTwin:** implement generic document pipes triggers ([#321](#321)) ([60a21fc](60a21fc)) * **engine:** add mappings update conflict ask call ([#340](#340)) ([5cd6aca](5cd6aca)) * implement comprehensible metadata ([#332](#332)) ([e80ef8e](e80ef8e)) * improve export ([#316](#316)) ([51db84a](51db84a)) * improve search model request types ([#364](#364)) ([5310ad1](5310ad1)) * **measure:** add device metadata to origin ([#356](#356)) ([1ed9d8b](1ed9d8b)) * **measuremodels:** add optional local names and unit to measure definitions ([#343](#343)) ([50038b2](50038b2)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53)) * **metadata:** add option field readOnly to prevent editing ([#358](#358)) ([e0641b4](e0641b4)) * **metadata:** add optional editor hints for metadata ([#370](#370)) ([afc5129](afc5129)) * **metadata:** add support nested metadata properties ([#348](#348)) ([17eda8b](17eda8b)) * **metadata:** improve asset metadata ([#346](#346)) ([2eb5887](2eb5887)) * **roles:** add specific roles for assets and devices ([#365](#365)) ([a01982c](a01982c)) * upsert devices ([#355](#355)) ([c75abef](c75abef))

# [2.4.0](v2.3.2...v2.4.0) (2024-10-01) ### Bug Fixes * add specific dev branch in releaserc.js ([e11a823](e11a823)) * **asset_migration:** fixes multiple assets migration and optimize ([fd546cf](fd546cf)) * **assetsGroups:** disable checkRights on impersonate ([#325](#325)) ([eaa3edc](eaa3edc)) * **assets:** migrate tenant only if user is admin ([c24f942](c24f942)) * **assetsMigrateTenant:** add type for frontend and fix admin only ([e249f13](e249f13)) * catch errors when updating the mappings on startup ([#338](#338)) ([e09c5a3](e09c5a3)) * **ci:** correct github token in ci ([a5a6b69](a5a6b69)) * **ci:** fix npm glob not working anymore ([18e79aa](18e79aa)) * copy-version script ([72b5e49](72b5e49)) * correct api types ([#333](#333)) ([05c426f](05c426f)) * correct KuzzleRequest for pipes ([ef932d1](ef932d1)) * correct release script ([f02c285](f02c285)) * **device-metadata:** fetch device's metadata from assigned tenant when available ([#374](#374)) ([acfd8a0](acfd8a0)) * **device:** correct device HTTP API ([#354](#354)) ([d9fe9e1](d9fe9e1)) * **engine:** update conflicts ask return values ([0f817c3](0f817c3)) * **export:** correct scroll in getNamedMeasures ([05fe4fd](05fe4fd)) * improve get internal devices documents ([03ccb29](03ccb29)) * improve measure export to be usable with multiple measures in one measure ([#330](#330)) ([dab8d57](dab8d57)) * **migrateTenant:** clear groups of newly created assets ([79f6a5a](79f6a5a)) * **models:** list asset returns commons assets ([#345](#345)) ([2844437](2844437)) * **npm:** fix repository url ([514dc59](514dc59)) * only list asset models for the requested engine group ([#367](#367)) ([8a14207](8a14207)) * update semantic-release ([380c8b3](380c8b3)) ### Features * add actions to get last measures on digital twins ([#363](#363)) ([b8005a2](b8005a2)) * add dev builds when pushing on 2-dev ([#375](#375)) ([aa11885](aa11885)) * add group descriptions to models ([#335](#335)) ([de00fb7](de00fb7)) * add log in decoders ([#339](#339)) ([fe08b7c](fe08b7c)) * add search actions for the models ([#362](#362)) ([9990e57](9990e57)) * add tooltip model by asset model ([#341](#341)) ([865ae6d](865ae6d)) * add UTC export by default for devices and assets ([#342](#342)) ([878ca58](878ca58)) * **asset:** add upsert method ([#323](#323)) ([5f87787](5f87787)) * **assetrole:** add link / unlink device action to asset admin role ([#357](#357)) ([febf3a6](febf3a6)) * **assets:** assets migrate tenant ([5f05101](5f05101)) * **assets:** remove assetId in upsert ([#326](#326)) ([2f286de](2f286de)) * **configuration:** allow to specify Elasticsearch index settings ([#337](#337)) ([6d22fa9](6d22fa9)) * digitaltwin exports ([#315](#315)) ([aab65af](aab65af)) * **digitalTwin:** allow to modify mapping from application ([#322](#322)) ([89d375a](89d375a)) * **digitalTwin:** implement generic document pipes triggers ([#321](#321)) ([60a21fc](60a21fc)) * **engine:** add mappings update conflict ask call ([#340](#340)) ([5cd6aca](5cd6aca)) * implement comprehensible metadata ([#332](#332)) ([e80ef8e](e80ef8e)) * improve export ([#316](#316)) ([51db84a](51db84a)) * improve search model request types ([#364](#364)) ([5310ad1](5310ad1)) * **measure:** add device metadata to origin ([#356](#356)) ([1ed9d8b](1ed9d8b)) * **measuremodels:** add optional local names and unit to measure definitions ([#343](#343)) ([50038b2](50038b2)) * **measures:** add lastMeasuredAt on DigitalTwin ([#314](#314)) ([05b8a53](05b8a53)) * **metadata:** add option field readOnly to prevent editing ([#358](#358)) ([e0641b4](e0641b4)) * **metadata:** add optional editor hints for metadata ([#370](#370)) ([afc5129](afc5129)) * **metadata:** add support nested metadata properties ([#348](#348)) ([17eda8b](17eda8b)) * **metadata:** improve asset metadata ([#346](#346)) ([2eb5887](2eb5887)) * **roles:** add specific roles for assets and devices ([#365](#365)) ([a01982c](a01982c)) * upsert devices ([#355](#355)) ([c75abef](c75abef))

sebtiz13 added 3 commits August 30, 2023 14:08

feat(export): add support elasticsearch lang

66e8017

feat: improve handle error

d72ba2d

feat(export): limit export to 10k documents

6d448a1

prevent segfault with more than 25k results

sebtiz13 requested review from rolljee and tdislay August 30, 2023 14:25

sebtiz13 self-assigned this Aug 30, 2023

fix(export): prevent error duplicate result

beacb1a

rolljee requested changes Aug 30, 2023

View reviewed changes

fix: correct test functional

58e0a0b

rolljee approved these changes Aug 30, 2023

View reviewed changes

fix: correct prepareMeasureSearch for ES requests

9ccba3f

sebtiz13 force-pushed the fix/digitaltwin-export branch from 485a82d to 9ccba3f Compare August 31, 2023 13:20

sebtiz13 added 2 commits August 31, 2023 16:14

fix: measures search lang

d5fc128

chore: add exporte response explaination [skip ci]

199b3f5

sebtiz13 merged commit e4e746e into 2-dev Aug 31, 2023

sebtiz13 deleted the fix/digitaltwin-export branch August 31, 2023 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improve export #316

feat: improve export #316

sebtiz13 commented Aug 30, 2023 •

edited

Loading

rolljee Aug 30, 2023

sebtiz13 Aug 30, 2023 •

edited

Loading

tdislay Aug 31, 2023

rolljee Aug 30, 2023

sebtiz13 Aug 30, 2023

tdislay Aug 31, 2023

feat: improve export #316

feat: improve export #316

Conversation

sebtiz13 commented Aug 30, 2023 • edited Loading

What does this PR do ?

rolljee Aug 30, 2023

Choose a reason for hiding this comment

sebtiz13 Aug 30, 2023 • edited Loading

Choose a reason for hiding this comment

tdislay Aug 31, 2023

Choose a reason for hiding this comment

rolljee Aug 30, 2023

Choose a reason for hiding this comment

sebtiz13 Aug 30, 2023

Choose a reason for hiding this comment

tdislay Aug 31, 2023

Choose a reason for hiding this comment

sebtiz13 commented Aug 30, 2023 •

edited

Loading

sebtiz13 Aug 30, 2023 •

edited

Loading