CUMULUS-3859:Update @Cumulus packages and elasticsearch removal (#1155)

* CUMULUS-3859:Update @Cumulus packages and elasticsearch removal * enable skipped test * see asyncOperations * Update api to 19.2.0-alpha.1 * Updated packages * update audit-ci allowedlist * fix lint * update package for errors
nasa · Nov 25, 2024 · d2f2b3f · d2f2b3f
1 parent 00285e8
commit d2f2b3f
Show file tree

Hide file tree

Showing 25 changed files with 17,178 additions and 42,151 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,17 +7,26 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+This version of the dashboard requires Cumulus API >= v19.2.0-alpha.1 (TBD API release)
+
 ### Changed
 
 - **CUMULUS-3748**
   - Removed reconciliationReports aggregate stats call from Granules Page
 - **CUMULUS-3821**
   - Removed fake-granuleid-refresh-connection/refreshCumulusDbConnection
+- **CUMULUS-3859**
+  - Removed ElasticSearch from localAPI docker-compose files
+  - Updated the dashboard to use alpha version `@cumulus/[email protected]` and
+    `@cumulus/[email protected]`
+  - Updated the integration tests to work with updated API
+  - Updated packages to address [CVE-2024-21538] (https://github.com/advisories/GHSA-3xgq-45jj-v275)
 - **CUMULUS-3870**
   - Remove launchpad security key information from cypress fixture for `valid-execution.json`
   - Add placeholders for security information with `fakePassword` and `userName`
 - **CUMULUS-3892**
   - Updated Metrics Overview page executions and rules query to only get count from the past 24 hours
+
 ### Fixed
 
 - **CUMULUS-3905**

diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
@@ -194,7 +194,7 @@ The basic steps to follow are:
 
     This is a two step process, where you use [`npm link`](https://docs.npmjs.com/cli-commands/link.html) to prepare the @cumulus/api by running `npm link` in the `cumulus/packages/api` directory and then use it in the dashboard project by running `npm link @cumulus/api` in the dashboard root directory.
 
-    When you have done this, you need to make sure that you only use the LocalStack/Elasticsearch portion of the docker-compose containers, e.g. run `npm run start-localstack` and `npm run serve-api` in order to run the linked version of the @cumulus/api.  Do all of your development locally including running unit and integration tests.  When you are satisfied with the changes to the Cumulus core code (and when that is merged) you can do an alpha release of the code by building an alpha release of the package `npm version prerelease --preid=alpha` and then publishing to npm with a tag of `next`. `npm publish --tag next`.  This will upload the contents of your `cumulus/packages/api` to npm with a version that looks like `1.18.1-alpha.0` and a tag of `next`.
+    When you have done this, you need to make sure that you only use the LocalStack portion of the docker-compose containers, e.g. run `npm run start-localstack` and `npm run serve-api` in order to run the linked version of the @cumulus/api.  Do all of your development locally including running unit and integration tests.  When you are satisfied with the changes to the Cumulus core code (and when that is merged) you can do an alpha release of the code by building an alpha release of the package `npm version prerelease --preid=alpha` and then publishing to npm with a tag of `next`. `npm publish --tag next`.  This will upload the contents of your `cumulus/packages/api` to npm with a version that looks like `1.18.1-alpha.0` and a tag of `next`.
 
 2. Update the dashboard to use the alpha package for CI.
 

diff --git a/README.md b/README.md
@@ -207,7 +207,7 @@ If you're just testing dashboard code, you can generally run all of the above co
 ```bash
   $ npm run start-dashboard
 ```
-This brings up LocalStack, Elasticsearch, the Cumulus localAPI, and the dashboard.
+This brings up LocalStack, the Cumulus localAPI, and the dashboard.
 
 Run the test suite (yet another terminal window)
 ```bash
@@ -223,7 +223,7 @@ For **development** and **testing** purposes only, you can run a Cumulus API loc
 
 *Important Note: These `docker compose` commands do not build distributable containers, but are a provided as testing conveniences.  The docker-compose[-\*].yml files show that they work by linking your local directories into the container.*
 
-In order to run the Cumulus API locally you must first [build the dashboard](#build-the-dashboard) and then run the containers that provide LocalStack and Elasticsearch services.
+In order to run the Cumulus API locally you must first [build the dashboard](#build-the-dashboard) and then run the containers that provide LocalStack services.
 
 These are started and stopped with the commands:
 ```bash
@@ -253,7 +253,6 @@ The output looks like this:
 > docker compose -f ./localAPI/docker-compose.yml -f ./localAPI/docker-compose-serve-api.yml up -d
 
 Creating localapi_shim_1 ... done
-Creating localapi_elasticsearch_1 ... done
 Creating localapi_localstack_1    ... done
 Creating localapi_serve_api_1     ... done
 ```
@@ -277,7 +276,7 @@ Bring up and down the entire stack (the localAPI and the dashboard) with:
   $ npm run start-dashboard
   $ npm run stop-dashboard
 ```
-This runs everything, the backing Localstack and Elasticsearch containers, the local Cumulus API and dashboard.  Edits to your code will be reflected in the running dashboard.  You can run cypress tests still with `npm run cypress`.  As a warning, this command takes a very long time to start up because the containers come up in a specific order and generally this should be reserved for use by Earthdata Bamboo or some other continuous integration service.  But, if you are using it locally, **be sure to wait until all containers are fully up** before trying to visit the dashboard which is exposed at http://localhost:3000
+This runs everything, the backing Localstack container, the local Cumulus API and dashboard.  Edits to your code will be reflected in the running dashboard.  You can run cypress tests still with `npm run cypress`.  As a warning, this command takes a very long time to start up because the containers come up in a specific order and generally this should be reserved for use by Earthdata Bamboo or some other continuous integration service.  But, if you are using it locally, **be sure to wait until all containers are fully up** before trying to visit the dashboard which is exposed at http://localhost:3000
 The stack is ready when the `view-docker-logs` task shows:
 ```bash
 dashboard_1      | > NODE_ENV=production http-server dist -p 3000 --proxy http://localhost:3000?

diff --git a/TABLES.md b/TABLES.md
@@ -44,7 +44,7 @@ Wraps `sortable-table` and implements auto-update and smart sort. When a new sor
 
 ## Dumb vs smart sort
 
-Dumb sorting uses react-table's built in sort functionality to sort table data that has **already** been received from the API. Smart sorting initiates a new API request, passing the sort parameter to the server (elasticsearch) which returns a sorted response. The `manualSortBy` option passed to `useTable()` tells react-table whether we are doing server-side sorting (`true`) or letting react-table sort (`false`).
+Dumb sorting uses react-table's built in sort functionality to sort table data that has **already** been received from the API. Smart sorting initiates a new API request, passing the sort parameter to the server which returns a sorted response. The `manualSortBy` option passed to `useTable()` tells react-table whether we are doing server-side sorting (`true`) or letting react-table sort (`false`).
 
 When sorting on multiple columns, select the first column, hold down the shift key, and select another column.
 

diff --git a/app/src/js/utils/format.js b/app/src/js/utils/format.js
@@ -335,9 +335,9 @@ export const truncate = (string, to = 100) => {
 const collectionIdSeparator = '___';
 /**
  * Returns the name and version of a collection based on
- * the collectionId used in elasticsearch indexing
+ * the collectionId
  *
- * @param {string} collectionId - collectionId used in elasticsearch index
+ * @param {string} collectionId - collectionId
  * @returns {Object} name and version as object
  */
 export const deconstructCollectionId = (collectionId) => {

diff --git a/app/src/js/utils/table-config/reconciliation-reports.js b/app/src/js/utils/table-config/reconciliation-reports.js
@@ -82,8 +82,8 @@ const handleDeleteClick = (e, value, dispatch, query) => {
   });
 };
 
-// It can take an extra second or two to delete a report from
-// Elasticsearch. This function keeps checking the reports to make sure it was
+// It can take an extra second or two to delete a report.
+// This function keeps checking the reports to make sure it was
 // deleted before displaying the delete result.
 const refreshReports = (dispatch, value, query, retries) => {
   if (retries > 0) {

diff --git a/app/src/js/utils/type.js b/app/src/js/utils/type.js
@@ -4,11 +4,6 @@ export const reconciliationReportTypes = [
     label: 'Granule Not Found',
     description: 'This report displays the granules that were not found in each bucket.',
   },
-  {
-    id: 'Internal',
-    label: 'Internal',
-    description: 'This report compares the collections and granules that are in DynamoDB versus ElasticSearch.',
-  },
   {
     id: 'Inventory',
     label: 'Inventory',

diff --git a/audit-ci.json b/audit-ci.json
@@ -2,5 +2,5 @@
   "high": true,
   "pass-enoaudit": true,
   "retry-count": 20,
-  "allowlist": ["braces", "d3-color", "webpack-dev-middleware", "ws", "jsonpath-plus"]
+  "allowlist": ["d3-color" ]
 }
diff --git a/cypress/fixtures/seeds/asyncOperationsFixture.json b/cypress/fixtures/seeds/asyncOperationsFixture.json
@@ -0,0 +1,89 @@
+{
+    "meta": {
+        "name": "cumulus-api",
+        "stack": "test-source-integration",
+        "table": "asyncOperation",
+        "limit": 10,
+        "page": 1,
+        "count": 7
+    },
+    "results": [
+        {
+            "output": "{\"createdAt\":1600888528207,\"location\":\"s3://cumulus-test-sandbox-internal/test-source-integration/reconciliation-reports/InternalReportc89bf1d37c.json\",\"name\":\"InternalReportc89bf1d37c\",\"updatedAt\":1600888542261,\"type\":\"Internal\",\"status\":\"Generated\"}",
+            "createdAt": 1600888510551,
+            "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/a5bea509-a6d4-4bba-b64a-01477b413c5f",
+            "description": "Create Reconciliation Report",
+            "operationType": "Reconciliation Report",
+            "id": "17907019-81fb-464a-bc76-0ad0490fdeec",
+            "status": "SUCCEEDED",
+            "updatedAt": 1600888542374,
+            "timestamp": 1600888542710
+        },
+        {
+            "output": "\"Processed 2 kinesis records from stream test-source-integration-KinesisReplayTest-1600888397315-ReplayTestStream\"",
+            "createdAt": 1600888447378,
+            "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/a00c2955-8ba9-4603-b9b7-a6691f9bbd50",
+            "description": "Kinesis Replay",
+            "operationType": "Kinesis Replay",
+            "id": "fefa46dc-14e5-4dd8-b3de-bccd192da9c8",
+            "status": "SUCCEEDED",
+            "updatedAt": 1600888480998,
+            "timestamp": 1600888485639
+        },
+        {
+            "output": "{\"deletedGranules\":[\"granule-id-45cde3b57f\"]}",
+            "createdAt": 1600888430615,
+            "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/2e71969d-41a9-453b-827f-1b3f42d8db47",
+            "description": "Bulk granule deletion",
+            "operationType": "Bulk Granule Delete",
+            "id": "452e8941-9d18-4c40-9962-2ff38c179a39",
+            "status": "SUCCEEDED",
+            "updatedAt": 1600888461269,
+            "timestamp": 1600888466116
+        },
+        {
+            "output": "{\"name\":\"AggregateError\",\"message\":\"\\n    DeletePublishedGranule: You cannot delete a granule that is published to CMR. Remove it from CMR first\\n        at Granule.delete (/home/task/lambda-function/webpack:/models/granules.js:542:13)\\n        at pMap (/home/task/lambda-function/webpack:/lambdas/bulk-operation.js:131:17)\",\"stack\":\"AggregateError: \\n    DeletePublishedGranule: You cannot delete a granule that is published to CMR. Remove it from CMR first\\n        at Granule.delete (/home/task/lambda-function/webpack:/models/granules.js:542:13)\\n        at pMap (/home/task/lambda-function/webpack:/lambdas/bulk-operation.js:131:17)\\n    at next (/home/task/lambda-function/webpack:/node_modules/p-map/index.js:43:1)\\n    at /home/task/lambda-function/webpack:/node_modules/p-map/index.js:67:1\\n    at process._tickCallback (internal/process/next_tick.js:68:7)\"}",
+            "createdAt": 1600888398792,
+            "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/0c8dfd89-ac9a-422f-b7a7-d02607c368ce",
+            "description": "Bulk granule deletion",
+            "operationType": "Bulk Granule Delete",
+            "id": "c81fa7ad-d83b-4ed0-bfbf-fca16a329836",
+            "status": "TASK_FAILED",
+            "updatedAt": 1600888420958,
+            "timestamp": 1600888425708
+        },
+        {
+            "output": "[1,2,3]",
+            "createdAt": 1600888260818,
+            "taskArn": "f0f6f6dbc7984c4bd93f268816184406b6e118c1",
+            "description": "Some description",
+            "operationType": "ES Index",
+            "id": "76558240-5417-4bfd-9a12-b2ae439ff44f",
+            "status": "SUCCEEDED",
+            "updatedAt": 1600888277922,
+            "timestamp": 1600888282659
+        },
+        {
+            "output": "{\"name\":\"JSONParsingError\",\"message\":\"Unable to parse payload: Unexpected token i in JSON at position 0\",\"stack\":\"JSONParsingError: Unable to parse payload: Unexpected token i in JSON at position 0\\n    at fetchPayload (/home/task/index.js:53:22)\\n    at process._tickCallback (internal/process/next_tick.js:68:7)\"}",
+            "createdAt": 1600888235257,
+            "taskArn": "fca90ba101555599cf2444295283cb6bf7da79e4",
+            "description": "Some description",
+            "operationType": "ES Index",
+            "id": "296e94d3-466e-474e-85b1-a09bddd69331",
+            "status": "TASK_FAILED",
+            "updatedAt": 1600888248842,
+            "timestamp": 1600888253342
+        },
+        {
+            "output": "{\"name\":\"Error\",\"message\":\"Failed to fetch s3://cumulus-test-sandbox-internal/da741270286ef3b6fb76ca28f67605ad8d11602a: The specified key does not exist.\",\"stack\":\"Error: Failed to fetch s3://cumulus-test-sandbox-internal/da741270286ef3b6fb76ca28f67605ad8d11602a: The specified key does not exist.\\n    at fetchPayload (/home/task/index.js:45:11)\\n    at process._tickCallback (internal/process/next_tick.js:68:7)\"}",
+            "createdAt": 1600888215835,
+            "taskArn": "0fe3105514718109e629f788a6533b0c4c94eaf3",
+            "description": "Some description",
+            "operationType": "ES Index",
+            "id": "1783f739-742d-4095-9283-3f65f37b2a7e",
+            "status": "RUNNER_FAILED",
+            "updatedAt": 1600888228564,
+            "timestamp": 1600888233309
+        }
+    ]
+}
diff --git a/cypress/fixtures/seeds/executionsFixture.json b/cypress/fixtures/seeds/executionsFixture.json
@@ -304,13 +304,13 @@
     {
       "arn": "arn:aws:states:us-east-1:123456789012:execution:test-source-integration-IngestAndPublishGranule:fd0bc7b5-56c3-4e37-b445-8a25f27c06b8",
       "status": "failed",
-      "timestamp": 1629488546679,
+      "timestamp": 1539488546679,
       "tasks": {},
-      "createdAt": 1629488527340,
+      "createdAt": 1539488527340,
       "cumulusVersion": "9.4.0",
       "name": "fd0bc7b5-56c3-4e37-b445-8a25f27c06b8",
       "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:test-source-integration-IngestAndPublishGranule:fd0bc7b5-56c3-4e37-b445-8a25f27c06b8",
-      "updatedAt": 1632110409176,
+      "updatedAt": 1542110409176,
       "collectionId": "MOD09GQ___006",
       "duration": 18.498,
       "error": {
@@ -323,18 +323,18 @@
     {
       "arn": "arn:aws:states:us-east-1:123456789012:execution:test-source-integration-HelloWorldWorkflow:79ba430a-0616-4458-ad5e-1bf2e979cfe4",
       "status": "completed",
-      "timestamp": 1580327992306,
+      "timestamp": 1540327992306,
       "tasks": {
         "0": {
           "name": "test-source-integration-HelloWorld",
           "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-source-integration-HelloWorld",
           "version": "$LATEST"
         }
       },
-      "createdAt": 1580327980133,
+      "createdAt": 1540327980133,
       "name": "79ba430a-0616-4458-ad5e-1bf2e979cfe4",
       "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:test-source-integration-HelloWorldWorkflow:79ba430a-0616-4458-ad5e-1bf2e979cfe4",
-      "updatedAt": 1581220884436,
+      "updatedAt": 1541220884436,
       "duration": 11.821,
       "error": {
         "Error": "Unknown Error",
@@ -345,22 +345,22 @@
     {
       "arn": "arn:aws:states:us-east-1:596205514787:execution:test-source-integration-HelloWorldWorkflow:1a5aaf01-835b-431a-924f-96a7feb3a3fb",
       "status": "completed",
-      "timestamp": 1632351781244,
+      "timestamp": 1542351781244,
       "tasks": {
        "0": {
         "name": "test-source-integration-HelloWorld",
         "arn": "arn:aws:lambda:us-east-1:596205514787:function:test-source-integration-HelloWorld",
         "version": "$LATEST"
        }
       },
-      "createdAt": 1632351778964,
+      "createdAt": 1542351778964,
       "cumulusVersion": "9.5.0",
       "name": "1a5aaf01-835b-431a-924f-96a7feb3a3fb",
       "finalPayload": {
        "hello": "Hello World"
       },
       "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:596205514787:execution:test-source-integration-HelloWorldWorkflow:1a5aaf01-835b-431a-924f-96a7feb3a3fb",
-      "updatedAt": 1632351781244,
+      "updatedAt": 1542351781244,
       "collectionId": "MOD09GQ___006",
       "duration": 1.691,
       "error": {},

diff --git a/cypress/fixtures/seeds/granulesFixture.json b/cypress/fixtures/seeds/granulesFixture.json
@@ -205,8 +205,8 @@
       ],
       "collectionId": "MOD09GQ___006",
       "status": "completed",
-      "timestamp": 1596203544626,
-      "updatedAt": 1596203544157
+      "timestamp": 1537803544626,
+      "updatedAt": 1537803544626
     },
     {
       "granuleId": "MOD09GQ.A5456658.rso6Y4.006.4979096122140",
@@ -440,7 +440,7 @@
     {
       "beginningDateTime": "2017-10-24T00:00:00Z",
       "collectionId": "MOD09GQ___006",
-      "createdAt": 1576106363823,
+      "createdAt": 1537806363823,
       "duration": 7.546,
       "endingDateTime": "2017-11-08T23:59:59Z",
       "error": {
@@ -468,10 +468,10 @@
       "published": false,
       "cmrLink": null,
       "status": "completed",
-      "timestamp": 1576106371369,
+      "timestamp": 1537806371369,
       "timeToArchive": 0.01,
       "timeToPreprocess": 0.318,
-      "updatedAt": 1576106371468
+      "updatedAt": 1537806371468
     }
   ]
 }
diff --git a/cypress/integration/bulk_granules_spec.js b/cypress/integration/bulk_granules_spec.js
@@ -201,7 +201,7 @@ describe('Dashboard Bulk Granules', () => {
         cy.contains('.error__report', errorMessage);
         cy.contains('button', 'Cancel Bulk Operations').click();
 
-        cy.contains('button', 'Run Bulk Granules').click();
+        cy.contains('button', 'Run Bulk Granules').click({ force: true });
 
         cy.get('.bulk_granules')
           .within(() => {
@@ -336,7 +336,7 @@ describe('Dashboard Bulk Granules', () => {
         cy.contains('.error__report', errorMessage);
         cy.contains('button', 'Cancel Bulk Recovery').click();
 
-        cy.contains('button', 'Run Bulk Granules').click();
+        cy.contains('button', 'Run Bulk Granules').click({ force: true });
 
         cy.get('.bulk_granules')
           .within(() => {