BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence #12392

ilija42 · 2024-03-12T11:46:01Z

Add KV store that uses psql and is tied to jobs
Use KV store for persisted backup value for inMemoryDataSourceCache (used for juelsFeePerCoin datasource)
Improve log severity errors in inMemoryDataSourceCache. Now log ERR severity when there are continuous cache update errors. Also log ERR severity if value failed to get stored in KV store

github-actions · 2024-03-12T11:46:21Z

I see that you haven't updated any README files. Would it make sense to do so?

.changeset/lemon-ladybugs-doubt.md

jmank88 · 2024-03-12T14:28:40Z

.changeset/lemon-ladybugs-doubt.md

+"chainlink": patch
+---
+
+Add kv store tied to jobs and use it for juels fee per coin cache to store persisted values for backup


Can we phrase this in a way that node ops will understand? e.g. by referencing the relevant job spec field

Job spec changes are already refferenced in previous changeset when the cache was added

Do I even need to add changesets that aren't useful for NOPs?

Yea, according to the new process every change needs a changeset. Releng will filter out NOP relevant changes to add to the changelog.

We now almost always require changeset because of new CI

core/services/job/kv_orm_test.go

cl-sonarqube-production · 2024-03-12T16:35:29Z

Quality Gate passed

Issues
0 New issues
1 Fixed issue
0 Accepted issues

Measures
0 Security Hotspots
88.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

…12392) * Add kv store migration * Add kv store implementation * Init kv store in ocr2 delegate and pass into median service * Update ds cache to have kv store fallback for final observation value * Prettify ds cache updateCache, add ERR log severity on consecutive errs * Add ds cache test for cache value persistence * Remove unused field in jobKVStore * Make sonar SQL migration lint happy * Rename TestJobKVStore * Add kv store mock * Add changeset file * Fix sonar sql lint * Change kv orm to use raw json message instead of jsonText * minor change * minor change * Fix SQ SQL lint * Add comments in KVStore * Rename jobKVStore to kVStore and return struct from constructor * Update core/store/migrate/migrations/0227_kv_store_table.sql Co-authored-by: Sam <[email protected]> * Update kVStore sql to match migration * Add more kv_orm tests --------- Co-authored-by: Sam <[email protected]>

* Add MetricsRegistry to automation (#12359) * Pin to metrics registry automation PR * Add missing go.sum * Pin to latest automation version * BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence (#12392) * Add kv store migration * Add kv store implementation * Init kv store in ocr2 delegate and pass into median service * Update ds cache to have kv store fallback for final observation value * Prettify ds cache updateCache, add ERR log severity on consecutive errs * Add ds cache test for cache value persistence * Remove unused field in jobKVStore * Make sonar SQL migration lint happy * Rename TestJobKVStore * Add kv store mock * Add changeset file * Fix sonar sql lint * Change kv orm to use raw json message instead of jsonText * minor change * minor change * Fix SQ SQL lint * Add comments in KVStore * Rename jobKVStore to kVStore and return struct from constructor * Update core/store/migrate/migrations/0227_kv_store_table.sql Co-authored-by: Sam <[email protected]> * Update kVStore sql to match migration * Add more kv_orm tests --------- Co-authored-by: Sam <[email protected]> * Fix in memory data source cache error logging and add err log when cache is over 24h old (#12586) * Fix in memory data source cache error logging * Add stale jfcp cache error log and lower other logs to warn * Update logger warnw to warnf function Co-authored-by: Jordan Krage <[email protected]> --------- Co-authored-by: Jordan Krage <[email protected]> * Handle zkSync "known transaction" error (SHIP-1233) (#12338) * Handle zkSync "known transaction" error (SHIP-1233) * add test case for zkSync specific known transaction error --------- Co-authored-by: Jim W <[email protected]> Co-authored-by: Prashant Yadav <[email protected]> * fix: trigger operator-ui-ci with specific tag (#12601) * fix: trigger operator-ui-ci with specific tag * fix path * add checkout step * update operator-ui tag * make operator-ui * update operator-ui-ci tag conditional (#12607) * update operator-ui-ci tag conditional * use main branch as default tag * fix: check for target branch name instead of current for operator-ui-ci (#12618) * add rebalancer support for feeds manager ocr2 plugins (#12344) * [Functions] Fix bug in comparing subscriptions (#12379) (#12576) Co-authored-by: Sneha Agnihotri <[email protected]> * Add cascade delete to kv store table fk (#12629) * Add cascade delete to kv store table fk * Add changeset (cherry picked from commit 3ec8cc9) * Fix InMemoryDataSourceCache cleanup (#12647) * Fix InMemoryDataSourceCache cleanup * Add changeset * Make linter happy * Handle data source cache unit test cleanup * Use services.StopChan in inMemoryDataSourceCache * Move ctxWithTimeout into updateCache Co-authored-by: Jordan Krage <[email protected]> --------- Co-authored-by: Jordan Krage <[email protected]> * Bump operator-ui (#12684) (#12687) * Finalize date on changelog for 2.10.0 * fix spacing Signed-off-by: Sneha Agnihotri <[email protected]> * Remove old operator-ui assets --------- Signed-off-by: Sneha Agnihotri <[email protected]> Co-authored-by: george-dorin <[email protected]> Co-authored-by: ilija42 <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Jordan Krage <[email protected]> Co-authored-by: Friedemann Fürst <[email protected]> Co-authored-by: Jim W <[email protected]> Co-authored-by: Prashant Yadav <[email protected]> Co-authored-by: frank zhu <[email protected]> Co-authored-by: Margaret Ma <[email protected]> Co-authored-by: Bolek <[email protected]> Co-authored-by: chainchad <[email protected]> Co-authored-by: george-dorin <[email protected]> Co-authored-by: krehermann <[email protected]>

ilija42 added 5 commits March 11, 2024 22:09

Add kv store migration

859b97b

Add kv store implementation

8742a21

Init kv store in ocr2 delegate and pass into median service

b817e41

Update ds cache to have kv store fallback for final observation value

d46279a

Prettify ds cache updateCache, add ERR log severity on consecutive errs

c98a990

ilija42 temporarily deployed to sdlc March 12, 2024 11:46 — with GitHub Actions Inactive

ilija42 changed the title ~~Job Based key value store and juelsFeePerCoin reboot persistence~~ BCF-3052 - Job Based key value store and juelsFeePerCoin reboot persistence Mar 12, 2024

Add ds cache test for cache value persistence

4d65dac

ilija42 temporarily deployed to sdlc March 12, 2024 12:45 — with GitHub Actions Inactive

Remove unused field in jobKVStore

23eabfb

ilija42 temporarily deployed to sdlc March 12, 2024 12:47 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/develop' into jfpc-caching

af4ccd5

ilija42 temporarily deployed to sdlc March 12, 2024 13:01 — with GitHub Actions Inactive

Make sonar SQL migration lint happy

1e0a49a

ilija42 temporarily deployed to sdlc March 12, 2024 13:04 — with GitHub Actions Inactive

Rename TestJobKVStore

18c3858

ilija42 temporarily deployed to sdlc March 12, 2024 13:06 — with GitHub Actions Inactive

ilija42 added 2 commits March 12, 2024 14:17

Add kv store mock

d2c90ef

Add changeset file

2409b01

ilija42 temporarily deployed to sdlc March 12, 2024 13:17 — with GitHub Actions Inactive

Fix sonar sql lint

c3d08d9

ilija42 temporarily deployed to sdlc March 12, 2024 13:23 — with GitHub Actions Inactive

ilija42 changed the title ~~BCF-3052 - Job Based key value store and juelsFeePerCoin reboot persistence~~ BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence Mar 12, 2024

Change kv orm to use raw json message instead of jsonText

c6d27b1

ilija42 temporarily deployed to sdlc March 12, 2024 13:30 — with GitHub Actions Inactive

minor change

7fe407c

ilija42 temporarily deployed to sdlc March 12, 2024 13:33 — with GitHub Actions Inactive

minor change

850606b

ilija42 temporarily deployed to sdlc March 12, 2024 13:35 — with GitHub Actions Inactive

Rename jobKVStore to kVStore and return struct from constructor

2dff468

ilija42 dismissed cedric-cordenier’s stale review via 2dff468 March 12, 2024 14:19

ilija42 temporarily deployed to sdlc March 12, 2024 14:19 — with GitHub Actions Inactive

Update core/store/migrate/migrations/0227_kv_store_table.sql

0c65925

Co-authored-by: Sam <[email protected]>

ilija42 temporarily deployed to sdlc March 12, 2024 14:19 — with GitHub Actions Inactive

krehermann reviewed Mar 12, 2024

View reviewed changes

.changeset/lemon-ladybugs-doubt.md Show resolved Hide resolved

ilija42 requested a review from samsondav March 12, 2024 14:26

jmank88 reviewed Mar 12, 2024

View reviewed changes

samsondav previously approved these changes Mar 12, 2024

View reviewed changes

core/services/job/kv_orm_test.go Show resolved Hide resolved

Update kVStore sql to match migration

d082747

ilija42 dismissed samsondav’s stale review via d082747 March 12, 2024 15:28

ilija42 temporarily deployed to sdlc March 12, 2024 15:28 — with GitHub Actions Inactive

Add more kv_orm tests

91a61eb

ilija42 temporarily deployed to sdlc March 12, 2024 16:23 — with GitHub Actions Inactive

ilija42 requested review from samsondav, jmank88 and cedric-cordenier March 12, 2024 16:26

ilija42 enabled auto-merge March 12, 2024 16:35

cedric-cordenier approved these changes Mar 12, 2024

View reviewed changes

ilija42 added this pull request to the merge queue Mar 12, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 12, 2024

ilija42 added this pull request to the merge queue Mar 12, 2024

Merged via the queue into develop with commit 8626f1b Mar 12, 2024
97 checks passed

ilija42 deleted the jfpc-caching branch March 12, 2024 17:53

snehaagni mentioned this pull request Apr 8, 2024

chore/release 2.10.0 to develop #12740

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence #12392

BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence #12392

ilija42 commented Mar 12, 2024 •

edited

Loading

github-actions bot commented Mar 12, 2024

jmank88 Mar 12, 2024

ilija42 Mar 12, 2024

ilija42 Mar 12, 2024

patrickhuie19 Mar 12, 2024

ilija42 Mar 12, 2024

cl-sonarqube-production bot commented Mar 12, 2024

BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence #12392

BCF-3052 - Job Based KV Store and juelsFeePerCoin reboot persistence #12392

Conversation

ilija42 commented Mar 12, 2024 • edited Loading

github-actions bot commented Mar 12, 2024

jmank88 Mar 12, 2024

Choose a reason for hiding this comment

ilija42 Mar 12, 2024

Choose a reason for hiding this comment

ilija42 Mar 12, 2024

Choose a reason for hiding this comment

patrickhuie19 Mar 12, 2024

Choose a reason for hiding this comment

ilija42 Mar 12, 2024

Choose a reason for hiding this comment

cl-sonarqube-production bot commented Mar 12, 2024

Quality Gate passed

ilija42 commented Mar 12, 2024 •

edited

Loading