improvement: capture and retry database concurrent update errors

[jfcalvo]

Description

After investigate timeouts for PostgreSQL I have found that timeouts should not affect errors when a SERIALIZABLE transactions is rollbacked because another concurrent update error is raised.

So the only way to support concurrent updates with PostgreSQL and SERIALIZABLE transactions is to capture errors and retry the transaction.

This code has the following changes:

• Start using backoff library to retry any of the possible CRUD context functions updating responses and record statuses, using SERIALIZABLE database sessions.
• This change has the side effect of working with PostgreSQL and SQLite at the same time.
• I have set a fixed time of 15 seconds as maximum time for retrying with exponential backoff.
• I have moved search engine updates outside of the transaction block.
• This should mitigate errors on high concurrency scenarios for PostgreSQL and SQLite:
  • For SQLite we have the additional setting to set a timeout if necessary.
• I have changed DEFAULT_DATABASE_SQLITE_TIMEOUT value to 5 seconds so the backoff logic will handle possible problems with locked database errors and SQLite.

Refs #5000

Type of change

• Improvement (change adding some improvement to an existing functionality)

How Has This Been Tested

• Manually testing with PostgreSQL and SQLite, running benchmarks using 20 concurrent requests.
• Running test suite for PostgreSQL and SQLite.

Checklist

• I added relevant documentation
• I followed the style guidelines of this project
• I did a self-review of my code
• I made corresponding changes to the documentation
• I confirm My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.61%. Comparing base (f77341e) to head (15950ab).

Additional details and impacted files

@@                              Coverage Diff                              @@
##           feat/add-dataset-automatic-task-distribution    #5227   +/-   ##
=============================================================================
  Coverage                                         90.60%   90.61%           
=============================================================================
  Files                                               137      137           
  Lines                                              5746     5752    +6     
=============================================================================
+ Hits                                               5206     5212    +6     
  Misses                                              540      540           

Flag	Coverage Δ
argilla-server	90.61% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.