StorageDuplicateError Multiple ConnRecord records load testing with askar/reusable invitation

[ianco]

Sometimes following error occurs when the load is very high in aca-py 0.7.4 and askar storage:

ERROR aries_cloudagent.core.conductor | DON'T shutdown on StorageDuplicateError Multiple ConnRecord records located for {'invitation_key': '***'}, {'state': 'invitation', 'their_role': 'invitee'}

Application is dotnet based agents with mediator for load test. dotnet agents make continuous iterations of three types of tasks (connection / issue-credential / present-proof) for a period of time. and agent scale out.

on each iteration, new agent/wallet is made on dotnet side. and all dotnet agent use single multi-use invitation from aca-py. therefore, total connections will grow over time on aca-py side.

In the beginning of load test, the connection takes about a second or so. However, as the loop runs and Askar's SCAN_QUERY returns an average of 5000 results, the connection latency increases to 3 seconds. This delay is reset when aca-py create a new invitation .

@kukgini

[kukgini]

I wonder the user who requested the connection request not be able to connect? or It just internal error and the requester eventually will be connected? And If the above error occurs and aca-py killed before cleanup is done properly, will the data consistency be broken?

[kukgini]

And it's weird that when I query with AdminAPI about this connection record. there is only one connection for this invitation_key with state invitation. I wonder where is come from duplication.

[KimEbert42]

See #2099 and #2116

[swcurran]

Closing via #2116