Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust error for already locked db and prevent level db lock on malformed connstr #18923

Merged
merged 5 commits into from
Feb 27, 2022

Conversation

zeripath
Copy link
Contributor

This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.

Reference #18921
Reference #18917

Signed-off-by: Andrew Thornton [email protected]

…rmed connstr

This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.

Reference go-gitea#18921
Reference go-gitea#18917

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath added the type/enhancement An improvement of existing functionality label Feb 26, 2022
@zeripath zeripath added this to the 1.17.0 milestone Feb 26, 2022
@zeripath
Copy link
Contributor Author

is backportable

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 26, 2022
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 26, 2022
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 26, 2022
Comment on lines +73 to +74
db.name = append(db.name, connection)
m.LevelDBConnections[connection] = db
Copy link
Contributor

@wxiaoguang wxiaoguang Feb 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
db.name = append(db.name, connection)
m.LevelDBConnections[connection] = db

If we run into this if, then neither connection nor uri exists in the LevelDBConnections.

But here the code only append the connection to the name list, but no uri, is it by purpose?

In my opinion:

  • Either we keep consistent that we append both non-existing connection and uri into the name list
  • Or append nothing (remove these 2 lines), because the logic is sill correct, no performance degrade, and more simple.

Since the old code works, this comment is not a blocker.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It prevents the repeated logging of the same message

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's not complete.

Current code It only prevents repeated logging for the new connection, but not for the new uri with the same directory. (that the case 1)

And I do not think there will be repeated loggings, the queues call GetLevelDB for open the db, there seems no repeated calls (so that why case 2 is also fine because there seems to be no repeated log messages)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it absolutely does prevent repeated logging of the same message. The log message doesn't contain the inferred URI. The point is to tell the server user which strings in their configuration are incorrect.

There are multiple queues. Depending on the order of initialisation and state of the incorrect configuration without this you face repeatedly seeing the same warning message.

@lunny
Copy link
Member

lunny commented Feb 27, 2022

2022/02/27 21:42:01 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: unique-level for repo-archive cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo-archive-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:01 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: level for push_update cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"push_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:01 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: level for notification-service cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"notification-service-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable

2022/02/27 21:42:02 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: unique-level for repo_stats_update cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo_stats_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:02 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: level for task cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"task-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:02 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: unique-level for mirror cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"mirror-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:02 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 2] Failed to create queue: unique-level for pr_patch_checker cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"pr_patch_checker-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:04 ...ource/manager_run.go:40:Run() [T] Event source has no listeners
2022/02/27 21:42:04 ...ource/manager_run.go:50:Run() [T] Pausing the eventsource
2022/02/27 21:42:10 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: level for push_update cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"push_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:10 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: unique-level for repo-archive cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo-archive-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:10 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: level for notification-service cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"notification-service-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:11 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: unique-level for repo_stats_update cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo_stats_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:11 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: unique-level for mirror cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"mirror-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:11 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: unique-level for pr_patch_checker cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"pr_patch_checker-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:11 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 3] Failed to create queue: level for task cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"task-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:19 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: level for push_update cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"push_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:19 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: unique-level for repo-archive cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo-archive-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:19 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: level for notification-service cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"notification-service-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:20 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: unique-level for repo_stats_update cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:20, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"repo_stats_update-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:20 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: unique-level for pr_patch_checker cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"pr_patch_checker-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:20 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: unique-level for mirror cfg: queue.LevelUniqueQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"mirror-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable
2022/02/27 21:42:20 ...eue/queue_wrapped.go:73:setInternal() [W] [Attempt: 4] Failed to create queue: level for task cfg: queue.LevelQueueConfiguration{ByteFIFOQueueConfiguration:queue.ByteFIFOQueueConfiguration{WorkerPoolConfiguration:queue.WorkerPoolConfiguration{QueueLength:1000, BatchLength:20, BlockTimeout:1000000000, BoostTimeout:300000000000, BoostWorkers:1, MaxWorkers:5}, Workers:0, Name:"task-level", WaitOnEmpty:false}, DataDir:"/gitea/gitea/data/queues/common", ConnectionString:"", QueueName:""} error: unable to lock level db at /gitea/gitea/data/queues/common: resource temporarily unavailable

@lunny
Copy link
Member

lunny commented Feb 27, 2022

The error resource temporarily unavailable means leveldb is not closed correctly.

@zeripath
Copy link
Contributor Author

I've adjusted the CloseLevelDB to also look for the datadir too.

@zeripath
Copy link
Contributor Author

make lgtm work

@zeripath zeripath merged commit 4697735 into go-gitea:main Feb 27, 2022
@zeripath zeripath deleted the improve-warning-for-wrapped-queues branch February 27, 2022 19:45
zeripath added a commit to zeripath/gitea that referenced this pull request Feb 27, 2022
…rmed connstr (go-gitea#18923)

Backport go-gitea#18923

This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.

Reference go-gitea#18921
Reference go-gitea#18917

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath added backport/done All backports for this PR have been created backport/v1.0 and removed backport/v1.0 labels Feb 27, 2022
zjjhot added a commit to zjjhot/gitea that referenced this pull request Feb 28, 2022
* giteaofficial/main:
  Adjust error for already locked db and prevent level db lock on malformed connstr (go-gitea#18923)
  Avoid database lookups for `DescriptionHTML` (go-gitea#18924)
  Update go-org to v1.6.1 (go-gitea#18932)
  Fix trace log to show value instead of pointers (go-gitea#18926)
  Add announcement steps to release-howto (go-gitea#18896)
  Fix `<strong>` html in translation (go-gitea#18929)
zeripath added a commit that referenced this pull request Feb 28, 2022
…rmed connstr (#18923) (#18938)

Backport #18923

This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.

Reference #18921
Reference #18917

Signed-off-by: Andrew Thornton <[email protected]>
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
…rmed connstr (go-gitea#18923)

This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.

Reference go-gitea#18921
Reference go-gitea#18917

Signed-off-by: Andrew Thornton <[email protected]>
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/done All backports for this PR have been created lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/enhancement An improvement of existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants