Check storage readiness and show meaningful error message when DB is busy

[aku]

Feature Description

I've tried to install Gitea with PostgreSQL storage via Helm chart and faced the following problem.
When you try to access Gitea right after it has been installed, some actions will fail with 500 errors because PostgreSQL is doing some replication/recovery.

It would be nice to add some sort of DB readiness check and show a meaningful error explaining that we need to wait for some time until the DB is ready

Example of error in a log file:

2023/07/20 08:09:19 ...ers/web/auth/auth.go:543:createUserInContext() [E] CreateUser: pq: cannot set transaction read-write mode during recovery
2023/07/20 08:09:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /user/sign_up for 10.244.0.1:1504, 500 Internal Server Error in 0.8ms @ auth/auth.go:405(auth.SignUpPost)

Screenshots

No response

[willzhang]

same here, v1.20.3 with helm install:

helm install gitea gitea-charts/gitea \
-n gitea --create-namespace \
--set gitea.admin.password=gitea_admin \
--set ingress.enabled=true \
--set ingress.className=nginx

i must wait some time and pods will running

root@ubuntu:~# kubectl -n gitea get pods
NAME                                          READY   STATUS             RESTARTS        AGE
gitea-9cf775dfb-xmd4s                         0/1     CrashLoopBackOff   4 (77s ago)     5m
gitea-postgresql-ha-pgpool-55ddb7cbf9-8v4qq   1/1     Running            0               5m
gitea-postgresql-ha-postgresql-0              1/1     Running            0               5m
gitea-postgresql-ha-postgresql-1              1/1     Running            0               5m
gitea-postgresql-ha-postgresql-2              1/1     Running            0               5m
gitea-redis-cluster-0                         1/1     Running            2 (3m51s ago)   5m
gitea-redis-cluster-1                         1/1     Running            3 (4m4s ago)    5m
gitea-redis-cluster-2                         1/1     Running            2 (4m4s ago)    5m
gitea-redis-cluster-3                         1/1     Running            2 (3m58s ago)   5m
gitea-redis-cluster-4                         1/1     Running            2 (3m18s ago)   5m
gitea-redis-cluster-5                         1/1     Running            1 (3m17s ago)   5m

root@ubuntu:~# kubectl -n gitea logs -f gitea-9cf775dfb-xmd4s
Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/08/25 06:40:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/08/25 06:40:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.3 built with GNU Make 4.4.1, go1.20.7 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/08/25 06:40:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/08/25 06:40:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/08/25 06:40:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/08/25 06:40:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/08/25 06:40:00 cmd/web.go:153:serveInstalled() [I] Run mode: prod
2023/08/25 06:40:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/08/25 06:40:00 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/08/25 06:40:01 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/08/25 06:40:01 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/08/25 06:40:01 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/08/25 06:40:01 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/attachments
2023/08/25 06:40:01 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/avatars
2023/08/25 06:40:01 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-avatars
2023/08/25 06:40:01 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/repo-archive
2023/08/25 06:40:01 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2023/08/25 06:40:01 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /data/packages
2023/08/25 06:40:01 routers/init.go:130:InitWebInstalled() [I] SQLite3 support is enabled
2023/08/25 06:40:01 routers/common/db.go:21:InitDBEngine() [I] Beginning ORM engine initialization.
2023/08/25 06:40:01 routers/common/db.go:28:InitDBEngine() [I] ORM engine initialization attempt #1/10...
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [I] PING DATABASE postgres
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table oauth2_application Column confidential_client db default is true, struct default is TRUE
2023/08/25 06:40:01 cmd/web.go:175:serveInstalled() [W] Table user Column max_repo_creation db default is '-1', struct default is -1
2023/08/25 06:40:01 routers/init.go:136:InitWebInstalled() [I] ORM engine initialization successful!
2023/08/25 06:40:01 ...er/issues/indexer.go:156:func2() [I] PID 7: Initializing Issue Indexer: db
2023/08/25 06:40:01 ...er/issues/indexer.go:246:func3() [I] Issue Indexer Initialization took 48.389µs
2023/08/25 06:40:01 ...xer/stats/indexer.go:38:populateRepoIndexer() [I] Populating the repo stats indexer with existing repositories
2023/08/25 06:40:01 routers/init.go:81:syncAppConfForGit() [I] AppPath changed from '' to '/usr/local/bin/gitea'
2023/08/25 06:40:01 routers/init.go:86:syncAppConfForGit() [I] CustomConf changed from '' to '/data/gitea/conf/app.ini'
2023/08/25 06:40:01 routers/init.go:92:syncAppConfForGit() [I] re-sync repository hooks ...
2023/08/25 06:40:01 routers/init.go:95:syncAppConfForGit() [I] re-write ssh public keys ...
2023/08/25 06:40:01 routers/init.go:69:mustInitCtx() [F] code.gitea.io/gitea/routers.syncAppConfForGit(ctx) failed: pq: cannot set transaction read-write mode during recovery
root@ubuntu:~# 

but UI still have 500 error

[wxiaoguang]

Are your Gitea connecting to a read-only replica instance?

https://stackoverflow.com/questions/24384407/cannot-set-transaction-read-write-mode-during-recovery-error

The answer is: Connect to the master server instead.

[phardelain-kouassi]

Check the logs of one of the posgres-db and restart the pod where you see the error message in the log. this help to solve the issue.
Did it and it works
zalando/postgres-operator#2354 (comment)

[willzhang]

Thanks, it work , i delete two standby pg to restart them, and login again, no 500 error

kubectl -n gitea delete pods gitea-postgresql-ha-postgresql-1  gitea-postgresql-ha-postgresql-2

[SchisandraX]

I also encountered the same problem, and my expectation was to deploy automatically through helm without human intervention

[GiteaBot]

We close issues that need feedback from the author if there were no new comments for a month. 🍵