Backup fails against Nextcloud WebDAV backend with Caddy proxy: HTTP 411

[andscape-dev]

Context

I've been trying with no success to integrate Seedvault with my Nextcloud instance, but both integration options (Nextcloud client and WebDAV) have been basically unusable since I started trying on the previous Seedvault version.

The Nextcloud client integration is ridiculously slow (seemingly backing up and uploading each chunk of each APK individually and synchronously) and suffers from the native Nextcloud client's upload function's fragility: uploads fail without retry at the first sign of network instability and that in turn fails the entire backup.

The WebDAV integration was entirely broken for me in the previous version: it would just get stuck on "Setting up the backend" eventually showing an error. I was excited for the new version 15 to hopefully fix this. Indeed, setting up the backend now works smoothly, but backups still fail when it gets to the uploading step.

For context: I'm using an app password for Nextcloud WebDAV as detailed here. My Seedvault WebDAV configs are:

Name	Value
WebDAV URL	https://nextcloud.example.com/remote.php/dav/files/myuser
Username	myuser
Password	myapppassword

Relevant Logs

01-18 19:05:37.701  5035 12259 I KVBackup: Finish K/V Backup of @pm@
01-18 19:05:37.725  5035 12259 D MemoryLogger: 79 MiB free - 52 of 131 (max 256)
01-18 19:05:37.951  5035 12259 D o.c.s.c.b.w.WebDavBackend: [binder:5035_6       ] https://nextcloud.example.com/remote.php/dav/files/myuser/.SeedVaultAndroidBackup/ae6530b8bd128d630ec9e707737778fc2c1e357230fc25471f034e00c20d2e16/6e not found, creating...
01-18 19:05:38.257  5035 12259 D o.c.s.c.b.w.WebDavBackend: [binder:5035_6       ] mkColCreateMissing(https://nextcloud.example.com/remote.php/dav/files/myuser/.SeedVaultAndroidBackup/ae6530b8bd128d630ec9e707737778fc2c1e357230fc25471f034e00c20d2e16/6e) = Response{protocol=h2, code=201, message=, url=https://nextcloud.example.comremote.php/dav/files/myuser/.SeedVaultAndroidBackup/ae6530b8bd128d630ec9e707737778fc2c1e357230fc25471f034e00c20d2e16/6e/}
01-18 19:05:38.386  5035 12259 E BackupCoordinator: Error finishing K/V backup for @pm@
01-18 19:05:38.386  5035 12259 E BackupCoordinator: java.io.IOException: at.bitfire.dav4jvm.exception.HttpException: HTTP 411 
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at org.calyxos.seedvault.core.backends.webdav.PipedCloseActionOutputStream.close(PipedCloseActionOutputStream.kt:51)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlin.io.CloseableKt.closeFinally(Closeable.kt:56)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.repo.BlobCreator.createNewBlob(BlobCreator.kt:72)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.repo.BackupReceiver.onNewChunk(BackupReceiver.kt:119)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.repo.BackupReceiver.finalize(BackupReceiver.kt:104)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.repo.BackupReceiver.readFromStream(BackupReceiver.kt:87)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.transport.backup.KVBackup.finishBackup(KVBackup.kt:165)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.transport.backup.BackupCoordinator.finishBackup(BackupCoordinator.kt:322)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.transport.ConfigurableBackupTransport$finishBackup$1.invokeSuspend(ConfigurableBackupTransport.kt:139)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:280)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source:1)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.stevesoltys.seedvault.transport.ConfigurableBackupTransport.finishBackup(ConfigurableBackupTransport.kt:138)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at android.app.backup.BackupTransport$TransportImpl.finishBackup(BackupTransport.java:790)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at com.android.internal.backup.IBackupTransport$Stub.onTransact(IBackupTransport.java:682)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at android.os.Binder.execTransactInternal(Binder.java:1396)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at android.os.Binder.execTransact(Binder.java:1335)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: Caused by: at.bitfire.dav4jvm.exception.HttpException: HTTP 411 
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at at.bitfire.dav4jvm.DavResource.checkStatus(DavResource.kt:650)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at at.bitfire.dav4jvm.DavResource.checkStatus(DavResource.kt:624)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at at.bitfire.dav4jvm.DavResource.put(DavResource.kt:467)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at at.bitfire.dav4jvm.DavResource.put$default(DavResource.kt:448)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at org.calyxos.seedvault.core.backends.webdav.WebDavBackend$save$deferred$1.invokeSuspend(WebDavBackend.kt:144)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
01-18 19:05:38.386  5035 12259 E BackupCoordinator: 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
01-18 19:05:38.390  5035 12259 I NotificationBackupObserver: Completed. Target: @pm@, status: -1002
01-18 19:05:38.390  5035 12259 I NotificationBackupObserver: Showing progress notification for org.lineageos.recorder 77/338
01-18 19:05:38.390  5035 12259 I NotificationBackupObserver: 77/338 - System package manager (@pm@)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: Error during PM metadata backup
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: com.android.server.backup.keyvalue.AgentException
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at com.android.server.backup.keyvalue.KeyValueBackupTask.handleTransportStatus(KeyValueBackupTask.java:1131)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at com.android.server.backup.keyvalue.KeyValueBackupTask.sendDataToTransport(KeyValueBackupTask.java:1074)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at com.android.server.backup.keyvalue.KeyValueBackupTask.sendDataToTransport(KeyValueBackupTask.java:578)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at com.android.server.backup.keyvalue.KeyValueBackupTask.backupPm(KeyValueBackupTask.java:683)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at com.android.server.backup.keyvalue.KeyValueBackupTask.run(KeyValueBackupTask.java:369)
01-18 19:05:38.390  1599 14654 E KeyValueBackupTask: 	at java.lang.Thread.run(Thread.java:1117)
01-18 19:05:38.412  1599 14654 V BackupManagerService: [UserID:0] Ancestral packages:  200
01-18 19:05:38.514  5035 12259 I BackupCoordinator: Request incremental backup time. Returned 0
01-18 19:05:38.515  1599 14654 I KeyValueBackupTask: K/V backup pass finished

Environment

Name	Value
Device	OnePlus 7T (hotdogb)
OS	LineageOS 22
Seedvault version	15-5.2
Nextcloud version	Nextcloud Hub 9 (30.0.5)

[grote]

The Nextcloud client integration is ridiculously slow

Yeah please don't use that.

HttpException: HTTP 411

HTTP 411 is Length Required. Could it be that your server doesn't support chunked transfer encoding?
See #806
Also https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding#chunked and https://datatracker.ietf.org/doc/html/rfc9112#section-7.1-4

[andscape-dev]

The Nextcloud client integration is ridiculously slow

Yeah please don't use that.

Oh I didn't know it was explicitly not recommended, might wanna put a warning about that somewhere...

HttpException: HTTP 411

HTTP 411 is Length Required. Could it be that your server doesn't support chunked transfer encoding?

I'm running stock Nextcloud. Based on the info here it seems like it uses a separate endpoint from the usual DAV one, specifically for chunked uploads: (<server>/remote.php/dav/uploads/<userid> instead of <server>/remote.php/dav/files/<userid>). I tried using that endpoint instead of the standard one for the seedvault config, but the backup failed immediately (probably because other DAV requests should still go to /files/). I don't know much about WebDAV, but I'm guessing this is NC using a non-standard implementation? Does WebDAV specify anything about chunked uploads?

[grote]

I think the issue is not Nextloud itself, because I ran tests against various Nextcloud instances. The issue is most likely with the HTTP server you run Nextcloud on.

[andscape-dev]

I think the issue is not Nextloud itself, because I ran tests against various Nextcloud instances. The issue is most likely with the HTTP server you run Nextcloud on.

You're probably right, I hadn't considered the reverse proxy. I'm using Caddy (latest) and it seems to have a bunch of problems handling chunked encoding and content lengths. This issue seems like the most relevant in this case. I tried one of the suggested fixes (setting request_buffers), the backup still fails but now I can't find any useful exception in the logs.

[andscape-dev]

Update: actually turns out after applying the Caddy request_buffers fix I was running into this issue. Excluding my VPN app finally got my backups to finish correctly.

I did some manual testing trying out different buffer sizes for the request proxying and it seems the minimum needed for the backups not to hang is in the order of 10MB. Do you have any suggestions for that? Is there a maximum size for the chunks sent by Seedvault, or the files themselves?

[grote]

Great, you made some progress. The maximum file size tries to write should be 15MB maybe a few bytes more with overhead.

[andscape-dev]

Perfect, thanks! I'll set the buffer size to just above that. I think this issue can be closed now, or turned to a discussion in case others run into the same issue.