Syncs are failing with the latest pihole update

[mike12806]

What happened?

My pihole docker containers auto-upgraded to 2023.3.0 overnight and I noticed this morning that orbital syncs were failing to apply to other instances. It seems like the backup was fine, but the replication to other pihole instances was failing.

I reverted pihole to 2023.2.0 and it started working again.

Version

1.5.1-0

Log output

3/23/2023, 11:16:32 AM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 11:16:32 AM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 11:16:32 AM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 11:16:32 AM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 11:16:32 AM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 11:16:32 AM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 11:16:37 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".
3/23/2023, 11:16:40 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 11:16:40 AM: ⚠ Failure: 0/2 hosts synced.
3/23/2023, 11:16:42 AM: Waiting 15 minutes...
3/23/2023, 11:31:42 AM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 11:31:42 AM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 11:31:42 AM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 11:31:42 AM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 11:31:42 AM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 11:31:42 AM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 11:31:42 AM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 11:31:42 AM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 11:31:42 AM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 11:31:42 AM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 11:31:47 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".
3/23/2023, 11:31:49 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 11:31:49 AM: ⚠ Failure: 0/2 hosts synced.
3/23/2023, 11:31:50 AM: Waiting 15 minutes...
3/23/2023, 11:46:50 AM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 11:46:50 AM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 11:46:50 AM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 11:46:50 AM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 11:46:50 AM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 11:46:50 AM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 11:46:50 AM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 11:46:50 AM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 11:46:50 AM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 11:46:50 AM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 11:46:56 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".
3/23/2023, 11:46:57 AM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 11:46:57 AM: ⚠ Failure: 0/2 hosts synced.
3/23/2023, 11:46:58 AM: Waiting 15 minutes...
3/23/2023, 12:01:58 PM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 12:01:58 PM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 12:01:58 PM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 12:01:58 PM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 12:01:58 PM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 12:01:58 PM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 12:01:58 PM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 12:01:58 PM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 12:01:58 PM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 12:01:58 PM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 12:02:04 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".
3/23/2023, 12:02:05 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 12:02:05 PM: ⚠ Failure: 0/2 hosts synced.
3/23/2023, 12:02:06 PM: Waiting 15 minutes...
3/23/2023, 12:17:06 PM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 12:17:06 PM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 12:17:06 PM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 12:17:06 PM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 12:17:06 PM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 12:17:06 PM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 12:17:06 PM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 12:17:06 PM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 12:17:06 PM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 12:17:06 PM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 12:17:12 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".
3/23/2023, 12:17:13 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 12:17:13 PM: ⚠ Failure: 0/2 hosts synced.
3/23/2023, 12:17:14 PM: Waiting 15 minutes...
3/23/2023, 12:32:15 PM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 12:32:15 PM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 12:32:15 PM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 12:32:15 PM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 12:32:15 PM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 12:32:15 PM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 12:32:15 PM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 12:32:15 PM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 12:32:15 PM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 12:32:15 PM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 12:32:20 PM: ✔️ Backup uploaded to http://192.168.1.112:90/admin!
3/23/2023, 12:32:20 PM: ➡️ Updating gravity on http://192.168.1.112:90/admin...
3/23/2023, 12:32:23 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.86:90/admin".
3/23/2023, 12:32:27 PM: ✔️ Gravity updated on http://192.168.1.112:90/admin!
3/23/2023, 12:32:27 PM: ⚠ Failure: 1/2 hosts synced.
3/23/2023, 12:32:29 PM: Waiting 15 minutes...
3/23/2023, 12:47:29 PM: ➡️ Signing in to http://192.168.1.166/admin...
3/23/2023, 12:47:29 PM: ✔️ Successfully signed in to http://192.168.1.166/admin!
3/23/2023, 12:47:29 PM: ➡️ Downloading backup from http://192.168.1.166/admin...
3/23/2023, 12:47:30 PM: ✔️ Backup from http://192.168.1.166/admin completed!
3/23/2023, 12:47:30 PM: ➡️ Signing in to http://192.168.1.112:90/admin...
3/23/2023, 12:47:30 PM: ➡️ Signing in to http://192.168.1.86:90/admin...
3/23/2023, 12:47:30 PM: ✔️ Successfully signed in to http://192.168.1.112:90/admin!
3/23/2023, 12:47:30 PM: ➡️ Uploading backup to http://192.168.1.112:90/admin...
3/23/2023, 12:47:30 PM: ✔️ Successfully signed in to http://192.168.1.86:90/admin!
3/23/2023, 12:47:30 PM: ➡️ Uploading backup to http://192.168.1.86:90/admin...
3/23/2023, 12:47:35 PM: ✔️ Backup uploaded to http://192.168.1.112:90/admin!
3/23/2023, 12:47:35 PM: ➡️ Updating gravity on http://192.168.1.112:90/admin...
3/23/2023, 12:47:37 PM: ✔️ Backup uploaded to http://192.168.1.86:90/admin!
3/23/2023, 12:47:37 PM: ➡️ Updating gravity on http://192.168.1.86:90/admin...
3/23/2023, 12:47:42 PM: ✔️ Gravity updated on http://192.168.1.112:90/admin!
3/23/2023, 12:47:44 PM: ✔️ Gravity updated on http://192.168.1.86:90/admin!
3/23/2023, 12:47:44 PM: ✔️ Success: 2/2 hosts synced.
3/23/2023, 12:47:44 PM: Waiting 15 minutes...

[mattwebbio]

Yeah, mine too. If someone else reading this figures out what's changed: please let me know! Otherwise I'll investigate as soon as I have time

[mike12806]

@mattwebbio i'll try to turn on verbose logging and see if I can get any more deets from orbitalsync

[mike12806]

3/23/2023, 3:22:55 PM: {"host":"http://192.168.1.112:90","path":"/admin","status":200,"responseBody":"Start importing...<br>\nProcessed adlist (5 entries)<br>\nProcessed adlist group assignments (5 entries)<br>\nProcessed blacklist (exact) (0 entries)<br>\nProcessed blacklist (regex) (0 entries)<br>\nProcessed client (0 entries)<br>\nProcessed client group assignments (0 entries)<br>\nProcessed local DNS records (20 entries)<br>\nProcessed local CNAME records (0 entries)<br>\nProcessed black-/whitelist group assignments (217 entries)<br>\nProcessed group (1 entry)<br>\nProcessed whitelist (exact) (206 entries)<br>\nProcessed whitelist (regex) (11 entries)<br>\nDone importing"}
3/23/2023, 3:22:55 PM: ⚠ Error: Failed to upload backup to "http://192.168.1.112:90/admin".

[mike12806]

@mattwebbio - weird - either error'ing silently or it's working but failing to respond, properly? Not really sure

[mattwebbio]

Looks like they updated the message that a successful 'restore' completes with. Currently, Orbital is expecting it to end with 'OK':

orbital-sync/src/client.ts

Line 125 in cd2869e

if (uploadResponse.status !== 200 || !uploadText.endsWith('OK'))

[mattwebbio]

Fixed with version v1.5.2! Let me know if you continue to see this :)

[mike12806]

Thx for the quick fix!