Local DNS records are synced when they shouldn't be #80

MJVerhulst · 2022-11-10T10:00:44Z

What happened?

I'm syncing two piholes (two docker containers), but do not want them to sync local DNS records, because of the different networks they operate in.

However, the SYNC_LOCALDNSRECORDS is being ignored. I've tried using
SYNC_LOCALDNSRECORDS: 'false' and SYNC_LOCALDNSRECORDS: 0, to no avail.

This is my config:

  orbital-sync-2:
    image: mattwebbio/orbital-sync:1
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1440

Version

latest

Log output

10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Signing in to http://192.168.1.200/admin...
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ✔️ Successfully signed in to http://192.168.1.200/admin!
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Downloading backup from http://192.168.1.200/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Backup from http://192.168.1.200/admin completed!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Signing in to http://192.168.1.202/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Successfully signed in to http://192.168.1.202/admin!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Uploading backup to http://192.168.1.202/admin...
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Success: 1/1 hosts synced.
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: Waiting 1400 minutes...

The text was updated successfully, but these errors were encountered:

cbundy · 2022-11-10T23:38:46Z

Has this changed for you in a recent version?

If you run with v.1.3.0 do you have the same issue?

mattwebbio · 2022-11-11T00:14:06Z

In addition to @cbundy's comment, if you could add VERBOSE: true to your configuration and re-post the log output it would be super helpful :)

The server's response from the upload should get logged when verbose mode is enabled.

MJVerhulst · 2022-11-11T06:55:06Z

I copied and pasted from the homepage so I was apparently running version 1. Nevertheless, there seems to be no difference between versions. See below.

Just to be sure, I've been running two instances/containers of this image, since I want to sync my primary pi-hole completely to a second pihole, and then without to DNS-records to another pihole. Maybe things (like the backup.zip) are getting mixed up on the host file-system? I suppose this shouldn't happen, but just to make sure.
Complete docker-compose:

  orbital-sync:
    image: mattwebbio/orbital-sync:1.4.0
    container_name: pihole-sync
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.20'
      SECONDARY_HOST_1_PASSWORD: ''
      UPDATE_GRAVITY: 'false'
      INTERVAL_MINUTES: 1440
      TZ: 'Europe/Amsterdam'
  orbital-sync-2:
    image: mattwebbio/orbital-sync:1.3.0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output for 1.3.0

11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Backup uploaded to http://192.168.1.202!
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: Result:
11-11-2022 7:51:32
Processed adlist (31 entries)<br>
11-11-2022 7:51:32
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:51:32
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:51:32
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:51:32
Processed client (3 entries)<br>
11-11-2022 7:51:32
Processed client group assignments (5 entries)<br>
11-11-2022 7:51:32
Processed local DNS records (16 entries)<br>
11-11-2022 7:51:32
Processed local CNAME records (23 entries)<br>
11-11-2022 7:51:32
Processed domain_audit (0 entries)<br>
11-11-2022 7:51:32
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:51:32
Processed group (2 entries)<br>
11-11-2022 7:51:32
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:51:32
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:51:32
OK
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Success: 1/1 hosts synced.

Output for 1.4.0.

11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Result:
11-11-2022 7:45:34
Processed adlist (31 entries)<br>
11-11-2022 7:45:34
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:45:34
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:45:34
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:45:34
Processed client (3 entries)<br>
11-11-2022 7:45:34
Processed client group assignments (5 entries)<br>
11-11-2022 7:45:34
Processed local DNS records (16 entries)<br>
11-11-2022 7:45:34
Processed local CNAME records (23 entries)<br>
11-11-2022 7:45:34
Processed domain_audit (0 entries)<br>
11-11-2022 7:45:34
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:45:34
Processed group (2 entries)<br>
11-11-2022 7:45:34
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:45:34
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:45:34
OK
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Success: 1/1 hosts synced.
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Waiting 1400 minutes...

MJVerhulst · 2022-11-11T07:12:09Z

I've been experimenting a bit more, and it seems that also Environment variables like SYNC_LOCALCNAMERECORDS, SYNC_WHITELIST, are not getting picked up on. Which is interesting because UPDATE_GRAVITY: 'false', is getting handled accordingly.

No differences between versions 1.3.0 and 1.4.0.

mattwebbio · 2022-11-19T20:48:39Z

So sorry for the delay on this! I'm in the process of moving and so I've not had time to look into this further.

Orbital just tries to emulate the form inputs on the "restore" section of the teleporter - it's possible these have either changed or I just didn't set them up correctly in the first place.

My default troubleshooting step would be to manually run a backup and then try unchecking the things you don't want synced. If it works by hand, it's definitely an issue with Orbital's handling of the checkboxes and I can fix (or PRs welcome!)

cbundy · 2022-11-20T00:41:50Z

I'm not sure if specifying false for those form properties is intended to work in pihole, but what I can see is that when I untick options for restore, the entire property is gone.

All ticked

Untick localdns and localcname

MJVerhulst · 2022-11-20T18:30:54Z

Forgive me if this doesn't make any sense, I'm a bit guessing on how this should work. In my understanding, interpreting @cbundy response:

Pihole does not want a variable=false when the form is submitted, it expects the value instead of being 'true', to be gone from posted form.
Orbital-sync processes the variables in the generateForm() function:

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

    form.append('whitelist', Config.syncOptions.whitelist);
    form.append('regex_whitelist', Config.syncOptions.regexWhitelist);
    form.append('blacklist', Config.syncOptions.blacklist);
    form.append('regexlist', Config.syncOptions.regexlist);
    form.append('adlist', Config.syncOptions.adlist);
    form.append('client', Config.syncOptions.client);
    form.append('group', Config.syncOptions.group);
    form.append('auditlog', Config.syncOptions.auditlog);
    form.append('staticdhcpleases', Config.syncOptions.staticdhcpleases);
    form.append('localdnsrecords', Config.syncOptions.localdnsrecords);
    form.append('localcnamerecords', Config.syncOptions.localcnamerecords);
    form.append('flushtables', Config.syncOptions.flushtables);

It may be the case that pihole does not even check the contents of the localdnsrecords variable, but simple checks whether the variable is present.

So, to solve the issue, the generateForm() function should be modified as to only add the variables that are not disabled through the environment-variables passed on from docker-compose.

In pseudo-code (I only know PHP a bit) something like this?

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

 variables = array ('whitelist','regex_whitelist','blacklist','...');
 foreach (variables as $variable) {
    if(Config.syncOptions$variable!== 'false) {
          form.append('$variable', Config.syncOptions.$variable);
    }
 }

cbundy · 2022-11-23T13:09:25Z

Yep, I think you're spot on. I will give it a test this weekend and can put a PR together if you can't.

If you want to have a crack, there is a dev container so you can just start coding if you have docker installed, or you can code in the cloud as well without needing anything installed, just click this button above the code!

MJVerhulst · 2022-11-23T19:23:22Z

I actually managed to use this cloud coding and created a pull request. Not sure if I should have done that, as I'm not able to use the variable name for the config.Syncoptions, it creates an "Identifier expected" error. In PHP using variables as variable names is relatively easy, in JS I've read that adding brackets should yield the same effect, but no luck so far.

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);
    const variables = ['whitelist','regex_whitelist','blacklist','regexlist','adlist','client','group','auditlog','staticdhcpleases','localdnsrecords','localcnamerecords','flushtables'];
    let fLen = variables.length;

    for (let i = 0; i < fLen; i++) {
      var configVariable = variables[i];
      if(Config.syncOptions.[configVariable] != false) {
        form.append(variables[i],Config.syncOptions.[configVariable]);
      }
    }

    return form;
  }

See also 684386c

mattwebbio · 2022-11-25T16:28:43Z

Thanks so much @cbundy and @MJVerhulst for looking in to this while I've been away! I really, really appreciate it. I saw you opened a PR @MJVerhulst but it didn't look like I had write access to the branch to update the tests. I've pushed up a similar PR that omits the form fields (rather than setting them to false), and I've pushed up a prerelease image:

mattwebbio/orbital-sync:1.4.1-0

@MJVerhulst if you have the opportunity to test, that would be amazing!

MJVerhulst · 2022-11-25T18:39:22Z

Thanks @cbundy for your feedback on my JS "code" that you posted on the pull-request, very helpful :).

I did some testing and I think it's now working as expected :).

Thanks for the new version @mattwebbio!

Fiddled around a bit, output/config below. Apologies for the messed up/double time-stamps, dozzle does that appearently,

Check 1
Config

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'true'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'
      SYNC_WHITELIST: 'false'

Output

25-11-2022 19:32:06
11/25/2022, 7:32:06 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Result:
25-11-2022 19:32:30
Processed adlist (33 entries)<br>
25-11-2022 19:32:30
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:32:30
Processed blacklist (exact) (7 entries)<br>
25-11-2022 19:32:30
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:32:30
Processed client (26 entries)<br>
25-11-2022 19:32:30
Processed client group assignments (28 entries)<br>
25-11-2022 19:32:30
Processed local CNAME records (24 entries)<br>
25-11-2022 19:32:30
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:32:30
Processed group (2 entries)<br>
25-11-2022 19:32:30
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:32:30
OK
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Waiting 1400 minutes...

Check 2

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'false'
      SYNC_WHITELIST: 'false'
      SYNC_REGEXLIST: 'true'
      SYNC_BLACKLIST: 'false'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output

25-11-2022 19:38:46
11/25/2022, 7:38:46 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Result:
25-11-2022 19:38:56
Processed adlist (33 entries)<br>
25-11-2022 19:38:56
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:38:56
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:38:56
Processed client (26 entries)<br>
25-11-2022 19:38:56
Processed client group assignments (28 entries)<br>
25-11-2022 19:38:56
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:38:56
Processed group (2 entries)<br>
25-11-2022 19:38:56
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:38:56
OK
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Waiting 1400 minutes...

mattwebbio · 2022-11-30T02:42:36Z

Wonderful! Thanks so much, @MJVerhulst and @cbundy :)

I've released this change with v1.5

MJVerhulst assigned mattwebbio Nov 10, 2022

mattwebbio mentioned this issue Nov 25, 2022

Omit false form fields when creating backup #87

Merged

mattwebbio closed this as completed in #87 Nov 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local DNS records are synced when they shouldn't be #80

Local DNS records are synced when they shouldn't be #80

MJVerhulst commented Nov 10, 2022 •

edited

Loading

cbundy commented Nov 10, 2022

mattwebbio commented Nov 11, 2022 •

edited

Loading

MJVerhulst commented Nov 11, 2022

MJVerhulst commented Nov 11, 2022

mattwebbio commented Nov 19, 2022

cbundy commented Nov 20, 2022

MJVerhulst commented Nov 20, 2022

cbundy commented Nov 23, 2022 •

edited

Loading

MJVerhulst commented Nov 23, 2022 •

edited

Loading

mattwebbio commented Nov 25, 2022 •

edited

Loading

MJVerhulst commented Nov 25, 2022 •

edited

Loading

mattwebbio commented Nov 30, 2022

Local DNS records are synced when they shouldn't be #80

Local DNS records are synced when they shouldn't be #80

Comments

MJVerhulst commented Nov 10, 2022 • edited Loading

What happened?

Version

Log output

cbundy commented Nov 10, 2022

mattwebbio commented Nov 11, 2022 • edited Loading

MJVerhulst commented Nov 11, 2022

MJVerhulst commented Nov 11, 2022

mattwebbio commented Nov 19, 2022

cbundy commented Nov 20, 2022

MJVerhulst commented Nov 20, 2022

cbundy commented Nov 23, 2022 • edited Loading

MJVerhulst commented Nov 23, 2022 • edited Loading

mattwebbio commented Nov 25, 2022 • edited Loading

MJVerhulst commented Nov 25, 2022 • edited Loading

mattwebbio commented Nov 30, 2022

MJVerhulst commented Nov 10, 2022 •

edited

Loading

mattwebbio commented Nov 11, 2022 •

edited

Loading

cbundy commented Nov 23, 2022 •

edited

Loading

MJVerhulst commented Nov 23, 2022 •

edited

Loading

mattwebbio commented Nov 25, 2022 •

edited

Loading

MJVerhulst commented Nov 25, 2022 •

edited

Loading