Cannot get this to work/run in my Docker setup

[reddwarf666]

Hi

Trying to setup icloudpd on docker at home and I fail, horribly.
I tried to follow the guidelines but I must have missed some crucial part of instructions or I'm an idiot for not seeing the obvious.
I understand the env variables moved from compose file to that new conf file icloudpd.conf but it is unclear if this needs to be prepared up front or is created by icloudpd itself.

The user running all commands is "steven" with ID 1000 and I make sure all folders and files have this set as the user with rw attributes.
I run several containers in this Docker setup so I know at it's core my Docker setup seems to be fine.
I created the docker compose file as per the example given and I get these issues:

• No configuration files are created. I created the folders icloudpd_steven_config and icloudpd_esther_config under the iCloud folder. I restarted the containers, no config files are to be seen.
  I expected the file icloudpd.conf to be present in both users folders but perhaps that is part of my misunderstanding?

• Looking at the logs after startup I see:

2025-01-29 17:27:48 INFO     Initialising container...
2025-01-29 17:27:48 INFO      - Creating temporary directory
2025-01-29 17:27:53 INFO      - Checking configuration file permissions
2025-01-29 17:27:53 INFO      - Create/update configuration file: /config/icloudpd.conf
2025-01-29 17:27:54 INFO      - Checking /config/icloudpd.conf for errors
2025-01-29 17:27:54 ERROR       | Apple ID not set
2025-01-29 17:27:54 ERROR       ! Waiting for it to be added to: /config/icloudpd.conf

• When I try to start to initialize with docker exec -it icloudpd_steven sync-icloud.sh --Initialise I get:

2025-01-29 17:43:00 INFO     ***** boredazfcuk/icloudpd container v1.0.1179_25-01-2025 started *****
2025-01-29 17:43:00 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2025-01-29 17:43:00 INFO     Alpine Linux 3.21.2
2025-01-29 17:43:00 INFO     Python version: 3.12.8
2025-01-29 17:43:00 INFO     icloud-photos-downloader version: 
2025-01-29 17:43:00 INFO     Loading configuration from: /config/icloudpd.conf
2025-01-29 17:43:00 INFO     Local user: steven_xxx:1000
2025-01-29 17:43:00 INFO     Local group: group:1000
2025-01-29 17:43:00 INFO     Debug logging: Disabled
2025-01-29 17:43:00 INFO     Apple ID: ******@******.
2025-01-29 17:43:00 INFO     Cookie path: /config/************
2025-01-29 17:43:00 INFO     Cookie expiry notification period: 7
2025-01-29 17:43:00 INFO     Download destination directory: /home/user/iCloud
2025-01-29 17:43:00 INFO     Folder structure: {:%Y/%m/%d}
2025-01-29 17:43:00 INFO     Keep Unicode: false
2025-01-29 17:43:00 INFO     Live Photo MOV Filename Policy: suffix
2025-01-29 17:43:01 INFO     File Match Policy: name-size-dedup-with-suffix
2025-01-29 17:43:01 INFO     Synchronisation interval: 86400
2025-01-29 17:43:01 INFO     Synchronisation delay (minutes): 0
2025-01-29 17:43:01 INFO     Set EXIF date/time: false
2025-01-29 17:43:01 INFO     Auto delete: false
2025-01-29 17:43:01 INFO     Delete after download: false
2025-01-29 17:43:01 INFO     Delete empty directories: false
2025-01-29 17:43:01 INFO     Photo size: original
2025-01-29 17:43:01 INFO     Align RAW: as-is
2025-01-29 17:43:01 INFO     Single pass mode: false
2025-01-29 17:43:01 INFO     Skip download check: false
2025-01-29 17:43:01 INFO     Skip live photos: false
2025-01-29 17:43:01 INFO     Number of most recently added photos to download: Download All Photos
2025-01-29 17:43:01 INFO     Downloading photos from: Download All Photos
2025-01-29 17:43:01 INFO     Stop downloading when prexisiting files count is: Download All Photos
2025-01-29 17:43:01 INFO     Live photo size: original
2025-01-29 17:43:01 INFO     Skip videos: false
2025-01-29 17:43:01 INFO     Convert HEIC to JPEG: false
2025-01-29 17:43:01 INFO     Downloading from: icloud.com
2025-01-29 17:43:01 INFO     Authentication domain: com
2025-01-29 17:43:01 INFO     Script launch parameters: --Initialise
su: unknown user steven
2025-01-29 17:43:01 ERROR    Keyring file does not exist. Please try again

• I create the "iCloud" folder in my user folder /home/steven/iCloud and created the file .mounted in it, did not seem to help

My compose file loks like this:

networks:
   icloudpd:
      name: icloudpd
      driver: bridge
      ipam:
         driver: default
      driver_opts:
         com.docker.network.bridge.name: icloudpd

volumes:
   icloudpd_steven_config:
      name: icloudpd_steven_config
   icloudpd_esther_config:
      name: icloudpd_esther_config

services:

   icloudpd_steven:
      hostname: icloudpd_steven
      container_name: icloudpd_steven
      image: boredazfcuk/icloudpd
      restart: unless-stopped
      networks:
         icloudpd:
            aliases:
               - icloudpd_steven
      environment:
         - TZ=Europe/Amsterdam
         - user=steven
      healthcheck:
         test: /usr/local/bin/healthcheck.sh
         start_period: 30s
      volumes:
         - icloudpd_steven_config:/config
         - ./iCloud/:/home/steven/iCloud/

   icloudpd_esther:
      hostname: icloudpd_esther
      container_name: icloudpd_esther
      image: boredazfcuk/icloudpd
      restart: unless-stopped
      networks:
         icloudpd:
            aliases:
               - icloudpd_esther
      environment:
         - TZ=Europe/Amsterdam
         - user=esther
      healthcheck:
         test: /usr/local/bin/healthcheck.sh
         start_period: 30s
      volumes:
         - icloudpd_esther_config:/config
         - ./iCloud/:/home/esther/iCloud/

Any ideas what could be wrong? Again, I must have missed something simple because it should not be this problematic I think. Many people run this so I do seem to be missing some info or insight here.

[boredazfcuk]

If your version of docker puts it's volumes in the default location, your config files will be in /var/lib/docker/volumes/icloudpd__config/_data/ on the host operating system.

Alternatively, you can run docker exec -it icloudpd_<username> ash to get to the terminal prompt for each user. Then run ```
editconfig

Awesome username matey

[reddwarf666]

Thanks! I see that folder structure:

/var/lib/docker/volumes/icloudpd_steven_config/_data/icloudpd.conf  
/var/lib/docker/volumes/icloudpd_esther_config/_data/icloudpd.conf  

Aaannndd I see what I did wrong, doh! My volume mapping in compose is not how it should have been.

This:

         - icloudpd_steven_config:/config

Should have been this:

         - ./data/config/icloudpd_steven_config:/config

As that maintains my folder structure as I have for all other containers.

Now this line in the volumes section of the compose has me confused now that I started to cast my lazy eyes on it:

 - ./iCloud/:/home/esther/iCloud/

I think that should be:

 - /home/steven/iCloud/:./iCloud/

What seems to be wrong there is that local relative folder ./iCloud is mapped to container folder /home/steven/iCloud/, this should be the other way around as far as I know as that folder /home/steven/iCloud/ does not exist? And I also think that in the container there is a folder mapped /iCloud and not relative ./iCloud, am I correct in that understanding?

Going to try it out now!

Awesome username matey

Thanks! 😁

[reddwarf666]

Ok, all is running now as I would expect it, at least regarding logs and folders created. I did have to create a file called .mounted in the container itself in folder /home/user/iCloud. Downloads started to happen but I see nothing in the docker/host machine folder I mapped in volumes.

icloudpd.conf content which might be relevant:

download_path=/home/user/iCloud
group=group
group_id=1000
user=user
user_id=1000

compose on volumes as I now have:

volumes:
   icloudpd_steven_config:
      name: icloudpd_steven_config

      volumes:
         - ./data/config/icloudpd_steven_config:/config
         - /home/steven/iCloud:/iCloud

So the questions I now have are "where are these downloaded files?" and "why did I have to go into the container and create a file before the downloads started to happen?"

[boredazfcuk]

This is a pretty much how I define my containers in my docker-compose file (I've a removed a few bits for clarity and changed names to protect the guilty).

networks:
   icloudpd_network:
      name: icloudpd_network
      external: true
      driver: bridge
      ipam:
         driver: default
         config:
            - subnet: 192.168.144.0/24
      driver_opts:
         com.docker.network.bridge.name: icloudpd_network0

volumes:
   icloudpd_ipad_config:
      name: icloudpd_ipad_config
   icloudpd_user1_config:
      name: icloudpd_user1_config
   icloudpd_user2_config:
      name: icloudpd_user2_config
   icloudpd_user3_config:
      name: icloudpd_user3_config
   icloudpd_user4_config:
      name: icloudpd_user4_config

services:
   icloudpd_ipad:
      hostname: icloudpd_ipad
      container_name: icloudpd_ipad
      networks:
         - icloudpd_network
      environment:
         - TZ=Europe/London
      image: boredazfcuk/icloudpd
      restart: always
      volumes:
         - icloudpd_ipad_config:/config/
         - /home/ipad/iCloud/:/home/ipad/iCloud/
   icloudpd_user1:
      hostname: icloudpd_user1
      container_name: icloudpd_user1
      networks:
         - icloudpd_network
      environment:
         - TZ=Europe/London
      image: boredazfcuk/icloudpd
      restart: always
      volumes:
         - icloudpd_user1_config:/config/
         - /home/user1/iCloud/:/home/user1/iCloud/
   icloudpd_user2:
      hostname: icloudpd_user2
      container_name: icloudpd_user2
      networks:
         - icloudpd_network
      environment:
         - TZ=Europe/London
      image: boredazfcuk/icloudpd
      restart: always
      volumes:
         - icloudpd_user2_config:/config/
         - /home/user2/iCloud/:/home/user2/iCloud/
   icloudpd_user3:
      hostname: icloudpd_user3
      container_name: icloudpd_user3
      networks:
         - icloudpd_network
      environment:
         - TZ=Europe/London
      image: boredazfcuk/icloudpd
      restart: always
      volumes:
         - icloudpd_user3_config:/config/
         - /home/user3/iCloud/:/home/user3/iCloud/
   icloudpd_user4:
      hostname: icloudpd_user4
      container_name: icloudpd_user4
      networks:
         - icloudpd_network
      environment:
         - TZ=Europe/London
      image: boredazfcuk/icloudpd
      restart: always
      volumes:
         - icloudpd_user4_config:/config/
         - /home/user4/iCloud/:/home/user4/iCloud/

All the config directories are mapped to docker volumes, but the data directories are absolute paths. That way I can move this compose file anywhere on my system and it will still map the directories where I want them to be (I'm not a fan of using relative directories starting ./)

[boredazfcuk]

To elaborate... the volume mappings are "directory on the host server:directory in the container" so:

/home/user4/iCloud/:/home/user4/iCloud/

maps the /home/user4/iCloud/ directory on the host computer to /home/user4/iCloud inside the container (in this instance, you want also want your download_path variable set to /home/user4/iCloud

Some people like to have the download path inside the container set to /downloads/ or maybe /icloud/. In that case your compose file volume mapping would be:

/home/user4/iCloud/:/downloads/

and you'd have you download_path=/downloads/ (or maybe without the trailing slash on the variable, I don't remember it if handles that or not lol)

[boredazfcuk]

and "why did I have to go into the container and create a file before the downloads started to happen?"

You should not be creating that from inside the container. This must be created from outside the container. It's how the container knows the volume is correctly mounted.

It's to avoid this scenario...
In your docker-compose file you set:

      volumes:
         - icloudpd_user1_config:/config/
         - /home/user1/iCloud/:/home/user1/iCloud/

You set user=user2 and download_path=/home/user2/iCloud in your icloudpd.conf file.
From inside the container you create /home/user2/iCloud/.mounted
You start the container, it finds /home/user2/iCloud/.mounted so believes the volume is mounted correctly.
You leave it overnight, it downloads 50GB of photos from your iCloud.
You upgrade the container, deleting the original.
50GB of photos are gone.

Also, if you create it in the correct location inside the container, but with an incorrect volume mapping in your docker compose file, then correct the docker compose file, the .mounted file will seem to have disappeared (it's actually underneath the volume mapping).

Then say your mapped volume dismounts, because there is a disk fault. The original .mounted file will now be exposed, but the container will not find any downloaded photos, so will go and download 50GB of photos again... this could potentially fill your server's boot directory and crash the whole system.

[reddwarf666]

Got it and thanks for the explanation. I kinda knew this (the folder part that is) but it is good to be reminded.

I kept the config to what I pasted above but I had to adjust compose config like so:

- /home/steven/iCloud:/home/user/iCloud

This way the container keeps using the user user and I map that per container to the correct user/folder on my docker machine drive.

Files are downloading as we type!

I was wondering, is there a reset you can do, to start the downloading all over? Where does it register what was downloaded or does it do a compare of the files? Curious minds...
Anyway, I could always destroy the container and start again but the local folder with all the pictures/files is already there, of course.

Are there functions to control icloudpd, something like that sync-icloud.sh ?

[boredazfcuk]

Yeah, I have user=ipad in my icloudpd.conf so that I can map to /home/ipad/iCloud/:/home/ipad/iCloud and also so I can differentiate between users inside the containers. Plus I match the ipad uid and gid inside the container to the host user/gid too. just to make sure permissions are all ok. The "user" inside the container is just cosmetic though. As long as the user_id and group_id match the user's uid and gid on the host it should all be fine.

I was wondering, is there a reset you can do, to start the downloading all over?
If you want to re-download the library, just map your dockerfile to a new location and create the .mounted file there. That will download your whole library again. Or just delete everything from /home/steven/iCloud/* and restart the container.

Where does it register what was downloaded or does it do a compare of the files?

There's no database, it just looks at what's in icloud and then copies it to the local filesystem. When I first started using it, I had folder structure set to 'none' as I liked them all in a flat file structure so it matched the photo library on my phone. After a while I realised that there's a limit of 10k pictures per directory, so setting it to 'none' limited me to 10k photos in total. Setting it to 'Year' allowed 10k photos per year... so now I set it to Year/Month so I can have 10k photos per month. Having Year/Month/Day allows 10k photos per day lol

Are there functions to control icloudpd, something like that sync-icloud.sh?

The sync-icloud.sh script does all the heavy lifting. It supports a few command line options to upload your library to a nextcloud server, also for HEIC to JPEG conversion.

Telegram integration is the only bit of it that I use frequently. I have a chat bot set up in Telegram and that's how I get the notifications about what has been sync'd/deleted, but I can also message that bot with "ipad" and the ipad container will spot that message and trigger a download run. Messaging it with "user1" and the "user1" container spots it and triggers a download run.

I can also message with "user2 auth" and it will start the MFA re-authentication process. It will then message back with the MFA options... Apple MFA code, SMS to main number, or SMS to trusted number and I message it back with the option I want. Once I get the MFA code on my phone, I'll send "user 123654" or whatever, and the container will pick that up and use it to authenticate. Saves me having to faff about with the laptop to re-confirm MFA. I can remotely control all 5 (soon to be 6) containers this way.

[reddwarf666]

Thanks for the extensive explanation! And good luck managing even more containers in the future 😄

So I guess restarting the container will trigger a sync and download action.

And a big Thanks for creating this valuable tool!

[boredazfcuk]

So I guess restarting the container will trigger a sync and download action.

Yeah, or it will download as per the configured sync interval... plus the manual syncs which can be forced via Telegram.

If you want the container to be switched off all the time, you can set single_pass=true and the container will exit after it is run. If you have restart: no in your docker-compose file it will not come back on.

That way you can use the host's cron jobs to launch the container as per your desired schedule.

[reddwarf666]

One more question (starting to sound like Columbo...)

I do not use telegram but I am interested in automating the MFA part of it all. I do use Signal and Whatsapp though.

Say I am able to create a REST API endpoint that I could call remotely. And you can assume that the endpoint is able to connect to the icloudpd container and start a script of some sort. What would I need to do to get the MFA generated on the container?
I see "reauth.sh" and "authenticate.exp" are used but in the end it is "icloudpd" that is doing the work it seems?

Or if I can create that authentication somewhere else and copy the cookie/etc to the icloudpd container, would that work?

Any insight you have or can point me to will help!

[boredazfcuk]

The .exp file is where all the magic happens. It's an Expect script which interacts with the icloudpd download utility. The icoudpd utility is what performs the MFA authentication, so that is what you need to interact with. Basically the expect script scrapes the output of icloudpd and looks for text patterns. It will then do different things based on those patterns, such as send the message listing the mobile numbers available, or to let the user know that the process failed/was successful.

You'll need to write something similar, either using expect, or something else which can do the same...

I can't comment on whether cookie copying will work or not. I do believe that I've seen someone say that they've had success with this before, but I've not attempted it myself, so cannot confirm.

[boredazfcuk]

One more question (starting to sound like Columbo...)

I'd not heard his name in a decade, but now it's twice in one week... We were talking about him at work, but then had to explain to the new kids who he was coz they are too young to have a clue!

[reddwarf666]

Thanks for the explanation!

And yeah, I have to explain Columbo to people as well. Brilliant show.