Container Size on Unraid

[KyleJFellin]

Container size on Unraid is nearly 30GB. Any advice for offloading some of this to the array vs the docker image?

[AzrelixJ]

Container size on Unraid is nearly 30GB. Any advice for offloading some of this to the array vs the docker image?

I was wondering the same thing, hoping that most of this can be saved the app drive in the main array instead of the docker image.

[DrewThomasson]

no idea about the unraid someone out there made it out of the existing ebook2audiobook docker image

If I were to guess though Have you selected multiple fine-tuned models so far?

Cause they are saved to the models folder after you use each fine-tuned voice from the dropdown

So check the size of the models folder, I'm curious...

[janstadt]

@DrewThomasson I think we should map your volumes to local drives instead of storing it all in your container.

Something like this:

    volumes:
      - $/path/to/config/ebook2audio:/home/user/app

[DrewThomasson]

@janstadt
So I should set it to the folder the 'docker-compose.yml' is located in right?

like this?

volumes:
  - ./:/home/user/app

[DrewThomasson]

cause the app dir is just a dir containing the ebook2audiobook git main repo anyway

[janstadt]

Correct, the left hand of the volume is on your host and can be wherever you want. This will allow that content to be outside of the container. I typically use a config directory as a sibling of the docker compose file and then a subdirectory for each of my containers for keeping things clean. I thought about doing the models and voices folders specifically since those are the beefy directories, but i ended up just doing the entire app directory.

[DrewThomasson]

kk I'll add that to the existing docker-compose.yml

[DrewThomasson]

Just added to the docker-compose.yml👌
@janstadt

[DrewThomasson]

gona try a test on my end with it

hit me up if you have any issue with the new update

@janstadt

[DrewThomasson]

hm changed it to a optional comment you can "uncomment" in the docker-compose.yml because some people are running into permission issues

[janstadt]

Yeah typically when u mount volumes youll need to pass user and group ids in the env vars (or set them in the dockerfile). Im curious if you can also clean up the repo after whatever artifact/binary is built to save space as well.

[DrewThomasson]

Hm

Interesting...

I didn't know about any of this, still pretty new at docker tbh

[DrewThomasson]

@janstadt

Do you think you might be able to help me re-build the Dockerfile?

Cause right now it's been broken and I can't figure this out :/

[janstadt]

I'll take a quick look

[DrewThomasson]

@janstadt

Thx

Heres where my latest broken Dockerfileattempt is located

https://github.com/DrewThomasson/ebook2audiobook/blob/v2.0_Dockerfile/Dockerfile

but I would really like to be able to just have it launch the ebook2audiobook.sh

At the moment the current image is built by just pulling this hugginface as a docker image
https://huggingface.co/spaces/drewThomasson/ebook2audiobook_v2.0_Beta

which isn't ideal for development 😅

(edit: I was also looking to just straight up make the ebook2audiobook huggingface space based on the dockerfile instead to make it easier to update as well

[VlarpNL]

I'm running into the same issue and was already trying to work this out on my own. Happy to see people are already working on solving this :)

[jaycedk]

When trying to fix my docker image / container size issues, so it does not run out of space.
I converted my docker vDisk into a folder on my Nvme, where the vDisk also is.
This will not save space, you just don't have to run out of space in your vDisk.

In unRaid goto setttings -> docker -> stop docker edit from vDisk to library.

**Warring **

Doing this you will have to recreate all you dockers.
That is done, from the docker tab, at the button "Add container" and select your template.
It should keep all your settings, when done this way.

But I would like not, having to do this.
So a fix would be appreciated.

Happy New Years, to you all 🎆🎇

[DrewThomasson]

Got a fully working Dockerfile now so

That should make it easier to workout more of this stuff...

[DrewThomasson]

So does the new Volumes line you can activate in the docker-compose link it to the repo dir works for everyone here?

if so I'm closing out this issue 👍

[janstadt]

It looks like you moved from the ~/app directory to ~/ebook2audiobook which is totally fine. I think i figured out why the volumes werent being mounted properly though. Since you're entrypoint is being fed the CMD (python app.py) from your Dockerfile and the app pulls things down every time it starts the app, when i was mapping my volume to the root, it was expecting things to be there (like the app.py file itself), but since my host directory didnt have anything in it, it would fail. Essentially you're not supposed to map the entire working dir as a vol so i can map subdirectories and that works fine, but i still see something odd going on.

In the dockerfile you run pip install requirements.txt which includes unidic, but then in the app.py here: https://github.com/DrewThomasson/ebook2audiobook/blob/main/app.py#L61 it checks for unidic and it doesnt find it so it redownloads it which takes a while to get done (its over a gb) so i wonder why thats happening? Shouldnt pip install that via the dockerfile and requirements.txt? Is it because you're passing --no-cache-dir? I think thats correct since you want to keep your container as small as possible, but im not a python dev.

Finally, permissions are always a pain in docker and the dockerd will run as root on the host server and typically will write mapped volumes as root which causes all sorts of permissioning issues. i thought you could pass -u 1000:1000 to tell the dockerd to run as the current user, but it still creates the volumes incorrectly. If they exist beforehand and are owned by a non root user, they'll stay like that, but there are other dockerfiles that somehow manage to create the volumes as the user when passed the PGID and GUID but i think it has more to do with the fact that they map a known volume in the container where they typically store configs and data that needs to be preserved between restarts. the linuxserver dockerfiles are a good example of allowing a user to map a config directory where all the persistent stuff can be put (dbs, configs, outputs, etc) https://github.com/linuxserver/docker-swag/blob/master/Dockerfile#L188 and then allow a user to run the container under a known host user via environment variables like this: https://github.com/linuxserver/docker-swag/tree/master?tab=readme-ov-file#docker-compose-recommended-click-here-for-more-info

Foof, thats a long post.....

[citadella]

Can someone give me a final update on where this sits/what we can do to mitigate the issue? Because the unraid app (docker template) appears to be from the initial release on December 27th, which didn't work and I'm not really clear on how to get this setup correctly based on the above comments so I'm (hopefully) waiting for an update to that docker template...

[VlarpNL]

For now I've used the Unraid template (don't think that is absolutely necessary but it's a bit easier), but during the Add Container part I changed the Repository to athomasson2/ebook2audiobook:latest

I then added 3 Path mounts:
/home/user/app/tmp > /mnt/user/appdata/ebook2audiobook/app/tmp/
/home/user/app/audiobooks > /mnt/user/appdata/ebook2audiobook/app/audiobooks/
/home/user/app/models > /mnt/user/appdata/ebook2audiobook/app/models/

Make sure that you create these directories first before starting the container! I did that using Midnight Commander because I'm lazy and no Linux wizard.
Then after you did that use chown to change the owner of /mnt/user/appdata/ebook2audiobook/app and it's subdirectories so the permissions are correct: chown 1000:1000 -R /mnt/user/appdata/ebook2audiobook/app/ (do this from within Unraid)

After this you should be able to Apply the container settings and start the container.

This at least gets some of the more data heavy parts outside of the docker.img

edit: This is not the right way to solve this issue though. This is just a dirty workaround. I hope Drew and others are able to solve this in a nice way :).

[citadella]

Right, thanks, think this did it for me!

[citadella]

Well...workeed, but unRAID still crashed. No idea what for, though, but definitely related to this (the only thing I've added in months)

[janstadt]

You can spin down your array and edit your docker settings to expand your docker.img file. Seems like its still pulling down things inside the container when it starts that would be great to have that located elsewhere or mounted on the host.

[DrewThomasson]

It looks like you moved from the ~/app directory to ~/ebook2audiobook which is totally fine. I think i figured out why the volumes werent being mounted properly though. ....#162 (comment)

oh yeah forgot to post here in response to this...

1. The default file structure In the compose and the Dockerfile in the repo now both reference ~/app and not ~/ebook2audiobook so that should fix that part

2. I also added comments in the Dockerfile that will make the Dockerfile run through a test.txt file so then the unidict and base model will be baked into the image you make form it, if un-commented that is :)

[citadella]

I completely removed the image, created the directories, and re-added it with the custom mappings and confirmed files were going into those respective directories. Then I locked down the CPUs to only 3/8. But still, after about 3-4h, my unRAID completely locked up. Unfortunately I don't run an outside syslog server so I don't know why-I could still see port 22 open, but couldn't even connect to that to get in to see so had to just reboot it.