Media Uploads fail with "Error uploading file: no such file or directory" (Docker)

[domsson]

Version:

• listmonk: v2.3.0
• docker.io: v20.10.5
• OS: Debian

Description of the bug and steps to reproduce:

Uploading media files fails with:

Error uploading file: open <path>: no such file or directory

• Installed Docker via apt install docker.io
• Installed Listmonk via the docker specific installation instructions
• Listmonk seems to run fine, but trying to upload files gives above error

This is with everything left at the default after install, hence in docker-compose.yml we can see (reduced to relevant):

app:
  volumes:
    - ./config/toml:/listmonk/config.toml

volumes:
  listmonk-data

Also, docker volume inspect listmonk_listmonk-data shows:

[
    {
        "CreatedAt": "2023-02-02T11:38:14+09:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "listmonk",
            "com.docker.compose.version": "1.25.0",
            "com.docker.compose.volume": "listmonk-data"
        },
        "Mountpoint": "/var/lib/docker/volumes/listmonk_listmonk-data/_data",
        "Name": "listmonk_listmonk-data",
        "Options": null,
        "Scope": "local"
    }
]

After trying many combinations, I forgot what the original setting for the upload path within Listmonk was, but I believe it was uploads.

Screenshots:

Additional notes

This is my firs time using docker and I realize some issues might come from my lack of understanding of volumes, binds and what-have-you. I also understand that there have been bug reports on this already, namely #620, #555 and #30.

The reason I'm still opening this issue are three-fold:

1. I believe I've tried every suggestion in the other reports, and more, yet the issue persists
2. After a clean install via the provided sh script, I feel like this should work out of the box without further tinkering, no?
3. The other reports are 2+ years old, maybe this is a regression or new bug?

[mr-karan]

Hi @domsson. The default docker-compose.yml doesn't ship with persistence for uploads directory, because Listmonk also supports AWS S3 uploads. Maybe we can change this in future.

However, you can read the docs on how to add volume bind mount for uploads: https://listmonk.app/docs/configuration/#filesystem. You can use either of the two options.

[domsson]

Thank you @mr-karan - unfortunately I'm still stuck with this.

The documentation seems somewhat incomplete, as docker volume ls shows that there is no listmonk_listmonk-uploads, only listmonk_listmonk-data. However, that's obviously easily fixed by creating the volume accordingly:

$ docker volume inspect listmonk_listmonk-uploads
[
    {
        "CreatedAt": "2023-02-03T10:44:52+09:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/listmonk_listmonk-uploads/_data",
        "Name": "listmonk_listmonk-uploads",
        "Options": {},
        "Scope": "local"
    }
]

I've amended the docker-compose.yml according to the documentation:

app:
  <<: *db-default
  container_name: listmonk_app
  depends_on:
    - db
  volumes:
    - type: volume
       source: listmonk-uploads
       target: /listmonk/uploads
    - ./config.toml:/listmonk/config.toml

volumes:
  listmonk-data:
  listmonk-uploads:

And the upload path in Listmonk is set to /listmonk/uploads.

Does that look correct?

---

The default docker-compose.yml doesn't ship with persistence for uploads directory, because Listmonk also supports AWS S3 uploads. Maybe we can change this in future.

I see. Yeah, with S3 it makes sense, as there is no way to put in working default. When it comes to the filesystem solution, however, I think it would be great to have a working default, as currently, installing the system basically presents with a system that feels a little broken. Getting this to work out of the box would definitely be appreciated!

[mr-karan]

Does that look correct?

Seems correct. docker-compose automatically creates volume for you, so no need to use docker CLI to create it manually.

The indentation was off in your snippet:

  volumes:
    - type: volume
      source: listmonk-uploads
      target: /listmonk/uploads

When you added this volume, you'll need to docker-compose stop ; docker-compose up so that the container gets the new volume endpoint.

If you did that as well, check if /listmonk/uploads folder actually exists in the container:

docker-compose exec app sh
ls -laht /listmonk/uploads

Hope this helps.

[domsson]

The indentation was off in your snippet:

Ah, that was a copy-paste error, sorry.

When you added this volume, you'll need to docker-compose stop ; docker-compose up so that the container gets the new volume endpoint.

Interesting. Guess my lack of docker knowledge really put a stick in my spokes - I had used docker restart listmonk_app, assuming that should reload the config. Now that I issued the suggested commands, the upload feature is finally working!

Thank you very much for your swift assistance, appreciated!

[MaximilianKohler]

Per the docs https://listmonk.app/docs/configuration/#media-uploads, and all the instructions I could find, in my docker-compose.yml I was using:

services:
  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - type: volume
        source: listmonk-uploads
        target: /listmonk/uploads

volumes:
  listmonk-data:
  listmonk-uploads:

I also tried:

    volumes:
      - ./config.toml:/listmonk/config.toml
      - type: volume
        source: listmonk-uploads
        target: /home/ubuntu/listmonk/uploads

I tested the commands above that @mr-karan gave:

ubuntu@listm:~$ sudo docker-compose exec app sh
/listmonk # ls -laht /listmonk/uploads
total 8K
drwxr-xr-x    1 root     root        4.0K Aug  5 18:30 ..
drwxr-xr-x    2 root     root        4.0K Feb 18 17:34 .
/listmonk # ls -laht /home/ubuntu/listmonk/uploads
ls: /home/ubuntu/listmonk/uploads: No such file or directory
/listmonk # 

Does this identify the problem? It shouldn't be root root right? And it should show my test.jpg right? Here's my actual file system:

If I run any of the docker commands without sudo I get Permission denied errors.

ubuntu@listm:~$ sudo docker volume ls
DRIVER    VOLUME NAME
local     <snip>
local     <snip>
local     <snip>
local     <snip>
local     ubuntu_listmonk-data
local     ubuntu_listmonk-uploads

[MaximilianKohler]

I think I mostly figured it out. From what I can tell, if you need to use sudo before any docker commands, then you can't use:

services:
  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - type: volume
        source: listmonk-uploads
        target: /listmonk/uploads

volumes:
  listmonk-data:
  listmonk-uploads:

You have to use:

  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - ./uploads:/listmonk/uploads

or - ./listmonk/uploads:/listmonk/uploads in the last line. Depending which folder you want it in.

I also confirmed that you have to run sudo docker-compose stop ; sudo docker-compose up after making any changes.

And under https://listm.mysite.com/admin/settings you put /listmonk/uploads.

Despite the UI saying:
Upload URI that is visible to the outside world. The media uploaded to upload_path will be publicly accessible under {root_url}, for instance, https://listmonk.yoursite.com/uploads.

https://listm.mysite.com/uploads is still not reachable. But the the files are, IE: https://listm.mysite.com/uploads/test2.jpg.

I think the only remaining issue I have is that Listmonk doesn't detect existing files. IE: I ran:

ubuntu@listm:~$ sudo docker-compose exec app sh
/listmonk # ls -laht /listmonk/uploads
total 48K
drwxr-xr-x    2 1000     1000        4.0K Aug 11 01:20 .
-rw-r--r--    1 1000     1000        8.7K Aug 11 01:20 test2.jpg
-rw-r--r--    1 1000     1000        4.6K Aug 11 01:20 thumb_test2.jpg
drwxr-xr-x    1 root     root        4.0K Aug 11 01:20 ..
-rw-r--r--    1 1000     1000        9.8K Aug 11 01:14 test4.jpg
-rw-r--r--    1 1000     1000        2.6K Aug 11 01:14 thumb_test4.jpg
-rwxr-xr-x    1 1000     1000        3.4K Aug  5 18:02 test.jpg
/listmonk #

It shows correctly that there are 3+ test files in the directory, but at https://listm.mysite.com/admin/campaigns/media the only one that shows is the one I uploaded directly from the UI (test2).

[ovicko]

I think I mostly figured it out. From what I can tell, if you need to use sudo before any docker commands, then you can't use:

services:
  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - type: volume
        source: listmonk-uploads
        target: /listmonk/uploads

volumes:
  listmonk-data:
  listmonk-uploads:

You have to use:

  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - ./uploads:/listmonk/uploads

or - ./listmonk/uploads:/listmonk/uploads in the last line. Depending which folder you want it in.

I also confirmed that you have to run sudo docker-compose stop ; sudo docker-compose up after making any changes.

And under https://listm.mysite.com/admin/settings you put /listmonk/uploads.

Despite the UI saying: Upload URI that is visible to the outside world. The media uploaded to upload_path will be publicly accessible under {root_url}, for instance, https://listmonk.yoursite.com/uploads.

https://listm.mysite.com/uploads is still not reachable. But the the files are, IE: https://listm.mysite.com/uploads/test2.jpg.

I think the only remaining issue I have is that Listmonk doesn't detect existing files. IE: I ran:

ubuntu@listm:~$ sudo docker-compose exec app sh
/listmonk # ls -laht /listmonk/uploads
total 48K
drwxr-xr-x    2 1000     1000        4.0K Aug 11 01:20 .
-rw-r--r--    1 1000     1000        8.7K Aug 11 01:20 test2.jpg
-rw-r--r--    1 1000     1000        4.6K Aug 11 01:20 thumb_test2.jpg
drwxr-xr-x    1 root     root        4.0K Aug 11 01:20 ..
-rw-r--r--    1 1000     1000        9.8K Aug 11 01:14 test4.jpg
-rw-r--r--    1 1000     1000        2.6K Aug 11 01:14 thumb_test4.jpg
-rwxr-xr-x    1 1000     1000        3.4K Aug  5 18:02 test.jpg
/listmonk #

It shows correctly that there are 3+ test files in the directory, but at https://listm.mysite.com/admin/campaigns/media the only one that shows is the one I uploaded directly from the UI (test2).

To newbies like me, create a folder in your listmonk directory then provide the absolute path to that dir otherwise it won't work.
in my case listmonk is in /opt/listmonk, created uploads folder /opt/listmonk/uploads, my media upload path settings as /opt/listmonk/uploads
Also this commands are a MUST : sudo docker-compose stop ; sudo docker-compose up

[PaulMcF1987]

@MaximilianKohler @ovicko @knadh
There really is no need for how difficult this is to set up the media folder.

I have tried countless things looking at all of the treads on here and I have had no joy. I was hoping someone could help and see where I might have went wrong...

YAML

version: "3.7"

x-app-defaults: &app-defaults
  restart: unless-stopped
  image: listmonk/listmonk:latest
  ports:
    - "8100:9000"
  networks:
    - listmonk
  environment:
    - TZ=Etc/UTC

x-db-defaults: &db-defaults
  image: postgres:13
  ports:
    - "8101:5432"
  networks:
    - listmonk
  environment:
    - POSTGRES_PASSWORD=xyz
    - POSTGRES_USER=xyz
    - POSTGRES_DB=xyz
  restart: unless-stopped
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U listmonk"]
    interval: 10s
    timeout: 5s
    retries: 6

services:
  db:
    <<: *db-defaults
    container_name: listmonk_db
    volumes:
      - type: volume
        source: listmonk-data
        target: /var/lib/postgresql/data

  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - ./uploads:/listmonk/uploads

  demo-db:
    container_name: listmonk_demo_db
    <<: *db-defaults

  demo-app:
    <<: *app-defaults
    container_name: listmonk_demo_app
    command: [sh, -c, "yes | ./listmonk --install --config config-demo.toml && ./listmonk --config config-demo.toml"]
    depends_on:
      - demo-db

networks:
  listmonk:

volumes:
  listmonk-data:

I have also tried

volumes:
  listmonk-data:
  listmonk-uploads:

I have then tried different commands to remove/restart the container

sudo docker-compose stop ; sudo docker-compose up, sudo docker-compose down ; sudo docker-compose up -d and sudo docker-compose up --force-recreate -d app

Every time I get the same errors... either
ERROR: for demo-db Cannot start service demo-db: driver failed programming external connectivity on endpoint listmonk_demo_db (7d071581a28d31f27bbe01e3659b6716ccb868948f69abdb61c9fb18ada6787d): Bind for 0.0.0.0:8101 failed: port is already allocated ERROR: Encountered errors while bringing up the project.

or

ERROR: for aa32054912fd_listmonk_db Cannot start service db: driver failed programming external connectivity on endpoint listmonk_db (224087eb7277894c910dc3d338fa350af20c4f3de03480f601564a0b1e6e48b8): Bind for 0.0.0.0:8101 failed: port is already allocated ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint listmonk_db (224087eb7277894c910dc3d338fa350af20c4f3de03480f601564a0b1e6e48b8): Bind for 0.0.0.0:8101 failed: port is already allocated

With these errors, the app still seems to work, but the uploads do not work and havent since the first install

Do I need the Demo section? All of the install tutorials I have seen seem to keep it.

Whenever I try to upload media I get

[MaximilianKohler]

Your errors are Bind for 0.0.0.0:8101 failed: port is already allocated. Did you try something other than 8101?

[PaulMcF1987]

@MaximilianKohler I think this is due to this part:

services:
  x-db-defaults: &db-defaults
  image: postgres:13
  ports:
    - "8101:5432"
    - 
  db:
    <<: *db-defaults
    container_name: listmonk_db
    volumes:
      - type: volume
        source: listmonk-data
        target: /var/lib/postgresql/data

  demo-db:
    container_name: listmonk_demo_db
    <<: *db-default

Both db and demo-db use <<: *db-default If I remove demo-db I dont get the error, so I was wondering if this was required?

But, also, my main query is surrounding the Media upload. I cannot get this to work regardless of what I try

[MaximilianKohler]

The docs have what worked for me: https://listmonk.app/docs/configuration/#media-uploads

I think you can remove the demo sections without issue.

[PaulMcF1987]

@MaximilianKohler
Thanks
Ive tried both

app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - type: volume
        source: listmonk-uploads
        target: /listmonk/uploads

volumes:
  listmonk-uploads:

and

    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      - ./uploads:/listmonk/uploads

networks:
  listmonk:

volumes:
  listmonk-data:
  listmonk-uploads:

Neither seems to work :(

[PaulMcF1987]

@MaximilianKohler sorted it. Thanks for your help :)
I had made an error in the settings page with the upload path