Adding Packages in Docker Compose within Portainer Stack #476

frabau1 · 2022-06-01T16:28:32Z

Is it possible to add packages via a Docker Compose statement in a Portainer stack? I typically deploy my docker containers as stacks and I’ve written the following code to deploy this stack:

version: '3'

services:
  shiny:
    image: rocker/shiny-verse:latest
    environment: 
      #RUN: "install.packages('sf')"
      RUN: "install2.R sf"
    restart: always
    networks: 
       - shinyservers
    ports:
       - "3838:3838"
       - "8787:8787"
    volumes:
       - "/volume1/docker/Shiny/Logs:/var/log/shiny-server"
       - "/volume1/docker/Shiny/Apps:/srv/shiny-server"
networks:
   shinyservers:
     driver: bridge

My issue revolves around the environmental variables with RUN commands above. I’m not familiar with the syntax to install additional packages when deploying Rocker as a stack, and have been unable to find any examples. The documentation on the Rocker website references command line code https://www.rocker-project.org/use/extending/, which is what I attempted to insert above in a failed experiment.

I have successfully deployed test programs that do not require any additional packages using the rest of the code above, however the project I’m working on needs the ’sf’ package in addition to those bundled with the shiny-verse image. I’ve experimented with additional syntax beyond what I inserted above without success.

Is there a way to deploy a Rocker stack and include additional packages? Any guidance would be greatly appreciated.

cboettig · 2022-06-01T16:50:55Z

I'm not familiar with Portainer, but I think the standard approach to extending Rocker images would be to write a Dockerfile that is derived from the desired Rocker image with the desired additional RUN commands, and then use the build directive in your docker compose yml.

eitsupi · 2022-06-02T11:27:19Z

Creating the following compose.yaml and Dockerfile should work (not tested).

services:
  shiny:
    build: .
    ports:
      - 3838:3838

FROM rocker/shiny-verse:latest
RUN install2.r --error -n -1 sf

Please check https://docs.docker.com/compose/gettingstarted/.

(Note that we are currently considering adding examples of such usage to the Rocker Project site)

frabau1 · 2022-06-03T15:17:14Z

I have successfully built a package using the docker file code provided by @eitsupi , however I am getting an error in my log stating that the package isn’t available when I run my shiny app:

── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
✔ ggplot2 3.3.6     ✔ purrr   0.3.4
✔ tibble  3.1.7     ✔ dplyr   1.0.9
✔ tidyr   1.2.0     ✔ stringr 1.4.0
✔ readr   2.1.2     ✔ forcats 0.5.1
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
Error: package or namespace load failed for ‘sf’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/usr/local/lib/R/site-library/units/libs/units.so':
  libudunits2.so.0: cannot open shared object file: No such file or directory

It sounds like it may be a similar issue to this one: r-spatial/sf#1158

I’m not sure how to move forward at this point given the GDAL package is already packaged into Rocker. Any suggestions?

—————————

As background for anyone who comes across this thread and is attempting something similar: I am using Portainer to manage Docker on a Synology NAS. The Docker implementation on Synology is more limited than in other environments, however Portainer augments the functionality and allows for a nice UI to do most things.

Portainer does allow for building images, but at this moment it is not possible to incorporate the build process into stack deployment - they are working on it (portainer/portainer#6639).

To build a rocker image with additional R packages within this framework:

In Portainer select

images from the left hand menu
-- click + Build a new image
--- paste 'rocker:yourtag’ into the name box
---- paste into the web editor:

FROM rocker/shiny-verse:latest
RUN install2.r --error --skipinstalled \
Rcpp \
sf \
readr \
xlsx \
sqldf \
sf \
tmap \
DT \
shiny \

(note: in the code above the list of items after '-skip installed' are the R package I am adding to the base rocker/shiny-verse image).
----- click build the image.

Once you have your custom image you can build the stack. To build a stack in Portioner:

select
-stack from the left hand menu
— click ‘+ add a stack’
—— insert your compose code for example:

version: '3'

services:
  shiny:
    image: rocker:yourtag
    restart: always
    networks: 
       - shinyservers
    ports:
       - "3838:3838"
       - "8787:8787"
    volumes:
       - "/volume1/docker/Shiny/Logs:/var/log/shiny-server"
       - "/volume1/docker/Shiny/Apps:/srv/shiny-server"
networks:
   shinyservers:
     driver: bridge

In the code above volumes references the volumes you are making available to the package on your file system. If you are using Synology - use file station to make folders for your log and apps and then insert the path to both folders in the code above before the :.

eitsupi · 2022-06-03T15:28:37Z

Seems the dependent package are not installed; try installing libudunits2-dev with the apt command.

FROM rocker/shiny-verse:latest
RUN apt-get update -y && apt-get install -y libudunits2-dev && install2.r --error -n -1 sf

eddelbuettel · 2022-06-03T16:04:50Z

(As an aside if you wanted to make it easier you could rely on r2u to install all packages as binaries easily by having install.packages() automagically (thanks to bspm) divert to r-cran-* binaries and their dependencies -- all of which is easier and foregoes these complications. You could do that based on eddelbuettel/r2u:22.04 and then just download and install the shiny .deb package... Still no idea about portainer and your other constraints; plain Docker works fine for me.)

frabau1 · 2022-06-06T12:57:04Z

I tried:
FROM rocker/shiny-verse:latest
RUN apt update && apt install libudunits2-dev && install2.r --error -n -1
sf

in the log throws an error - it looks like the program expects input 'Do you want to continue? [Y/n] Abort.' however since I'm not in command line it isn't available.

Here is the readout from the log:
20 packages can be upgraded. Run 'apt list --upgradable' to see them.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...

Building dependency tree...

Reading state information...

The following additional packages will be installed:

libudunits2-0 libudunits2-data

The following NEW packages will be installed:

libudunits2-0 libudunits2-data libudunits2-dev

0 upgraded, 3 newly installed, 0 to remove and 20 not upgraded.
Need to get 429 kB of archives.
After this operation, 1,026 kB of additional disk space will be used.
Do you want to continue? [Y/n] Abort.

The command '/bin/sh -c apt update && apt install libudunits2-dev && install2.r --error -n -1 sf' returned a non-zero code: 1

eitsupi · 2022-06-06T13:07:47Z

@frabau1 Sorry, I used the wrong apt command in my post above.
Also, I have not tested this command. We volunteers cannot support the way for you to reliably install the sf package.

FROM rocker/shiny-verse:latest
RUN apt-get update -y && apt-get install -y libudunits2-dev && install2.r --error -n -1 sf

If you want to ensure a reliable installation, the /rocker_scripts/install_geospatial.sh script included in this repository should be able to install the sf package, so you can use the following Dockerfile.
However, it will take about 20 minutes to build time.

rocker-versioned2/scripts/install_geospatial.sh

Line 65 in ae74156

sf \

FROM rocker/shiny-verse:latest
RUN /rocker_scripts/install_geospatial.sh

eitsupi · 2022-06-06T14:12:46Z

@frabau1I have tested sf package on rocker/shiny-verse:latest and it seems to need additional packages libudunits2-dev, libproj-dev, and libgdal-dev.

FROM rocker/shiny-verse:latest
RUN apt-get update -y && apt-get install -y libudunits2-dev libproj-dev libgdal-dev && install2.r --error -n -1 sf

frabau1 · 2022-06-06T17:54:14Z

@eitsupi Thanks - that last block of code worked for me! I was able to build my image and deploy my app successfully using:

FROM rocker/shiny-verse:latest
RUN apt-get update -y && apt-get install -y libudunits2-dev libproj-dev libgdal-dev && install2.r --error -n -1
sf
readr
xlsx
sqldf
sf
tmap
DT

I totally get you guys are volunteers and I have to say the level of support you all provide is amazing. I really appreciate all of your help. I have an unusual implementation, so ultimately testing for my environment is going to have to fall to me simply because of that.

eddelbuettel · 2022-06-06T18:42:30Z

Installing from source is not easy, especially when you have build dependencies. But there are alternatives to consider:

use a container that can use binaries such as r2u with all of CRAN for Ubuntu
install the packages in one go
then maybe manually / via shell command install the shiny-server .deb package and customize

This quick demo launches the r2u container (not yet a Rocker product, but related, and we do have some for the alternate / smaller c2d4u setup) and I then launch one install.r to install all your desired package and their dependencies as binaries.

No tricks. You should be able to reproduce and customize as needed.

frabau1 · 2022-12-24T14:17:29Z

As further documentation for anyone else trying to run packages on a NAS via Portainer. You can build the custom image natively within Portainer. I was unaware this was an option in Portainer previously as I had only deployed pre-built packages.

In Portainer

Select Image from the left hand menu.
In the main panel below the pull image box in the header of the images box select "Build image" in Portainer
Name your image
Put the code to build your image in the web editor - mine is below and adds a couple of packages to geoshinyverse

FROM andbarker/geoshinyverse:latest
MAINTAINER “XXXX” [email protected]

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
      && install2.r --error \
  	tidycensus \ 
	DT \ 
	sqldf \ 
	xlsx

After building your image select stacks from the left hand menu.
Click add stack from the header menu
name your stack
Define your container using the web editor using docker compose

version: '3'

services:
  shiny:
    image: img_custom_geo_Shiny
    container_name: container_custom_geo_Shiny
    restart: always
    networks: 
       - shinyservers
    ports:
       - "3838:3838"
       #- "8787:8787"
    volumes:
       - "/volume1/docker/Shiny/Logs:/var/log/shiny-server"
       - "/volume1/docker/Shiny/Apps:/srv/shiny-server"
    environment:
       - PASSWORD=test
networks:
   shinyservers:
     driver: bridge

Click deploy stack.
The container will now be listed within the stack - the published ports are listed next to the container.
Any shiny program in the shiny apps volume will be available.

Additional options specific to Synology

To access the shiny server outside your network you need to first enable DDNS.
https://kb.synology.com/en-br/DSM/help/DSM/AdminCenter/connection_ddns?version=7
After enabling DDNS, you can enable Reverse Proxy to make the port specified for your shiny server accessible outside your network.

In Control Panel
Select System -> Login Portal -> Advanced -> Reverse Proxy -> Create
Name it whatever you want
Under source set your protocol HTTP or HTTPS
Set a hostname - for example shiny.XXXX.synology.me - where XXXX is what you selected for your DDNS
Set the port - if you used https for the protocol you need to put 443, if http then 80.
Under destination set hostname to your NAS's IP ie 192.168.1.XXX
Set port to 3838
Hit save

You should be able to navigate to your shiny server using the shiny.XXXX.synology.me hostname you specified.

frabau1 · 2023-05-25T17:16:14Z

Preventing index access in Rocker/Using a custom configuration file in Rocker

By default it is possible to edit the url for a shiny app and view an index which lists all apps available on the shiny server. This index access can be disabled by altering the shiny config file.

This took me a while to figure out so I wanted to document it for anyone else looking to do something similar.

How to block access to the index.

You must edit the config file and copy this edited config file into your image when building it.

For information regarding the configuration file see https://docs.posit.co/shiny-server/#default-configuration

Before editing the config file, you have to get a copy of the current file. This can be done by copying the current config file out of the container and saving it somewhere you can access it.

My setup is on a Synology running docker/synology container manager with Portainer. I used the container console window (https://docs.portainer.io/user/docker/containers/console) within Portainer and used this code to copy the file from the container to a directory I could access
cp /etc/shiny-server/shiny-server.conf /srv/shiny-server/shiny-customized.config

NOTE: /srv/shiny-server is mapped to the root app folder so I copied it there.

You could do something similar with ssh:

sudo docker cp shiny:/etc/shiny-server/shiny-server.conf /volume1/docker/Shiny/config/shiny-customized.config

Once you have a copy of the config file you can open it with a text editor and altered the file to say “directory_index off” instead of the default “directory_index on”.

Save your custom config file

How to use a custom configuration file in Rocker

To use your custom configuration file you have to copy it into your image when building it. I submitted documentation on the process of building an image using Portainer previously here: #476 (comment). This line of code copies your custom configuration file into the image being built "COPY shiny-customized.config /etc/shiny-server/shiny-server.conf"

Example:

FROM rocker/shiny:4.0.2
COPY shiny-customized.config /etc/shiny-server/shiny-server.conf
MAINTAINER “XXXX” [email protected]

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
      && install2.r --error \
  	tidycensus \ 
	DT \ 
	sqldf \ 
	xlsx

Once the image is built, you do not need to do anything special with the docker compose statement when deploying the container.

Documentation issues:

The documentation found here: https://github.com/rocker-org/shiny#custom-configuration mentions a commented block of code in the dockerfile that can be uncommented in order to use a custom configuration file.

Looking at the docker file within the GitHub page here: https://github.com/rocker-org/shiny/blob/master/Dockerfile the referenced commented line of code with “COPY shiny-customized.config /etc/shiny-server/shiny-server.conf” does not exist.

My initial question was where should this line of code appear in the dockerfile as it wasn’t included in the example code.
I found this solution in an issue from November 2020 - #360

frabau1 · 2024-08-01T15:59:59Z

Dirk/Carl, I had emailed you in June of 2022 regarding a custom docker file that I was building. My desire was to create a combination of the Rocker geospatial image with the shiny-verse image. I picked this back up last month and I’m now using the method your team employs using helper functions to build the image. I created a git repository for my docker file here: https://github.com/frabau1/GeoShinyVerse/blob/main/Dockerfile.dockerfile NOTE: the ‘RUN chmod +x’ was necessary due to permissions issues when trying to run the *.sh files I’m pulling from the geospatial and shiny-verse repositories for the vast majority of the build, but I also created a helper function based on your helper functions to install a few additional R packages. My helper function is saved here: https://github.com/frabau1/GeoShinyVerse/blob/main/install_additionalpackages.sh When running the code the docker build will successfully complete, however I’m getting an error when trying build a container with the image: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/init": stat /init: no such file or directory: unknown Have you encountered this type of error before? I assume the init error is regarding line 56 of my docker file "CMD ["/init”]”. Any information your could provide would be greatly appreciated. Sincerely, Frank Baumgardner

…

On Jun 6, 2022, at 2:42 PM, Dirk Eddelbuettel ***@***.***> wrote: Installing from source is not easy, especially when you have build dependencies. But there are alternatives to consider: use a container that can use binaries such as r2u with all of CRAN for Ubuntu install the packages in one go then maybe manually / via shell command install the shiny-server .deb package and customize <https://user-images.githubusercontent.com/673121/172225347-b31feb15-eca5-4d11-9e9f-bf77ec4c5bbf.gif> This quick demo launches the r2u container (not yet a Rocker product, but related, and we do have some for the alternate / smaller c2d4u setup) and I then launch one install.r to install all your desired package and their dependencies as binaries. No tricks. You should be able to reproduce and customize as needed. — Reply to this email directly, view it on GitHub <#476 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZN7WNOSKKGHIOQ77TQ2OMLVNZBCFANCNFSM5XR4U3PA>. You are receiving this because you were mentioned.

eitsupi added the question label Jun 2, 2022

eitsupi closed this as completed Jun 2, 2022

eitsupi mentioned this issue Jun 6, 2022

More contents... rocker-org/website#35

Closed

12 tasks

eitsupi mentioned this issue Jun 10, 2022

Add test for install_shiny_server.sh on rocker/r-bspm:20.04 #483

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Packages in Docker Compose within Portainer Stack #476

Adding Packages in Docker Compose within Portainer Stack #476

frabau1 commented Jun 1, 2022 •

edited by eitsupi

Loading

cboettig commented Jun 1, 2022

eitsupi commented Jun 2, 2022

frabau1 commented Jun 3, 2022 •

edited by eitsupi

Loading

eitsupi commented Jun 3, 2022 •

edited

Loading

eddelbuettel commented Jun 3, 2022

frabau1 commented Jun 6, 2022

eitsupi commented Jun 6, 2022 •

edited

Loading

eitsupi commented Jun 6, 2022 •

edited

Loading

frabau1 commented Jun 6, 2022

eddelbuettel commented Jun 6, 2022

frabau1 commented Dec 24, 2022 •

edited

Loading

frabau1 commented May 25, 2023

frabau1 commented Aug 1, 2024 via email

Adding Packages in Docker Compose within Portainer Stack #476

Adding Packages in Docker Compose within Portainer Stack #476

Comments

frabau1 commented Jun 1, 2022 • edited by eitsupi Loading

cboettig commented Jun 1, 2022

eitsupi commented Jun 2, 2022

frabau1 commented Jun 3, 2022 • edited by eitsupi Loading

eitsupi commented Jun 3, 2022 • edited Loading

eddelbuettel commented Jun 3, 2022

frabau1 commented Jun 6, 2022

eitsupi commented Jun 6, 2022 • edited Loading

eitsupi commented Jun 6, 2022 • edited Loading

frabau1 commented Jun 6, 2022

eddelbuettel commented Jun 6, 2022

frabau1 commented Dec 24, 2022 • edited Loading

frabau1 commented May 25, 2023

Preventing index access in Rocker/Using a custom configuration file in Rocker

How to block access to the index.

How to use a custom configuration file in Rocker

frabau1 commented Aug 1, 2024 via email

frabau1 commented Jun 1, 2022 •

edited by eitsupi

Loading

frabau1 commented Jun 3, 2022 •

edited by eitsupi

Loading

eitsupi commented Jun 3, 2022 •

edited

Loading

eitsupi commented Jun 6, 2022 •

edited

Loading

eitsupi commented Jun 6, 2022 •

edited

Loading

frabau1 commented Dec 24, 2022 •

edited

Loading