Docker Image - PyRFC

[Idan-fishman-old]

Hi There,
I'm trying to develop an automations system, and i thought im not the first so why should i invent the wheel?
do you have any pyrfc docker image ready to production?

[jfilak]

I did some experiments https://hub.docker.com/r/filaksap/abapops
but you have to provide NW RFC SDK libraries anyways.

[bsrdjan]

Hi Dan,

there is no standard docker because users run PyRFC on different Linux platforms and Python versions.

Here the one based on Ubuntu and if you search for "docker" in PyRFC and node-rfc issues, more can be found, like: #176 (comment)

If you use Linux the build from source is recommended and, as @jfilak mentioned, NWRFC SDK libraries are not included, must be downloaded from SAP Service Portal.

#
# Build:
# docker build -f Dockerfile-node -t rfcqa-node .
# docker run --name rfcqa-node -v /home/test/src:/home/www-admin/src -it rfcqa-node
#
# Run:
# docker start -ai rfcqa
#

FROM ubuntu:latest

LABEL maintainer="[email protected]"
LABEL version="1.0"
LABEL description="RFM Connectors QA"

ARG adminuser=www-admin
ARG nwrfc_local=/usr/local/sap
ARG venv_base=/home/${adminuser}/.virtualenvs
ARG dev_tools="sudo curl wget git unzip vim tree tmux"
ARG dev_libs="build-essential make libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev"

# os update and packages
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt update && apt install -y locales ${dev_tools}

# timezone # https://serverfault.com/questions/683605/docker-container-time-timezone-will-not-reflect-changes
ENV TZ=Europe/Berlin
RUN locale-gen de_DE && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Add admin user
RUN \
  adduser --disabled-password --gecos "" ${adminuser} && \
  usermod -aG www-data,sudo ${adminuser} && \
  echo "${adminuser} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

USER ${adminuser}
WORKDIR /home/${adminuser}
RUN printf "alias e=exit\nalias ..=cd..\nalias :q=exit\nalias ll='ls -l'\nalias la='ls -la'\n" > .bash_aliases
RUN printf "\n# colors\nexport TERM=xterm-256color\n" >> ~/.bashrc

# git configuration
RUN \
  git config --global http.sslVerify false && \
  git config --global user.name bsrdjan && \
  git config --global user.email [email protected]

# essentials
RUN sudo apt install -y ${dev_libs}

# cmake
RUN cd /tmp && \
  CMAKE_VERSION=$(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest | grep '"tag_name"' | sed -E 's/.*"v([^"]+)".*/\1/') && \
  wget -q https://github.com/Kitware/CMake/releases/latest/download/cmake-${CMAKE_VERSION}.tar.gz && \
  tar -xzvf cmake-${CMAKE_VERSION}.tar.gz && \
  cd cmake-${CMAKE_VERSION} && \
  ./bootstrap && \
  make -j4 && sudo make install

# sap nw rfc lib
RUN printf "\n# nwrfc sdk \n" >> ~/.bashrc
RUN printf "export SAPNWRFC_HOME=${nwrfc_local}/nwrfcsdk \n" >> ~/.bashrc
USER root
RUN mkdir -p ${nwrfc_local}
COPY /SAP/NWRFCSDK-750-PL6/linux/nwrfcsdk ${nwrfc_local}/nwrfcsdk
RUN chmod -R a+r ${nwrfc_local}/nwrfcsdk && chmod -R a+x ${nwrfc_local}/nwrfcsdk/bin
RUN printf "# include nwrfcsdk\n${nwrfc_local}/nwrfcsdk/lib\n" |  tee /etc/ld.so.conf.d/nwrfcsdk.conf
RUN ldconfig && ldconfig -p | grep sap

# pyenv
RUN git clone https://github.com/pyenv/pyenv.git .pyenv
RUN printf 'export PYENV_ROOT="$HOME/.pyenv"\n' >> .bashrc
RUN printf 'export PATH="$PYENV_ROOT/bin:$PATH"\n' >> .bashrc
RUN printf 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi\n' >> .bashrc
ENV PYENV_ROOT /home/${adminuser}/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
RUN pyenv install ${py38}
RUN pyenv install ${py37}
RUN pyenv install ${py36}

# pyenv-virtualenv
RUN git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
RUN printf 'eval "$(pyenv virtualenv-init -)"\n' >> ~/.bashrc
RUN printf 'export PYENV_VIRTUALENV_DISABLE_PROMPT=1\n' >> ~/.bashrc
RUN bash -ic "pyenv virtualenv ${py36} ${py36venv} && pyenv activate ${py36venv} && pip install --upgrade ${dev_python}"
RUN bash -ic "pyenv virtualenv ${py37} ${py37venv} && pyenv activate ${py37venv} && pip install --upgrade ${dev_python}"
RUN bash -ic "pyenv virtualenv ${py38} ${py38venv} && pyenv activate ${py38venv} && pip install --upgrade ${dev_python}"
RUN printf "pyenv activate ${py38venv}\n" >> ~/.bashrc

# nvm
#USER ${adminuser}
#RUN printf "\n# nvm" >> ~/.bashrc
#RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.35.3/install.sh | bash
#RUN bash -ic "nvm install lts/dubnium && nvm install lts/erbium && nvm install node && nvm alias default node"
#RUN printf "export PATH=node_modules/.bin:\$PATH\nnvm use default\n\n" >> ~/.bashrc

# yarn
# RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
# RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
# RUN sudo apt-get update && sudo apt-get install --no-install-recommends yarn

# cleanup
RUN sudo rm -rf /var/lib/apt/lists/*
RUN rm -rf /tmp/*

[jfilak]

The point creating a standard docker image is that users will no longer need to run different GNU/Linux and different PyRFC versions. They would just do "docker run pyrfc ...". Docker images are being used because you can run them anywhere with compatible Linux Kernel - I haven't run into any case of Linux Kernel incompatibility yet. I seriously doubt PyRFC uses Linux Kernel calls directly and in the way that PyRFC would work for example on Ubuntu Kernel but not on Fedora Kernel.

The only problem with docker image is NW RFC SDK libraries. I provide the libraries via a bind mount in my image but users can take my image, copy the libraries, create a new image and publish it in their internal Docker registry - then they can use the docker image on whatever Linux/Windows/Mac machine to execute RFC functions.

[Idan-fishman-old]

Hey Guys thanks for replying,
I have already created a Simple Docker image, that sized 442MB including anything.

FROM ubuntu
ENV SAPNWRFC_HOME=/usr/local/sap/nwrfcsdk
COPY ./nwrfcsdk.conf /etc/ld.so.conf.d/nwrfcsdk.conf
COPY ./nwrfc750P_6-70002752 /usr/local/sap
RUN apt-get update && apt-get install -y python3.8 python3-pip && ldconfig

Just fine for a run, but i want to decrease the size.
I would like to try using python:3.8-slim or python:3.8-alpine

Anyone tried?

[bsrdjan]

Alpine is based on musl libc and not supported by PyRFC because SAP NWRFC SDK not supported on musl: SAP-archive/node-rfc#148

The python:3.8-slim if debian/ubuntu/glibc based, should work, but you have to invent the wheel :)

Due to subtle differences among Linux platforms, the "universal Linux" PyRFC would require the manylinux build. The package would include the SAP NWRFC SDK and could not be distributed as such. Therefore the (many)Linux wheel is not available for PyRFC on PyPI.

[Idan-fishman-old]

Actually I have done some experiments with python:alpine-3.8 and I was able to build pynwrfc
But I wasn’t able to perform the ldconfig command ,with the dynamic linker.

If you have some experience with the alpine ldconfig command i would like you to help me

[bsrdjan]

Does this help https://stackoverflow.com/questions/36990951/ldconfig-seems-no-functional-under-alpine-3-3 ?

[Idan-fishman-old]

trying to understand how to use ldd.so / ldconfig on alpine.

[jfilak]

exporting LD_LIBRARY_PATH is not enough? It's cool to have ldconfig in place, but if it does not work out of the box, then why not just using the environment variable.

[Idan-fishman-old]

What do you mean with using LD_LIBRARY_PATH?
To add?
ENV LD_LIBRARY_PATH=/usr/local/sap/nwrfcsdk
and than?
use ldd? ldconfig?

[jfilak]

Actually, the environment variable LD_LIBRARY_PATH must point to a directory where ldd can look up for libraries.

ENV LD_LIBRARY_PATH=/usr/local/sap/nwrfcsdk/lib

[Idan-fishman-old]

okay, and than should i add:

RUN ldd
or 
RUN ldconfig

[jfilak]

Nothing else. The environment variable is used in runtime:
https://github.com/filak-sap/abapops/blob/master/Dockerfile#L16

[Idan-fishman-old]

doesnt work

/ # python
Python 3.8.5 (default, Aug  4 2020, 04:11:56) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyrfc import Connection
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/pyrfc/__init__.py", line 39, in <module>
    from .pyrfc import (
ImportError: Error relocating /usr/local/sap/nwrfcsdk/lib/libsapnwrfc.so: __strdup: symbol not found
>>> exit()
/ # 

[jfilak]

Actually, it worked quite well. The symbol __strdup is from libc and since you have musl on alpine you have no __strdup symbol.

$ readelf -ds /usr/lib64/libc.so.6 | grep __strdup
   270: 00000000000902c0    64 FUNC    GLOBAL DEFAULT   15 __strdup@@GLIBC_2.2.5
  8349: 00000000000902c0     0 NOTYPE  LOCAL  HIDDEN    15 .annobin___GI___strdup.st
  8350: 0000000000090300     0 NOTYPE  LOCAL  HIDDEN    15 .annobin___GI___strdup.en
 22895: 00000000000902c0    64 FUNC    LOCAL  DEFAULT   15 __GI___strdup
 27732: 00000000000902c0    64 FUNC    GLOBAL DEFAULT   15 __strdup

^^ from my Fedora box.

[Idan-fishman-old]

Yea, i understand that. which cause me to disqualify Alpine

[bsrdjan]

Just as info, a new tool, eventually helpful in your work: #167 (comment)

[Idan-fishman-old]

Finally I have successfully created a Docker image of pynwrfc sized only 169MB.
have done this using, python:3.8.5-slim-buster.

[bsrdjan]

Great! f you want to share the working image, you could create the PR, helping other users as well. If published on docker hub, we could add the hyperlink in the README file, in a (new) PyRFC docker folder? The contribution agreement is simplified recently.

[jfilak]

@bsrdjan Well, the image must not include the SAP NWRFC SDK libraries, does it?

[Idan-fishman-old]

@bsrdjan Well, the image must not include the SAP NWRFC SDK libraries, does it?

Including the SAP NWRFC SDK :)

[Idan-fishman-old]

Great! f you want to share the working image, you could create the PR, helping other users as well. If published on docker hub, we could add the hyperlink in the README file, in a (new) PyRFC docker folder? The contribution agreement is simplified recently.

I will share it on docker hub, and let you know

[bsrdjan]

It must not include SAP NWRFC SDK but for example a COPY statement, copying from local folder to image, or wget/curl from SAP Service Portal ...

[Idan-fishman-old]

What do you mean, on your image you have locally copied the files.

On my image i do it too.

[jfilak]

@Dan-Fishman You can build an image which contains the SDK but you must not publish the image free to download by public because the SDK's license does not allow that. However, you can distribute that image in your company as you wish, just don't publish it any public site as long as it contains the SDK.

[Idan-fishman-old]

Understood.

[Idan-fishman-old]

Great! f you want to share the working image, you could create the PR, helping other users as well. If published on docker hub, we could add the hyperlink in the README file, in a (new) PyRFC docker folder? The contribution agreement is simplified recently.

Hey, lets do it 👍🏼, i prepared the image.

[Idan-fishman-old]

bsrdjan,
I want to contribute to your PYRFC Project, with the dockerfile - updated it to the current 2.3 version.
contact me at [email protected].
Thanks.

[mviniciius29]

Where did they put the files to upload the image ?