Problem running builder

[OrKoN]

Hi!

I have encountered a problem running the builder:

cd examples/echo
docker run --rm -v $(pwd):/code:ro ilianaw/crowbar-builder
error: failed to load source for a dependency on `crowbar`

Caused by:
  Unable to update file:///

Caused by:
  failed to read `/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

I tried to fix it but it looks like a glitch in cargo. Perhaps someone could help to solve this problem.

Best regards,
Alex

[euank]

I think I see what's going on here.

The example's Cargo.toml depends on the current checkout of rust-crowbar. Building it in a docker container like that ends up losing the parent directory, and thus that dependency fails.

I'm specifically referring to the lines:

https://github.com/ilianaw/rust-crowbar/blob/efab795ac505ec12ea60b0bfe368e258844906d1/examples/echo/Cargo.toml#L11-L13

It's possible that editing the dependency to be crowbar = "0.1" will be enough for the docker method of building to work.

It does make sense for the examples to be compiled against the local copy of rust-crowbar, especially for development purposes... but that leads to it not working with the builder and a confusing error.

Maybe the builder documentation should mention it doesn't work on the examples, or maybe there's some cleverness would allow a single Cargo.toml to work for both use-cases?

Let me know if the above makes sense!

[OrKoN]

@euank Thanks! You are right. This was causing the problem. Actually I have seen these lines in Cargo.toml but I have failed to correlate the error message with them. I think it might help if examples also have a readme file which explains how to build them.

Nevertheless, after I changed the cargo.toml like this:

[package]
name = "lambda-echo"
version = "0.2.0"
authors = ["Iliana Weller <[email protected]>"]
publish = false

[lib]
name = "lambda"
crate-type = ["cdylib"]

[dependencies]
# Normally you'd write: crowbar = "0.2"
crowbar = "0.1"
cpython = { version = "0.1", default-features = false }

[features]
default = ["cpython/python3-sys"]

I get the following error:

error[E0428]: the name `PYTHONSYS_ENV_VAR` is defined multiple times
  --> /root/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/cpython-0.1.0/build.rs:10:1
   |
7  | const PYTHONSYS_ENV_VAR: &'static str = "DEP_PYTHON27_PYTHON_FLAGS";
   | -------------------------------------------------------------------- previous definition of the value `PYTHONSYS_ENV_VAR` here
...
10 | const PYTHONSYS_ENV_VAR: &'static str = "DEP_PYTHON3_PYTHON_FLAGS";
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `PYTHONSYS_ENV_VAR` redefined here
   |
   = note: `PYTHONSYS_ENV_VAR` must be defined only once in the value namespace of this module

error: aborting due to previous error

error: Could not compile `cpython`.
warning: build failed, waiting for other jobs to finish...
error: build failed

[OrKoN]

It seems to be building if I change python3-sys to default = ["cpython/python27-sys"] in cargo.toml. But when I try to run produced lambda.zip on AWS I get an error: Fatal Python error: PyThreadState_Get: no current thread

[ChristopherMacGown]

Hi @OrKoN,

The Fatal Python error: PyThreadState_Get: no current thread error message means that the module was compiled against an incompatible version of Python. In my testing on US East (N. Virginia) and US West (N. California) the python version for the python 2.7 runtime is 2.7.12 but @ilianaw's builder image compiles a 2.7.13 pyenv.

I was able to successfully build and deploy a lambda function using rust-crowbar with the following Dockerfile:

FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y python27-devel bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]

with the following modified build.sh:

#!/bin/bash
set -euo pipefail
export CARGO_TARGET_DIR=$(mktemp -d)
(
    if [[ $# -gt 0 ]]; then
        yum install -y "$@"
    fi
    . $HOME/.cargo/env
    cargo build ${CARGO_FLAGS:-} --release
) 1>&2
cd $CARGO_TARGET_DIR/release
(
    strip liblambda.so
    zip lambda.zip liblambda.so
) 1>&2
exec cat lambda.zip

[OrKoN]

Thanks @ChristopherMacGown I didn't that even minor versions of Python are not compatible :-) that worked fine for me too.

[ChristopherMacGown]

I wasn't sure if that was the case either, so I've run a test with just the pyenv 2.7.13 and it fails the same way:

FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ENV PYENV_ROOT=/usr/local/pyenv PYTHON_CONFIGURE_OPTS="OPT=\"-fPIC\""
RUN git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT
RUN /usr/local/pyenv/bin/pyenv install 2.7.13
#RUN /usr/local/pyenv/bin/pyenv install 3.6.1
#RUN /usr/local/pyenv/bin/pyenv global 3.6.1 2.7.13
RUN /usr/local/pyenv/bin/pyenv global 2.7.13
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]

with the base build.sh results in:

docker run -it -v $(pwd):/code:ro amazonlinux:latest /bin/bash

bash-4.2# yum install -y unzip
# elided yum stuff... 
bash-4.2# unzip code/lambda.zip  
Archive:  code/lambda.zip
  inflating: liblambda.so            
bash-4.2# python
Python 2.7.12 (default, Sep  1 2016, 22:14:00) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblambda
Fatal Python error: PyThreadState_Get: no current thread
Aborted

[iliana]

I used pyenv because at the time of writing that Amazon Linux didn't have a 3.6 build available yet.

I wonder if it works fine if we use 2.7.12 in pyenv...?

[ChristopherMacGown]

Not as far as my testing shows:

FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ENV PYENV_ROOT=/usr/local/pyenv PYTHON_CONFIGURE_OPTS="OPT=\"-fPIC\""
RUN git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT
RUN /usr/local/pyenv/bin/pyenv install 2.7.12
#RUN /usr/local/pyenv/bin/pyenv install 3.6.1
#RUN /usr/local/pyenv/bin/pyenv global 3.6.1 2.7.13
RUN /usr/local/pyenv/bin/pyenv global 2.7.12
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]

-- ELIDED -- Successfully built 91d63bed5bc0

$ docker run --rm -v $(pwd):/code:ro 91d63bed5bc0 > lambda.zip
... ELIDED ...
   Compiling lambda-test v0.1.0 (file:///code)
    Finished release [optimized] target(s) in 32.5 secs
  adding: liblambda.so (deflated 61%)

Installed:
  unzip.x86_64 0:6.0-4.10.amzn1                                                                                                                                                                                                              

Complete!
bash-4.2# unzip code/
.git/         .gitignore    Cargo.lock    Cargo.toml    butts         lambda-2.zip  lambda.zip    src/          target/       
bash-4.2# unzip code/lambda.zip 
Archive:  code/lambda.zip
  inflating: liblambda.so            
bash-4.2# python
Python 2.7.12 (default, Sep  1 2016, 22:14:00) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblambda
Fatal Python error: PyThreadState_Get: no current thread
Aborted
bash-4.2# 

[iliana]

Well, sounds like pyenv is out for 2.7 then.

[iliana]

pyenv is also out for 3.6. It fails with Fatal Python error: take_gil: NULL tstate, which is somehow even more obtuse than the 2.7 error :)

I'm prepping a commit which just uses python27-devel and python36-devel from Amazon Linux, now that 2017.09 is out and Python 3.6 is available in the repos...

[iliana]

For @OrKoN's original issue, you can run examples in the builder like this, from the root of the project:

docker run --rm -v $(pwd):/code:ro -w /code/examples/echo ilianaw/crowbar-builder > lambda.zip

as the builder script just execs cargo build from the workdir.

[iliana]

https://hub.docker.com/r/ilianaw/crowbar-builder/builds/bvd6fywfnfzzpnpedr8rtqm/