Allow use of OpenSSL 1.1.1 when building Pony

[srenatus]

This introduces an array of valid versions, and only errors out if the
passed default_ssl version is not among them.

Unfortunately, the list of valid default_ssl version in the help output
does not use that variable.

Also switches the default_ssl debug output to "info" (was "error").

Fixes #3150, I hope.

[srenatus]

❓ Shall we add some test build that actually uses openssl v1.1.1?

[srenatus]

Looks like we could either

• copy the ./.ci-dockerfiles/openssl-1.1.0/Dockerfile and build openssl-1.1.1; or
• copy .ci-dockerfiles/ubuntu-18.04-base/Dockerfile for using ubuntu 19.04 to be closer to the original issue, Ubuntu 19.04 provides only openssl1.1.1 #3148.

What do you think? Also besides adding a Dockerfile, and a line to the CircleCI config, is there anything else that needs to be done to wire up the new container image?

[jemc]

Maybe we should just replace the current testing of openssl-1.1.0 to test with openssl-1.1.1 instead?

[EpicEric]

IMO it would make sense to support Ubuntu LTS releases and the latest non-LTS, so I'd go with @srenatus' second option.

[srenatus]

Happy to add what's required here! I'm still not sure if the Dockerfile and the mention in the circleci config is sufficient, but I guess I'll find out. 😃

[srenatus]

💡 https://github.com/ponylang/ponyc/blob/master/.ci-dockerfiles/ubuntu-18.04-base/README.md#push-to-dockerhub so, I'll talk to @jemc and @SeanTAllen when this is ready 😉

[srenatus]

Ok, it seems the existing ubuntu base images are used with the various LLVM versions we're supporting. Would we want to multiply that -- i.e. replicate the LLVM dockerfiles that are built with FROM ponylang/ponyc-ci:ubuntu-16.04-base and do FROM ponylang/ponyc-ci:ubuntu-19.04-base? It would be a bit much extra work for each PR, I suppose.

Happy to help, though; but I think I'd rather get this merged and look into adding ubuntu 19.04 ci coverage separately. 🤔

[srenatus]

ℹ️ Please don't merge -- I'm not done reproducing the original issue, and verifying that this solves it. (In the narrowest scope, at least, this will make openssl_1.1.1 an accepted version.)

[srenatus]

😃 It's not resolved by this change alone, but I'm onto it. 🤞

[srenatus]

I've taken a bit of a detour in 0de999d -- but since what all the ifdefs in the stdlib packages care about seems to be 1.1.x, that seemed like a viable approach. The remaining question is:

• Do we want to keep the Makefile options compatible? I.e. we could have openssl_1.1.0 and openssl_1.1.1 be accepted default_ssl settings.

However, this seemed a bit disingenuous since that version part of the option really doesn't have much of an effect: It doesn't matter if you provide openssl_1.1.0 or openssl_1.1.1. So, I think I'd rather expose an option that reflects that better.

What do you think? 😃

[srenatus]

I've tried verifying that this now runs tests successfully on Ubuntu 19.04 using a docker container, but I keep running into #3099. Just spent some time trying with the vendored LLVM, but the same thing occurred. I might have used the wrong commands, anyways.

My dockerfile:

FROM ubuntu:19.04

RUN apt-get update \
 && apt-get install -y \
  apt-transport-https \
  build-essential \
  g++-6 \
  git \
  libncurses5-dev \
  libssl-dev \
  libpcre2-dev \
  make \
  wget \
  xz-utils \
  zlib1g-dev \
  cmake \
  python \
 && rm -rf /var/lib/apt/lists/* \
 && apt-get -y autoremove --purge \
 && apt-get -y clean

# add user pony in order to not run tests as root
RUN useradd -ms /bin/bash -d /home/pony -g root pony
USER pony
WORKDIR /home/pony

The command I've run:

make -f Makefile-lib-llvm test-ci default_ssl=openssl_1.1.x default_pic=true config=release

The output I see:

--- 8< ---

[----------] Global test environment tear-down
[==========] 42 tests from 7 test cases ran. (36 ms total)
[  PASSED  ] 42 tests.
Building builtin -> /ponyc/packages/builtin
Building packages/stdlib -> /ponyc/packages/stdlib
Building ponytest -> /ponyc/packages/ponytest
Building time -> /ponyc/packages/time
Building collections -> /ponyc/packages/collections
Building random -> /ponyc/packages/random
Building assert -> /ponyc/packages/assert
Building backpressure -> /ponyc/packages/backpressure
Building encode/base64 -> /ponyc/packages/encode/base64
Building buffered -> /ponyc/packages/buffered
Building builtin_test -> /ponyc/packages/builtin_test
Building bureaucracy -> /ponyc/packages/bureaucracy
Building promises -> /ponyc/packages/promises
Building capsicum -> /ponyc/packages/capsicum
Building files -> /ponyc/packages/files
Building term -> /ponyc/packages/term
Building strings -> /ponyc/packages/strings
Building signals -> /ponyc/packages/signals
Building cli -> /ponyc/packages/cli
Building collections/persistent -> /ponyc/packages/collections/persistent
Building crypto -> /ponyc/packages/crypto
Building format -> /ponyc/packages/format
Building debug -> /ponyc/packages/debug
Building glob -> /ponyc/packages/glob
Building regex -> /ponyc/packages/regex
Building ini -> /ponyc/packages/ini
Building itertools -> /ponyc/packages/itertools
Building json -> /ponyc/packages/json
Building logger -> /ponyc/packages/logger
Building math -> /ponyc/packages/math
Building net -> /ponyc/packages/net
Building options -> /ponyc/packages/options
Building ponybench -> /ponyc/packages/ponybench
Building process -> /ponyc/packages/process
Building serialise -> /ponyc/packages/serialise
Building net/ssl -> /ponyc/packages/net/ssl
Generating
 Reachability
 Selector painting
 Data prototypes
 Data types
 Function prototypes
 Functions
 Descriptors
Verifying
Writing ./stdlib.o
Linking ./stdlib
/usr/bin/ld.gold: error: ./stdlib.o: requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/bin/ld.gold: error: ./stdlib.o: requires dynamic R_X86_64_32 reloc against 'ponyint_personality_v0' which may overflow at runtime; recompile with -fPIC
collect2: error: ld returned 1 exit status
Warning: environment variable $CC undefined, using cc as the linker
Error:
unable to link: cc -o ./stdlib -O3 -march=native -mcx16 -latomic  -fuse-ld=gold ./stdlib.o -L"." -Wl,-rpath,"." -L"/ponyc/build/release/" -Wl,-rpath,"/ponyc/build/release/" -L"/ponyc/build/release/lib/native" -Wl,-rpath,"/ponyc/build/release/lib/native" -L"/ponyc/build/release/../../packages" -Wl,-rpath,"/ponyc/build/release/../../packages" -L"/usr/local/lib" -Wl,-rpath,"/usr/local/lib" -Wl,--start-group -l"rt" -l"crypto" -l"pcre2-8" -l"ssl" -Wl,--end-group  -lpthread  -lponyrt-pic -ldl -lm
make[1]: *** [Makefile-ponyc:963: stdlib-debug] Error 255
make[1]: Leaving directory '/ponyc'
make: *** [Makefile-lib-llvm:33: test-ci] Error 2

[SeanTAllen]

We've previously decided that we support Ubuntu LTS releases. That goes for CI so supporting Disco via would be a change that needs to be discussed.

[srenatus]

@SeanTAllen 👍 I'm fine with not pulling 19.04 into our CI. I'm just conflicted with this PR itself:

• It should address the issue insofar as you now can provide default_ssl=openssl_1.1.1 (or default_ssl=openssl_1.1.x, since it's only a string used in build flags, and has no other use)
• I haven't been able to verify that this makes it compile on Ubuntu 19.04 -- which was the root issue for the desired Makefile change, I suppose.

So, as a way forward, we could either merge this, to allow folks to provide default_ssl=openssl_1.1.1, or document that openssl_1.1.0 really means any 1.1.x version, and that it doesn't matter which 1.1.x you have, default_ssl=openssl_1.1.0 will flip the right switches in stdlib. (On a related note, I've got to be doubtful that the proposed fix in #3148 (comment) really made this work -- it's not going to affect the relevant ifdefs if its set to openssl_1.1.1 with the current code...)

[SeanTAllen]

@srenatus I think hardcoding in 1.1.1 for now and then we can figure out a longer term change makes sense.

[SeanTAllen]

@srenatus yeah we discussed at sync and adding in hardcoded 1.1.1 as another option in Makefile is fine. Please let us know when this is ready for merging.

[srenatus]

I think this is good to go. I'll rebase, and see what CI thinks.

[SeanTAllen]

Thanks @srenatus