Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Builds #2558

Merged
merged 2 commits into from
May 26, 2021
Merged

Builds #2558

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e4a6d0e
chore: nightly builds, now with rust for flux.
docmerlin May 20, 2021
8096289
Merge branch 'master' into builds
docmerlin May 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 7 additions & 18 deletions Dockerfile_build_ubuntu64
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
FROM ubuntu:18.04
ARG GO_VERSION
FROM quay.io/influxdb/cross-builder:go${GO_VERSION}-latest

# This dockerfile is capabable of performing all
# build/test/package/deploy actions needed for Kapacitor.
Expand Down Expand Up @@ -41,35 +42,23 @@ RUN wget -q https://github.com/google/protobuf/releases/download/v${PROTO_VERSIO
# Install protobuf3 python library
RUN wget -q https://github.com/google/protobuf/releases/download/v${PROTO_VERSION}/protobuf-python-${PROTO_VERSION}.tar.gz \
&& tar -xf protobuf-python-${PROTO_VERSION}.tar.gz \
&& cd /protobuf-${PROTO_VERSION}/python \
&& cd protobuf-${PROTO_VERSION}/python \
&& python2 setup.py install \
&& python3 setup.py install \
&& cd ../../ \
&& rm -rf /protobuf-${PROTO_VERSION} protobuf-python-${PROTO_VERSION}.tar.gz

# Install rust
ENV PATH "/root/.cargo/bin:$PATH"
RUN curl --proto '=https' --tlsv1.2 --tls-max 1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable -y && \
cargo --help

# Install go
ENV GOPATH /root/go
ENV GO_VERSION 1.15.5
ENV GO_ARCH amd64
RUN wget -q https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${GO_ARCH}.tar.gz; \
tar -C /usr/local/ -xf /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz ; \
rm /go${GO_VERSION}.linux-${GO_ARCH}.tar.gz
ENV PATH /usr/local/go/bin:$PATH
ENV PROJECT_DIR $GOPATH/src/github.com/influxdata/kapacitor
ENV PKG_CONFIG $PROJECT_DIR/pkg-config.sh
#ENV PKG_CONFIG $PROJECT_DIR/pkg-config.sh
ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR

VOLUME $PROJECT_DIR
VOLUME /root/go/src
VOLUME /go/src

# Configure local git
RUN git config --global user.email "[email protected]"
RUN git config --global user.Name "Docker Builder"

ENTRYPOINT [ "/root/go/src/github.com/influxdata/kapacitor/build.py" ]
ENTRYPOINT [ "/go/src/github.com/influxdata/kapacitor/build.py" ]
62 changes: 31 additions & 31 deletions build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@

supported_builds = {
'darwin': [ "amd64" ],
'linux': [ "amd64", "i386", "armhf", "arm64", "armel", "static_i386", "static_amd64" ],
'windows': [ "amd64", "i386" ]
'linux': [ "arm64", "amd64" ],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did arm64 before amd64 because it is more likely to fail

'windows': [ "amd64" ]
}

supported_packages = {
Expand Down Expand Up @@ -158,7 +158,11 @@ def run_generate():
"""Run 'go generate' to rebuild any static assets.
"""
logging.info("Running generate...")
run("go install -mod=mod github.com/golang/protobuf/protoc-gen-go github.com/benbjohnson/tmpl github.com/mailru/easyjson/easyjson")
run("""go install -mod=mod
github.com/golang/protobuf/protoc-gen-go \
github.com/benbjohnson/tmpl \
github.com/mailru/easyjson/easyjson \
github.com/influxdata/pkg-config""")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case the kapa has a newer version of pkg-config it wants.

try:
subprocess.check_output(["go", "generate", "./..."])
except subprocess.CalledProcessError as exc:
Expand Down Expand Up @@ -382,13 +386,11 @@ def get_system_arch():
arch = os.uname()[4]
if arch == "x86_64":
arch = "amd64"
elif arch == "386":
arch = "i386"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing 32 bit support (sorry!), because flux doesn't support 32 bit.

elif arch == "aarch64":
arch = "arm64"
elif 'arm' in arch:
# Prevent uname from reporting full ARM arch (eg 'armv7l')
arch = "arm"
arch = "arm64"
return arch

def get_system_platform():
Expand Down Expand Up @@ -491,6 +493,7 @@ def build(version=None,
nightly=False,
race=False,
clean=False,
cc="",
outdir=".",
tags=[],
static=False):
Expand Down Expand Up @@ -522,42 +525,39 @@ def build(version=None,
logging.info("Building target: {}".format(target))
build_command = ""

# Handle static binary output
if static is True or "static_" in arch:
if "static_" in arch:
static = True
arch = arch.replace("static_", "")
build_command += "CGO_ENABLED=0 "
build_command += "CGO_ENABLED=1 "
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We always need cgo now.


# Handle variations in architecture output
fullarch = arch
if arch == "i386" or arch == "i686":
arch = "386"
elif arch == "aarch64" or arch == "arm64":
if arch == "aarch64" or arch == "arm64":
arch = "arm64"
elif "arm" in arch:
arch = "arm"
build_command += "GOOS={} GOARCH={} ".format(platform, arch)

if platform == "linux":
if arch == "amd64":
tags += ["netgo", "osusergo", "static_build"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are defaulting to static building now, because it eliminates a whole class of problems

if arch == "arm64":
cc = "aarch64-unknown-linux-musl-cc"
tags += ["netgo", "osusergo", "static_build", "noasm"]
elif platform == "darwin" and arch == "amd64":
cc = "x86_64-apple-darwin15-clang"
tags += [ "netgo", "osusergo"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't build static on mac.

elif platform == "windows" and arch == "amd64":
cc = "x86_64-w64-mingw32-gcc"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While Influx doesn't officially support the windows build, it would be awesome if someone wanted to test it for me (I don't have a windows machine).

build_command += "CC={} GOOS={} GOARCH={} ".format(cc, platform, arch)

if "arm" in fullarch:
if fullarch == "armel":
build_command += "GOARM=5 "
elif fullarch == "armhf" or fullarch == "arm":
build_command += "GOARM=6 "
elif fullarch == "arm64":
# GOARM not used - see https://github.com/golang/go/wiki/GoArm
pass
else:
logging.error("Invalid ARM architecture specified: {}".format(arch))
logging.error("Please specify either 'armel', 'armhf', or 'arm64'.")
if fullarch != "arm64":
logging.error("Invalid ARM architecture specified: {} only arm64 is supported".format(arch))
return False
if platform == 'windows':
target = target + '.exe'
build_command += "go build -o {} ".format(os.path.join(outdir, target))
build_command += "go build -buildmode=exe -o {} ".format(os.path.join(outdir, target))
else:
build_command += "go build -o {} ".format(os.path.join(outdir, target))
if race:
build_command += "-race "
if len(tags) > 0:
build_command += "-tags {} ".format(','.join(tags))
build_command += "-tags \"{}\" ".format(' '.join(tags))
if "1.4" in get_go_version():
if static:
build_command += "-ldflags=\"-s -X main.version {} -X main.branch {} -X main.commit {} -X main.platform OSS\" ".format(version,
Expand Down Expand Up @@ -913,7 +913,7 @@ def main(args):
type=str,
help='Name to use for package name (when package is specified)')
parser.add_argument('--arch',
metavar='<amd64|i386|armhf|arm64|armel|all>',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No more 32 bit support :-(

metavar='<amd64|arm64|all>',
type=str,
default=get_system_arch(),
help='Target architecture for build output')
Expand Down
13 changes: 8 additions & 5 deletions build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,24 @@ BUILD_NUM=${BUILD_NUM-$RANDOM}
# Home dir of the docker user
HOME_DIR=/root

GO_VERSION=1.15.10

imagename=kapacitor-builder-img-$BUILD_NUM
dataname=kapacitor-data-$BUILD_NUM

# Build new docker image
docker build -f Dockerfile_build_ubuntu64 -t $imagename $DIR
docker build -f Dockerfile_build_ubuntu64 -t $imagename --build-arg GO_VERSION=${GO_VERSION} $DIR

# Build new docker image
docker build -f Dockerfile_build_ubuntu64 -t influxdata/kapacitor-builder $DIR
docker build -f Dockerfile_build_ubuntu64 -t influxdata/kapacitor-builder --build-arg GO_VERSION=${GO_VERSION} $DIR

# Create data volume with code
docker create \
--name $dataname \
-v "$HOME_DIR/go/src/github.com/influxdata/kapacitor" \
-v "/go/src/github.com/influxdata/kapacitor" \
$imagename /bin/true
docker cp "$DIR/" "$dataname:$HOME_DIR/go/src/github.com/influxdata/"

docker cp "$DIR/" "$dataname:/go/src/github.com/influxdata/"

echo "Running build.py"
# Run docker
Expand All @@ -38,6 +41,6 @@ docker run \
$imagename \
"$@"

docker cp "$dataname:$HOME_DIR/go/src/github.com/influxdata/kapacitor/build" \
docker cp "$dataname:/go/src/github.com/influxdata/kapacitor/build" \
./
docker rm -v $dataname
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ require (
github.com/influxdata/httprouter v1.3.1-0.20191122104820-ee83e2772f69
github.com/influxdata/influxdb v1.8.4
github.com/influxdata/influxdb/v2 v2.0.1-alpha.10.0.20210507184756-dc72dc3f0c07
github.com/influxdata/pkg-config v0.2.7
github.com/influxdata/usage-client v0.0.0-20160829180054-6d3895376368
github.com/influxdata/wlog v0.0.0-20160411224016-7c63b0a71ef8
github.com/k-sone/snmpgo v3.2.0+incompatible
Expand Down
Loading