OOM/Heap Overflow: cli/sidecar ct gets node resources instead of limit/requests #18540
Labels
kind/bug
Outline of a bug - must adhere to the bug report template.
lifecycle/stale
Denotes an issue or PR has remained open with no activity and has become stale.
status/analyzing
An issue has been proposed and it is currently being analyzed for effort and implementation approach
status/duplicate
Issue identified as a duplicate of another issue
Comments
gattytto
added
the
kind/bug
7 tasks
che-bot
added
the
status/need-triage
ibuziuk
added
status/duplicate
ibuziuk
added
the
status/analyzing
|
this issue persists in che osio after setting the limits in devfile, so cabal build tool is still getting the wrong specs? locally the same happens but I don't have any method host-side to kill the overprovisioned processes so it goes on, uses what it needs (meaning it bypasses namespace limits) and finishes the build process. But maybe there is a need of some bind mounts for cpu/meminfo to inform ct's apps about the actually available resources. [INFO] Consulting the cradle for "test/AppSpec.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/projects/src/github.com/che-samples/haskell-demos/example-servant-minimal", cradleOptsProg = CradleAction: Cabal}
> Warning: The package list for 'hackage.haskell.org' is 25 days old.
> Run 'cabal update' to get the latest list of available packages.
> Resolving dependencies...
> Build profile: -w ghc-8.10.2 -O1
> In order, the following will be built (use -v for more details):
> - asn1-parse-0.9.5 (lib:asn1-parse) (requires build)
> - hspec-core-2.7.4 (lib) (requires download & build)
> - hspec-discover-2.7.4 (lib) (requires download & build)
> - network-3.1.2.0 (lib:network) (requires build)
> - servant-client-core-0.18.1 (lib) (requires download & build)
> - unix-time-0.4.7 (lib:unix-time) (requires build)
> - x509-1.7.5 (lib) (requires build)
> - hspec-2.7.4 (lib) (requires download & build)
> - wai-3.2.2.1 (lib) (requires build)
> - streaming-commons-0.2.2.1 (lib) (requires build)
> - simple-sendfile-0.2.30 (lib) (requires build)
> - iproute-1.7.9 (lib) (requires build)
> - http2-2.0.5 (lib) (requires build)
> - fast-logger-3.0.2 (lib) (requires build)
> - http-client-0.6.4.1 (lib) (requires download & build)
> - warp-3.3.13 (lib) (requires build)
> - wai-logger-2.3.6 (lib:wai-logger) (requires build)
> - servant-client-0.18.1 (lib) (requires download & build)
> - wai-extra-3.1.2 (lib) (requires build)
> - wai-app-static-3.1.7.2 (lib) (requires build)
> - servant-server-0.18.1 (lib) (requires build)
> - example-servant-minimal-0.0.0 (lib) (first run)
> - example-servant-minimal-0.0.0 (test:spec) (first run)
> Downloading hspec-discover-2.7.4
> Starting asn1-parse-0.9.5 (all, legacy fallback)
> Downloaded hspec-discover-2.7.4
> Downloading http-client-0.6.4.1
> Downloaded http-client-0.6.4.1
> Downloading hspec-core-2.7.4
> Downloaded hspec-core-2.7.4
> Downloading hspec-2.7.4
> Downloaded hspec-2.7.4
> Downloading servant-client-core-0.18.1
> Starting hspec-discover-2.7.4 (lib)
> Building asn1-parse-0.9.5 (all, legacy fallback)
> Downloaded servant-client-core-0.18.1
> Downloading servant-client-0.18.1
> Building hspec-discover-2.7.4 (lib)
> Downloaded servant-client-0.18.1
> Installing asn1-parse-0.9.5 (all, legacy fallback)
> Starting network-3.1.2.0 (all, legacy fallback)
> Completed asn1-parse-0.9.5 (all, legacy fallback)
> Starting hspec-core-2.7.4 (lib)
> Starting servant-client-core-0.18.1 (lib)
> Building hspec-core-2.7.4 (lib)
> Installing hspec-discover-2.7.4 (lib)
> Completed hspec-discover-2.7.4 (lib)
> Building servant-client-core-0.18.1 (lib)
> Starting unix-time-0.4.7 (all, legacy fallback)
> Installing servant-client-core-0.18.1 (lib)
> Completed servant-client-core-0.18.1 (lib)
> Building unix-time-0.4.7 (all, legacy fallback)
> Starting x509-1.7.5 (lib)
> Building x509-1.7.5 (lib)
> Building network-3.1.2.0 (all, legacy fallback)
> Installing unix-time-0.4.7 (all, legacy fallback)
> Completed unix-time-0.4.7 (all, legacy fallback)
> Installing hspec-core-2.7.4 (lib)
> Completed hspec-core-2.7.4 (lib)
> Installing x509-1.7.5 (lib)
> Completed x509-1.7.5 (lib)
>
> Failed to build network-3.1.2.0.
> Build log (
> /home/theia/.cabal/logs/ghc-8.10.2/network-3.1.2.0-f313f458cfb9472e49923061663feda9781129bd7a55b5c51b8bf13fb9eff621.log
> ):
> Configuring network-3.1.2.0...
> configure: WARNING: unrecognized options: --with-compiler
> checking build system type... x86_64-pc-linux-gnu
> checking host system type... x86_64-pc-linux-gnu
> checking for gcc... /usr/bin/cc
> checking whether the C compiler works... yes
> checking for C compiler default output file name... a.out
> checking for suffix of executables...
> checking whether we are cross compiling... no
> checking for suffix of object files... o
> checking whether we are using the GNU C compiler... yes
> checking whether /usr/bin/cc accepts -g... yes
> checking for /usr/bin/cc option to accept ISO C89... none needed
> checking for an ANSI C-conforming const... yes
> checking how to run the C preprocessor... /usr/bin/cc -E
> checking for grep that handles long lines and -e... /bin/grep
> checking for egrep... /bin/grep -E
> checking for ANSI C header files... yes
> checking for sys/types.h... yes
> checking for sys/stat.h... yes
> checking for stdlib.h... yes
> checking for string.h... yes
> checking for memory.h... yes
> checking for strings.h... yes
> checking for inttypes.h... yes
> checking for stdint.h... yes
> checking for unistd.h... yes
> checking limits.h usability... yes
> checking limits.h presence... yes
> checking for limits.h... yes
> checking for stdlib.h... (cached) yes
> checking for unistd.h... (cached) yes
> checking for sys/types.h... (cached) yes
> checking fcntl.h usability... yes
> checking fcntl.h presence... yes
> checking for fcntl.h... yes
> checking sys/uio.h usability... yes
> checking sys/uio.h presence... yes
> checking for sys/uio.h... yes
> checking sys/socket.h usability... yes
> checking sys/socket.h presence... yes
> checking for sys/socket.h... yes
> checking netinet/in.h usability... yes
> checking netinet/in.h presence... yes
> checking for netinet/in.h... yes
> checking netinet/tcp.h usability... yes
> checking netinet/tcp.h presence... yes
> checking for netinet/tcp.h... yes
> checking sys/un.h usability... yes
> checking sys/un.h presence... yes
> checking for sys/un.h... yes
> checking arpa/inet.h usability... yes
> checking arpa/inet.h presence... yes
> checking for arpa/inet.h... yes
> checking netdb.h usability... yes
> checking netdb.h presence... yes
> checking for netdb.h... yes
> cabal: Failed to build network-3.1.2.0 (which is required by test:spec from
> example-servant-minimal-0.0.0). See the build log above for details. |
|
cabal build gives more memory related kind of error (again, this in osio with cpulimits in devfile and memory limits in meta.yaml spec:
containers:
- image: quay.io/eclipse/che-plugin-sidecar:haskell-b64039f
name: vscode-haskell
memoryLimit: 3072Mi$ cabal build
Warning: The package list for 'hackage.haskell.org' is 25 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following will be built (use -v for more details):
- aeson-1.5.4.1 (lib) (requires build)
- blaze-markup-0.8.2.7 (lib) (requires build)
- fast-logger-3.0.2 (lib) (requires build)
- http-api-data-0.4.2 (lib) (requires build)
- http-media-0.8.0.0 (lib) (requires build)
- http2-2.0.5 (lib) (requires build)
- iproute-1.7.9 (lib) (requires build)
- memory-0.15.0 (lib) (requires build)
- simple-sendfile-0.2.30 (lib) (requires build)
- singleton-bool-0.1.5 (lib) (requires build)
- streaming-commons-0.2.2.1 (lib) (requires build)
- string-conversions-0.4.0.1 (lib) (requires build)
- wai-3.2.2.1 (lib) (requires build)
- blaze-html-0.9.1.2 (lib) (requires build)
- pem-0.2.4 (lib) (requires build)
- cryptonite-0.27 (lib) (requires build)
- asn1-types-0.3.4 (lib:asn1-types) (requires build)
- servant-0.18.1 (lib) (requires build)
- wai-logger-2.3.6 (lib:wai-logger) (requires build)
- asn1-encoding-0.9.6 (lib) (requires build)
- wai-extra-3.1.2 (lib) (requires build)
- asn1-parse-0.9.5 (lib:asn1-parse) (requires build)
- x509-1.7.5 (lib) (requires build)
- warp-3.3.13 (lib) (requires build)
- wai-app-static-3.1.7.2 (lib) (requires build)
- servant-server-0.18.1 (lib) (requires build)
- example-servant-minimal-0.0.0 (lib) (first run)
- example-servant-minimal-0.0.0 (exe:example-servant-minimal) (first run)
Starting http-api-data-0.4.2 (lib)
Starting blaze-markup-0.8.2.7 (lib)
Starting fast-logger-3.0.2 (lib)
Building blaze-markup-0.8.2.7 (lib)
Building http-api-data-0.4.2 (lib)
Building fast-logger-3.0.2 (lib)
Installing fast-logger-3.0.2 (lib)
Completed fast-logger-3.0.2 (lib)
Installing blaze-markup-0.8.2.7 (lib)
Completed blaze-markup-0.8.2.7 (lib)
Installing http-api-data-0.4.2 (lib)
Completed http-api-data-0.4.2 (lib)
Starting http-media-0.8.0.0 (lib)
Building http-media-0.8.0.0 (lib)
Starting memory-0.15.0 (lib)
Building memory-0.15.0 (lib)
Installing http-media-0.8.0.0 (lib)
Completed http-media-0.8.0.0 (lib)
Starting aeson-1.5.4.1 (lib)
Building aeson-1.5.4.1 (lib)
Installing memory-0.15.0 (lib)
Completed memory-0.15.0 (lib)
Starting http2-2.0.5 (lib)
Building http2-2.0.5 (lib)
Failed to build http2-2.0.5.
Build log (
/home/theia/.cabal/logs/ghc-8.10.2/http2-2.0.5-6f7b1faa5ea1d369837e907fb4edd7f2ba1c1d2dd1d010240ed2a81e71eda830.log
):
Configuring library for http2-2.0.5..
Preprocessing library for http2-2.0.5..
Building library for http2-2.0.5..
[ 1 of 45] Compiling Imports ( Imports.hs, dist/build/Imports.o, dist/build/Imports.dyn_o )
[ 2 of 45] Compiling Network.HPACK.Builder ( Network/HPACK/Builder.hs, dist/build/Network/HPACK/Builder.o, dist/build/Network/HPACK/Builder.dyn_o )
[ 3 of 45] Compiling Network.HPACK.HeaderBlock.Integer ( Network/HPACK/HeaderBlock/Integer.hs, dist/build/Network/HPACK/HeaderBlock/Integer.o, dist/build/Network/HPACK/HeaderBlock/Integer.dyn_o )
[ 4 of 45] Compiling Network.HPACK.Huffman.Bit ( Network/HPACK/Huffman/Bit.hs, dist/build/Network/HPACK/Huffman/Bit.o, dist/build/Network/HPACK/Huffman/Bit.dyn_o )
[ 5 of 45] Compiling Network.HPACK.Huffman.ByteString ( Network/HPACK/Huffman/ByteString.hs, dist/build/Network/HPACK/Huffman/ByteString.o, dist/build/Network/HPACK/Huffman/ByteString.dyn_o )
[ 6 of 45] Compiling Network.HPACK.Huffman.Params ( Network/HPACK/Huffman/Params.hs, dist/build/Network/HPACK/Huffman/Params.o, dist/build/Network/HPACK/Huffman/Params.dyn_o )
[ 7 of 45] Compiling Network.HPACK.Huffman.Table ( Network/HPACK/Huffman/Table.hs, dist/build/Network/HPACK/Huffman/Table.o, dist/build/Network/HPACK/Huffman/Table.dyn_o )
[ 8 of 45] Compiling Network.HPACK.Huffman.Encode ( Network/HPACK/Huffman/Encode.hs, dist/build/Network/HPACK/Huffman/Encode.o, dist/build/Network/HPACK/Huffman/Encode.dyn_o )
[ 9 of 45] Compiling Network.HPACK.Huffman.Tree ( Network/HPACK/Huffman/Tree.hs, dist/build/Network/HPACK/Huffman/Tree.o, dist/build/Network/HPACK/Huffman/Tree.dyn_o )
[10 of 45] Compiling Network.HPACK.Token ( Network/HPACK/Token.hs, dist/build/Network/HPACK/Token.o, dist/build/Network/HPACK/Token.dyn_o )
[11 of 45] Compiling Network.HPACK.Types ( Network/HPACK/Types.hs, dist/build/Network/HPACK/Types.o, dist/build/Network/HPACK/Types.dyn_o )
cabal: Failed to build aeson-1.5.4.1 (which is required by
exe:example-servant-minimal from example-servant-minimal-0.0.0). The build
process was killed (i.e. SIGKILL). The typical reason for this is that there
is not enough memory available (e.g. the OS killed a process using lots of
memory).
Failed to build http2-2.0.5 (which is required by exe:example-servant-minimal
from example-servant-minimal-0.0.0). See the build log above for details.
Failed to unpack iproute-1.7.9 (which is required by
exe:example-servant-minimal from example-servant-minimal-0.0.0). The exception
was:
/projects/src/github.com/che-samples/haskell-demo/dist-newstyle/tmp/src-6436/iproute-1.7.9:
createDirectory: does not exist (No such file or directory) |
|
Issues go stale after Mark the issue as fresh with If this issue is safe to close now please do so. Moderators: Add |
che-bot
added
the
lifecycle/stale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
when opening a shell to my haskell sidecar in che osio and issuing the "cat /proc/meminfo" command, I get:
when the value I should see for memory resides in:
meaning: most apps don't distinguish if they are running inside a container and most have the same code base for fetching the available cpu(cores)/memory limits available.
with this background, when running "cabal build" in a project folder with many includes, I get "compiler process got terminated unexpectedly" meaning something "killed" my process for overprovisioning its resources.
####################################LOCAL TEST####################################
#################################### ####################################
in this local test I am using a LXD container to limit overall resources for my kubernetes single-node cluster (6GB ram and 5 cores), cri-o is chosen as the container engine, then set up minikube on top of it and then I set up che using che-operator and the mentioned patches for limiting the resources within the workspaces namespace. The result is as follows:
RESULT: as shown by the screenshot, getting as root into the sidecar container using runc -u 0 to install procps and htop, I can see the sidecar container getting mem/cpu info from the node, instead of the container's resources or even the LXC container limited resources from the host.
running the htop command as non-root user inside the sidecar:
The text was updated successfully, but these errors were encountered: