Java VM for Docker allocates 1GB memory even if less is specified, unless bootstrap.memory_lock=false

[pszabop]

Describe the bug
The Java VM for Docker allocates 1GB of memory if bootstrap.memory_lock=true no matter the settings for "OPENSEARCH_JAVA_OPTS=-Xms128m -Xmx128m"

To Reproduce
Steps to reproduce the behavior:

1. Use a t2.micro or other machine with only 1GB of memory
2. run

docker run -d -p 9200:9200 -p 9600:9600 -e "bootstrap.memory_lock=true" -e "OPENSEARCH_JAVA_OPTS=-Xms128m -Xmx128m" -e "discovery.type=single-node" opensearchproject/opensearch:latest

3. You will get a message of an error when attempting to allocate 1073741824 bytes of memory. This is far larger than the memory specified under OPENSEARCH_JAVA_OPTS above

[Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory]

4. If you set bootstrap.memory_lock=false the error doesn't occur

Expected behavior
The bootstrap should allocate no more memory than OPENSEARCH_JAVA_OPTS specifies

Plugins
None

Screenshots
See error messages noted above

Host/Environment (please complete the following information):
AWS t2.micro using recent wizard
Linux 5.10.157-139.675.amzn2.x86_64 #1 SMP Thu Dec 8 01:29:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Docker version 20.10.17, build 100c701
Docker Compose version v2.15.1

Additional context
docker-compose fails without even the above message. Separate issue was filed.

I'm using a static index for a small-ish document (KJV Bible), and there aren't really good instructions for running opensearch in a small VM. If this is a weird corner case for you, feel free to close, but posting the workaround here for small VMs may be useful to others

[saratvemulapalli]

Feels like a bug. The recommendation is to have half of instance's memory allocated for OpenSearch[1] but looks like OPENSEARCH_JAVA_OPTS is not really configuring the memory with JVM.

[1] https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/

[dbwiddis]

The error shows up when memory is locked because that prevents paging to expand the addressable space. The same memory is likely being requested with memory_lock false, but results in paging.

Note that -Xmx is a limit for heap memory, while the error message calls out nmap which is direct mapped native memory. I believe limiting the permgen space could help here. -XX:MaxPermSize=512m is the older version, in newer JDK's it is -XX:MaxMetaspaceSize=512m.

See the "Class Metadata" section here

I'll test this a bit, as well as look into the minimum requirements and we should definitely update our documentation to include this information.

[dbwiddis]

Hey @pszabop I've done various experiments over the weekend and narrowed it down to the JVM's Compressed Class Space. The space is "reserved" but not used, which when combined with bootstrap.memory_lock=true makes a hard requirement to actually allocate this memory.

You can use the -XX:CompressedClassSpaceSize command line switch (in the OPENSEARCH_JAVA_OPTS parameter) to reduce this value. It's a subset of the metaspace size mentioned in my previous comment, so limiting that could also help. It doesn't look like either of these needs to be much larger than your heap in any case.

Let me know if this works for you and I'll look for ways to improve our documentation.

[dbwiddis]

Closing this as it's been documented in opensearch-project/documentation-website#3266