Leaking/bloating memory for v1

[bradleyd]

We have started using SQS in production and noticed that our consumers memory is growing unbound. We have tried it on ruby 2.0,2.1.5, 2.0, rbx-2.5.2 and the results are the same. I provided a test script to show what we are doing. The memory goes to 90MB within 1 hour. This is with no messages in queue..just polling.
If the daemons are left unchecked the memory will keep on growing until the box is out of memory.

Here is the script

gem 'aws-sdk', '~> 1.0'

require 'aws-sdk'

AWS.config(:sqs_region => ENV['AWS_REGION'] || 'us-west-1')

sqs = AWS::SQS.new(
        :access_key_id => ENV["ACCESS_KEY"],
        :secret_access_key => ENV["SECRET_KEY"])

queue = sqs.queues[ENV["SQS_URL"]]

p dbg: sqs
puts "polling for messages"
queue.poll(wait_time_seconds: 10) { |m| p m }

Gem list

*** LOCAL GEMS ***

aws-sdk (1.62.0, 1.61.0)
aws-sdk-v1 (1.62.0, 1.61.0)
bigdecimal (1.2.4)
bundler (1.8.1)
daedalus-core (0.2.0)
ewes-db (0.0.1)
ewes-mailmessage (0.0.1)
ewes-sqs (0.0.1)
io-console (0.4.2)
json (1.8.2, 1.8.1)
mini_portile (0.6.2)
minitest (4.7.5)
nokogiri (1.6.6.2)
pg (0.18.1)
psych (2.0.5)
rake (10.4.2, 10.1.0)
rdoc (4.1.0)
redcard (1.1.0)
rubinius-ast (2.3.1)
rubinius-bridge (1.1.0)
rubinius-build_tools (2.0.0)
rubinius-compiler (2.3.1)
rubinius-melbourne (2.3.1.0)
rubinius-processor (2.3.0)
rubinius-toolset (2.3.1)
sequel (4.19.0)
test-unit (2.1.5.0)

Not sure where to pivot from here..any help is greatly appreciated.

[trevorrowe]

I'll see what I can do about reproducing this locally. Can you run the same test in Ruby 1.9.3?

[bradleyd]

will do.

[bradleyd]

Same issue. No messages and it keeps growing. It has been running for 10 minutes. Starts at 14MB.

Here is a memory map of the process.

Thanks for the help!

Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000       4       4       0 r-x-- ruby1.9.1
0000000000600000       4       4       4 r---- ruby1.9.1
0000000000601000       4       4       4 rw--- ruby1.9.1
00000000007f8000   44436   44264   44264 rw---   [ anon ]
00007fbfdc000000     132       4       4 rw---   [ anon ]
00007fbfdc021000   65404       0       0 -----   [ anon ]
00007fbfe2c67000      92      12       0 r-x-- CP932.so
00007fbfe2c7e000    2044       0       0 ----- CP932.so
00007fbfe2e7d000       4       4       4 r---- CP932.so
00007fbfe2e7e000       4       4       4 rw--- CP932.so
00007fbfe2e7f000      16      16       0 r-x-- cparse.so
00007fbfe2e83000    2044       0       0 ----- cparse.so
00007fbfe3082000       4       4       4 r---- cparse.so
00007fbfe3083000       4       4       4 rw--- cparse.so
00007fbfe3084000    2148     468       0 r-x-- nokogiri.so
00007fbfe329d000    2044       0       0 ----- nokogiri.so
00007fbfe349c000      32      32      32 r---- nokogiri.so
00007fbfe34a4000      12      12      12 rw--- nokogiri.so
00007fbfe34a7000       4       4       4 rw---   [ anon ]
00007fbfe34a8000      92      52       0 r-x-- libresolv-2.19.so
00007fbfe34bf000    2048       0       0 ----- libresolv-2.19.so
00007fbfe36bf000       4       4       4 r---- libresolv-2.19.so
00007fbfe36c0000       4       4       4 rw--- libresolv-2.19.so
00007fbfe36c1000       8       4       4 rw---   [ anon ]
00007fbfe36c3000      20      12       0 r-x-- libnss_dns-2.19.so
00007fbfe36c8000    2044       0       0 ----- libnss_dns-2.19.so
00007fbfe38c7000       4       4       4 r---- libnss_dns-2.19.so
00007fbfe38c8000       4       4       4 rw--- libnss_dns-2.19.so
00007fbfe38c9000      44      24       0 r-x-- libnss_files-2.19.so
00007fbfe38d4000    2044       0       0 ----- libnss_files-2.19.so
00007fbfe3ad3000       4       4       4 r---- libnss_files-2.19.so
00007fbfe3ad4000       4       4       4 rw--- libnss_files-2.19.so
00007fbfe3ad5000    1028      12      12 rw---   [ anon ]
00007fbfe3bd6000      60      32       0 r-x-- bigdecimal.so
00007fbfe3be5000    2044       0       0 ----- bigdecimal.so
00007fbfe3de4000       4       4       4 r---- bigdecimal.so
00007fbfe3de5000       4       4       4 rw--- bigdecimal.so
00007fbfe3de6000      28      20       0 r-x-- generator.so
00007fbfe3ded000    2048       0       0 ----- generator.so
00007fbfe3fed000       4       4       4 r---- generator.so
00007fbfe3fee000       4       4       4 rw--- generator.so
00007fbfe3fef000       4       4       0 r-x-- utf_32le.so
00007fbfe3ff0000    2044       0       0 ----- utf_32le.so
00007fbfe41ef000       4       4       4 r---- utf_32le.so
00007fbfe41f0000       4       4       4 rw--- utf_32le.so
00007fbfe41f1000       4       4       0 r-x-- utf_32be.so
00007fbfe41f2000    2044       0       0 ----- utf_32be.so
00007fbfe43f1000       4       4       4 r---- utf_32be.so
00007fbfe43f2000       4       4       4 rw--- utf_32be.so
00007fbfe43f3000       4       4       0 r-x-- utf_16le.so
00007fbfe43f4000    2048       0       0 ----- utf_16le.so
00007fbfe45f4000       4       4       4 r---- utf_16le.so
00007fbfe45f5000       4       4       4 rw--- utf_16le.so
00007fbfe45f6000       4       4       0 r-x-- utf_16be.so
00007fbfe45f7000    2048       0       0 ----- utf_16be.so
00007fbfe47f7000       4       4       4 r---- utf_16be.so
00007fbfe47f8000       4       4       4 rw--- utf_16be.so
00007fbfe47f9000      24      20       0 r-x-- parser.so
00007fbfe47ff000    2044       0       0 ----- parser.so
00007fbfe49fe000       4       4       4 r---- parser.so
00007fbfe49ff000       4       4       4 rw--- parser.so
00007fbfe4a00000     184      36       0 r-x-- date_core.so
00007fbfe4a2e000    2048       0       0 ----- date_core.so
00007fbfe4c2e000       4       4       4 r---- date_core.so
00007fbfe4c2f000       4       4       4 rw--- date_core.so
00007fbfe4c30000       4       0       0 rw---   [ anon ]
00007fbfe4c31000      20      16       0 r-x-- strscan.so
00007fbfe4c36000    2044       0       0 ----- strscan.so
00007fbfe4e35000       4       4       4 r---- strscan.so
00007fbfe4e36000       4       4       4 rw--- strscan.so
00007fbfe4e37000     124      80       0 r-x-- libyaml-0.so.2.0.2
00007fbfe4e56000    2044       0       0 ----- libyaml-0.so.2.0.2
00007fbfe5055000       4       4       4 r---- libyaml-0.so.2.0.2
00007fbfe5056000       4       4       4 rw--- libyaml-0.so.2.0.2
00007fbfe5057000      24      20       0 r-x-- psych.so
00007fbfe505d000    2044       0       0 ----- psych.so
00007fbfe525c000       4       4       4 r---- psych.so
00007fbfe525d000       4       4       4 rw--- psych.so
00007fbfe525e000      24      16       0 r-x-- pathname.so
00007fbfe5264000    2044       0       0 ----- pathname.so
00007fbfe5463000       4       4       4 r---- pathname.so
00007fbfe5464000       4       4       4 rw--- pathname.so
00007fbfe5465000      96       8       0 r-x-- libz.so.1.2.8
00007fbfe547d000    2044       0       0 ----- libz.so.1.2.8
00007fbfe567c000       4       4       4 r---- libz.so.1.2.8
00007fbfe567d000       4       4       4 rw--- libz.so.1.2.8
00007fbfe567e000      48      24       0 r-x-- zlib.so
00007fbfe568a000    2044       0       0 ----- zlib.so
00007fbfe5889000       4       4       4 r---- zlib.so
00007fbfe588a000       4       4       4 rw--- zlib.so
00007fbfe588b000     132     100       0 r-x-- socket.so
00007fbfe58ac000    2048       0       0 ----- socket.so
00007fbfe5aac000       4       4       4 r---- socket.so
00007fbfe5aad000       4       4       4 rw--- socket.so
00007fbfe5aae000       4       4       0 r-x-- fcntl.so
00007fbfe5aaf000    2044       0       0 ----- fcntl.so
00007fbfe5cae000       4       4       4 r---- fcntl.so
00007fbfe5caf000       4       4       4 rw--- fcntl.so
00007fbfe5cb0000      24      20       0 r-x-- stringio.so
00007fbfe5cb6000    2044       0       0 ----- stringio.so
00007fbfe5eb5000       4       4       4 r---- stringio.so
00007fbfe5eb6000       4       4       4 rw--- stringio.so
00007fbfe5eb7000      12      12       0 r-x-- digest.so
00007fbfe5eba000    2044       0       0 ----- digest.so
00007fbfe60b9000       4       4       4 r---- digest.so
00007fbfe60ba000       4       4       4 rw--- digest.so
00007fbfe60bb000    1732     976       0 r-x-- libcrypto.so.1.0.0
00007fbfe626c000    2044       0       0 ----- libcrypto.so.1.0.0
00007fbfe646b000     108     108     108 r---- libcrypto.so.1.0.0
00007fbfe6486000      44      44      44 rw--- libcrypto.so.1.0.0
00007fbfe6491000      16      16      16 rw---   [ anon ]
00007fbfe6495000     336     220       0 r-x-- libssl.so.1.0.0
00007fbfe64e9000    2048       0       0 ----- libssl.so.1.0.0
00007fbfe66e9000      12      12      12 r---- libssl.so.1.0.0
00007fbfe66ec000      28      28      28 rw--- libssl.so.1.0.0
00007fbfe66f3000     292     224       0 r-x-- openssl.so
00007fbfe673c000    2048       0       0 ----- openssl.so
00007fbfe693c000       4       4       4 r---- openssl.so
00007fbfe693d000       8       8       8 rw--- openssl.so
00007fbfe693f000       4       4       4 rw---   [ anon ]
00007fbfe6940000       8       8       0 r-x-- transdb.so
00007fbfe6942000    2048       0       0 ----- transdb.so
00007fbfe6b42000       4       4       4 r---- transdb.so
00007fbfe6b43000       4       4       4 rw--- transdb.so
00007fbfe6b44000       8       8       0 r-x-- encdb.so
00007fbfe6b46000    2044       0       0 ----- encdb.so
00007fbfe6d45000       4       4       4 r---- encdb.so
00007fbfe6d46000       4       4       4 rw--- encdb.so
00007fbfe6d47000    2852      48       0 r---- locale-archive
00007fbfe7010000    1044      64       0 r-x-- libm-2.19.so
00007fbfe7115000    2044       0       0 ----- libm-2.19.so
00007fbfe7314000       4       4       4 r---- libm-2.19.so
00007fbfe7315000       4       4       4 rw--- libm-2.19.so
00007fbfe7316000      36       4       0 r-x-- libcrypt-2.19.so
00007fbfe731f000    2048       0       0 ----- libcrypt-2.19.so
00007fbfe751f000       4       4       4 r---- libcrypt-2.19.so
00007fbfe7520000       4       4       4 rw--- libcrypt-2.19.so
00007fbfe7521000     184       0       0 rw---   [ anon ]
00007fbfe754f000      12       8       0 r-x-- libdl-2.19.so
00007fbfe7552000    2044       0       0 ----- libdl-2.19.so
00007fbfe7751000       4       4       4 r---- libdl-2.19.so
00007fbfe7752000       4       4       4 rw--- libdl-2.19.so
00007fbfe7753000      28      16       0 r-x-- librt-2.19.so
00007fbfe775a000    2044       0       0 ----- librt-2.19.so
00007fbfe7959000       4       4       4 r---- librt-2.19.so
00007fbfe795a000       4       4       4 rw--- librt-2.19.so
00007fbfe795b000     100      76       0 r-x-- libpthread-2.19.so
00007fbfe7974000    2044       0       0 ----- libpthread-2.19.so
00007fbfe7b73000       4       4       4 r---- libpthread-2.19.so
00007fbfe7b74000       4       4       4 rw--- libpthread-2.19.so
00007fbfe7b75000      16       4       4 rw---   [ anon ]
00007fbfe7b79000    1772     656       0 r-x-- libc-2.19.so
00007fbfe7d34000    2048       0       0 ----- libc-2.19.so
00007fbfe7f34000      16      16      16 r---- libc-2.19.so
00007fbfe7f38000       8       8       8 rw--- libc-2.19.so
00007fbfe7f3a000      20      16      16 rw---   [ anon ]
00007fbfe7f3f000    2024    1476       0 r-x-- libruby-1.9.1.so.1.9.1
00007fbfe8139000    2044       0       0 ----- libruby-1.9.1.so.1.9.1
00007fbfe8338000      20      20      20 r---- libruby-1.9.1.so.1.9.1
00007fbfe833d000      16      16      16 rw--- libruby-1.9.1.so.1.9.1
00007fbfe8341000     112     108     108 rw---   [ anon ]
00007fbfe835d000     140     120       0 r-x-- ld-2.19.so
00007fbfe83e5000      28      28       0 r--s- gconv-modules.cache
00007fbfe83ec000       4       0       0 -----   [ anon ]
00007fbfe83ed000    1560      60      60 rw---   [ anon ]
00007fbfe8579000       4       0       0 -----   [ anon ]
00007fbfe857a000      20      16      16 rw---   [ anon ]
00007fbfe857f000       4       4       4 r---- ld-2.19.so
00007fbfe8580000       4       4       4 rw--- ld-2.19.so
00007fbfe8581000       4       4       4 rw---   [ anon ]
00007fff34143000     136      48      48 rw---   [ stack ]
00007fff34175000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB          200976   50076   45128

[ph]

I've been trying to reproduce a similar issue in the logstash-input-sqs plugin for logstash 1.4.2.
From what I understand it seems to be a leak in the sqs ``#receive_message` method.

If we let the process run for more than a few hours, all the memory available in the JVM will be consumed and
the performance will start to drop because the garbage collector will kick really often to clean the process.
The JVM never go OOM in the test, we only see a huge performance drop events processing.

I am able to consistently reproduce the issue with the 1.35 version of the aws-sdk and the latest 1.63.

Scenario:

Logstash is running on JRuby 1.7.11

1. Used a large ec2 instance for the tests to replicate the issue.
2. Create two SQS queue using the default settings, assume we have an infinite number of evens in the queues.
3. Create two SQS input with a settings of 20 workers per threads (Most people don't go that far with the workers, this is really an empiric settings.)
4. After 1h30 the events rate will drop from 400 events per seconds to 80 events per seconds.

If I look at some visualvm graph we will see this kind of behavior on the memory and the GC process.

I have a also have a thread dump of the JVM.

Anything else I could provide to help debug this issue?

[trevorrowe]

Watching Ruby issue: https://bugs.ruby-lang.org/issues/10942. This appears not to be an issue in other versions of Ruby, might be a Ruby 2.2 bug.

[bradleyd]

fwiw, I see the same behavior in 1.9 and 2.0. Though it is not as profound.

[pickhardt]

I found this issue because I suspect that the aws gem is leaking memory. Anybody know if upgrading to v2 of the aws gem fixes this?

[smeyfroi]

I started seeing a significant memory leak after upgrading to v2, but I also upgraded to ruby2.2 and a bunch of other stuff at the same time, so take that observation with a pinch of salt. It's on my list to investigate further.

[bradleyd]

There appears to be movement on this.

[trevorrowe]

Yup, it appears to have been fixed in trunk. I suspect this will go out with the next release of Ruby and will be back-ported into supported version. For users looking to work around this issue, see this related issue: #785

[trevorrowe]

I should add that the work-around is given as an example to the v2 SDK, modifications would be required to make this work in the v1 SDK.

[bradleyd]

gotcha..thanks!