>4.4.0 <=4.13.0 causing RuntimeError: can't add a new key into hash during iteration

[tardate]

upgrading airbrake in a Rails 3.2.21 (old, yes I know) app caused frequent "RuntimeError: can't add a new key into hash during iteration" errors, usually pinpointing active_record modules e.g. connection_pool.rb:102 or abstract_adapter.rb:71 . Ruby processes also spinning very high CPU

prior to upgrade, app was using these versions without issue:

airbrake (9.2.2)
airbrake-ruby (4.4.0)

initial upgrade that started causing problems updated to these versions:

airbrake (10.0.1)
airbrake-ruby (4.13.0)

downgraded airbrake, but didn't force airbrake-ruby down. Problem persisted:

airbrake (9.2.2)
airbrake-ruby (4.13.0)

after pegging both gem versions back to the original versions (9.2.2, 4.4.0), problem resolved.

I tried to examine the diff, but can't see a smoking gun - but new sql grouping and threading draw my attention: v4.4.0...v4.13.0

 

[kyrylo]

This sounds like a thread safety issue to me. One possible place which may cause this is our rbtree3 dependency. Specifically, this commit: kyrylo/rbtree3@38e7277

I am not sure how exactly this could trigger the issue but the fact is that the error you see is related to hashes and we did make a change regarding hashes.

airbrake-ruby v4.13.0 is the first airbrake-ruby that is on rbtree3 v0.6.0. Could you please try to test again with airbrake (10.0.1) and airbrake-ruby (4.12.0)?

For example:

gem 'airbrake', '~> 10.0'
gem 'airbrake-ruby', '~> 4.12'
gem 'rbtree3', '~> 0.5'

[tardate]

@kyrylo thanks for taking a look. yes threading + rbtree may do it. I'll try with 4.12.0 and feedback

[kyrylo]

Hey @tardate, have you been able to downgrade rbtree yet?

I just released v4.13.1, which includes one important thread-safety fix (specifically, #554). Could you give it a shot?

[tardate]

@kyrylo aiming to schedule a slot on one of the environments where we can reproduce the issue in the next week. Thanks for the tip about v4.13.1, can try that also

[vitalinfo]

does TypeError: can't modify rbtree in iteration problem related to this ticket as well? (v4.13.2)

[kyrylo]

@vitalinfo it does sound related.

Have you tried downgrading rbtree?
Do you have the full backtrace?

[vitalinfo]

Just did downgrade today, will keep on eye on it.
backtraces:
1)

/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:236 in clear
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:236 in reset!
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:131 in compress!
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/stat.rb:32 in to_h
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:176 in block in serialize_resources
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:175 in map
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:175 in serialize_resources
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:167 in block in with_grouped_payload
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:165 in each
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:165 in with_grouped_payload
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:145 in send
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:93 in send_resource
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:37 in notify_sync
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby.rb:504 in notify_queue_sync
/gems/airbrake-10.0.1/lib/airbrake/rails/active_job.rb:40 in perform
/gems/airbrake-10.0.1/lib/airbrake/rails/active_job.rb:53 in block (2 levels) in <module:ActiveJob>
....

/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:358 in []=
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:358 in _digest
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:227 in block in push
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:227 in each
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/tdigest.rb:227 in push
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/stat.rb:60 in increment_ms
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:117 in update_total
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:107 in update_payload
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:91 in send_resource
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby/performance_notifier.rb:37 in notify_sync
/gems/airbrake-ruby-4.13.2/lib/airbrake-ruby.rb:504 in notify_queue_sync
/gems/airbrake-10.0.1/lib/airbrake/rails/active_job.rb:40 in perform
/gems/airbrake-10.0.1/lib/airbrake/rails/active_job.rb:53 in block (2 levels) in <module:ActiveJob>
....

[vitalinfo]

@kyrylo
I did downgrade to

airbrake (9.5.5)
airbrake-ruby (4.11.1)
rbtree3 (0.5.0)

it helped

Also I've tried downgrade only rbtree3 and it didn't help

[kyrylo]

Thanks for checking & backtraces. That was useful for me.

I published #560 as a possible fix.

Could you please give it a try? Don't worry, it won't cause any harm to your app and it's basically airbrake-ruby v4.13.2 with that change (there are no new changes on master).

If you can reproduce that reliably, I would appreciate if you could tell me the steps.

[vitalinfo]

@kyrylo I'm worry about update to the version more than 4.11
because, if you check the memory usage graph

you will see, that combination of

airbrake (10.0.1)
airbrake-ruby (4.13.2)

definitely has a memory leaks

[kyrylo]

Well, I wasn't aware of memory leaks you mentioned. That sounds like a separate issue to me. Would you mind filing a new issue?