'Rails/Heroku, switching between databases multiple times yields "can't create Thread: Resource temporarily unavailable" error
Setup: Ruby 2.5.9, Rails 5.2.2, Heroku-18, Postgres
Issue: I'm running a task on my Heroku staging database which compares data from two different databases (both hosted on Heroku as Heroku Postgres resources).
I switch between old and new databases like so:
def switch_to_old_db()
ActiveRecord::Base.connection.close
ActiveRecord::Base.connection_pool.release_connection
ActiveRecord::Base.clear_all_connections!
ActiveRecord::Base.establish_connection(ENV["OLD_DB_URL"])
end
def switch_to_new_db()
ActiveRecord::Base.connection.close
ActiveRecord::Base.connection_pool.release_connection
ActiveRecord::Base.clear_all_connections!
ActiveRecord::Base.establish_connection(Rails.env.to_sym)
end
To simplify I created a barebones test which yields the same error.
def test
switch_to_old_db()
switch_to_new_db()
450.times do |i|
puts "Test reading #{i}"
switch_to_old_db()
switch_to_new_db()
end
end
Running this in console yields an error on the 125th iteration of db switching. Here is the error in full
can't create Thread: Resource temporarily unavailable /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.26.0/lib/rollbar/plugins/thread.rb:6:in
initialize' /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.26.0/lib/rollbar/plugins/thread.rb:6:in
initialize_with_rollbar' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:299:innew' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:299:in
run' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:365:ininitialize' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:989:in
new' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:989:inblock in establish_connection' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in
instrument' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:988:inestablish_connection' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_handling.rb:60:in
establish_connection' (irb):5:inswitch_to_old_db' (irb):32:in
block in run' (irb):30:intimes' (irb):30:in
run' (irb):42:inirb_binding' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/workspace.rb:85:in
eval' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/workspace.rb:85:inevaluate' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/context.rb:380:in
evaluate' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:491:inblock (2 levels) in eval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:623:in
signal_status' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:488:inblock in eval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:246:in
block (2 levels) in each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:232:inloop' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:232:in
block in each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:231:incatch' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:231:in
each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:487:ineval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:428:in
block in run' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:427:incatch' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:427:in
run' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:383:instart' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:64:in
start' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:19:instart' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:96:in
perform' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:inrun' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in
invoke_command' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:indispatch' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/command/base.rb:69:in
perform' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/command.rb:46:ininvoke' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands.rb:18:in
<top (required)>' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:inrequire' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in
block in require_with_bootsnap_lfi' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:inregister' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in
require_with_bootsnap_lfi' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:inrequire' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in
block in require' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:257:inload_dependency' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in
require'
This code runs successfully locally and only has this error running on my Heroku instance after 125 different iterations.
What I've tried so far:
- I've increased the db threadpool to the maximum limit
- I've tried taking threadpool size down to 1
- I've cleared / closed the database connection in multiple different ways before switching connection
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|