'Rails/Heroku, switching between databases multiple times yields "can't create Thread: Resource temporarily unavailable" error

Setup: Ruby 2.5.9, Rails 5.2.2, Heroku-18, Postgres

Issue: I'm running a task on my Heroku staging database which compares data from two different databases (both hosted on Heroku as Heroku Postgres resources).

I switch between old and new databases like so:

def switch_to_old_db()
    ActiveRecord::Base.connection.close
    ActiveRecord::Base.connection_pool.release_connection
    ActiveRecord::Base.clear_all_connections!
    ActiveRecord::Base.establish_connection(ENV["OLD_DB_URL"]) 
end 

def switch_to_new_db()
   ActiveRecord::Base.connection.close
   ActiveRecord::Base.connection_pool.release_connection
   ActiveRecord::Base.clear_all_connections!
   ActiveRecord::Base.establish_connection(Rails.env.to_sym)
end 

To simplify I created a barebones test which yields the same error.

def test 
  switch_to_old_db()
  switch_to_new_db()
   450.times do |i|
    puts "Test reading #{i}"
    switch_to_old_db()
    switch_to_new_db()
  end
end 

Running this in console yields an error on the 125th iteration of db switching. Here is the error in full

can't create Thread: Resource temporarily unavailable /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.26.0/lib/rollbar/plugins/thread.rb:6:in initialize' /app/vendor/bundle/ruby/2.5.0/gems/rollbar-2.26.0/lib/rollbar/plugins/thread.rb:6:in initialize_with_rollbar' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:299:in new' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:299:in run' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:365:in initialize' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:989:in new' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:989:in block in establish_connection' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in instrument' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:988:in establish_connection' /app/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.4.3/lib/active_record/connection_handling.rb:60:in establish_connection' (irb):5:in switch_to_old_db' (irb):32:in block in run' (irb):30:in times' (irb):30:in run' (irb):42:in irb_binding' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/workspace.rb:85:in eval' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/workspace.rb:85:in evaluate' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/context.rb:380:in evaluate' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:491:in block (2 levels) in eval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:623:in signal_status' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:488:in block in eval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:246:in block (2 levels) in each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:232:in loop' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:232:in block in each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:231:in catch' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb/ruby-lex.rb:231:in each_top_level_statement' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:487:in eval_input' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:428:in block in run' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:427:in catch' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:427:in run' /app/vendor/ruby-2.5.9/lib/ruby/2.5.0/irb.rb:383:in start' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:64:in start' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:19:in start' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands/console/console_command.rb:96:in perform' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command' /app/vendor/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/command/base.rb:69:in perform' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/command.rb:46:in invoke' /app/vendor/bundle/ruby/2.5.0/gems/railties-5.2.4.3/lib/rails/commands.rb:18:in <top (required)>' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi' /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in block in require' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:257:in load_dependency' /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/dependencies.rb:291:in require'

This code runs successfully locally and only has this error running on my Heroku instance after 125 different iterations.

What I've tried so far:

  1. I've increased the db threadpool to the maximum limit
  2. I've tried taking threadpool size down to 1
  3. I've cleared / closed the database connection in multiple different ways before switching connection


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source