Running migrations with Rails in a Docker container with multiple container instances

Question

I've seen lots of examples of making Docker containers for Rails applications. Typically they run a rails server and have a CMD that runs migrations/setup then brings up the Rails server.

If I'm spawning 5 of these containers at the same time, how does Rails handle multiple processes trying to initiate the migrations? I can see Rails checking the current schema version in the general query log (it's a MySQL database):

 SELECT `schema_migrations`.`version` FROM `schema_migrations`

But I can see a race condition here if this happens at the same time on different Rails instances.

Considering that DDL is not transactional in MySQL and I don't see any locks happening in the general query log while running migrations (other than the per-migration transactions), it would seem that kicking them off in parallel would be a bad idea. In fact if I kick this off three times locally I can see two of the rails instances crashing when trying to create a table because it already exists while the third rails instance completes the migrations happily. If this was a migration that inserted something into the database it would be quite unsafe.

Is it then a better idea to run a single container that runs migrations/setup then spawns (for example) a Unicorn instance which in turn spawns multiple rails workers?

Should I be spawning N rails containers and one 'migration container' that runs the migration then exits?

Is there a better option?

Answer 1

Especially with Rails I don't have any experience, but let's look from a docker and software engineering point of view.

The Docker team advocates, sometimes quite aggressively, that containers are about shipping applications. In this really great statement, Jerome Petazzoni says that it is all about separation of concerns. I feel that this is exactly the point you already figured out.

Running a rails container which starts a migration or setup might be good for initial deployment and probably often required during development. However, when going into production, you really should consider separating the concerns.

Thus I would say have one image, which you use to run N rails container and add a tools/migration/setup whatever container, which you use to do administrative tasks. Have a look what the developers from the official rails image say about this:

It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as the base to build other images off of.

When you look at that image there is no setup or migration command. It is totally up to the user how to use it. So when you need to run several containers just go ahead.

From my experience with mysql this works fine. You can run a data-only container to host the data, run a container with the mysql server and finally run a container for administrative tasks like backup and restore. For all three containers you can use the same image. Now you are free to access your database from let's say several Wordpress containers. This means clear separation of concerns. When you use docker-compose it is not that difficult to manage all those containers. Certainly there are already many third party containers and tools to also support you with setting up a complex application consisting of several containers.

Finally, you should decide whether docker and the micro-service architecture is right for your problem. As outlined in this article there are some reasons against. One of the core problems being that it adds a whole new layer of complexity. However, that is the case with many solutions and I guess you are aware of this and willing to except it.

Answer 2

docker run <container name> rake db:migrate

Starts you standard application container but don't run the CMD (rails server), but rake db:migrate

UPDATE: Suggested by Roman, the command would now be:

docker exec <container> rake db:migrate

Answer 3

Having the same pb publishing to a docker swarm, I put here a solution partially grabbed from others.

Rails has already a mechanism to detect concurrent migrations by using a lock on the database. But it triggers ConcurrentException where it should just wait.

One solution is then to have a loop, that whenever a ConcurrentException is thrown, just wait for 5s et then redo the migration. This is especially important that all containers perform the migration as the migration fails, all containers must fails.

Solution from coffejumper

  namespace :db do
    namespace :migrate do
      desc 'Run db:migrate and monitor ActiveRecord::ConcurrentMigrationError errors'
      task monitor_concurrent: :environment do
        loop do
          puts 'Invoking Migrations'
          Rake::Task['db:migrate'].reenable
          Rake::Task['db:migrate'].invoke
          puts 'Migrations Successful'
          break
        rescue ActiveRecord::ConcurrentMigrationError
          puts 'Migrations Sleeping 5' 
          sleep(5)
        end
      end
    end
  end

And sometimes you have other processes you want to execute also one by one to perform the migration like after_party, cron setup, etc... The solution is then to use the same mechanism as Rails to embed rake tasks around a database lock:

Below, based on Rails 6 code, the migrate_without_lock performs the needed migrations while with_advisory_lock gets database lock (triggering ConcurrentMigrationError if lock cannot be acquired).

module Swarm
  class Migration
    def migrate
      with_advisory_lock { migrate_without_lock }
    end

    private

    def migrate_without_lock
      **puts "Database migration"
      Rake::Task['db:migrate'].invoke
      puts "After_party migration"
      Rake::Task['after_party:run'].invoke
      ...
      puts "Migrations successful"**
    end

    def with_advisory_lock
      lock_id = generate_migrator_advisory_lock_id
      MyAdvisoryLockBase.establish_connection(ActiveRecord::Base.connection_config) unless MyAdvisoryLockBase.connected?
      connection = MDAdvisoryLockBase.connection
      got_lock = connection.get_advisory_lock(lock_id)
      raise ActiveRecord::ConcurrentMigrationError unless got_lock
      yield
    ensure
      if got_lock && !connection.release_advisory_lock(lock_id)
        raise ActiveRecord::ConcurrentMigrationError.new(
          ActiveRecord::ConcurrentMigrationError::RELEASE_LOCK_FAILED_MESSAGE
        )
      end
    end

    MIGRATOR_SALT = 1942351734

    def generate_migrator_advisory_lock_id
      db_name_hash = Zlib.crc32(ActiveRecord::Base.connection_config[:database])
      MIGRATOR_SALT * db_name_hash
    end
  end

  # based on rails 6.1 AdvisoryLockBase
  class MyAdvisoryLockBase < ActiveRecord::AdvisoryLockBase # :nodoc:
    self.connection_specification_name = "MDAdvisoryLockBase"
  end
end

Then as before, do a loop to wait

namespace :swarm do
  desc 'Run migrations tasks after acquisition of lock on database'
  task migrate: :environment do
    result = 1
    (1..10).each do |i|
      **Swarm::Migration.new.migrate**
      puts "Attempt #{i} sucessfully terminated"
      result = 0
      break
    rescue ActiveRecord::ConcurrentMigrationError
      seconds = rand(3..10)
      puts "Attempt #{i} another migration is running => sleeping #{seconds}s"
      sleep(seconds)
    rescue => e
      puts e
      e.backtrace.each { |m| puts m }
      break
    end
    exit(result)
  end
end

Then in your startup script just launch the rake tasks

set -e
bundle exec rails swarm:migrate
exec bundle exec rails server -b "0.0.0.0"

At the end, as your migrations tasks are run by all containers, they must have a mechanism to do nothing when it's already done. (like does db:migrate)

Using this solution, the order in which Swarm launches containers doesn't matter anymore AND if something goes wrong, all containers know the problem :-)

Answer 4

For single container id:

docker exec -it <container ID> bundle exec rails db:migrate

for multiple we can repeat the process for different container, if there number in 1000 the need to script to execute.