Background Workers using Crontab

Published On - September 27, 2017

tony

cron is the system process which will automatically perform tasks for you according to a set schedule. The schedule is called the crontab, which is also the name of the program used to edit that schedule. For example, let’s say you have a rake task which you want to run every hour. 

namespace :send_update_mail do
  desc "send_product_update_mails"
  task :send_mail => :environment do
    UserMailer.notify_product_updates
  end
end
To edit the crontab, use this command: 
crontab -e
Now let’s add our job to the crontab. Each job you add should take up a single line. The format is very simple: six pieces of information, each separated by a space; the first five pieces of information tell cron when to run the job, and the last piece of information tells cron what the job is. 
m h  dom mon dow   command
m, representing the minute of the hour, h, representing the hour of the day, dom, representing the day of the month, mon, representing the month of the year, dow, representing the day of the week and  command, which is the command to be run. For example in our case 
0 * * * * /home/myname/myapp/lib/tasks/send_mail.rb
The asterisks (“*“) will tell cron that for that unit of time, the job should run ‘every’. Save and close the file. And that’s it. But sometimes fiddling with crontab on the server can be very hectic and it would be much better if we can configure cron job in our rails application so we can keep the configuration in our source control. We have a gem called whenever that allows us to set up cron jobs from within our Rails apps using Ruby code. Let’s see how you can schedule our background jobs in Rails using Whenever to set up your schedule. Add in your Gemfile. 
gem 'whenever', :require => false
Run bundle install to install the gem. Run the wheneverize command in your app’s root folder to set up an initial configuration. 
wheneverize .
The wheneverize command will create an initial config/schedule.rb file. Now add following to schedule.rb 
every 1.hour do
  rake 'send_update_mail:send_mail'
end

A whenever plugin for mina

The gem mina-wheneveris a whenever plugin for mina. Add this line to your application’s Gemfile and run bundle install 
gem 'mina-whenever'
Modify your deploy.rb 
require 'mina/whenever'
desc "Deploys the current version to the server."
task :deploy do
  deploy do
    .........
    on :launch do
      invoke :'sidekiq:restart'
      .....
    end
  end
end

Drawback

This rake task-based approach can have a potential drawback of extra memory consumption. When crontab runs `rake` or `rails runner`, it is booting up a full instance of our rails application to access relevant models and associations. It is memory expensive especially when your code base is huge and using a considerable number of gems. In a nutshell, the entire app and dependencies are loaded for every task that runs. Imagine if you have scheduled `n` number of background workers to start at a given point of time in our main application server. This could end up being too resource expensive that our application might become unusable for the user.

How to fix?

1) Start another server to schedule jobs and another server to consume the jobs. 2) Create another Ruby project that would add the messages to our Redis queue to schedule the tasks. 3) Offload that to another language like Crystal, which is more efficient and schedule jobs independent of our Rails app. 4)  Use a sidekiq or other background job to schedule which takes less RAM 1st would be the easiest solution as it would keep our codebase same. (Heroku works like this) 2nd and 3rd are recommended if you are looking to keep it on a single server and keep the cost down until the business scale.  

References

]]>