tony – redpanthers.co

Configuring a GitLab CI pipeline for Rails, MongoDB, and Rspec

tony — Mon, 29 Apr 2019 11:28:12 +0000

Gitlab has a free private git repository service and an integrated CI service that can be configured to suit our needs. To know more about setting up Gitlab CI, check out our [blog post](link here)

Including MongoDB in CI build system

Instead of installing and starting MongoDB manually, we can use the Gitlab mongo service. A Gitlab service is just another Docker image that runs during your job and is linked to the main image. The first step is to include the latest MongoDB image to the build system. Add this to your .gitlab-ci.yml

services:
- mongo:latest

Add the MONGODB_URI variable

Add a variables section within .gitlab-ci.yml to set the MONGODB_URI variable.

variables:
MONGODB_URI: mongodb://mongo:27017/test_db

Add config/mongoid.yml ile

Add a file config/mongoid.yml.gitlab with the following contents

test:
  clients:
    default:
      uri: <%= ENV['MONGODB_URI'] %>

Override config/mongoid.yml with config/mongoid.yml.gitlab

We need to override the local test configuration with the above settings specific to CI. Add the following to the before_script section of .gitlab-ci.yml

before_script:
  - cp config/database.yml.gitlab config/database.yml

That’s it fellas, you are all set

Summary

The complete .gitlab-ci.yml

image: starefossen/ruby-node:latest
services:
  - mongo:latest
variables:
  MONGODB_URI: mongodb://mongo:27017/test_db
before_script:
  - RAILS_ENV=test bundle install --jobs $(nproc) "${FLAGS[@]}"
  - cp config/mongoid.yml.gitlab config/mongoid.yml
  - RAILS_ENV=test bundle exec rake db:create db:migrate
test:
  script:
    - bundle exec rspec

the contents of config/mongoid.yml

development:
  clients:
    default:
      database: dev_db
      hosts:
        - localhost:27017
      options:
        read:
          mode: :primary
        max_pool_size: 10
production:
  clients:
    default:
      uri: <%= ENV['MONGODB_URI'] %>
      pool_size: <%= ENV['DB_POOL_SIZE'] %>
test:
  clients:
    default:
      database: test_db
      hosts:
        - localhost:27017
      options:
        read:
          mode: :primary
        max_pool_size: 1

and config/mongoid.yml.gitlab

test:
  clients:
    default:
      uri: <%= ENV['MONGODB_URI'] %>

May you have a green build! :-)]]>

Upgrading to Rails 5.1x

tony — Wed, 21 Mar 2018 08:14:20 +0000

Yarn and webpack support

Dropped JQuery as a default dependency

Built-in support for writing system tests using Capybara

Encrypted secrets

and many more, you can find the detailed release notes here. Recently we upgraded one of our production apps to 5.1x and since we have good test coverage, upgrading was mostly painless. We still had to make some minor changes down the road

1. No more `before_filter`

One of our controllers still had the old-fashioned `before_filter` and we encountered the following error while booting up

undefined method 'before_filter' before_filter has been deprecated in Rails 5.0 and removed in 5.1.

This is because `before_filter` and after_filter are deprecated from Rails 5.0.0 onwards. As you might have guessed, changing from before_filter to before_action is the solution here.

2. `halt_callback_chains_on_return_false` is deprecated

We saw the following deprecation warning in our logs after the update

ActiveSupport.halt_callback_chains_on_return_false= is deprecated and will be removed in Rails 5.2.

From Rails 5.0x, the callback chain is not halted when a before callback returns false. We need to use throw(:abort) to explicitly halt callbacks Though we had no return false in our callbacks, we still had to remove the line, ActiveSupport.halt_callback_chains_on_return_false = false from config/initializers/new_framework_defaults.rb to get rid of the warning.

3. Passing a class to the `class_name` is deprecated

One of the deprecation warnings after the update was,

Passing a class to the 'class_name' is deprecated and will raise an ArgumentError in Rails 5.2.
It eagerloads more classes than necessary and potentially creates circular dependencies. Please pass the class name as a string:

This warning message was due to passing a class to `class_name` option when specifying associations – this is no longer recommended‘

- has_one :contact, class_name: UserContact
+ has_one :contact, class_name: 'UserContact

passing class name as a string is a standard now Happy Hacking

PS: If you need any help in upgrading your rails version drop an email to us at info@redpanthers.co or upgrade@redpanthers.co. We will be more than happy to go through your system and give a free upgrade action plan and estimation .]]>

Background Workers using Crontab

tony — Wed, 27 Sep 2017 06:17:21 +0000

cron is the system process which will automatically perform tasks for you according to a set schedule. The schedule is called the crontab, which is also the name of the program used to edit that schedule. For example, let’s say you have a rake task which you want to run every hour.

namespace :send_update_mail do
  desc "send_product_update_mails"
  task :send_mail => :environment do
    UserMailer.notify_product_updates
  end
end

To edit the crontab, use this command:

crontab -e

Now let’s add our job to the crontab. Each job you add should take up a single line. The format is very simple: six pieces of information, each separated by a space; the first five pieces of information tell cron when to run the job, and the last piece of information tells cron what the job is.

m h  dom mon dow   command

m, representing the minute of the hour, h, representing the hour of the day, dom, representing the day of the month, mon, representing the month of the year, dow, representing the day of the week and command, which is the command to be run. For example in our case

0 * * * * /home/myname/myapp/lib/tasks/send_mail.rb

The asterisks (“*“) will tell cron that for that unit of time, the job should run ‘every’. Save and close the file. And that’s it. But sometimes fiddling with crontab on the server can be very hectic and it would be much better if we can configure cron job in our rails application so we can keep the configuration in our source control. We have a gem called whenever that allows us to set up cron jobs from within our Rails apps using Ruby code. Let’s see how you can schedule our background jobs in Rails using Whenever to set up your schedule. Add in your Gemfile.

gem 'whenever', :require => false

Run bundle install to install the gem. Run the wheneverize command in your app’s root folder to set up an initial configuration.

wheneverize .

The wheneverize command will create an initial config/schedule.rb file. Now add following to schedule.rb

every 1.hour do
  rake 'send_update_mail:send_mail'
end

A whenever plugin for mina

The gem mina-wheneveris a whenever plugin for mina. Add this line to your application’s Gemfile and run bundle install

gem 'mina-whenever'

Modify your deploy.rb

require 'mina/whenever'
desc "Deploys the current version to the server."
task :deploy do
  deploy do
    .........
    on :launch do
      invoke :'sidekiq:restart'
      .....
    end
  end
end

Drawback

This rake task-based approach can have a potential drawback of extra memory consumption. When crontab runs `rake` or `rails runner`, it is booting up a full instance of our rails application to access relevant models and associations. It is memory expensive especially when your code base is huge and using a considerable number of gems. In a nutshell, the entire app and dependencies are loaded for every task that runs. Imagine if you have scheduled `n` number of background workers to start at a given point of time in our main application server. This could end up being too resource expensive that our application might become unusable for the user.

How to fix?

1) Start another server to schedule jobs and another server to consume the jobs. 2) Create another Ruby project that would add the messages to our Redis queue to schedule the tasks. 3) Offload that to another language like Crystal, which is more efficient and schedule jobs independent of our Rails app. 4) Use a sidekiq or other background job to schedule which takes less RAM 1st would be the easiest solution as it would keep our codebase same. (Heroku works like this) 2nd and 3rd are recommended if you are looking to keep it on a single server and keep the cost down until the business scale.

References

]]>

Painless Cron jobs in Crystal using Schedule

tony — Wed, 06 Sep 2017 03:15:23 +0000

whenever gem for scheduling tasks in their projects, so do we and with ‘whenever’ scheduling tasks become so effortless that we absolutely loved it. Recently we have been deploying Crytal apps to production and Kemal is our framework of choice. During the process, we felt the urge to have something similar to whenever in Crystal for cron jobs and that’s when we discovered Schedule – a Crystal shard that provides a clear DSL to write periodic or scheduled tasks and there was no turning back.

Getting started

Add Schedule to your `shard.yml` file

dependencies:
  schedule:
    github: hugoabonizio/schedule.cr

you are all set to schedule your tasks, all you gotta do is require the schedule module as require "schedule"

Examples

Schedul’s API defines 2 important methods, .every and .after for periodic and scheduled execution of tasks respectively.

Periodic execution

For running a task periodically we have to pass in a valid interval as well as a block to the .every method. For example,

require "schedule"
Schedule.every(3.seconds) do
  puts "I run every 3 seconds"
end

will print the message on every 3 seconds. Similarly

require "schedule"
Schedule.every(1.hour) do
  ClassName.hourly_job
end

will call the class method hourly_job every 1 hour

Scheduled execution

For scheduling tasks to run sometime in the future we can make use of the after method.

require "schedule"
# Execute a task after X interval
Schedule.after(10.seconds) do
  puts "I am going to run after 10 seconds"
end

Stop and Retry jobs

Call Schedule.retry to retry a task and Schedule.stop to stop the executing a task

MAX_COUNT = 3
Schedule.every(10.seconds) do
  count = get_count
  Schedule.retry if count == 0
  Schedule.stop if count >= MAX_COUNT
end

Exception handlers

We can use Schedule.exception_handler to set an exception handler in our tasks

require "schedule"
Schedule.exception_handler do |ex|
  puts "Exception recued! #{ex.message}"
end
Schedule.every(100.milliseconds) do
  raise "I'm an Exception"
end

it is also possible to pass a proc to `Schedule.exception_handler` directly to set the exception handler

handler = ->(ex : Exception) { puts "Exception recued! #{ex.message}" }
Schedule.exception_handler = handler

schedule.cr is still in the early stages of development and many options that we are familiar with using whenever – like running tasks on a specific day of the week are not available yet but are coming soon. Happy Hacking <3]]>

Deploying Crystal app to production using Heroku

tony — Tue, 05 Sep 2017 14:39:26 +0000

This article assumes that you have Crystal and Heroku CLI installed.

Create a sample application

Create a new application using

$ crystal init app sample-app

You should see the following on your terminal

$ crystal init app sample-app
      create  sample-app/.gitignore
      create  sample-app/.editorconfig
      create  sample-app/LICENSE
      create  sample-app/README.md
      create  sample-app/.travis.yml
      create  sample-app/shard.yml
      create  sample-app/src/sample-app.cr
      create  sample-app/src/sample-app/version.cr
      create  sample-app/spec/spec_helper.cr
      create  sample-app/spec/sample-app_spec.cr
Initialized empty Git repository in /Users/yourname/Desktop/sample-app/.git/

now cd to the root of the app, $ cd sample-app and add Kemal to shrad.yml file

name: sample-app
version: 0.1.0
authors:
  - John Doe 
dependencies:
  kemal:
    github: kemalcr/kemal
    branch: master
targets:
  sample-app:
    main: src/sample-app.cr
crystal: 0.23.1
license: MIT

and run $ crystal deps for installing dependencies.

Add a “/” route

Open sample-app.cr file and add replace the contents with the following code

require "kemal"
get "/" do
"Hello World"
end
Kemal.run

now run $ crystal src/kemal_sample.cr and go to http://localhost:3000/on your browser. You should see ur app running and the message ‘Hello World’.

Deploy to Heroku

So far so good Now let us deploy our app to Heroku. Run $ heroku create your-custom-name –buildpack https://github.com/crystal-lang/heroku-buildpack-crystal.git You should see something like the following on your terminal.

$ heroku create your-custom-name --buildpack https://github.com/crystal-lang/heroku-buildpack-crystal.git
Creating ⬢ your-custom-name... done
Setting buildpack to https://github.com/crystal-lang/heroku-buildpack-crystal.git... done
https://your-custom-name.herokuapp.com/ | https://git.heroku.com/your-custom-name.git

You are all set. Commit and push to heroku

$ git status
$ git add -A
$ git commit -m "commit message"
$ git push heroku master

To see your app running go to https://your-custom-name.herokuapp.com/ . Congratulations you just deployed your crystal app to production Happy Coding]]>

How we made our rspec test suite to run 2x faster

tony — Fri, 01 Sep 2017 09:24:27 +0000

Why on earth is my test suite taking so long to run?” If you are a developer, you might have asked this question at least once to yourself. So did we, our rails project’s test suite was taking a good 1 hour 30 minutes to run and we wanted to improve that time so badly that we eventually did exactly that, reducing nearly 1 and half hours to just minutes, and this is how we did it.

1. Database independent tests

DB operations are often time-consuming and most of the time we can do away with saving objects to the database to run our tests. Use test doubles, stubs and mocks instead of creating the real instance of a class and invoking methods on the created instance.

class Student < ActiveRecord::Base
  .
  .
  def name
    first_name +" "+ last_name
  end
end

Our test case

describe Student do
  let (:student) {create(:student, first_name: 'Red', last_name: 'Panther')}
  it 'should return name' do
    student.name.should == 'Red Panther'
  end
end

This test can be made faster by replacing

let(:student) {create(:student, first_name: 'Red', last_name: 'panther')}

with

let(:student) {build_stubbed(:student, first_name: 'Red', last_name: 'Panther')

2) Use gem group

Rails preload your gems before running tests. Using gem groups allow rails to load only the environment specific dependencies.

#Gemfile

group: production do
  gem 'activemerchent'
end
group :test, : development do
  gem 'capybara'
  gem 'rspec-rails'
  gem 'byebug'
end

3) Use `before(:each)` and `before(:all)` hooks carefully

Since before(:each) runs for every test, be careful what we include inside before(:each) hook. If the code inside before(:each) is slow every single test in your file is going to be slow. A workaround would be to refactor the code to have fewer dependencies or move them to a before(:all) block which runs only once. Let’s say you have

before(:each) do
  @article = create(:article)
  @author = create(:author)
end

moving them to a before(:all) block

before(:all) do
  @article = create(:article)
  @author = create(:author)
end

Should save you some time but with some drawbacks of its own, for example, the objects @article and @author are not recreated for each test as they in before(:all) block which means any test case that changes the attributes of these objects might affect the result of other following tests.

4. Use `build_stubbed` Instead of `build`

FactoryGirl.build is not suitable when we want our instance to behave as though it is persisted. In this scenario instead of creating a real class instance, we can use build_stubbed which makes the instance to behave as it is persisted by assigning an id.

FactoryGirl.build_stubbed(:student)

Also note that when we build instance using ` .build` it calls .create on the associated models, where as .build_stubbed calls nothing but .build_stubbed also on associated models as well.

5. Running tests parallelly

parallel_tests is a gem that allows us to run tests across multiple CPU cores. A very important thing to take into account when running tests in parallel is to make sure that the tests are independent of each other. Even though parallel_tests uses one database per thread, if there are any shared state between tests that live outside the DB such as Elastic search or Apache solar those dependencies should be taken into account when writing tests.

6. Use continuous integration

As our test suite grew into 3k test cases, it was no longer viable to run the entire suite on our local machines. That’s when we felt the urgency to switch to a CI. We chose Circle CI which supports parallel builds. We split out tests into multiple virtual machines that run parallelly and it was a huge win for us in terms of test times. Our developers wrote the code and pushed to the repo and the CI took care of the rest. Few popular CI tools are 1) Trvis CI 2) Jenkins 3) CircleCI 4) Codeship Automated tests with continuous integration also enhance code quality.

7. Database cleaner

We observed an increase in speed after tweaking our database_cleaner strategies a little bit. To start with, include gem database_cleaner in gemfile. Inside a separate file spec/support/database_cleaner.rb,

RSpec.configure do |config|
 
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end
 
  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end
 
  config.before(:each) do
    DatabaseCleaner.start
  end
 
  config.after(:each) do
    DatabaseCleaner.clean
  end
 
end

This will clear the test database out completely before the entire test suite runs.
Sets the default database cleaning strategy to be transactions, which are very fast.
DatabaseCleaner.start and DatabaseCleaner.clean hook up database_cleaner when each test begins and ends.

References

]]>

The future of Spree OSS and whats new in Spree 3.3

tony — Tue, 29 Aug 2017 05:36:14 +0000

The story so far Ever since its advent in 2007, Spree – originally developed by Sean Schofield, is the most popular open source Ruby On Rails E-commerce Software spanning over 17,703 commits, 725 contributes and 208 releases. Spree was acquired by First Data in the late 2015 and since then, there has been fair amount of speculation around Spree remaining as an opensource project and it’s long term future as outlined here in this blog post. As it turned out Spree didn’t die off, but revived under a new core team. After the acquisition announcement developers from Spark Solutions and Vinsol volunteered to keep spree alive and has been doing a fantastic job, putting shoulders to the wheel by taking Spree from version 3.0.4 to the current Spree 3.3

Whats new in Spree 3.3

On August 22, 2017, Spree released version 3.3 and is packed with the following major features.

Rails 5.1 support

Spree 3.3 is Rails 5.1 compatible whereas Spree 3.2 runs on Rails version 5.0. Which means managing JavaScript dependencies from NPM via `Yarn`, ability to use direct & resolved routes, parameterized mailers and many more goodies `Rails 5.1` has to offer

Ruby 2.4 support

Spree now works with Ruby versions `2.2 (>= 2.2.7)` , 2.2 (>= 2.2.7) , 2.2 (>= 2.2.7) , 2.3.x and the latest 2.4.x. So Ruby 2.4x ? Yay! that’s Faster Hashes, Unified Integers, and Better Rounding for you right out of the box

Segment tracker integration

Spree 3.3 is bundled with segment.com Tracker system which enables you to connect your store with over 200 analytics engines, CRMs, live chats, remarketing platforms and much more without any additional development. If that’s not cool enough for you stay tuned for next releases for more third party tracker and analytics integrations.

Bug fixes and performance enhancements

– Unique indexes and uniqueness validation on number fields for important models – Multiple regular & unique indexes that were missing have been added to improve data consistency and performance – Optimized Shipments and Inventory Units. – Better Store Credits management in customer Frontend and Admin Panel

Whats next?

Spree development has been slowly gaining traction recently. The future roadmap for spree development can be found here. But still, it is fair to say that more focus and a unified effort is necessary for the long term survival of Spree. Let’s join the folks at Spark solutions and Vinsol and start contributing to Spree OSS project]]>

tony – redpanthers.co

Configuring a GitLab CI pipeline for Rails, MongoDB, and Rspec

Including MongoDB in CI build system

Add the MONGODB_URI variable

Add config/mongoid.yml ile

Override config/mongoid.yml with config/mongoid.yml.gitlab

Summary

Upgrading to Rails 5.1x

1. No more before_filter

2. halt_callback_chains_on_return_false is deprecated

3. Passing a class to the class_name is deprecated

Background Workers using Crontab

A whenever plugin for mina

Drawback

How to fix?

References

Painless Cron jobs in Crystal using Schedule

Getting started

Examples

Periodic execution

Scheduled execution

Stop and Retry jobs

Exception handlers

Deploying Crystal app to production using Heroku

Create a sample application

Add a “/” route

Deploy to Heroku

How we made our rspec test suite to run 2x faster

1. Database independent tests

2) Use gem group

3) Use before(:each) and before(:all) hooks carefully

4. Use build_stubbed Instead of build

5. Running tests parallelly

6. Use continuous integration

7. Database cleaner

References

The future of Spree OSS and whats new in Spree 3.3

Whats new in Spree 3.3

Rails 5.1 support

Ruby 2.4 support

Segment tracker integration

Bug fixes and performance enhancements

Whats next?

1. No more `before_filter`

2. `halt_callback_chains_on_return_false` is deprecated

3. Passing a class to the `class_name` is deprecated

3) Use `before(:each)` and `before(:all)` hooks carefully

4. Use `build_stubbed` Instead of `build`