rajana – redpanthers.co

Readability of text using odyssey

rajana — Wed, 28 Jun 2017 12:07:19 +0000

Flesch-Kincaid readability test

Flesch Kincaid Grade Level

Gunning Fog Score

SMOG

Coleman Liau Index

Automated Readability Index (ARI)

Recently in a project that we worked on we were asked to find the readability of various pages of a website. We decided to start with Flesch-Kincaid test, as we found this to be a popular one in our research. Flesch-Kincaid readability test is designed to indicate how difficult a passage in English is to understand. In this test higher score indicates how easier to read and a lower score indicates how difficult it is to read.The formula to find Flesch-Kincaid reading-ease score is 206.835 – 1.015 * (total words / total sentences) – 84.6 * (total syllables / total words) The scores can be interrupted as

Score	School Level	Notes
100.00-90.00	5th grade	Very easy to read.
90.00-80.00	6th grade	Easy to read.
80.00-70.00	7th grade	Fairly easy to read.
70.00-60.00	8th & 9th grade	Plain English.
60.00-50.00	10th to 12th grade	Fairly difficult to read.
50.00-30.00	College	Difficult to read.
30.00-0.00	College Graduate	Very difficult to read.

Since we were not experts we wanted the ability to tweak and play around with it. We found an already build gem called Odyssey which had all these various tests and also provided the ability to extend this feature as well. So here in this article, we will discuss how to use Odyssey gem to find readability of an article and a web page.

Install Odyssey

Add in your Gemfile.

gem 'odyssey'

Usage

require 'odyssey'
Odyssey.formula_name(text, all_stats)

So if we want to use the Flesch-Kincaid test, we write the code as below.

require 'odyssey'
Odyssey.flesch_kincaid_re(text, all_stats)

To find the readability of a website we use Nokogiri and Odyssey together. Nokogiri to fetch the contents of the page and Odyssey to get the readability. Example of finding readability of our own website (https://redpanthers.co)/

url = "https://redpanthers.co/"
doc = Nokogiri::HTML(open(url))
# Get all the contents
paragraph = doc.css('p', 'article', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'a').map(&:text)
score = Odyssey.flesch_kincaid_re(para.join('. '), true)
p score

If all_stats is set to false, it returns score only. If it is true returns a hash like below

{
 "string_length"=>3024,
 "letter_count"=>2270,
 "syllable_count"=>808,
 "word_count"=>505,
 "sentence_count"=>75,
 "average_words_per_sentence"=>6.733333333333333,
 "average_syllables_per_word"=>1.6,
 "name"=>"Flesch-Kincaid Reading Ease",
 "formula"=>#,
 "score"=>64.6
}

We can perform multiple text analyses on the same text as shown below

url = "https://redpanthers.co/"
doc = Nokogiri::HTML(open(url))
para = doc.css('p', 'article', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'a').map(&:text)
score = Odyssey.analyze_multi(para.join('. ').gsub('\n', ' '),
          ['FleschKincaidRe', 'FleschKincaidGl', 'GunningFog', 'Smog','Ari','ColemanLiau'],
          true)

if all_stats is set to true it will return a hash like this

{
"string_length"=>19892,
 "letter_count"=>14932,
 "syllable_count"=>5079,
 "word_count"=>3325,
 "sentence_count"=>435,
 "average_words_per_sentence"=>7.64367816091954,
 "average_syllables_per_word"=>1.5275187969924813,
 "scores"=>
  {
   "FleschKincaidRe"=>69.8,
   "FleschKincaidGl"=>5.4,
   "GunningFog"=>3.1,
   "Smog"=>8.7,
   "Ari"=>3.5,
   "ColemanLiau"=>10.6
  }
}

Extending odyssey

To extending odyssey, you can create a class that inherit from formula

class NewFormula < Formula
  def score(passage, stats)
    p passage
    p stats
  end
  def sentence
    "Red Panthers is a Ruby on Rails development studio,
     based in the beautiful city of Cochin."
  end
end

To call your formula you just use

obj = NewFormula.new
Odyssey.new_formula(obj.sentence, false)

Resultant passage will be a Hash like this

{
 "raw"=>"Red Panthers is a Ruby on Rails development studio,
        based in the beautiful city of Cochin.",
 "words"=>["Red", "Panthers", "is", "a", "Ruby", "on", "Rails",
           "development", "studio", "based", "in", "the",
           "beautiful", "city", "of", "Cochin"],
 "sentences"=>["Red Panthers is a Ruby on Rails development studio,
               based in the beautiful city of Cochin."],
 "syllables"=>[1, 2, 1, 1, 2, 1, 1, 4, 2, 1, 1, 1, 4, 2, 1, 2]
}

and resultant status will be a Hash like this

{
 "string_length"=>90,
 "letter_count"=>73,
 "word_count"=>16,
 "syllable_count"=>27,
 "sentence_count"=>1,
 "average_words_per_sentence"=>16.0,
 "average_syllables_per_word"=>1.6875
}

Because we have access to formula’s class that is ‘status’ flag set to true then we have access to other methods or class formula. Thanks to Odyssey we were able to implement the feature quite easily and right now the algorithm we are using have evolved to new forms. But that’s another article. But if you want to build a simple readability checker then it’s quite easy and simple in Rails.

References

]]>

to_json vs as_json in Rails API

rajana — Wed, 14 Jun 2017 10:58:30 +0000

to_json in Rails API Let’s discuss how we started out building our APIs using ‘to_json’. to_json will return JSON string representing the hash. Option are passed to each element. In to_json without any option, the returned JSON string will include all the model attributes

user = User.find(1)
user.to_json

to_json had some great options of ActiveRecord objects. We could just tell the method to only render certain attributes, or to include association or method calls. We had:

only – Only show column names in the output as specified in the list
```
user.to_json(:only => [ :id, :email ])
```
except – Show all column names except the ones specified in the list
```
user.to_json(:except => [:id, :username])
```

to_json works fine and seamlessly ok, as long as the database schema is deeply coupled with the API output.When it takes the next level where we want to render a certain attribute in json it start to break.

render :json => { :sucess => true,
  :user => @user.to_json(:only => [:name]) }

This will start to generate a bit load to controllers. Such methods of generating json don’t feel quite right and begin to break down. This is because the to_json is interested in ‘serializing’ a database object while most of the developers try to put relevant representation for their API. Anytime to_json is called on an object, as_json is invoked to create the data structure, and then that hash is encoded as a JSON string using ActiveSupport::json.encode. This happens for all types: Object, Numeric, Data, String etc. Now the creation of the json is separated from the rendering of the json. as_json is used to create the structure of the json as a Hash, and the rendering of that hash into JSON string is left up-to ActiveSupport::json.encode.

as_json in Rails API

as_json Returns a hash representing the model. Like in to_json, option are passed to each element in as_json. The option include_root_in_json controls the top-level behavior of as_json. If true, as_json will emit a single root node named after the object’s type. The default value for include_root_in_json option is false. This behavior can also be achieved by setting the :root option to true.

user.as_json(root: true)

In as_json without any option, the returned HASH will include all the model attributes

user = User.find(1)
user.as_json

The :only and :except options can be used to limit the attributes included, and work similar to the attributes method.

user.as_json(:only => [ :id, :email ])
user.as_json(:except => [:id, :username])

Default as_json will create a hash which includes all the model attributes. We normally override the as_json to create custom JSON structure.

def as_json(options={})
 { :email => self.email }
end

As we were going through this we also came across a method called from_json. So we decided to write about it as well

from_json in Rails API

from_json will sets the model attributes from a JSON string. The default value for include_root option is false. We can change it to ‘true’ if the given JSON string includes a single root node.

json = { user: { email: 'adone@gmail.com', username: 'adone'} }.to_json
user = User.new
user.from_json(json, true)
user.email                 #=> "adone@gmail.com"
user.username              #=> "adone"

Do not call to_json directly, and let the render take it, in controllers. Incase of custom structured or multi level JSON structure override as_json in the model or call as_json. Using as_json will help to worry one less.

References

]]>

Fixtures in Rails Tests

rajana — Mon, 13 Feb 2017 10:32:47 +0000

Fixtures are very much relevant in Rails testing. Fixtures are a way of organizing your test data.Testing the application helps to debug it more efficiently and ensures the desired functionality to the application. Let’s have a look on them. It is used to manage the test data. It tests against the real data and is written in YAML files. For each model in the application, there is a .yml file in the test/fixtures directory. When we generate the model using rails g it will automatically create the .yml file also. Here you can see an example,

Matz:
 name: redpanthers
 message: hi

Here ‘Matz’ is the fixture name. And the name and message are the fields in the User model. Fixtures come in 3 flavours:

YAML fixtures: It is a file format which describes data structures in human-readable format.These are stored in a single file per model(above example is in yaml format).
CSV fixtures: Here values are kept in the Comma Separated Value (CSV) format. These are stored in a single file but instead end with the .csv file extension.

Eg:

  name, message
   redpanthers, hi

Single-file fixtures: These are the original format for Active Record.

Eg:

name => redpanthers
message => hi

Creating multiple/random items in fixtures

You can create multiple items as:

Adorn:
 name: redpanthers
 message: hi
Alex:
 name: google
 message: hello
.............

When you add fixtures, they get IDs based on a hash of the table name and the fixture name. To us humans, they just look like random numbers. So there is no need to define the id for this. It will ensure that the id is unique for every item. Fixtures are unordered. If you want ordered items, use the omap YAML type. Also, you can access the data in the test as:

test "find" do
  assert_equal "User", users(:Adorn).name
end

Where ‘users’ is the name of the yml file and ‘Adorn’ is the fixture name.

ERB Fixtures

You can add ERB with your YAML fixtures to create a bunch of fixtures as:

<% 1000.times do |i| %>
user_<%= i %>:
  name: <%= "user#{i}" %>
  message: <%= "message#{i}" %>
<% end %>

In the above example, the code generate 1000 users.

Writing belongs_to/has_many relationships

We can also define associations between fixtures. For example, if a user has so many customers there exists a has_many/belongs to a relationship. So we can get the value as:

# In fixtures/customers.yml
Jacob:
 name: Tiago
# In fixtures/users.yml
David:
 name: redpanthers
 message: hi
 customer: Jacob

Like above, you can get customer details in users.yml.

Why are they Special?

It allows you to populate your test database with values so that you can test more conveniently without having to put lots of data into it when setting up a test run.
These are the instances of Active Record. So you can access the object directly because it is automatically setup as a local variable of the test case.

Eg:

# this will return the User object for the fixture named David
Adam(:David)

We can set a value to nil/blank from the fixture.

Eg:

require 'yaml'
YAML.load "--- \n:private: null\n")
=> {:private=>nil}
# In fixture
Abner:
 name: john
 private: null

I hope this article helps you in a better way to know the basics about fixtures!

References

]]>