Lazy enumerator to handle huge files

coderhs — Fri, 12 Aug 2016 08:49:12 +0000

Lazy evaluation, or call-by-need is an evaluation strategy which delays the evaluation of an expression until its value is needed. It’s frequently seen in functional languages, ruby introduced the lazy method in Ruby 2.0. For those who don’t know what are enumerators: enumerators are something that can be counted. So a collection of elements, files (file is an collection of lines of string), etc can be treated as an enumerator. In ruby we need to make something countable into an enumerator object, which is done by applying .each and .map on it.

[].each
[1, 3, 4].each
file.each
{}.each

Ruby has a wide range of operations we can do over a collection, it’s one of those features that makes Ruby such a powerful dynamic language. An enumerator can be used to generate series like the Fibonacci series.

fib = Enumerator.new do |y|
  a = b = 1
  loop do
    y << a
    a, b = b, a + b
  end
end

But when we do a .map / .each with a code block, then it would try to realize the enumerator fully and then apply the block over it. That would be fine when we are working on something small like:

[1,2,3].map { |i| i*i }

But when we take the above fib enumerator, which will grow into an infinite series, adding a .map would lead the code to an infinite loop. If you are crazy enough to write an infinite loop, feel free to run the below code.

foo.map { |i| i * * }

When we try to use these operations on a huge 10 GB file,there it makes your program realize the entire file is close to impossible. So this is the case where the lazy enumerator comes to place. It would ask ruby to defer the realization of the enumerator until it is required, and not before you run your operation. So with respect to the above Fibonacci enumerator, you should change the code to as below.

foo.lazy.map { |i| i * i }

So now, the program is ready, all good it would run fast. But it won’t execute them until you call or access those elements. So only when we do foo.lazy.map { |i| i* i }. First the first element is calculated and given to us. Like wise if you want the first 10 elements of the fibonacci series then you have to do foo.lazy.map { |i| i* i }.take(1o).to_a To do the same with the files, as a file is a collection of lines, follow the below code snippets:

fille = File.open('OurHugeFile.log')

get the first three lines, lazily:

file.each_line.lazy.take(3).to_a

get the firs two lines with word “INFO”

 file.each_line.lazy.select { |line| line.match(/INFO/) }.first(2)

So if you want to print something with line numbers, just do a map from the lazy.

file.each_line.lazy.each_with_index.map { |l, i| "#{i}: #{l}" }.select { |i| match(/INFO/) }.first(3)

If you have anything more to add to this, kindly comment below.]]>

enumerator – redpanthers.co

Lazy enumerator to handle huge files