class Enumerator

Class Enumerator supports:

An Enumerator may be created by the following methods:

In addition, certain Ruby methods return Enumerator objects: a Ruby iterator method that accepts a block may return an Enumerator if no block is given. There are many such methods, for example, in classes Array and Hash. (In the documentation for those classes, search for ‘new_enumerator`.)

Internal Iteration¶ ↑

In _internal iteration_, an iterator method drives the iteration and the caller’s block handles the processing; this example uses method each_with_index:

words = %w[foo bar baz] # => ["foo", "bar", "baz"]
enumerator = words.each # => #<Enumerator: ...>
enumerator.each_with_index {|word, i| puts "#{i}: #{word}" }
0: foo
1: bar
2: baz

Iterator methods in class Enumerator include:

each: passes each item to the block.
each_with_index: passes each item and its index to the block.
each_with_object (aliased as with_object): passes each item and a given object to the block.
with_index: like each_with_index, but starting at a given offset (instead of zero).

Class Enumerator includes module Enumerable, which provides many more iterator methods.

External Iteration¶ ↑

In _external iteration_, the user’s program both drives the iteration and handles the processing in stream-like fashion; this example uses method next:

words = %w[foo bar baz]
enumerator = words.each
enumerator.next # => "foo"
enumerator.next # => "bar"
enumerator.next # => "baz"
enumerator.next # Raises StopIteration: iteration reached an end

External iteration methods in class Enumerator include:

feed: sets the value that is next to be returned.
next: returns the next value and increments the position.
next_values: returns the next value in a 1-element array and increments the position.
peek: returns the next value but does not increment the position.
peek_values: returns the next value in a 1-element array but does not increment the position.
rewind: sets the position to zero.

Each of these methods raises FrozenError if called from a frozen Enumerator.

External Iteration and Fiber¶ ↑

External iteration that uses Fiber differs significantly from internal iteration:

Using Fiber adds some overhead compared to internal enumeration.
The stacktrace will only include the stack from the Enumerator, not above.
Fiber-local variables are not inherited inside the Enumerator Fiber, which instead starts with no Fiber-local variables.
Fiber storage variables are inherited and are designed to handle Enumerator Fibers. Assigning to a Fiber storage variable only affects the current Fiber, so if you want to change state in the caller Fiber of the Enumerator Fiber, you need to use an extra indirection (e.g., use some object in the Fiber storage variable and mutate some ivar of it).

Concretely:

Thread.current[:fiber_local] = 1
Fiber[:storage_var] = 1
e = Enumerator.new do |y|
  p Thread.current[:fiber_local] # for external iteration: nil, for internal iteration: 1
  p Fiber[:storage_var] # => 1, inherited
  Fiber[:storage_var] += 1
  y << 42
end

p e.next # => 42
p Fiber[:storage_var] # => 1 (it ran in a different Fiber)

e.each { p _1 }
p Fiber[:storage_var] # => 2 (it ran in the same Fiber/"stack" as the current Fiber)

Converting External Iteration to Internal Iteration¶ ↑

You can use an external iterator to implement an internal iterator as follows:

def ext_each(e)
  while true
    begin
      vs = e.next_values
    rescue StopIteration
      return $!.result
    end
    y = yield(*vs)
    e.feed y
  end
end

o = Object.new

def o.each
  puts yield
  puts yield(1)
  puts yield(1, 2)
  3
end

# use o.each as an internal iterator directly.
puts o.each {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3

# convert o.each to an external iterator for
# implementing an internal iterator.
puts ext_each(o.to_enum) {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3

Public Class Methods

new(size = nil) {|yielder| ... }

Returns a new Enumerator object that can be used for iteration.

The given block defines the iteration; it is called with a “yielder” object that can yield an object via a call to method yielder.yield:

fib = Enumerator.new do |yielder|
  n = next_n = 1
  while true do
    yielder.yield(n)
    n, next_n = next_n, n + next_n
  end
end

fib.take(10) # => [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Parameter size specifies how the size is to be calculated (see size); it can either be a value or a callable object:

Enumerator.new{}.size          # => nil
Enumerator.new(42){}.size      # => 42
Enumerator.new(-> {42}){}.size # => 42

static VALUE
enumerator_initialize(int argc, VALUE *argv, VALUE obj)
{
    VALUE iter = rb_block_proc();
    VALUE recv = generator_init(generator_allocate(rb_cGenerator), iter);
    VALUE arg0 = rb_check_arity(argc, 0, 1) ? argv[0] : Qnil;
    VALUE size = convert_to_feasible_size_value(arg0);

    return enumerator_init(obj, recv, sym_each, 0, 0, 0, size, false);
}

produce(initial = nil, size: nil) { |prev| block } → enumerator

Creates an infinite enumerator from any block, just called over and over. The result of the previous iteration is passed to the next one. If initial is provided, it is passed to the first iteration, and becomes the first element of the enumerator; if it is not provided, the first iteration receives nil, and its result becomes the first element of the iterator.

Raising StopIteration from the block stops an iteration.

Enumerator.produce(1, &:succ)   # => enumerator of 1, 2, 3, 4, ....

Enumerator.produce { rand(10) } # => infinite random number sequence

ancestors = Enumerator.produce(node) { |prev| node = prev.parent or raise StopIteration }
enclosing_section = ancestors.find { |n| n.type == :section }

Using ::produce together with Enumerable methods like Enumerable#detect, Enumerable#slice_after, Enumerable#take_while can provide Enumerator-based alternatives for while and until cycles:

# Find next Tuesday
require "date"
Enumerator.produce(Date.today, &:succ).detect(&:tuesday?)

# Simple lexer:
require "strscan"
scanner = StringScanner.new("7+38/6")
PATTERN = %r{\d+|[-/+*]}
Enumerator.produce { scanner.scan(PATTERN) }.slice_after { scanner.eos? }.first
# => ["7", "+", "38", "/", "6"]

The optional size keyword argument specifies the size of the enumerator, which can be retrieved by Enumerator#size. It can be an integer, Float::INFINITY, a callable object (such as a lambda), or nil to indicate unknown size. When not specified, the size defaults to Float::INFINITY.

# Infinite enumerator
enum = Enumerator.produce(1, size: Float::INFINITY, &:succ)
enum.size  # => Float::INFINITY

# Finite enumerator with known/computable size
abs_dir = File.expand_path("./baz") # => "/foo/bar/baz"
traverser = Enumerator.produce(abs_dir, size: -> { abs_dir.count("/") + 1 }) {
  raise StopIteration if it == "/"
  File.dirname(it)
}
traverser.size  # => 4

# Finite enumerator with unknown size
calendar = Enumerator.produce(Date.today, size: nil) {
  it.monday? ? raise(StopIteration) : it + 1
}
calendar.size  # => nil

static VALUE
enumerator_s_produce(int argc, VALUE *argv, VALUE klass)
{
    VALUE init, producer, opts, size;
    ID keyword_ids[1];

    if (!rb_block_given_p()) rb_raise(rb_eArgError, "no block given");

    keyword_ids[0] = rb_intern("size");
    rb_scan_args_kw(RB_SCAN_ARGS_LAST_HASH_KEYWORDS, argc, argv, "01:", &init, &opts);
    rb_get_kwargs(opts, keyword_ids, 0, 1, &size);

    size = UNDEF_P(size) ? DBL2NUM(HUGE_VAL) : convert_to_feasible_size_value(size);

    if (argc == 0 || (argc == 1 && !NIL_P(opts))) {
        init = Qundef;
    }

    producer = producer_init(producer_allocate(rb_cEnumProducer), init, rb_block_proc(), size);

    return rb_enumeratorize_with_size_kw(producer, sym_each, 0, 0, producer_size, RB_NO_KEYWORDS);
}

product(*enums) → enumerator

product(*enums) { |elts| ... } → enumerator

Generates a new enumerator object that generates a Cartesian product of given enumerable objects. This is equivalent to Enumerator::Product.new.

e = Enumerator.product(1..3, [4, 5])
e.to_a #=> [[1, 4], [1, 5], [2, 4], [2, 5], [3, 4], [3, 5]]
e.size #=> 6

When a block is given, calls the block with each N-element array generated and returns nil.

static VALUE
enumerator_s_product(int argc, VALUE *argv, VALUE klass)
{
    VALUE enums = Qnil, options = Qnil, block = Qnil;

    rb_scan_args(argc, argv, "*:&", &enums, &options, &block);

    if (!NIL_P(options) && !RHASH_EMPTY_P(options)) {
        rb_exc_raise(rb_keyword_error_new("unknown", rb_hash_keys(options)));
    }

    VALUE obj = enum_product_initialize(argc, argv, enum_product_allocate(rb_cEnumProduct));

    if (!NIL_P(block)) {
        enum_product_run(obj, block);
        return Qnil;
    }

    return obj;
}

Public Instance Methods

e + enum → enumerator

Returns an enumerator object generated from this enumerator and a given enumerable.

e = (1..3).each + [4, 5]
e.to_a #=> [1, 2, 3, 4, 5]

static VALUE
enumerator_plus(VALUE obj, VALUE eobj)
{
    return new_enum_chain(rb_ary_new_from_args(2, obj, eobj));
}

each { |elm| block } → obj

each → enum

each(*appending_args) { |elm| block } → obj

each(*appending_args) → an_enumerator

Iterates over the block according to how this Enumerator was constructed. If no block and no arguments are given, returns self.

Examples¶ ↑

"Hello, world!".scan(/\w+/)                     #=> ["Hello", "world"]
"Hello, world!".to_enum(:scan, /\w+/).to_a      #=> ["Hello", "world"]
"Hello, world!".to_enum(:scan).each(/\w+/).to_a #=> ["Hello", "world"]

obj = Object.new

def obj.each_arg(a, b=:b, *rest)
  yield a
  yield b
  yield rest
  :method_returned
end

enum = obj.to_enum :each_arg, :a, :x

enum.each.to_a                  #=> [:a, :x, []]
enum.each.equal?(enum)          #=> true
enum.each { |elm| elm }         #=> :method_returned

enum.each(:y, :z).to_a          #=> [:a, :x, [:y, :z]]
enum.each(:y, :z).equal?(enum)  #=> false
enum.each(:y, :z) { |elm| elm } #=> :method_returned

static VALUE
enumerator_each(int argc, VALUE *argv, VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);

    if (argc > 0) {
        VALUE args = (e = enumerator_ptr(obj = rb_obj_dup(obj)))->args;
        if (args) {
#if SIZEOF_INT < SIZEOF_RB_LEN_T
            /* check int range overflow */
            rb_len2int(RARRAY_LEN(args) + argc);
#endif
            args = rb_ary_dup(args);
            rb_ary_cat(args, argv, argc);
        }
        else {
            args = rb_ary_new4(argc, argv);
        }
        RB_OBJ_WRITE(obj, &e->args, args);
        e->size = Qnil;
        e->size_fn = 0;
    }
    if (!rb_block_given_p()) return obj;

    if (!lazy_precheck(e->procs)) return Qnil;

    return enumerator_block_call(obj, 0, obj);
}

each_with_index {|(*args), idx| ... }

each_with_index

Same as Enumerator#with_index(0), i.e. there is no starting offset.

If no block is given, a new Enumerator is returned that includes the index.

static VALUE
enumerator_each_with_index(VALUE obj)
{
    return enumerator_with_index(0, NULL, obj);
}

each_with_object(obj) {|(*args), obj| ... }

each_with_object(obj)

Iterates the given block for each element with an arbitrary object, obj, and returns obj

If no block is given, returns a new Enumerator.

Example¶ ↑

to_three = Enumerator.new do |y|
  3.times do |x|
    y << x
  end
end

to_three_with_string = to_three.with_object("foo")
to_three_with_string.each do |x,string|
  puts "#{string}: #{x}"
end

# => foo: 0
# => foo: 1
# => foo: 2

static VALUE
enumerator_with_object(VALUE obj, VALUE memo)
{
    RETURN_SIZED_ENUMERATOR(obj, 1, &memo, enumerator_enum_size);
    enumerator_block_call(obj, enumerator_with_object_i, memo);

    return memo;
}

Also aliased as: with_object

feed obj → nil

Sets the value to be returned by the next yield inside e.

If the value is not set, the yield returns nil.

This value is cleared after being yielded.

# Array#map passes the array's elements to "yield" and collects the
# results of "yield" as an array.
# Following example shows that "next" returns the passed elements and
# values passed to "feed" are collected as an array which can be
# obtained by StopIteration#result.
e = [1,2,3].map
p e.next           #=> 1
e.feed "a"
p e.next           #=> 2
e.feed "b"
p e.next           #=> 3
e.feed "c"
begin
  e.next
rescue StopIteration
  p $!.result      #=> ["a", "b", "c"]
end

o = Object.new
def o.each
  x = yield         # (2) blocks
  p x               # (5) => "foo"
  x = yield         # (6) blocks
  p x               # (8) => nil
  x = yield         # (9) blocks
  p x               # not reached w/o another e.next
end

e = o.to_enum
e.next              # (1)
e.feed "foo"        # (3)
e.next              # (4)
e.next              # (7)
                    # (10)

static VALUE
enumerator_feed(VALUE obj, VALUE v)
{
    struct enumerator *e = enumerator_ptr(obj);

    rb_check_frozen(obj);

    if (!UNDEF_P(e->feedvalue)) {
        rb_raise(rb_eTypeError, "feed value already set");
    }
    RB_OBJ_WRITE(obj, &e->feedvalue, v);

    return Qnil;
}

inspect → string

Creates a printable version of e.

static VALUE
enumerator_inspect(VALUE obj)
{
    return rb_exec_recursive(inspect_enumerator, obj, 0);
}

next → object

Returns the next object in the enumerator, and move the internal position forward. When the position reached at the end, StopIteration is raised.

Example¶ ↑

a = [1,2,3]
e = a.to_enum
p e.next   #=> 1
p e.next   #=> 2
p e.next   #=> 3
p e.next   #raises StopIteration

See class-level notes about external iterators.

static VALUE
enumerator_next(VALUE obj)
{
    VALUE vs = enumerator_next_values(obj);
    return ary2sv(vs, 0);
}

next_values → array

Returns the next object as an array in the enumerator, and move the internal position forward. When the position reached at the end, StopIteration is raised.

See class-level notes about external iterators.

This method can be used to distinguish yield and yield nil.

Example¶ ↑

o = Object.new
def o.each
  yield
  yield 1
  yield 1, 2
  yield nil
  yield [1, 2]
end
e = o.to_enum
p e.next_values
p e.next_values
p e.next_values
p e.next_values
p e.next_values
e = o.to_enum
p e.next
p e.next
p e.next
p e.next
p e.next

## yield args       next_values      next
#  yield            []               nil
#  yield 1          [1]              1
#  yield 1, 2       [1, 2]           [1, 2]
#  yield nil        [nil]            nil
#  yield [1, 2]     [[1, 2]]         [1, 2]

static VALUE
enumerator_next_values(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);
    VALUE vs;

    rb_check_frozen(obj);

    if (!UNDEF_P(e->lookahead)) {
        vs = e->lookahead;
        e->lookahead = Qundef;
        return vs;
    }

    return get_next_values(obj, e);
}

peek → object

Returns the next object in the enumerator, but doesn’t move the internal position forward. If the position is already at the end, StopIteration is raised.

See class-level notes about external iterators.

Example¶ ↑

a = [1,2,3]
e = a.to_enum
p e.next   #=> 1
p e.peek   #=> 2
p e.peek   #=> 2
p e.peek   #=> 2
p e.next   #=> 2
p e.next   #=> 3
p e.peek   #raises StopIteration

static VALUE
enumerator_peek(VALUE obj)
{
    VALUE vs = enumerator_peek_values(obj);
    return ary2sv(vs, 1);
}

peek_values → array

Returns the next object as an array, similar to Enumerator#next_values, but doesn’t move the internal position forward. If the position is already at the end, StopIteration is raised.

See class-level notes about external iterators.

Example¶ ↑

o = Object.new
def o.each
  yield
  yield 1
  yield 1, 2
end
e = o.to_enum
p e.peek_values    #=> []
e.next
p e.peek_values    #=> [1]
p e.peek_values    #=> [1]
e.next
p e.peek_values    #=> [1, 2]
e.next
p e.peek_values    # raises StopIteration

static VALUE
enumerator_peek_values_m(VALUE obj)
{
    return rb_ary_dup(enumerator_peek_values(obj));
}

rewind → e

Rewinds the enumeration sequence to the beginning.

If the enclosed object responds to a “rewind” method, it is called.

static VALUE
enumerator_rewind(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);

    rb_check_frozen(obj);

    rb_check_funcall(e->obj, id_rewind, 0, 0);

    e->fib = 0;
    e->dst = Qnil;
    e->lookahead = Qundef;
    e->feedvalue = Qundef;
    e->stop_exc = Qfalse;
    return obj;
}

size → int, Float::INFINITY or nil

Returns the size of the enumerator, or nil if it can’t be calculated lazily.

(1..100).to_a.permutation(4).size # => 94109400
loop.size # => Float::INFINITY
(1..100).drop_while.size # => nil

Note that enumerator size might be inaccurate, and should be rather treated as a hint. For example, there is no check that the size provided to ::new is accurate:

e = Enumerator.new(5) { |y| 2.times { y << it} }
e.size # => 5
e.to_a.size # => 2

Another example is an enumerator created by ::produce without a size argument. Such enumerators return Infinity for size, but this is inaccurate if the passed block raises StopIteration:

e = Enumerator.produce(1) { it + 1 }
e.size # => Infinity

e = Enumerator.produce(1) { it > 3 ? raise(StopIteration) : it + 1 }
e.size # => Infinity
e.to_a.size # => 4

static VALUE
enumerator_size(VALUE obj)
{
    struct enumerator *e = enumerator_ptr(obj);
    int argc = 0;
    const VALUE *argv = NULL;
    VALUE size;

    if (e->procs) {
        struct generator *g = generator_ptr(e->obj);
        VALUE receiver = rb_check_funcall(g->obj, id_size, 0, 0);
        rb_len_t i = 0;

        for (i = 0; i < RARRAY_LEN(e->procs); i++) {
            VALUE proc = RARRAY_AREF(e->procs, i);
            struct proc_entry *entry = proc_entry_ptr(proc);
            lazyenum_size_func *size_fn = entry->fn->size;
            if (!size_fn) {
                return Qnil;
            }
            receiver = (*size_fn)(proc, receiver);
        }
        return receiver;
    }

    if (e->size_fn) {
        return (*e->size_fn)(e->obj, e->args, obj);
    }
    if (e->args) {
        argc = (int)RARRAY_LEN(e->args);
        argv = RARRAY_CONST_PTR(e->args);
    }
    size = rb_check_funcall_kw(e->size, id_call, argc, argv, e->kw_splat);
    if (!UNDEF_P(size)) return size;
    return e->size;
}

with_index(offset = 0) {|(*args), idx| ... }

with_index(offset = 0)

Iterates the given block for each element with an index, which starts from offset. If no block is given, returns a new Enumerator that includes the index, starting from offset

offset: the starting index to use

static VALUE
enumerator_with_index(int argc, VALUE *argv, VALUE obj)
{
    VALUE memo;

    rb_check_arity(argc, 0, 1);
    RETURN_SIZED_ENUMERATOR(obj, argc, argv, enumerator_enum_size);
    memo = (!argc || NIL_P(memo = argv[0])) ? INT2FIX(0) : rb_to_int(memo);
    return enumerator_block_call(obj, enumerator_with_index_i, (VALUE)rb_imemo_memo_new(memo, 0, 0));
}

with_object(obj) {|(*args), obj| ... }

with_object(obj)

Iterates the given block for each element with an arbitrary object, obj, and returns obj

If no block is given, returns a new Enumerator.

Example¶ ↑

to_three = Enumerator.new do |y|
  3.times do |x|
    y << x
  end
end

to_three_with_string = to_three.with_object("foo")
to_three_with_string.each do |x,string|
  puts "#{string}: #{x}"
end

# => foo: 0
# => foo: 1
# => foo: 2

Alias for: each_with_object