itertools.product causes an OOM crash on infinite generator

Why does itertools.product crash on an infinite generator even though it is an "iterator," and how do I process these combinations without loading the entire stream into RAM?

Given that an iterable can be a generator, shallow copying it won't result in a new iterable that will produce values starting from the beginning. Also note that generators cannot be deepcopied. Consider the following:

def stream():
    yield "a"
    yield "b"
    yield "c"
it = stream()
for i in itertools.product([0, 1], it):
    print(i)

The above will produce the outputs (0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), and (1, 'c'). Note how it, being an iterator, has produced two effective streams of output paired under both 0 and 1. At some point all the values produced by the iterator may need to be reproduced on subsequent runs. Given that shallow cloning an iterator will not reproduce outputs from beginning, memory will need to be allocated to store all values produced by the provided generator. The underlying C implementation of product simply front-load this step by collecting everything from every incoming iterable into their corresponding tuple before yielding the products, i.e. trigger OOM on infinite generator before a single value being yielded, rather than yielding values until memory exhaustion from tracking values from the infinite generator.

If you need to make use of infinite generator to produce some subset of values, just use nested for loops that yield values. For example, if you have something like:

def demo():
    for i in itertools.product(infinitestream(), stream()):
        yield i 

def demo(): 
    for a in infinitestream():
        for b in stream():
            yield (a, b)

This way works as every iteration of the outer loop, stream() will produce a brand new iterator to yield values in the inner loop so new values are produced, and without additional cached values eating up memory somewhere.

Of course, if product were to take callables that produce an iterable as an argument, there wouldn't be a need to cache values as every inner loop will produce a new iterable from the callable.

There is no obvious solution to that problem, even if you reimplemented your own function product.

What would you expect the output to be?

product('ab', [1,2,3])
yields [('a', 1), ('a', 2), ('a', 3), ('b', 1), ('b', 2), ('b', 3)]
product('ab', [1,2,3,...to infinity])
yields ('a', 1), ('a', 2), ('a', 3), ...to infinity... ('b', 1), ('b', 2), ('b', 3), ...to infinity again...

The combinations starting with 'b' are never yielded, because there would be an infinite stream of combinations starting with 'a' first.

There are ways to yield all combinations even with an infinite stream, but it requires a diagonal enumeration, and this would deviate significantly from the usual simple behaviour of product, and if more than one stream is infinite, it would still eventually cause memory problems as the diagonals grow longer and longer.