I have a function which processes binary data from a file using file.read(len) method. However, my file is huge and is cut into many smaller files 50 MBytes each. Is there some wrapper class that feeds many files into a buffered stream, and provides a read() method?

Class fileinput.FileInput can do such a thing, but it supports only line-by-line reading (method readline() with no arguments) and does not have read(len) with specifying number of bytes to read.

Instead of converting the list of streams into a generator - as some of the other answers do - you can chain the streams together and then use the file interface:

def chain_streams(streams, buffer_size=io.DEFAULT_BUFFER_SIZE):"""Chain an iterable of streams together into a single buffered stream.Usage:def generate_open_file_streams():for file in filenames:yield open(file, 'rb')f = chain_streams(generate_open_file_streams())f.read()"""class ChainStream(io.RawIOBase):def __init__(self):self.leftover = b''self.stream_iter = iter(streams)try:self.stream = next(self.stream_iter)except StopIteration:self.stream = Nonedef readable(self):return Truedef _read_next_chunk(self, max_length):# Return 0 or more bytes from the current stream, first returning all# leftover bytes. If the stream is closed returns b''if self.leftover:return self.leftoverelif self.stream is not None:return self.stream.read(max_length)else:return b''def readinto(self, b):buffer_length = len(b)chunk = self._read_next_chunk(buffer_length)while len(chunk) == 0:# move to next streamif self.stream is not None:self.stream.close()try:self.stream = next(self.stream_iter)chunk = self._read_next_chunk(buffer_length)except StopIteration:# No more streams to chain togetherself.stream = Nonereturn 0  # indicate EOFoutput, self.leftover = chunk[:buffer_length], chunk[buffer_length:]b[:len(output)] = outputreturn len(output)return io.BufferedReader(ChainStream(), buffer_size=buffer_size)

Then use it as any other file/stream:

f = chain_streams(open_files_or_chunks)
f.read(len)