I am using custom generator to pass my data. But i keep encountering an error which says i have run out of data and to use repeat() when passing the dataset. i am using plain generator therefore it is not possible to use repeat(). can someone help me to fix this issue

import os
import numpy as np
import cv2
def generator(idir,odir,batch_size,shuffle ):i_list=os.listdir(idir)o_list=os.listdir(odir)batch_index=0batch_size = batch_sizesample_count=len(i_list)while True:input_image_batch=[]output_image_batch=[]for i in range(batch_index * batch_size, (batch_index + 1) * batch_size  ): #iterate for  a batchj=i % sample_count # cycle j value over range of available  imagesk=j % batch_size  # cycle k value over batch sizeif shuffle == True: # if shuffle select a random integer between 0 and sample_count-1 to pick as the image=label pairm=np.random.randint(low=0, high=sample_count-1, size=None, dtype=int) else:m=jpath_to_in_img=os.path.join(idir,i_list[m])path_to_out_img=os.path.join(odir,o_list[m])print(path_to_in_img,path_to_out_img)input_image=cv2.imread(path_to_in_img)input_image=cv2.resize(input_image,(3200,3200))#create the target image from the input image output_image=cv2.imread(path_to_out_img)output_image=cv2.resize(output_image,(3200,3200))input_image_batch.append(input_image)output_image_batch.append(output_image)input_val1image_array=np.array(input_image_batch) input_val1image_array = input_val1image_array / 255.0print (input_val1image_array)output_val2image_array=np.array(output_image_batch)output_val2image_array = output_val2image_array / 255.0batch_index= batch_index + 1 yield (input_val1image_array, output_val2image_array)if batch_index * batch_size > sample_count:break

Calling the function

    idir = r"D:\\image\\"odir=r"D:\\image1\\"train = generator(idir,odir,4,True)model.compile(optimizer="adam", loss='mean_squared_error', metrics=['mean_squared_error'])model.fit(train,validation_data = (valin_images,valout_images),batch_size= 5,epochs = 20,steps_per_epoch = int(560/batch_size))

The error

Epoch 1/20
186/186 [==============================] - 475s 3s/step - loss: 1779.7604 - mean_squared_error: 1779.7601 - val_loss: 28278.5488 - val_mean_squared_error: 28278.5488
Epoch 2/201/186 [..............................] - ETA: 1:41 - loss: 275.7113 - mean_squared_error: 275.7113WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 3720 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 187 batches). You may need to use the repeat() function when building your dataset.
186/186 [==============================] - 1s 235us/step - loss: 275.7113 - mean_squared_error: 275.7113

If you aren't using repeat (and even if you are it's good for debugging) the first thing you need to check is how many elements your generator creates. a quick way to do that would be with something like

len([g for g in generator(idir,odir,4,True)])

Then you need to make sure that your steps per epoch times your batch size is less than that number.

You can use repeat even with that generator though - you just need to wrap it with a tf.dataset like this:

gen = lambda : generator(idir,odir,4,True)
dataset = tf.data.Dataset.from_generator(gen, output_types=(<types>), output_shapes=(<shapes>)).repeat()

you have to specify output type and shape but then it works fine.