I am trying to find the center point of the green rectangle which is behind the fish, but my approach is not working. Here is my code:

#Finding contours (almost always finds those 2 retangles + some noise):
_, conts, hierarchy = cv2.findContours(img_green, cv2.RETR_TREE , cv2.CHAIN_APPROX_SIMPLE)
for cnt in conts:area = cv2.contourArea(cnt)#filter noiseif area > 25:M = cv2.moments(cnt)x1, y1, w, h = cv2.boundingRect(cnt)x2 = x1 + w                           # (x1, y1) = top-left vertexy2 = y1 + h                           # (x2, y2) = bottom-right vertexcy = int(M['m01']/M['m00'])           # (cx, cy) = rect centercx = int(M['m10']/M['m00'])rect = cv2.rectangle(green_bar_win, (x1, y1), (x2, y2), (255,0,0), 2)center = cv2.circle(green_bar_win, (cx, cy), 2, (0,0,255), 4)

As you can see, it finds the rectangle's contour but divided where the fish is, making 2 different shapes. It also finds the center of this 2 shapes (the blue points), however I don't know how to find the middle of the big one. I thought about averaging all the found rectangle centers but I don't know how to write this out. I am finding the rectangles by hsv color. Help?

EDIT: I have 'y1' from the top rectangle, but don't know how to get y2 from the bottom one while inside the for loop. I tried this:

_, conts, hierarchy = cv2.findContours(img_green, cv2.RETR_TREE , cv2.CHAIN_APPROX_SIMPLE)
for cnt in conts:area = cv2.contourArea(cnt)#filter noiseif area > 25:x1, y1, w, h = cv2.boundingRect(cnt)x2 = x1 + w                           # (x1, y1) = top-left vertextry:rect_center = np.average([y1, y2])except:print("Failed to average")y2 = y1 + h                           # (x2, y2) = bottom-right vertexrect = cv2.rectangle(green_bar_win, (x1, y1), (x2, y2), (255,0,0), 2)

But it still fails, because y2 is used before assigment. So, how could I get y2 from the second loop itineration before y1 is overwritten by the 'for' loop?

It's time to break this to a full answer.

After you identify each rectangle, compare its corners to each existing shape. If it has the same color and a pair of corners in common (shared edge), then update the old rectangle: replace those two corners with the other corners of the new rectangle.

Add a couple of things to your code:

1. keep a list of the shapes found so far

2. as you find each shape, compare to the ones found already.

   rect_list = []

   for cnt in conts:area = cv2.contourArea(cnt)

   #filter noise
   if area > 25:M = cv2.moments(cnt)# ... several lines deletedrect = cv2.rectangle(green_bar_win, (x1, y1), (x2, y2), (255,0,0), 2)center = cv2.circle(green_bar_win, (cx, cy), 2, (0,0,255), 4)# Look for adjacent shapes herefor old_rect in rect_list:# fetch the corners of old_rect# if old_rect has a pair of corners in common with#     the current rect, then merge the two.#     Do this by expanding the corners of old_rect.else:rect_list.append(rect)