opencv - cropping handwritten lines (line segmentation) : <p>I'm trying to build a handwriting recognition system using python and opencv.
The recognition of the characters is not the problem but the segmentation.
I have successfully :</p>

<ul>
<li>segmented a word into single characters</li>
<li>segmented a <strong>single sentence</strong> into words in the required order.</li>
</ul>

<p>But I couldn't segment different lines in the document. I tried sorting the contours (to avoid line segmentation and use only word segmentation) but it didnt work.
I have used the following code to segment words contained in a handwritten document , but it returns the words out-of-order(it returns words in left-to-right sorted manner) :</p>

<pre><code>import cv2
import numpy as np
#import image
image = cv2.imread('input.jpg')
#cv2.imshow('orig',image)
#cv2.waitKey(0)

#grayscale
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
cv2.imshow('gray',gray)
cv2.waitKey(0)

#binary
ret,thresh = cv2.threshold(gray,127,255,cv2.THRESH_BINARY_INV)
cv2.imshow('second',thresh)
cv2.waitKey(0)

#dilation
kernel = np.ones((5,5), np.uint8)
img_dilation = cv2.dilate(thresh, kernel, iterations=1)
cv2.imshow('dilated',img_dilation)
cv2.waitKey(0)

#find contours
im2,ctrs, hier = cv2.findContours(img_dilation.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

#sort contours
sorted_ctrs = sorted(ctrs, key=lambda ctr: cv2.boundingRect(ctr)[0])

for i, ctr in enumerate(sorted_ctrs):
    # Get bounding box
    x, y, w, h = cv2.boundingRect(ctr)

    # Getting ROI
    roi = image[y:y+h, x:x+w]

    # show ROI
    cv2.imshow('segment no:'+str(i),roi)
    cv2.rectangle(image,(x,y),( x + w, y + h ),(90,0,255),2)
    cv2.waitKey(0)

cv2.imshow('marked areas',image)
cv2.waitKey(0)
</code></pre>

<p>Please note that i am able to segment all the words here <strong>but they appear out order.Is there any way to sort these contours in order of top to bottom</strong></p>

<p><strong>OR</strong></p>

<p><strong>segment the image into separate lines so that each line can be segmented into words using above code?</strong></p>
