Efficient string matching in Apache Spark : <p>Using an OCR tool I extracted texts from screenshots (about 1-5 sentences each). However, when manually verifying the extracted text, I noticed several errors that occur from time to time.</p>

<p>Given the text "Hello there 😊! I really like Spark ❤️!", I noticed that:</p>

<p>1) Letters like "I", "!", and "l" get replaced by "|".</p>

<p>2) Emojis are not correctly extracted and replaced by other characters or are left out.</p>

<p>3) Blank spaces are removed from time to time.</p>

<p>As a result, I might end up with a string like this: "Hello there 7l | real|y like Spark!"</p>

<p>Since I am trying to match these string against a dataset including the correct text (in thise case "Hello there 😊! I really like Spark ❤️!"), I am looking for an efficient way how to match the string in Spark.</p>

<p>Can anyone suggest an efficient algorithm for Spark which allows me to compare the extract texts (~100.000) against my dataset (~100 million)?</p>
