Navigating through links using Selenium in Python : <p>I'm trying to scrape data from a site with multiple pages linked via a <strong>NEXT</strong> button </p>

<p>The successive page URL has no correspondence with the previous page URL as one might assume</p>

<p>(In that case modifying the path would've solved the problem)</p>

<p>This is what I plan to do -</p>

<p>1.Start with an initial URL </p>

<p>2.Extract information</p>

<p>3.Click <strong>NEXT</strong></p>

<p>Repeat 2 and 3 <em>n</em> times</p>

<p>Specifically, I wanted to know how to get the new page URL on clicking </p>

<p>This is what I've come up with so far</p>

<pre><code>def startWebDriver():
    global driver
    options = Options()
    options.add_argument("--disable-extensions")
    driver = webdriver.Chrome(executable_path = '/path/to/driver/chromedriver_linux64/chromedriver',options=options)

#URL of the initial page
driver.get('https://openi.nlm.nih.gov/detailedresult.php?img=CXR1_1_IM-0001-3001&amp;query=&amp;coll=cxr&amp;req=4&amp;npos=1')

time.sleep(4)

#XPATH of the "NEXT" button
element = driver.find_element_by_xpath('//*[@id="imageClassM"]/div/a[2]/img').click()
</code></pre>

<p>Any help would be appreciated</p>
