Unable to exhaust the content of all the identical urls used within my scraper : <p>I've written a scraper in python using BeautifulSoup library to parse all the names traversing different pages of a website. I could manage it if it were not for more than one urls with different pagination, meaning some urls have pagination some does not as the content are few. </p>

<p>My question is: how could I manage to compile them within a function to handle whether they have pagination or not?</p>

<p>My initial attempt (it is able to parse the content from each url's first page only):</p>

<pre><code>import requests 
from bs4 import BeautifulSoup

urls = {
    'https://www.mobilehome.net/mobile-home-park-directory/maine/all',
    'https://www.mobilehome.net/mobile-home-park-directory/rhode-island/all',
    'https://www.mobilehome.net/mobile-home-park-directory/new-hampshire/all',
    'https://www.mobilehome.net/mobile-home-park-directory/vermont/all'
}

def get_names(link):
    r = requests.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    for items in soup.select("td[class='table-row-price']"):
        name = items.select_one("h2 a").text
        print(name)

if __name__ == '__main__':
    for url in urls:
        get_names(url)
</code></pre>

<p>I could have managed to do the whole thing, if there is a single url with pagination like below:</p>

<pre><code>from bs4 import BeautifulSoup 
import requests

page_no = 0
page_link = "https://www.mobilehome.net/mobile-home-park-directory/new-hampshire/all/page/{}"

while True:
    page_no+=1
    res = requests.get(page_link.format(page_no))
    soup = BeautifulSoup(res.text,'lxml')
    container = soup.select("td[class='table-row-price']")
    if len(container)&lt;=1:break 

    for content in container:
        title = content.select_one("h2 a").text
        print(title)
</code></pre>

<p>But, all the urls do not have pagination. So, how can i manage to grab all of them whether there is any pagination or not?</p>
