
# search engine architecture
# 1. Web Crawling
# 2. Index Building
# 3. Searching

# robots.txt is a file present on websites to provide web crawlers with data

# use the sample code in pagerank dir, and Beautiful Soup code is in bs4


2aeffe1310f5acf99abf12a257172ab78bf7399e

# search engine architecture
# 1. Web Crawling
# 2. Index Building
# 3. Searching

# robots.txt is a file present on websites to provide web crawlers with data

# use the sample code in pagerank dir, and Beautiful Soup code is in bs4

