How to build a powerful crawler like google's? : <p>I want to build a crawler which can update hundreds of thousands of links in several minutes. 
Is there any mature ways to do the scheduling? 
Is distributed system needed? 
What is the greatest barrier that limits the performance? 
Thx.</p>
