A web crawler downloads documents from among a plurality of host computers. The web crawler enqueues document addresses in a data structure called the Frontier. The Frontier generally includes a set of queues, with all document addresses sharing a respective common host component being stored in a respective...http://www.google.es/patents/US6263364?utm_source=gb-gplus-sharePatente US6263364 - Web crawler system using plurality of parallel priority level queues having distinct associated download priority levels for prioritizing document downloading and maintaining document freshness
Web crawler system using plurality of parallel priority level queues having ...
