Proceedings Article10.1145/1013367.1013529
Distributed location aware web crawling
Odysseas Papapetrou,George Samaras +1 more
- 19 May 2004
- pp 468-469
TL;DR: This work proposes a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner.
read more
Abstract: Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Journal Article
Parallel crawler architecture and web page change detection
TL;DR: The paper discuses a fresh approach for parallel crawling the web using multiple machines and integrates the trivial issues of crawling also, using a three-step algorithm for page refreshment.
33
•Journal Article
Topical web crawling using weighted anchor text and web page change detection techniques
TL;DR: A technique called weighted anchor text which uses the link structure to form the weighted directed graph of anchor texts which can be very useful when incorporated with other existing algorithms.
Web Crawlers: Taxonomy, Issues & Challenges
Rajender Nath,Khyati Chopra +1 more
- 01 Jan 2013
TL;DR: This paper makes an attempt to classify all the existing crawlers on certain parameters and also identifies the various challenges to web crawlers.
9
Challenges in Using Peer-to-Peer Structures in Order to Design a Large-Scale Web Search Engine
Hamid Mousavi,Ali Movaghar +1 more
- 09 Mar 2008
TL;DR: Challenges in using P2P structures to design a large-scale WSE are introduced and the best model may be the use of a special case of Super-Peer Networks which is yet conditioned on the peers’ active and trustful contributions.
2
References
UCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers
Odysseas Papapetrou,Stavros Papastavrou,George Samaras +2 more
- 03 Sep 2003
TL;DR: An alternative distributed crawling method with the use of mobile agents is suggested that minimizes network utilization, keeps up with document changes, employs time realization, and is easily upgradeable.
Distributed Indexing of the Web Using Migrating Crawlers.
Odysseas Papapetrou,Stavros Papastavrou,George Samaras +2 more
- 01 Jan 2003
TL;DR: In this article, the authors suggest an alternative distributed crawling method with the use of mobile agents, which minimizes network utilization, keeps up with document changes, employs time realization, and is easily upgradeable.
Related Papers (5)
Odysseas Papapetrou,George Samaras +1 more
- 25 Oct 2004
Odysseas Papapetrou,George Samaras +1 more
- 01 Jan 2004
Weizheng Gao,Hyun Chul Lee,Yingbo Miao +2 more
- 23 May 2006