Web crawler optimization system

TL;DR: In this article, an optimization method of a distributed vertical crawler service system is presented, where the original crawler services system is split into two parts of download service and page analysis logic, and a task queue is also split into a download task queue and an analysis task queue.

TL;DR: A distributed system for large volume deep web data extraction that is extremely scalable, allows multiple heterogeneous concurrent searches, has power web scrape result processing capabilities and uses a well defined, highly customizable, simplified, search agent configuration interface requiring minimal specialized programming knowledge is presented in this paper.

TL;DR: In this paper, a technique for monitoring performance of a computer system is presented, where bucket data is stored that indicates that multiple buckets are associated with a particular type of request. But the assignment may be further based on a complexity determined for each request.

TL;DR: In this article, a data collection method including collecting, by a configuration collector manager, configuration data, including configuration properties, from a plurality of data sources, creating, by model discovery component, a business application model using the configuration data collected by the configuration manager, a collecting, from an application model analysis user interface, edits and confirmations associated with the business application application model, and a analyzing, by feedback analyzer component, the edits and confirmsations associated to the application model and prioritizing the configuration properties based on the data sources.

TL;DR: In this paper, an adaptive data insertion mechanism conducts future available bandwidth prediction/estimation by analyzing recent bandwidth in the transport stream which is consumed by general programs and inserts data by replacement of selected packets within the transportstream.

TL;DR: In this article, the authors propose a method for controlling a management computer connected to a server for permitting communications therebetween, wherein the server transmits to a client the result of processing executed in response to each processing request sent from the client.

TL;DR: In this paper, the authors present a method for managing requests for service over a communications network, which comprises the steps of: receiving a request for service from a customer terminal at a queue server via the communications network; allocating a queue identifier to the request for services; sending the queue identifiers to the customer terminal; receiving the queue identifier from the client terminal at the queue server as part of a subsequent request for servi cation; performing a comparison between queue identifier and queue status information; and forwarding the request to the service host in accordance with the result of the comparison.