A key problem of retrieving, integrating and mining rich and high quality information from massive Deep Web Databases (WDBs) online is how to automatically and effectively discover and recognize domain-specific WDBs' entry points, i.e., searchable forms, in the Web. It has been a challenging task because domain-specific WDBs' forms with dynamic and heterogeneous properties are very sparsely distributed over several trillion Web pages. Although significant efforts have been made to address the problem and its special cases, more intelligent and effective solutions remain to be further explored. In this paper, a new architecture of an intelligent agent-based crawler (iCrawler) for domain-specific Deep Web databases has been proposed to address the limitations of the existing methods. The iCrawler, based on intelligent learning agents and domain ontology, and a series of novel and effective strategies, including a two-step page classifier, a link scoring strategy, etc, can improve the performance of the existing methods. Experiments of the iCrawler over a number of real Web pages in a set of representative domains have been conducted and the results show that the iCrawler outperforms the existing domain-specific Deep Web Form-Focused Crawlers (FFCs) in terms of the harvest rate, coverage rate and time performance.

A New Architecture of an Intelligent Agent-Based Crawler for Domain-Specific Deep Web Databases

The Web has been rapidly deepened with the prevalence of databases online. As sources proliferate, while there are often useful, alternative, and related sources for our needs, we are lacking an effective facility to explore this ”deep Web.” For “ad-hoc users” and “system integrators” alike, to enable access and integration to the multitude of sources, we often must answer semantic association questions– How sources relate to each other? What ”vocabularies” do they speak? Such semantic associativity is often revealed holistically through cooccurrence analysis of “schematic metadata,” which describes the nature of data at sources. We observe two interesting phenomena through the syntactic associativity of sources and their schematic metadata: The first phenomenon, occurrence localities, suggests syntactic associativity as a useful notion for discovering semantic associativity, and the second, fuzzy boundaries, suggests a query-driven rank-based mechanism as its realization. We thus propose to build an associativity search facility for systematic exploration of deep Web sources. In its realization, we combine occurrence analysis and link analysis by abstracting occurrence of metadata in sources as links in a graph, which effectively transforms associativity of entities into connectivity of nodes. To quantify the associativity, we propose a wave propagation model; to compute the associativity efficiently, we develop spatial and temporal optimization strategies. We validate the usefulness and efficiency with a real-world dataset of 30,000 sources. The experiments show that syntactic associativity is not only useful for semantic discovery, but also practical as an online search mechanism.

https://www.ideals.illinois.edu/bitstream/handle/2142/11170/Exploring%20the%20Deep%20Web%20Associativity%20Search%20over%20Schematic%20Metadata.pdf?sequence=2

Exploring the Deep Web: Associativity Search over Schematic Metadata

No other medium has taken a more meaningful place in our life in such a short time than the world-wide largest data network, the World Wide Web. However, when searching for information in the data network, the user is constantly exposed to an ever-growing flood of information. This is both a blessing and a curse at the same time. While many users only scratch at the surface of the Web via index-based search engines, an almost endless amount of sources providing high quality information often stored in databases can be found in the depths of the WWW. In order to take advantage of these sources when searching the Web novel techniques are required facilitating the integration of heterogeneous Deep Web with minimal effort and enabling a semantic interpretation of their published content. With respect to the Semantic Web, which focuses on computer-computer and human-computer interactions, the Web still resides in its early stages of development. It is, however, not to be expected that the expenditure of a complete semantic annotation of all web inventories can be done manually. Instead, the development of automatic extraction, annotation, and integration systems operating on representation-oriented HTML pages will play a significant role in this context. In this thesis, the first challenge that is faced is the development of a fully automated wrapper system. Various new approaches will be presented to solve specific problems in the context of web data extraction. The spectrum of the presented methods ranges from visual document analysis techniques for the localization and relevance ranking of similar structured data regions to the reorganization of extracted data records and the automatic annotation of aligned data items. Finally, by materializing derived extraction and annotation rules we are able to automatically generate a wrapper specification for congeneric web pages which can be used to efficiently transform web data into arbitrary target formats or just integrate the extracted web data into local database systems.

ViPER: visual perception based information extraction of structured web content.

Antecedentes: El acceso efectivo a la informacion geoespacial adquiere una importancia fundamental en una economia basada en el conocimiento. Es esencial para muchas actividades tener acceso a los recursos geoespaciales a traves de la Web. Desde la respuesta a un desastre a la decision de desarrollar un nuevo negocio, el acceso a informacion geoespacial actualizada ofrecida a traves de la Web Geoespacial puede ser la diferencia entre el exito y el fracaso. Se define la Web Geoespacial como la coleccion de servicios Web, datos geoespaciales y metadatos que permiten el uso de datos geoespaciales en una amplia gama de aplicaciones de dominio. Esta tesis trata tres problemas relacionados con la Web Geoespacial. En primer lugar, los servicios estandar de la Web Geoespacial forman parte de la Web invisible. La literatura utiliza el termino invisible para identificar la parte de la Web ignorada por los motores de busqueda (invisible Web). Los usuarios comunes pueden estar al tanto del contenido de la Web invisible solo por casualidad. A continuacion, esta tesis clasifica el contenido de la Web Geoespacial accesible a traves de servicios estandar como parte de la Web profunda (deep Web). El termino profunda destaca que un contenido invisible tiene suficiente valor en si mismo para ser tratado como si estuviera en la superficie de la Web, en otras palabras, merece ser indexado por un motor de busqueda. Los contenidos de la Web Geoespacial son parte de la Web profunda por su propia naturaleza. Es decir, los contenidos de la Web Geoespacial se ocultan detras de servicios Web. Por ultimo, los servicios estandar y contenidos de la Web Geoespacial estan socialmente desconectados del resto de la Web. Los servicios publicos de la Web Geoespacial no estan concebidos para servir a su funcionalidad a los consumidores de servicios y, al mismo tiempo, como herramienta social para la comunicacion entre los agentes relacionados con los servicios.
Actualmente, la vinculacion de los servicios Web con la iniciativa de Datos abiertos enlazados (Linked Open Data), una de las materializaciones de la Web Semantica, parece prometer dicha posibilidad.

Objetivo: Los problemas derivados de la invisibilidad de los servicios Web Geoespaciales podria mitigarse si es posible realizar un rastreo sistematico y enfocado de la Web en busca de dichos servicios en combinacion con la publicacion como Datos enlazados (Linked Data) de las descripciones y los contenidos de los servicios descubiertos. Esta tesis analiza la viabilidad de una solucion basada en el desarrollo de una arana Web (Web
crawler) enfocada a los servicios Web geoespaciales, y el acceso a la informacion sobre los servicios descubiertos como descripciones procesables por maquinas. Ademas, la solucion analizada debera permitir el acceso a contenido geoespacial remoto en el mismo formato. Este enfoque requiere el desarrollo de dos ontologias de dominio. La primera ontologia describe la arquitectura abstracta de los serviciosWeb geoespaciales. La segunda ontologia describe el contenido geoespacial.

Ambito. El rastreo se restringe a los servicios Web compatibles con las especificaciones de Open Geospatial Consortium (OGC). OGC lidera el desarrollo de especificaciones abiertas y estandarizadas interfaz de servicios Web para acceder a la informacion geoespacial desde 1994. Mas de 400 companias, agencias gubernamentales y universidades son miembros de OGC, y participan en los procesos de consenso para desarrollar a disposicion del publico las normas de interfaz de servicios Web para el acceso a los datos geoespaciales. Muchas especificaciones OGC que definen la interfaces de servicios Web se han convertido en estandares ISO (International Organization for the Standardization). Adicionalmente, el rastreo se limita a las especificaciones OGC de servicios Web relacionadas con la Directiva Europea INSPIRE. La Directiva INSPIRE es el marco mas grande y mas reciente de tamano continental que establece normas compartidas entre los paises europeos para el acceso a la informacion geografica digital. Los servicios Web de OGC han sido senalados como una de las posibles implementaciones de los servicios de INSPIRE. Esta restriccion garantiza que las contribuciones de esta tesis pueden beneficiar a investigadores y desarrolladores interesados en la aplicacion de la Directiva INSPIRE (2010-2019). La conceptualizacion de la arquitectura de servicios Web de OGC debe ser reutilizable, y debe evitar una fuerte dependencia de la terminologia de OGC. La arquitectura de servicios Web de OGC esta basada en la familia de normas RM-ODP (Reference Model of Open Distributed Processing, ISO / IEC 10746-2 Foundations, ISO / IEC 10746-3 Arquitectura, ISO 15414 Enterprise language). RM-ODP es un modelo de referencia que proporciona un marco para la normalizacion de los sistemas abiertos de procesamiento distribuido. La terminologia, los conceptos y las relaciones utilizadas para la conceptualizacion de esta tesis se derivan principalmente de la RM-ODP. Estos conceptos se aplican a conceptualizar el conocimiento del dominio que se encuentran en la especificacion abstracta de OGC para los servicios Web y en la recopilacion de normas especificas de OGC para interfaces de servicios Web. Cuando sea necesario, el conocimiento se formalizara mediante Logica Descriptiva, una familia de formalismos de representacion del conocimiento. Logica Descriptiva esta equipada con una semantica formal, basada en la logica, que proporciona los procedimientos de inferencia para derivar conocimiento implicito del conocimiento que esta explicitamente representada. El uso de Logica Descriptiva se ve obstaculizado por la complejidad computacional de algunos de sus algoritmos de inferencia. Sin embargo, se reconoce que sistemas relativamente expresivos de tamano real con problemas de razonamiento exponenciales en tiempo pueden ser procesados en un tiempo razonable. En esta tesis se formalizara el conocimiento tomando como
limite la Logica Descriptiva SROIQ(D). Esta Logica Descriptiva tiene propiedades interesantes de
computo. Razonadores populares, como Pellet y HermiT, soportan SROIQ(D). Ademas, el lenguaje de ontologias OWL 2 (Web Ontology Language 2) es compatible con algunas restricciones con SROIQ(D). Solo dos tipos de accesos a contenidos Geoespaciales se consideran en esta tesis: el acceso a los metadatos geoespaciales almacenados en OGC CSW y el acceso a los datos geoespaciales almacenados en OGC WFS. Por otra parte, el acceso a los metadatos geoespaciales almacenados en las instancias de servicio CSW se limita a los metadatos de los registros cuyo esquema de metadatos tiene una correspondencia conocida por el modelo abstracto de Dublin Core Metadata Initiative. El acceso a los datos geoespaciales se limita a los casos de servicios WFS que devuelvan fenomenos, es decir, representaciones de lugares, que se puedan corresponder con una extension del meta modelo GKB, que fue desarrollado en el proyecto GREASE.

Metodo: La aproximacion metodologia de esta tesis tiene dos aspectos: uno relacionado con la
ingenieria de software y otro relacionados con la ingenieria del conocimiento. La metodologia en la
parte relacionada con la ingenieria de software es un desarrollo clasico incremental de la solucion. La
metodologia en la parte de relacionada con la ingenieria del conocimiento se basa en Methontology.

Importancia social, experiencia previa, trabajo futuro. El desarrollo de una Web Geoespacial
publica, abierta e interoperable es consecuencia de las iniciativas para el desarrollo de Infraestructuras
de Datos Espaciales (IDEs). Por ejemplo, la iniciativa europea INSPIRE (materializado en
2007 como Directiva Europea) tiene por objeto crear una IDE europea. Una de las lineas de investigacion
del grupo de investigacion IAAA se centra en aspectos relacionados con las las IDEs como
la descripcion de los datos geoespaciales y los servicios, el descubrimiento de estos recursos a traves
de catalogos estandar, y los aspectos conceptuales y arquitectonicos relacionados con datos geoespaciales
y servicios. Algunos resultados de la investigacion de la linea de investigacion de IDEs, donde
ha participado el autor son la exploracion de nuevas alternativas para el descubrimiento de servicios
geoespaciales, y el estudio de nuevas formas de dar acceso a los datos geoespaciales y los metadatos. Este trabajo ha consistido en la caracterizacion de modelos de contenido, el desarrollo de conjuntos de datos geoespaciales, y la colaboracion en la formalizacion y la produccion de grandes geo-ontologias. Ademas, el autor ha participado en la elaboracion de normas para el contenido geoespacial, y servicios geoespaciales. Esta tesis es resultado de la investigacion citada. Las lineas de trabajo futuro estan orientadas a mejorar las contribuciones relacionadas con en el descubrimiento de serviciosWeb geoespaciales, la formalizacion de los modelos y el acceso a los datos geoespaciales.

Semantic linkage of the invisible geospatial web

A key problem of retrieving, integrating and mining rich and high quality information from massive Deep Web Databases (WDBs) online is how to automatically and effectively discover and recognize domain-specific WDBs' entry points, i.e., forms, in the Web. It has been a challenging task because domain-specific WDBs' forms with dynamic and heterogeneous properties are very sparsely distributed over several trillion Web pages. Although significant efforts have been made to address the problem and its special cases, more effective solutions remain to be further explored towards achieving both the satisfactory harvest rate and coverage rate of domain-specific WDBs' forms simultaneously. In this paper, an Enhanced Form-Focused Crawler for domain-specific WDBs (E-FFC) has been proposed as a novel framework to address existing solutions' limitations. The E-FFC, based on the divide and conquer strategy, employs a series of novel and effective strategies/algorithms, including a two-step page classifier, a link scoring strategy, classifiers for advanced searchable and domain-specific forms, crawling stopping criteria, etc. to its end achieving the optimized harvest rate and coverage rate of domain-specific WDBs' forms simultaneously. Experiments of the E-FFC over a number of real Web pages in a set of representative domains have been conducted and the results show that the E-FFC outperforms the existing domain-specific Deep Web Form-Focused Crawlers in terms of the harvest rate, coverage rate and crawling robustness.

E-FFC: an enhanced form-focused crawler for domain-specific deep web databases

The Web has been rapidly “deepened” by massive databases online: Recent surveys show that while the surface Web has linked billions of static HTML pages, a far more significant amount of information is “hidden” in the deep Web, behind the query forms of searchable databases. With its myriad databases and hidden content, this deep Web is an important frontier for information search. In this paper, we develop a novel Web Form Crawler to collect the “doors” of Web databases, i.e., query forms, to build a database for online databases in both efficient and comprehensive manners. Being object-focused, topic-neutral and coveragecomprehensive, such a crawler, while critical to searching and integrating online databases, has not been extensively studied. In particular, query forms, while many, when compared with the size of the Web, are sparsely scattered among pages, which brings new challenges for focused crawling: First, due to the topic-neutral nature of our crawling problem, we cannot rely on existing topicfocused crawling techniques. Second, existing focused crawling cannot achieve the comprehensiveness requirement because it is not able to be aware of the coverage of crawled content. As a new attempt, we propose a structure-driven crawling framework by observing structure locality of query forms– That is, query forms are often close to root pages of Web sites and accessible by following navigational links. Exploring this structure locality, we substantiate the structure-driven crawling framework into a site-based Web Form Crawler by first collecting the site entrances, as the Site Finder, and then searching for query forms within the scope of each site, as the Form Finder. Analytical justification and empirical evaluation of the Web Form Crawler both show that: 1) our crawler can maintain stable harvest and coverage throughout the crawling, and 2) compared to page-based crawling, our best harvest rate is about 10 to 400 times better, depending on the page traversal schemes used.

/pdf/a-structure-driven-yield-aware-web-form-crawler-building-a-5cf4jg277d.pdf

A Structure-Driven Yield-Aware Web Form Crawler: Building a Database of Online Databases

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

David Killian

Author Tools

Chat about Author

Papers

A Structure-Driven Yield-Aware Web Form Crawler: Building a Database of Online Databases