Abstract: This paper presents a novel end-to-end system for table understanding in document images called DeepDeSRT In particular, the contribution of DeepDeSRT is two-fold First, it presents a deep learning-based solution for table detection in document images Secondly, it proposes a novel deep learning-based approach for table structure recognition, ie identifying rows, columns, and cell positions in the detected tables In contrast to existing rule-based methods, which rely on heuristics or additional PDF metadata (like, for example, print instructions, character bounding boxes, or line segments), the presented system is data-driven and does not need any heuristics or metadata to detect as well as to recognize tabular structures in document images Furthermore, in contrast to most existing table detection and structure recognition methods, which are applicable only to PDFs, DeepDeSRT processes document images, which makes it equally suitable for born-digital PDFs (as they can automatically be converted into images) as well as even harder problems, eg scanned documents To gauge the performance of DeepDeSRT, the system is evaluated on the publicly available ICDAR 2013 table competition dataset containing 67 documents with 238 pages overall Evaluation results reveal that DeepDeSRT outperforms state-of-the-art methods for table detection and structure recognition and achieves F1-measures of 9677% and 9144% for table detection and structure recognition, respectively Additionally, DeepDeSRT is evaluated on a closed dataset from a real use case of a major European aviation company comprising documents which are highly unlike those in ICDAR 2013 Tested on a randomly selected sample from this dataset, DeepDeSRT achieves high detection accuracy for tables which demonstrates the sound generalization capabilities of our system

Abstract: This article introduces TableMiner+, a Semantic Table Interpretation method that annotates Web tables in a both effective and efficient way. Built on our previous work TableMiner, the extended version advances state-of-the-art in several ways. First, it improves annotation accuracy by making innovative use of various types of contextual information both inside and outside tables as features for inference. Second, it reduces computational overheads by adopting an incremental, bootstrapping approach that starts by creating preliminary and partial annotations of a table using ‘sample’ data in the table, then using the outcome as ‘seed’ to guide interpretation of remaining contents. This is then followed by a message passing process that iteratively refines results on the entire table to create the final optimal annotations. Third, it is able to handle all annotation tasks of Semantic Table Interpretation (e.g., annotating a column, or entity cells) while state-of-the-art methods are limited in different ways. We also compile the largest dataset known to date and extensively evaluate TableMiner+ against four baselines and two re-implemented (near-identical, as adaptations are needed due to the use of different knowledge bases) state-of-the-art methods. TableMiner+ consistently outperforms all models under all experimental settings. On the two most diverse datasets covering multiple domains and various table schemata, it achieves improvement in F1 by between 1 and 42 percentage points depending on specific annotation tasks. It also significantly reduces computational overheads in terms of wall-clock time when compared against classic methods that ‘exhaustively’ process the entire table content to build features for inference. As a concrete example, compared against a method based on joint inference implemented with parallel computation, the non-parallel implementation of TableMiner+ achieves significant improvement in learning accuracy and almost orders of magnitude of savings in wall-clock time.

Abstract: Generating texts from structured data (e.g., a table) is important for various natural language processing tasks such as question answering and dialog systems. In recent studies, researchers use neural language models and encoder-decoder frameworks for table-to-text generation. However, these neural network-based approaches do not model the order of contents during text generation. When a human writes a summary based on a given table, he or she would probably consider the content order before wording. In a biography, for example, the nationality of a person is typically mentioned before occupation in a biography. In this paper, we propose an order-planning text generation model to capture the relationship between different fields and use such relationship to make the generated text more fluent and smooth. We conducted experiments on the WikiBio dataset and achieve significantly higher performance than previous methods in terms of BLEU, ROUGE, and NIST scores.

Abstract: Probabilistic planners have improved recently to the point that they can solve difficult tasks with complex and expressive models. In contrast, learners cannot tackle yet the expressive models that planners do, which forces complex models to be mostly handcrafted. We propose a new learning approach that can learn relational probabilistic models with both action effects and exogenous effects. The proposed learning approach combines a multi-valued variant of inductive logic programming for the generation of candidate models, with an optimization method to select the best set of planning operators to model a problem. We also show how to combine this learner with reinforcement learning algorithms to solve complete problems. Finally, experimental validation is provided that shows improvements over previous work in both simulation and a robotic task. The robotic task involves a dynamic scenario with several agents where a manipulator robot has to clear the tableware on a table. We show that the exogenous effects learned by our approach allowed the robot to clear the table in a more efficient way.

Abstract: Feature engineering is one of the most important and time consuming tasks in predictive analytics projects It involves understanding domain knowledge and data exploration to discover relevant hand-crafted features from raw data In this paper, we introduce a system called One Button Machine, or OneBM for short, which automates feature discovery in relational databases OneBM automatically performs a key activity of data scientists, namely, joining of database tables and applying advanced data transformations to extract useful features from data We validated OneBM in Kaggle competitions in which OneBM achieved performance as good as top 16% to 24% data scientists in three Kaggle competitions More importantly, OneBM outperformed the state-of-the-art system in a Kaggle competition in terms of prediction accuracy and ranking on Kaggle leaderboard The results show that OneBM can be useful for both data scientists and non-experts It helps data scientists reduce data exploration time allowing them to try and error many ideas in short time On the other hand, it enables non-experts, who are not familiar with data science, to quickly extract value from their data with a little effort, time and cost

Abstract: Tables are among the most powerful and practical tools for organizing and working with data. Our motivation is to equip spreadsheet programs with smart assistance capabilities. We concentrate on one particular family of tables, namely, tables with an entity focus. We introduce and focus on two specific tasks: populating rows with additional instances (entities) and populating columns with new headings. We develop generative probabilistic models for both tasks. For estimating the components of these models, we consider a knowledge base as well as a large table corpus. Our experimental evaluation simulates the various stages of the user entering content into an actual table. A detailed analysis of the results shows that the models' components are complimentary and that our methods outperform existing approaches from the literature.

Abstract: Generating texts from structured data (e.g., a table) is important for various natural language processing tasks such as question answering and dialog systems. In recent studies, researchers use neural language models and encoder-decoder frameworks for table-to-text generation. However, these neural network-based approaches typically do not model the order of content during text generation. When a human writes a summary based on a given table, he or she would probably consider the content order before wording. In this paper, we propose an order-planning text generation model, where order information is explicitly captured by link-based attention. Then a self-adaptive gate combines the link-based attention with traditional content-based attention. We conducted experiments on the WikiBio dataset and achieve higher performance than previous methods in terms of BLEU, ROUGE, and NIST scores; we also performed ablation tests to analyze each component of our model.

Abstract: Tables are an easy way to represent information in a structural form Table recognition is important for the extraction of such information from document images Usually, modern OCR systems provide textual information coming from tables without recognizing actual table structure However, recognition of table structure is important to get the contextual meaning of the contents Table structure recognition in heterogeneous documents is challenging due to a variety of table layouts It becomes harder where no physical rulings are present in a table This work proposes a novel learning based methodology for the recognition of table contents in heterogeneous document images Textual contents of documents are classified as table or non-table elements using a pre-trained neural network model The output of the neural network is further enhanced by applying a contextual post processing on each element to correct the classifications errors if any The system is trained using a subset of UNLV and UW3 document images and depicted more than 97% accuracy on a test set in detection of table and non-table elements

Abstract: Tables are among the most powerful and practical tools for organizing and working with data. Our motivation is to equip spreadsheet programs with smart assistance capabilities. We concentrate on one particular family of tables, namely, tables with an entity focus. We introduce and focus on two specifc tasks: populating rows with additional instances (entities) and populating columns with new headings. We develop generative probabilistic models for both tasks. For estimating the components of these models, we consider a knowledge base as well as a large table corpus. Our experimental evaluation simulates the various stages of the user entering content into an actual table. A detailed analysis of the results shows that the models' components are complimentary and that our methods outperform existing approaches from the literature.

Abstract: With the growth of big data systems and ubiquitous computing, privacy has become a critical issue in security research. Purpose based access control model is a common approach for privacy preserving in data access for database management systems. However, previous works that are based on purposes ranging from the table level to the data cell level and that are extended with role-based access control model inherently suffer from the problem of role explosion and cumbersomeness in context aware policy specification. Besides, NoSQL databases have recently become increasingly popular as data platforms for big data and real-time web applications. Due to the simplicity in design but effectiveness in horizontal scaling and performance, using NoSQL databases are a better alternative approach in comparison with traditional relational databases. However, the lack of a fine-grained access control system with data privacy protection is one of the most important considerations in NoSQL databases. In this paper, we address this issue by proposing and implementing a comprehensive mechanism for enforcing document store attribute-based security policies together with an improved data privacy protection mechanism in the fine-grained level. We use Polish notation for modeling conditional expressions which are the combination of subject, resource, and environment attributes so that privacy policies are flexible, dynamic and fine grained. Furthermore, privacy rules are constrained not only by access and intended purposes but also by subject, resource, and environment attributes as well as levels of data disclosure. The experiments have been carried out to illustrate the execution time of evaluating access control policies and privacy policies in various data sizes.

Abstract: Context: Databases are an integral element of enterprise applications. Similarly to code, database schemas are also prone to smells - best practice violations. Objective: We aim to explore database schema quality, associated characteristics and their relationships with other software artifacts. Method: We present a catalog of 13 database schema smells and elicit developers' perspective through a survey. We extract embedded SQL statements and identify database schema smells by employing the DbDeo tool which we developed. We analyze 2925 production-quality systems (357 industrial and 2568 well-engineered open-source projects) and empirically study quality characteristics of their database schemas. In total, we analyze 629 million lines of code containing more than 393 thousand SQL statements. Results: We find that the index abuse smell occurs most frequently in database code, that the use of an ORM framework doesn't immune the application from database smells, and that some database smells, such as adjacency list, are more prone to occur in industrial projects compared to open-source projects. Our co-occurrence analysis shows that whenever the clone table smell in industrial projects and the values in attribute definition smell in open-source projects get spotted, it is very likely to find other database smells in the project. Conclusion: The awareness and knowledge of database smells are crucial for developing high-quality software systems and can be enhanced by the adoption of better tools helping developers to identify database smells early.

Abstract: This letter describes and applies generic methods for generating local measures from the correspondence table. These were developed by integrating the functionality of two existing R packages: gwxt...

Abstract: Entity matching, or record linkage, is the task of identifying records that refer to the same entity. Naive entity matching techniques (i.e., brute-force pairwise comparisons) have quadratic complexity. A typical shortcut to the problem is to employ blocking techniques to reduce the number of comparisons, i.e. to partition the data in several blocks and only compare records within the same block. While classic blocking methods are designed for data from relational databases with clearly defined schemas, they are not applicable to data from Web tables, which are more prone to noise and do not come with an explicit schema. At the same time, Web tables are an interesting data source for many knowledge intensive tasks, which makes record linkage on Web Tables an important challenge. In this work, we propose an unsupervised approach to partition the data, that does not exploit any external knowledge, but only relies on heuristics to select the blocking attributes. We compare different partitioning methods: we use (i) clustering on bagof-words, (ii) binning via Locality-Sensitive Hashing and (iii) clustering using word embeddings. In particular, the clustering methods show good results on a standard dataset of Web Tables, and, when combined with word embeddings, are a robust solution which allows for computing the clusters in a dense, low-dimensional space.

Abstract: Table-to-text generation aims to generate a description for a factual table which can be viewed as a set of field-value records. To encode both the content and the structure of a table, we propose a novel structure-aware seq2seq architecture which consists of field-gating encoder and description generator with dual attention. In the encoding phase, we update the cell memory of the LSTM unit by a field gate and its corresponding field value in order to incorporate field information into table representation. In the decoding phase, dual attention mechanism which contains word level attention and field level attention is proposed to model the semantic relevance between the generated description and the table. We conduct experiments on the \texttt{WIKIBIO} dataset which contains over 700k biographies and corresponding infoboxes from Wikipedia. The attention visualizations and case studies show that our model is capable of generating coherent and informative descriptions based on the comprehensive understanding of both the content and the structure of a table. Automatic evaluations also show our model outperforms the baselines by a great margin. Code for this work is available on this https URL.

Abstract: The present disclosure describes methods and systems for interpreting a table grouping input value associated with a table, wherein the table comprises a plurality of categories and a plurality of associated data sets corresponding to the plurality of categories, determining an aggregation value in response to the table grouping input value, wherein the aggregation value corresponds to at least one of the plurality of categories, and in response to the aggregation value, providing an aggregated table view.

Abstract: A data storage device includes a nonvolatile memory device including a main map table, the main map table including a plurality of map segments; and a controller comprising a sub map table including only some of the plurality of map segments of the main map table, the controller is suitable for updating access frequencies for the respective map segments of the main map table; and for determining whether to erase a map segment of the sub map table based on the updated access frequencies for the respective map segments.

Abstract: In this article,a random number of datasets was generated from random samples of used GSM(Global Systems for MobileCom- munications) rechargecards.Statisticalanalyseswereperformed to refine therawdatatorandomnumberdatasetsarrangedin table. Adetaileddescriptionofthemethodandrelevanttestsof randomness werealsodiscussed

Abstract: Disclosed is a data compression method of DeMura Table, a data decompression method of DeMura Table, and a Mura compensation method. The data compression method includes: acquiring image information of a display panel and obtaining an original DeMura Table; performing region extraction based on the original DeMura Table; performing edge detection based on the Mura region obtained from extraction; distributing each sub-pixel element included in the display panel as per results from the region extraction and the edge detection to determine a numerical value of each sampling point in the DeMura Table. The method may save storage costs of DeMura Table.

Abstract: Spreadsheets are one of the most successful content generation tools, used in almost every enterprise to perform data transformation, visualization, and analysis. The high degree of freedom provided by these tools results in very complex sheets, intermingling the actual data with formatting, formulas, layout artifacts, and textual metadata. To unlock the wealth of data contained in spreadsheets, a human analyst will often have to understand and transform the data manually. To overcome this cumbersome process, we propose a framework that is able to automatically infer the structure and extract the data from these documents in a canonical form. In this paper, we describe our heuristics-based method for discovering tables in spreadsheets, given that each cell is classified as either header, attribute, metadata, data, or derived. Experimental results on a real-world dataset of 439 worksheets (858 tables) show that our approach is feasible and effectively identifies tables within partially structured spreadsheets.

Abstract: The relevance of local knowledge in cultural heritage is by now acknowledged. It helps to determine many community-based projects by identifying the material to be digitally maintained in multimedia collections provided by communities of volunteers, rather than for-profit businesses or government entities. Considering that the search and browsing of texts, images, video, and 3D models related to places is more essential than using a simple text-based search, an interactive multimedia map was implemented in this study. The map, which is loaded on a single HyperText Markup Language (HTML) page using AJAX (Asynchronous JavaScript and XML), with a client-side control mechanism utilising jQuery components that are both freely available and ad-hoc developed, is updated according to user interaction. To simplify the publication of geo-referenced information, the application stores all the data in a Geographic JavaScript Object Notation (GeoJSON) file rather than in a database. The multimedia contents—associated with the selected Points of Interest (PoIs)—can be selected through text search and list browsing as well as by viewing their previews one by one in a sequence all together in a scrolling window (respectively: “Table”, “Folder”, and “Tile” functions). PoIs—visualised on the map with multi-shape markers using a set of unambiguous colours—can be filtered through their categories and types, accessibility status and timeline, thus improving the system usability. The map functions are illustrated using data collected in a Comenius project. Notes on the application software and architecture are also presented in this paper.

Abstract: Since a data warehouse deals with huge amounts of data and complex analytical queries, online processing and answering to users' queries in data warehouses can be a serious challenge. Materialized views are used to speed up query processing rather than direct access to the database in on-line analytical processing. Since the large number and high volume of views prevents all of the views from being stored, selection of a proper subset of views to materialization is inevitable. Proposing an appropriate method for selecting the optimal subset of views for materialization plays an essential role in increasing the efficiency of responding to data warehouse queries. In this paper, a greedy materialized view selection algorithm is represented, which selects a proper set of views for materialization from a novel table-like structure. The information in this table-like structure is extracted from a multivalue processing plan. This table-like structure-based greedy view selection (TSGV) method is evaluated using the queries of an analytical database, and the query-processing and view maintenance costs of the selected subset are both considered in this evaluation. The experimental results show that TSGV operates better than previously represented methods in terms of time.

Abstract: In this paper we propose several approaches for automatic annotation of natural science spreadsheets using a combination of structural properties of the tables and external vocabularies. During the design process of their spreadsheets, domain scientists implicitly include their domain model in the content and structure of the spreadsheet tables. However, this domain model is essential to unambiguously interpret the spreadsheet data. The overall objective of this research is to make the underlying domain model explicit, to facilitate evaluation and reuse of these data.We present our annotation approaches by describing five structural properties of natural science spreadsheets, that may pose challenges to annotation, and at the same time, provide additional information on the content. For example, the main property we describe is that, within a spreadsheet table, semantically related terms are grouped in rectangular blocks. For each of the five structural properties we suggest an annotation approach, that combines heuristics on the property with knowledge from external vocabularies. We evaluate our approaches in a case study, with a set of existing natural science spreadsheets, by comparing the annotation results with a baseline based on purely lexical matching.Our case study results show that combining information on structural properties of spreadsheet tables with lexical matching to external vocabularies results in higher precision and recall of annotation of individual terms. We show that the semantic characterization of blocks of spreadsheet terms is an essential first step in the identification of relations between cells in a table. As such, the annotation approaches presented in this study provide the basic information that is needed to construct the domain model of scientific spreadsheets. HighlightsWe describe five structural properties of scientific spreadsheet tables.Within a spreadsheet table, semantically related terms are grouped in blocks.We annotate tables using information on their structure and external vocabularies.Including information on table structure improves annotation of spreadsheet terms.We identify relations within a table by semantically categorizing blocks of terms.

Abstract: A system that implements a data storage service may store data for a database table in multiple replicated partitions on respective storage nodes. In response to a request to back up a table, the service may back up individual partitions of the table to a remote storage system independently and (in some cases) in parallel, and may update (or create) and store metadata about the table and its partitions on storage nodes of the data storage service and/or in the remote storage system. Backing up each partition may include exporting it from the database in which the table is stored, packaging and compressing the exported partition for upload, and uploading the exported, packaged, and compressed partition to the remote storage system. The remote storage system may be a key-value durable storage system in which each backed-up partition is accessible using its partition identifier as the key.

Abstract: The invention provides a data processing method and system. The method comprises the following steps: receiving a downloading request which is sent by a user and at least comprises a target serial number; obtaining result data information corresponding to a serial number consistent with the target serial number in a result table, and determining the target data information corresponding to the target serial number, the result table being obtained by screening a source data table based on a configuration table in a database in advance; and feeding back the target data information to the user. According to the scheme, the source data table is screened in the database based on the configuration table in advance to obtain the result table, screening work is completed in the database in advance, the screening time is shortened, occupied system resources are reduced, and the integrity of the screening result is guaranteed. When a download request containing the target serial number sent by auser is received, the target data information corresponding to the target serial number is obtained from the result table, the target data information is fed back to the user, the user does not needto wait for a screening result for a long time, and the use experience of the user is improved.

Abstract: Table constraints are very useful for modeling combinatorial constrained problems, and thus play an important role in Constraint Programming (CP). During the last decade, many algorithms have been proposed for enforcing the property known as Generalized Arc Consistency (GAC) on such constraints. A state-of-the art GAC algorithm called CompactTable (CT), which has been recently proposed, significantly outperforms all previously proposed algorithms. In this paper, we extend this algorithm in order to deal with both short supports and negative tables, i.e., tables that contain universal values and conflicts. Our experimental results show the interest of using this fast general algorithm. Introduction Table constraints, also called extension(al) constraints, explicitly express for the variables they involve, either the allowed combinations of values, called supports, or the forbidden combinations of values, called conflicts. Table constraints can theoretically encode any kind of restrictions and are consequently very important in Constraint Programming (CP). Indeed, as especially claimed by people from industry (e.g., IBM and Google), table constraints are often required when modeling combinatorial constrained problems in many application fields. The design of filtering algorithms for such constraints has generated a lot of research effort, see (Bessiere and Régin 1997; Lhomme and Régin 2005; Lecoutre and Szymanek 2006; Gent et al. 2007; Ullmann 2007; Lecoutre 2011; Lecoutre, Likitvivatanavong, and Yap 2015; J.-B. Mairy and Deville 2014; Perez and Régin 2014; Wang et al. 2016; Demeulenaere et al. 2016). On classical tables, i.e., sequences of ordinary tuples, the algorithmic progresses that have been made over the years for maintaining the property called GAC (Generalized Arc Consistency) are quite impressive. Roughly speaking, an algorithm such as Compact-Table (Demeulenaere et al. 2016) is about one order of magnitude faster than the best algorithm(s) proposed a decade ago (Lhomme and Régin 2005; Lecoutre and Szymanek 2006; Gent et al. 2007; Ullmann 2007). Unfortunately, table constraints admit practical boundaries because the memory space required to represent them may grow exponentially with their arity. To reduce Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. space complexity, researchers have focused on various forms of compression. For example, tries (Gent et al. 2007), Multivalued Decision Diagrams (MDDs) (Cheng and Yap 2010; Perez and Régin 2014) and Deterministic Finite Automaton (DFA) (Pesant 2004) are general structures used to represent table constraints in a compact way, so as to facilitate filtering process. Cartesian product is another classical mechanism to represent compactly large sets of tuples. This is the approach followed by works on compressed tuples (Katsirelos and Walsh 2007; Régin 2011; Xia and Yap 2013) and short supports and tuples (Jefferson and Nightingale 2013). A short tuple allows the presence of universal values, denoted by the symbol *, meaning that some variables can take any values from their domains. Other forms of compact representation are obtained by means of sliced tables (Gharbi et al. 2014) and smart tables (Mairy, Deville, and Lecoutre 2015). Compact-Table (CT) is a state-of-the-art GAC algorithm for positive (ordinary) table constraints, i.e., constraints defined by tables containing (uncompressed) supports. In this paper, we extend CT in order to be able to deal with: • negative tables (i.e., tables containing conflicts), • and/or short tuples (i.e., tuples containing the symbol *). Technical Background A constraint network (CN)N is composed of a set of n variables and a set of e constraints. Each variable x has an associated domain, denoted by dom(x), that contains the finite set of values that can be assigned to it. Each constraint c involves an ordered set of variables, called the scope of c and denoted by scp(c), and is semantically defined by a relation, denoted by rel(c), which contains the set of tuples allowed for the variables involved in c. The arity of a constraint c is |scp(c)|. For simplicity, a variable-value pair (x, a) such that x ∈ scp(c) and a ∈ dom(x) is called a value (of c). A table constraint c is a constraint such that rel(c) is defined explicitly by listing (in a table) the tuples that are allowed by c or the tuples that are disallowed by c. In the former case, the table constraint is said to be positive whereas in the latter case, it is said negative. Let τ = (a1, a2, . . . , ar) be a tuple of values associated with an ordered set of variables vars(τ) = {x1, x2, . . . , xr}. The ith value of τ is denoted by τ [i] or

Abstract: The invention provides a picking route generation method and device. The method comprises the steps that the number of standard cases needed for a wave order is calculated, and a standard case configuration table is generated; a standard case roadway distribution diagram is generated according to the standard case configuration table and a roadway number and a storage location number of goods SKU; a roadway is selected to start picking simulation, and picking simulation information corresponding to the serial number of each standard case is recorded and added into traversed standard case picking information; after picking simulation on the current roadway is completed, the serial numbers of the standard cases where picking is completed form a virtual trolley according to preset rules; if the trolley is successfully formed, whether picking is completed at all standard cases or not is judged, if yes, picking simulation is ended, and if not, the traversed standard case picking information is deleted, and a next roadway is entered to restart picking simulation; and if the trolley is not successfully formed, the next roadway is entered to continue picking. Through the picking route generation method and device, both breadth and precision of picking coverage are guaranteed, and picking efficiency is improved.