TL;DR: The preparation of the data sets, the creation of the background collections to allow systems to acquire the required knowledge, the metric used for the evaluation of the systems' submissions, and the results of this first attempt of the QA4MRE challenge are described.
Abstract: This paper describes the first steps towards developing a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011. This year a major innovation was introduced, as the traditional QA task was replaced by a new Machine Reading task whose intention was to ask questions which required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with the background collections provided by the organization. Beside the main task, also one pilot task was offered, namely, Processing Modality and Negation for Machine Reading. This task was aimed at evaluating whether systems were able to understand extra-propositional aspects of meaning like modality and negation. This paper describes the preparation of the data sets, the creation of the background collections to allow systems to acquire the required knowledge, the metric used for the evaluation of the systems' submissions, and the results of this first attempt. Twelve groups participated in the task submitting a total of 62 runs in three languages: English, German and Romanian.
TL;DR: Significant performance improvement over plain word-based retrieval, three other language-independent morphological normalizers, as well as rule-based stemmers is demonstrated.
Abstract: A novel graph-based language-independent stemming algorithm suitable for information retrieval is proposed in this article. The main features of the algorithm are retrieval effectiveness, generality, and computational efficiency. We test our approach on seven languages (using collections from the TREC, CLEF, and FIRE evaluation platforms) of varying morphological complexity. Significant performance improvement over plain word-based retrieval, three other language-independent morphological normalizers, as well as rule-based stemmers is demonstrated.
TL;DR: An overview of the resources and assessments of the plant identification task at ImageCLEF 2011 is presented, the retrieval approaches employed by the participating groups are summarized, and an analysis of the main evaluation results are provided.
Abstract: ImageCLEF’s plant identification task provides a testbed for the system-oriented evaluation of tree species identification based on leaf images. The aim is to investigate image retrieval approaches in the context of crowdsourced images of leaves collected in a collaborative manner. This paper presents an overview of the resources and assessments of the plant identification task at ImageCLEF 2011, summarizes the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results.
TL;DR: It is found that adding full descriptions to abstracts gives a clear improvement; the first 400 words of the description also improves classification but to a lesser degree; the most important finding however is the importance of the threshold on the class selection.
Abstract: We report the results of a series of classification experiments with the Linguistic Classification System LCS in the context of CLEF-IP 2011. We participated in the main classification task: classifying documents on the subclass level. We investigated (1) the use of different sections (abstract, description, metadata) from the patent documents; (2) adding dependency triples to the bag-of-words representation; (3) adding the WIPO corpus to the EPO training data; (4) the use of patent citations in the test data for reranking the classes; and (5) the threshold on the class scores for class selection. We found that adding full descriptions to abstracts gives a clear improvement; the first 400 words of the description also improves classification but to a lesser degree. Adding metadata (applicants, inventors en address) did not improve classification. Adding dependency triples to words gives a much higher recall at the cost of a lower precision but this effect is largely due to the class selection threshold. We did not find an effect from adding the WIPO corpus, nor from reranking with patent citations. In future work, we plan to investigate whether there are other methods for reranking with patent citations that does give an improvement, because we feel that the citations may still give valuable information. Our most important finding however is the importance of the threshold on the class selection. For the current work, we only compared two values for the threshold and the results are much better for 1.0 than for 0.5. The 0.5 threshold gives higher recall in all runs, which was the original motivation for submitting runs with a lower threshold. However, because the much lower precision, the F-scores are lower. We think that there is still some improvement to be gained from proper tuning of the class selection threshold, and the use of a flexible threshold (also taking into account the different text representations). This is part of our future work.
TL;DR: A new algorithm method for the detection of plagiarism is described, which removes numerous limitations of the older method, which has been used as part of a complex information system for the detected plagiarism.
Abstract: In this article we describe a new algorithm method for the detection of plagiarism. The method removes numerous limitations of our older method, which has been used as part of a complex information system for the detection of plagiarism. The method has been tested using multiple corpora mainly in Slovak language. With the PAN-09 and PAN-10 corpora it was of great advantage that we could compare our results with the results of other methods. The very good initial results gave us motivation to implement multiple algorithm and parameter improvements.
TL;DR: This paper aims to explain the performance of plagiarism detection system which can detect External as well as Intrinsic Plagia- rism in text and reports the results on PAN-PC-2011 test corpus.
Abstract: This paper aims to explain the performance of plagiarism detection system which can detect External as well as Intrinsic Plagia- rism in text. It reports the results on PAN-PC-2011 test corpus. We investigated Vector Space Model based techniques for detecting external plagiarism cases and discourse markers based features to detect intrinsic plagiarism cases.
TL;DR: This work participated both in the photo annotation and conceptbased retrieval tasks of CLEF 2011 and developed visual, textual and multi-modal approaches using multi-label learning algorithms from the Mulan open source library.
Abstract: We participated both in the photo annotation and conceptbased retrieval tasks of CLEF 2011. For the annotation task we developed visual, textual and multi-modal approaches using multi-label learning algorithms from the Mulan open source library. For the visual model we employed the ColorDescriptor software to extract visual features from the images using 7 descriptors and 2 detectors. For each combination of descriptor and detector a multi-label model is built using the Binary Relevance approach coupled with Random Forests as the base classifier. For the textual models we used the boolean bag-of-words representation, and applied stemming, stop words removal, and feature selection using the chi-squared-max method. The multi-label learning algorithm that yielded the best results in this case was Ensemble of Classifier Chains using Random Forests as base classifier. Our multi-modal approach was based on a hierarchical late-fusion scheme. For the concept based retrieval task we developed two different approaches. The first one is based on the concept relevance scores produced by the system we developed for the annotation task. It is a manual approach, because for each topic we manually selected the relevant topics and manually set the strength of their contribution to the final ranking produced by a general formula that combines topic relevance scores. The second approach is based solely on the sample images provided for each query and is therefore fully automated. In this approach only the textual information was used in a query-by-example framework.
TL;DR: The external plagiarism Detection problem has been solved with the help of Nutch, an open source Information Retrieval (IR) system and the system contains three phases – knowledge preparation, candidate retrieval and plagiarism detection.
Abstract: This paper reports about the development of a Plagiarism detection system as a part of the Plagiarism detection task in PAN 2011. The external plagiarism detection problem has been solved with the help of Nutch, an open source Information Retrieval (IR) system. The system contains three phases – knowledge preparation, candidate retrieval and plagiarism detection. From the source documents, knowledge base has been prepared for developing the Nutch index and the queries have been formed from the suspicious documents for submission to the Nutch IR system. The retrieved candidate source sentences are assigned similarity scores by Nutch. Dissimilarity score is assigned for each candidate sentence and the suspicious sentence. Each candidate source sentence is ranked based on these two scores. The top ranked candidate sentence is selected for each suspicious sentence.
TL;DR: An extended version of external CoReMo System (Contextual Reference Monotony, ranked 6th in PAN2010), now with crosslingual capability, with high reliability and speed, low computer requirements and its own internal translation system.
Abstract: This paper shows an extended version of external CoReMo System (Contextual Reference Monotony, ranked 6th in PAN2010), now with crosslingual capability (ranked 5th in PAN2011 / Plagdet 0,2340). It's not the best ranked system for translated plagiarism (ranked 3th / Plagdet 0,3587), but it has high reliability and speed (global results in 30 minutes), low computer requirements and its own internal translation system.
TL;DR: Invited talks will address further challenges and possibilities of alternative evaluation approaches from different perspectives or applications regarding Cultural Heritage materials and interaction patterns of users with CH information systems.
Abstract: Digital libraries and other information systems that access Cultural Heritage (CH) materials are becoming increasingly complex. They must often manage a diverse range of content from different CH institutions – such as libraries, museums, written and audiovisual archives – and have to provide access to them in a unified and coherent way. The content from CH institutions is often multilingual and multimedia (e.g. text, photographs, images, audio recordings, and videos), usually described with metadata in multiple formats and of different levels of complexity. CH institutions have different approaches to managing information and serve diverse user communities, often with specialized needs. This makes the meaning of “search and browse” quite different for users of a library or archive and non-specialist users may not be able to successfully retrieve relevant results or may be disoriented by the kind of results they obtain. Interaction patterns of users with CH information systems do not represent clear separated and isolated use cases but alternate with each other thus representing possible sequences of user interactions with a CH information system. Invited talks will address further challenges and possibilities of alternative evaluation approaches from different perspectives or applications. Participants are asked to bring in statements dealing with the following topics:
TL;DR: This chapter reports on activities undertaken to provide a set of topics for the two tasks, to extract the relevance assessments for the provided topics, and on evaluating the effectiveness of the employed retrieval methods.
Abstract: The Clef–Ip track ran for the first time within the Clef 2009 campaign. The purpose of the track was twofold: (a) to encourage and facilitate research in the area of patent retrieval by providing a large clean data set for experimentation; (b) to create a large test collection of patents in the three main European languages for the evaluation of cross-lingual information access. The track focused on the task of prior art search, to which a second task was added in 2010, the patent classification task. The participating teams deployed a variety of Information Retrieval techniques, adapted or custom-made, to tackle with this specific domain and tasks. This chapter reports on activities undertaken to provide a set of topics for the two tasks, to extract the relevance assessments for the provided topics, and on evaluating the effectiveness of the employed retrieval methods.
TL;DR: This book constitutes the refereed proceedings of the Second International Conference on Multilingual and Multimodal Information Access Evaluation, in continuation of the popular CLEF campaigns and workshops that have run for the last decade, CLEF 2011, held in Amsterdem, The Netherlands, in September 2011.
Abstract: This book constitutes the refereed proceedings of the Second International Conference on Multilingual and Multimodal Information Access Evaluation, in continuation of the popular CLEF campaigns and workshops that have run for the last decade, CLEF 2011, held in Amsterdem, The Netherlands, in September 2011. The 14 revised full papers presented together with 2 keynote talks were carefully reviewed and selected from numerous submissions. The papers accepted for the conference included research on evaluation methods and settings, natural language processing within different domains and languages, multimedia and reflections on CLEF. Two keynote speakers highlighted important developments in the field of evaluation: the role of users in evaluation and a framework for the use of crowdsourcing experiments in the setting of retrieval evaluation.
TL;DR: This system is able to process the PAN plagiarism corpus for the external plagiarism detection task within relatively short timescales in contrast to previously reported state-of-the-art, and still produce a reasonable degree of performance.
Abstract: In this paper we report on our high-performance plagiarism detection system which is able to process the PAN plagiarism corpus for the external plagiarism detection task within relatively short timescales in contrast to previously reported state-of-the-art, and still produce a reasonable degree of performance (PAN 11, 4 place, PlagDet=0.2467329, Recall=0.1500480, Precision=0.7106536, Granularity=1.0058894). At the core of our system is a simple method which avoids the use of hash-type approaches, but about which we are unable to disclose too many details due to a patent application in progress. We optimised our performance using the PAN10 collection, and used the best parameters for the final submission. We anticipated a relatively similar performance at PAN11, modulo changes to the plagiarism cases, and 4 place this year put us between participants who had been 5 and 6 in PAN 10.
TL;DR: IPL's participation to the image CLEF ad-hoc textual and visual medical retrieval for 2011 is described and the approaches and methods from a systematic experimental investigation on fusion from visual and textual sources of images are reported on.
Abstract: This article describes IPL's participation to the image CLEF ad-hoc textual and visual medical retrieval for 2011. We report on our approaches and methods from a systematic experimental investigation on fusion from visual and textual sources of images. We also explore ways to enrich our searches using external sources like Wikipedia.
TL;DR: The LogCLEF lab - "A benchmarking activity on Multilingual Log File Analysis: Language identification, query classification, success of a query" deals with information contained in query logs of search engines and digital libraries from which knowledge can be mined to understand search behavior in multilingual context.
Abstract: Interactions between users and information access systems can be analyzed and studied to gather user preferences and to learn what a user likes the most, and to use this information to adapt the search to users and personalize the presentation of results. The LogCLEF lab - "A benchmarking activity on Multilingual Log File Analysis: Language identification, query classification, success of a query" deals with information contained in query logs of search engines and digital libraries from which knowledge can be mined to understand search behavior in multilingual context. LogCLEF has created the first long-term standard collection for evaluation purposes in the area of log analysis. The LogCLEF 2011 lab is the continuation of the past two editions: as a pilot task in CLEF 2009, and a workshop in CLEF 2010. The Cross-Language Evaluation Forum (CLEF) promotes research and development in multilingual information access and is an activity of the PROMISE Network of Excellence.
TL;DR: In this article, a pasteboard showing a keyboard of a keyboard instrument such as a piano with note-name/pitch-name (or syllable-name) was provided to reduce the effort and time for practicing musical score reading and to enjoy performance.
Abstract: PROBLEM TO BE SOLVED: To provide a freely pastable and releasable sheet with a keyboard pasteboard of a keyboard instrument such as a piano with note-name/pitch-name (or syllable-name) for practicing musical score reading to enable a user to reduce the effort and time for practicing musical score reading and to enjoy performance. SOLUTION: A pasteboard 1 showing a keyboard of a keyboard instrument such as a piano and, for white keys, a sheet 2a in G clef notation and a sheet 2b in F clef notation are prepared. A sheet 3a for black keys in #-system notation is shown in G clef notation for the range of C and higher, and is shown in F clef notation for the range of B and lower. A sheet 3b for black keys in flat-system notation is shown in G clef notation for the range of C and higher, and is shown in F clef notation for the range of B and lower. Further, the sheets showing pitch names and syllable names in German, Italian, Russian, Chinese, and Korean are prepared so that the sheets can also be used in these countries. COPYRIGHT: (C)2011,JPO&INPIT
TL;DR: This book constitutes the refereed proceedings of the Second International Conference on Multilingual and Multimodal Information Access Evaluation, CLEF 2011, held in Amsterdem, The Netherlands, in September 2011, and highlighted important developments in the field of evaluation.
Abstract: This book constitutes the refereed proceedings of the Second International Conference on Multilingual and Multimodal Information Access Evaluation,in continuation of the popular CLEF campaigns and workshops that have run for the last decade,CLEF 2011, held in Amsterdem, The Netherlands, in September 2011. The 14 revised full papers presented together with 2 keynote talks were carefully reviewed and selected from numerous submissions. The papers accepted for the conference included research on evaluation methods and settings, natural language processing within different domains and languages, multimedia and reflections on CLEF. Two keynote speakers highlighted important developments in the field of evaluation: the role of users in evaluation and a framework for the use of crowdsourcing experiments in the setting of retrieval evaluation.
TL;DR: The CLEF Labs are a continuation of tracks from previous CLEF workshops and follow the traditional (“campaign-style”) cycle of activities in a large-scale information retrieval evaluation experiment set-up.
Abstract: The CLEF Labs are a continuation of tracks from previous CLEF workshops. In 2010, CLEF went from being a workshop collocated with an existing conference to a conference in its own right. The CLEF Labs are an integral part of the conference and two different types of lab are offered: (1) benchmarking or “campaign-style” and (2) workshop-style. The benchmarking labs on the whole follow the traditional (“campaign-style”) cycle of activities in a large-scale information retrieval evaluation experiment set-up:
TL;DR: In this paper, a novel optimal music score note rapid positioning algorithm device and a novel optical music score point rapid positioning method were presented, which are accurate in note positioning in musical notation.
Abstract: The invention discloses a novel optimal music score note rapid positioning algorithm device and a novel optical music score note rapid positioning algorithm method. Staff is currently a universal musical work musical notation method in the world. A note value of a note means indicates the corresponding position of a note head in five lines of the staff according to a sound specifically produced by the note. The number of factors determining the specific position of the note head of the note in the five lines of the staff is great, and the factors commonly determine the final positioning of the note head according to certain combined computational relationships. The novel optimal music score note rapid positioning algorithm device and the novel optical music score note rapid positioning algorithm method provided by the invention are accurate in note positioning. The method comprises the following steps of: establishing a coordinate system according to the staff, and dividing the coordinate ranges of each musical clef and each tone mark; determining the coordinate range of the musical clef according to the musical clef; determining the coordinate range of the tone mark according to the tone mark; and determining a final note position according to the ranges of the musical clef and the tone mark. The invention is a technical innovation in the field of electronic musical score development.
TL;DR: This work presents an approach to the Automatic Speech Recognition component of a Voice-Activated Question Answering system, focusing on building a language model able to include as many relevant words from the document repository as possible, but also representing the general syntactic structure of typical questions.
Abstract: The interest of the incorporation of voice interfaces to the Question Answering systems has increased in recent years. In this work, we present an approach to the Automatic Speech Recognition component of a Voice-Activated Question Answering system, focusing our interest in building a language model able to include as many relevant words from the document repository as possible, but also representing the general syntactic structure of typical questions. We have applied these technique to the recognition of questions of the CLEF QA 2003-2006 contests.
TL;DR: The main goal of the Clef Ip remained the same to foster research in the patent retrieval area, and provide a large clean data set, but the number of tasks in the track was increased and the data set was enlarged.
Abstract: The patent system is designed to encourage disclosure of new technologies and novel ideas by granting exclusive rights on the use of inventions to their inventors, for a limited period of time. Before a patent can be granted, patent o ces around the world perform thorough searches to ensure that no previous similar disclosures were made. In the intellectual property terminology, such kind of searches are called prior art searches. In some industries, the number of granted patents a company owns has a high impact on the market value of the company. This underlines the importance of well-performed prior art searches. Together with the Trec Chem track [5], also organized by our institution, the Clef Ip e ort comes to complete the work that is being done in the series of Ntcir workshops (see for example [4]). The rst Clef Ip track ran within Clef 2009. The purpose of the track was twofold: to encourage and facilitate research in the area of patent retrieval by providing a large clean data set for experimentation; to create a large test collection of patents in the three main European languages for the evaluation of cross lingual information access. The Clef Ip data set includes documents published by the European Patent O ce (Epo) which contain a mixture of English, German and French content. The track focused on the task of prior art search. In 2010 and 2011, the Clef Ip track was organized as a benchmarking activity (lab) in the Clef conference. In these years, the main goal of the Clef Ip e ort remained the same to foster research in the patent retrieval area, and provide a large clean data set. To this end, the number of tasks in the track was increased and the data set was enlarged. Recognizing the importance of patent classi cations in the daily activity of an intellectual property professional, in 2010 the Clef Ip benchmarking activity included a patent classi cation task. The participants were asked to classify