This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users The primary goal of this article is to catalog and compile material related to this topic into a single source This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR) Finally, this article concludes with a discussion of outstanding challenges and future research directions

https://www.nowpublishers.com/article/DownloadSummary/INR-012

Methods for Evaluating Interactive Information Retrieval Systems with Users

In this paper, we review 300 references on video retrieval, indicating when text-only solutions are unsatisfactory and showing the promising alternatives which are in majority concept-based. Therefore, central to our discussion is the notion of a semantic concept: an objective linguistic description of an observable entity. Specifically, we present our view on how its automated detection, selection under uncertainty, and interactive usage might solve the major scientific problem for video retrieval: the semantic gap. To bridge the gap, we lay down the anatomy of a concept-based video search engine. We present a component-wise decomposition of such an interdisciplinary multimedia system, covering influences from information retrieval, computer vision, machine learning, and human–computer interaction. For each of the components we review state-of-the-art solutions in the literature, each having different characteristics and merits. Because of these differences, we cannot understand the progress in video retrieval without serious evaluation efforts such as carried out in the NIST TRECVID benchmark. We discuss its data, tasks, results, and the many derived community initiatives in creating annotations and baselines for repeatable experiments. We conclude with our perspective on future challenges and opportunities.

/pdf/concept-based-video-retrieval-1iy430iazc.pdf

Concept-Based Video Retrieval

Opinions are ubiquitous in text, and readers of on-line text — from consumers to sports fans to news addicts to governments — can benefit from automatic methods that synthesise useful opinion-orientated information from the sea of data In this chapter on opinion mining and sentiment analysis, we introduce an idealised, end-to-end opinion analysis system and describe its components, including constructing opinion lexica, performing sentiment analysis, and producing opinion summaries

/pdf/39-opinion-mining-and-sentiment-analysis-2wo5n16jvv.pdf

39. Opinion mining and sentiment analysis

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

/pdf/bridging-the-gap-between-social-animal-and-unsocial-machine-28vclv3t3o.pdf

Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

In an expert search task, the users' need is to identify people who have relevant expertise to a topic of interest. An expert search system predicts and ranks the expertise of a set of candidate persons with respect to the users' query. In this paper, we propose a novel approach for predicting and ranking candidate expertise with respect to a query. We see the problem of ranking experts as a voting problem, which we model by adapting eleven data fusion techniques.We investigate the effectiveness of the voting approach and the associated data fusion techniques across a range of document weighting models, in the context of the TREC 2005 Enterprise track. The evaluation results show that the voting paradigm is very effective, without using any collection specific heuristics. Moreover, we show that improving the quality of the underlying document representation can significantly improve the retrieval performance of the data fusion techniques on an expert search task. In particular, we demonstrate that applying field-based weighting models improves the ranking of candidates. Finally, we demonstrate that the relative performance of the adapted data fusion techniques for the proposed approach is stable regardless of the used weighting models.

/pdf/voting-for-candidates-adapting-data-fusion-techniques-for-an-239z8gtan4.pdf

Voting for candidates: adapting data fusion techniques for an expert search task

In TREC 2007, we participate in four tasks of the Blog and Enterprise tracks. We continue experiments using Terrier 1 [14], our modular and scalable Information Retrieval (IR) platform, and the Divergence From Randomness (DFR) framework. In particular, for the Blog track opinion finding task, we propose a statisti cal term weighting approach to identify opinionated documents. An alternative approach based on an opinion identification too l is also utilised. Overall, a 15% improvement over a non-opinionated baseline is observed in applying the statistical term weighting approach. In the Expert Search task of the Enterprise track, we investi gate the use of proximity between query terms and candidate name occurrences in documents.

/pdf/university-of-glasgow-at-trec-2005-experiments-in-terabyte-4rusxlqtqy.pdf

University of Glasgow at TREC 2005: Experiments in Terabyte and Enterprise Tracks with Terrier

In an Enterprise setting, an expert search system can assist users with their "expertise need" by suggesting people with relevant expertise to the topic of interest. These systems typically work by associating documentary evidence of expertise to each candidate expert, and then ranking the candidates by the extent to which the documents in their profile are about the query. There are three important factors that affect the retrieval performance of an expert search system - firstly, the selection of the candidate profiles (the documents associated with each candidate), secondly, how the topicality of the documents is measured, and thirdly how the evidence of expertise from the associated documents is combined. In this work, we investigate a new dimension to expert finding, namely whether some documents are better indicators of expertise than others in each candidate's profile. We apply five techniques to predict the quality documents in candidate profiles, which are likely to be good indicators of expertise. The techniques applied include the identification of possible candidate homepages, and of clustering the documents in each profile to determine the candidate's main areas of expertise. The proposed approaches are evaluated on three expert search task from recent TREC Enterprise tracks and provide conclusions.

/pdf/high-quality-expertise-evidence-for-expert-search-5ercrcfrbi.pdf

High quality expertise evidence for expert search

Recommender systems have been systematically applied in industry and academia to help users cope with information uncertainty. However, given the multiplicity of the preferences and needs it has been shown that no approach is suitable for all users in all situations. Thus, it is believed that an effective recommender system should incorporate a variety of techniques and features to offer valuable recommendations and enhance the search experience. In this paper we propose a novel video search interface that employs a multimodal recommender system, which can predict topical relevance. The multimodal recommender accounts for interaction data, contextual information, as well as users' affective responses, and exploits these information channels to provide meaningful recommendations of unseen videos. Our experiment shows that the multimodal interaction feature is a promising way to improve the performance of recommendation.

Enriching user profiling with affective features for the improvement of a multimodal recommender system

Search interfaces are mainly designed to support a single searcher at a time. We therefore have a limited understanding of how an interface can support search where more than one searcher concurrently pursues a shared information need. This paper investigated the performance and user behaviour of concurrent search. Based on a recall-oriented search task, a user study was carried out to compare an independent search condition to collaborative search conditions. The results show that the collaborative conditions helped searchers diversify search vocabulary while reducing redundant documents to be bookmarked within teams. However, these effects were found to be insufficient to improve the retrieval effectiveness. We discussed the implications for concurrent search support based on our findings.

/pdf/comparing-collaborative-and-independent-search-in-a-recall-5b4gr6tzu7.pdf

Comparing collaborative and independent search in a recall-oriented task

There are a number of multimedia tasks and environments that can be collaborative in nature and involve contributions from more than one individual. Examples of such tasks include organising photographs or videos from multiple people from a large event, students working together to complete a class project, or artists and/or animators working on a production. Despite this, current state of the art applications that have been created to assist in multimedia search and organisation focus on a single user searching alone and do not take into consideration the collaborative nature of a large number of multimedia tasks. The limited work in collaborative search for multimedia applications has concentrated mostly on synchronous, and quite often co-located, collaboration between persons. However, these collaborative scenarios are not always practical or feasible. In order to overcome these shortcomings we have created an innovative system for online video search, which provides mechanisms for groups of users to collaborate both asynchronously and remotely on video search tasks. In order to evaluate our system an user evaluation was conducted. This evaluation simulated multiple conditions and scenarios for collaboration, varying on awareness, division of labour, sense making and persistence. The outcome of this evaluation demonstrates the benefit and usability of our system for asynchronous and remote collaboration between users. In addition the results of this evaluation provide a comparison between implicit and explicit collaboration in the same search system.

/pdf/an-asynchronous-collaborative-search-system-for-online-video-m9xccgaq8d.pdf

An asynchronous collaborative search system for online video search

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

David Hannah

Author Tools

Chat about Author