Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd work will fail to achieve its potential, focusing on assembly-line piecework. Can we foresee a future crowd workplace in which we would want our children to participate? This paper frames the major challenges that stand in the way of this goal. Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.

/pdf/the-future-of-crowd-work-2houizqbg3.pdf

The future of crowd work

The Future of Crowd Work

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.

/pdf/automated-feedback-generation-for-introductory-programming-5g15kzw2th.pdf

Automated feedback generation for introductory programming assignments

The growing demand for structured knowledge has led to great interest in relation extraction, especially in cases with limited supervision. However, existing distance supervision approaches only extract relations expressed in single sentences. In general, cross-sentence relation extraction is under-explored, even in the supervised-learning setting. In this paper, we propose the first approach for applying distant supervision to cross-sentence relation extraction. At the core of our approach is a graph representation that can incorporate both standard dependencies and discourse relations, thus providing a unifying way to model relations within and across sentences. We extract features from multiple paths in this graph, increasing accuracy and robustness when confronted with linguistic variation and analysis error. Experiments on an important extraction task for precision medicine show that our approach can learn an accurate cross-sentence extractor, using only a small existing knowledge base and unlabeled text from biomedical research articles. Compared to the existing distant supervision paradigm, our approach extracted twice as many relations at similar precision, thus demonstrating the prevalence of cross-sentence relations and the promise of our approach.

/pdf/distant-supervision-for-relation-extraction-beyond-the-4hf7kjenuk.pdf

Distant Supervision for Relation Extraction beyond the Sentence Boundary

Recent research on entity linking (EL) has introduced a plethora of promising techniques, ranging from deep neural networks to joint inference. But despite numerous papers there is surprisingly little understanding of the state of the art in EL. We attack this confusion by analyzing differences between several versions of the EL problem and presenting a simple yet effective, modular, unsupervised system, called Vinculum, for entity linking. We conduct an extensive evaluation on nine data sets, comparing Vinculum with two state-of-the-art systems, and elucidate key aspects of the system that include mention extraction, candidate generation, entity type prediction, entity coreference, and coherence.

/pdf/design-challenges-for-entity-linking-2k9t16wme4.pdf

Design Challenges for Entity Linking

Interest in online education is surging, as dramatized bythe success of Khan Academy and recent Stanford online courses, but the technology for online education isin its infancy. Crowdsourcing mechanisms will likelybe essential in order to reach the full potential of thismedium. This paper sketches some of the challengesand directions we hope HCOMP researchers will address.

https://homes.cs.washington.edu/~weld/papers/weld-hcomp12.pdf

Personalized Online Education — A Crowdsourcing Challenge

Distant supervision has become the leading method for training large-scale relation extractors, with nearly universal adoption in recent TAC knowledge-base population competitions. However, there are still many questions about the best way to learn such extractors. In this paper we investigate four orthogonal improvements: integrating named entity linking (NEL) and coreference resolution into argument identification for training and extraction, enforcing type constraints of linked arguments, and partitioning the model by relation type signature. We evaluate sentential extraction performance on two datasets: the popular set of NY Times articles partially annotated by Hoffmann et al. (2011) and a new dataset, called GORECO, that is comprehensively annotated for 48 common relations. We find that using NEL for argument identification boosts performance over the traditional approach (named entity recognition with string match), and there is further improvement from using argument types. Our best system boosts precision by 44% and recall by 70%.

/pdf/type-aware-distantly-supervised-relation-extraction-with-4ytxabbh3h.pdf

Type-Aware Distantly Supervised Relation Extraction with Linked Arguments

We propose a framework for learning robust Bayesian network models of cell signalling from high-throughput proteomic data. We show that model averaging using Bayesian bootstrap resampling generates more robust structures than procedures that learn structures using all of the data. We also develop an algorithm for ranking the importance of network features using bootstrap resample data. We apply our algorithms to derive the T-cell signalling network from the flow cytometry data of Sachs et al. (2005). Our learning algorithm has identified, with high confidence, several new crosstalk mechanisms in the T-cell signalling network. Many of them have already been confirmed experimentally in the recent literature and six new crosstalk mechanisms await experimental validation.

Learning robust cell signalling models from high throughput proteomic data

We apply a search-based technique for learning high-quality Bayesian networks to proteomic flowcytometry data for a portion of the human T-cell signaling network. Although Bayesian network models have been learned from this data using methods such as MCMC that sample from a posterior distribution [1], [2], [3], we demonstrate that our more comprehensible searchbased technique, which uses model averaging by Bayesian bootstrap replicates, provides comparable results. We also show that additional edges not in the consensus model, but identified by both our technique and that of Eaton and Murphy are very unlikely to be artifacts of the network learning technique, and should be investigated further.

A Search-Based Approach for Bayesian Inference of the T-cell Signaling Network

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Mitchell Koch

Author Tools

Chat about Author

Papers

Personalized Online Education — A Crowdsourcing Challenge

Type-Aware Distantly Supervised Relation Extraction with Linked Arguments

Learning robust cell signalling models from high throughput proteomic data

A Search-Based Approach for Bayesian Inference of the T-cell Signaling Network