Abstract: A novel touch screen technology is presented. TouchLight uses simple image processing techniques to combine the output of two video cameras placed behind a semi-transparent plane in front of the user. The resulting image shows objects that are on the plane. This technique is well suited for application with a commercially available projection screen material (DNP HoloScreen) which permits projection onto a transparent sheet of acrylic plastic in normal indoor lighting conditions. The resulting touch screen display system transforms an otherwise normal sheet of acrylic plastic into a high bandwidth input/output surface suitable for gesture-based interaction.

Abstract: In this paper, we describe The Designer's Augmented Reality Toolkit (DART). DART is built on top of Macromedia Director, a widely used multimedia development environment. We summarize the most significant problems faced by designers working with AR in the real world, and discuss how DART addresses them. Most of DART is implemented in an interpreted scripting language, and can be modified by designers to suit their needs. Our work focuses on supporting early design activities, especially a rapid transition from story-boards to working experience, so that the experiential part of a design can be tested early and often. DART allows designers to specify complex relationships between the physical and virtual worlds, and supports 3D animatic actors (informal, sketch-based content) in addition to more polished content. Designers can capture and replay synchronized video and sensor data, allowing them to work off-site and to test specific parts of their experience more effectively.

Abstract: In this paper, we explore the concept of dual-purpose speech: speech that is socially appropriate in the context of a human-to-human conversation which also provides meaningful input to a computer. We motivate the use of dual-purpose speech and explore issues of privacy and technological challenges related to mobile speech recognition. We present three applications that utilize dual-purpose speech to assist a user in conversational tasks: the Calendar Navigator Agent, DialogTabs, and Speech Courier. The Calendar Navigator Agent navigates a user's calendar based on socially appropriate speech used while scheduling appointments. DialogTabs allows a user to postpone cognitive processing of conversational material by proving short-term capture of transient information. Finally, Speech Courier allows asynchronous delivery of relevant conversational information to a third party.

Abstract: We present a variant of hierarchical marking menus where items are selected using a series of inflection-free simple marks, rather than the single "zig-zag" compound mark used in the traditional design. Theoretical analysis indicates that this simple mark approach has the potential to significantly increase the number of items in a marking menu that can be selected efficiently and accurately. A user experiment is presented that compares the simple and compound mark techniques. Results show that the simple mark technique allows for significantly more accurate and faster menu selections overall, but most importantly also in menus with a large number of items where performance of the compound mark technique is particularly poor. The simple mark technique also requires significantly less physical input space to perform the selections, making it particularly suitable for small footprint pen-based input devices. Visual design alternatives are also discussed.

Abstract: We present Citrine, a system that extends the widespread copy-and-paste interaction technique with intelligent transformations, making it useful in more situations. Citrine uses text parsing to find the structure in copied text and allows users to paste the structured information, which might have many pieces, in a single paste operation. For example, using Citrine, a user can copy the text of a meeting request and add it to the Outlook calendar with a single paste. In applications such as Excel, users can teach Citrine by example how to copy and paste data by showing it which fields go into which columns, and can use this to copy or paste many items at a time in a user-defined manner. Citrine can be used with a wide variety of applications and types of data and can be easily extended to work with more. It currently includes parsers that recognize contact information, calendar appointments and bibliographic citations. It works with Internet Explorer, Outlook, Excel, Palm Desktop, EndNote and other applications. Citrine is available to download on the internet.

Abstract: Projection technology typically places several constraints on the geometric relationship between the projector and the projection surface to obtain an undistorted, properly sized image. In this paper we describe a simple, robust, fast, and low-cost method for automatic projector calibration that eliminates many of these constraints. We embed light sensors in the target surface, project Gray-coded binary patterns to discover the sensor locations, and then prewarp the image to accurately fit the physical features of the projection surface. This technique can be expanded to automatically stitch multiple projectors, calibrate onto non-planar surfaces for object decoration, and provide a method for simple geometry acquisition.

Abstract: ScreenCrayons is a system for collecting annotations on any type of document or visual information from any application. The basis for the system is a screen capture upon which the user can highlight the relevant portions of the image. The user can define any number of topics for organizing notes. Each topic is associated with a highlighting "crayon." In addition the user can supply annotations in digital ink or text. Algorithms are described that summarize captured images based on the highlight strokes so as to provide overviews of many annotations as well as being able to "zoom in" on particular information about a given note and the context of that note.

Abstract: This paper presents an approach for tracking paper documents on the desk over time and automatically linking them to the corresponding electronic documents using an overhead video camera. We demonstrate our system in the context of two scenarios, paper tracking and photo sorting. In the paper tracking scenario, the system tracks changes in the stacks of printed documents and books on the desk and builds a complete representation of the spatial structure of the desktop. When users want to find a printed document buried in the stacks, they can query the system based on appearance, keywords, or access time. The system also provides a remote desktop interface for directly browsing the physical desktop from a remote location. In the photo sorting scenario, users sort printed photographs into physical stacks on the desk. The systemautomatically recognizes the photographs and organizes the corresponding digital photographs into separate folders according to the physical arrangement. Our framework provides a way to unify the physical and electronic desktops without the need for a specialized physical infrastructure except for a video camera.

Abstract: This article presents MaggLite, a toolkit and sketch-based interface builder allowing fast and interactive design of post-WIMP user interfaces. MaggLite improves design of advanced UIs thanks to its novel mixed-graph architecture that dynamically combines scene-graphs with interaction-graphs. Scene-graphs provide mechanisms to describe and produce rich graphical effects, whereas interaction-graphs allow expressive and fine-grained description of advanced interaction techniques and behaviors such as multiple pointers management, toolglasses, bimanual interaction, gesture, and speech recognition. Both graphs can be built interactively by sketching the UI and specifying the interaction using a dataflow visual language. Communication between the two graphs is managed at runtime by components we call Interaction Access Points. While developers can extend the toolkit by refining built-in generic mechanisms, UI designers can quickly and interactively design, prototype and test advanced user interfaces by applying the MaggLite principle: "draw it, connect it and run it".

Abstract: We are building a multimedia conversation system to facilitate information seeking in large and complex data spaces. To provide tailored responses to diverse user queries introduced during a conversation, we automate the generation of a system response. Here we focus on the problem of determining the data content of a response. Specifically, we develop an optimization-based approach to content selection. Compared to existing rule-based or plan-based approaches, our work offers three unique contributions. First, our approach provides a general framework that effectively addresses content selection for various interaction situations by balancing a comprehensive set of constraints (e.g., content quality and quantity constraints). Second, our method is easily extensible, since it uses feature-based metrics to systematically model selection constraints. Third, our method improves selection results by incorporating content organization and media allocation effects, which otherwise are treated separately. Preliminary studies show that our method can handle most of the user situations identified in a Wizard-of-Oz study, and achieves results similar to those produced by human designers.

Abstract: We present a new screen space widget, the IBar, for effective camera control in 3D graphics environments. The IBar provides a compelling interface for controlling scene perspective based on the artistic concept of vanishing points. Various handles on the widget manipulate multiple camera parameters simultaneously to create a single perceived projection change. For example, changing just the perspective distortion is accomplished by simultaneously decreasing the camera's distance to the scene while increasing focal length. We demonstrate that the IBar is easier to learn for novice users and improves their understanding of camera perspective.

Abstract: This paper presents tangible interaction techniques for fine-tuning one-to-one scale NURBS curves on a large display for automotive design. We developed a new graspable handle with a transparent groove that allows designers to manipulate virtual curves on a display screen directly. The use of the proposed handle leads naturally to a rich vocabulary of terms describing interaction techniques that reflect existing shape styling methods. A user test raised various issues related to the graspable user interface, two-handed input, and large-display interaction.

Abstract: This paper describes a domain-specific debugger for one-way constraint solvers. The debugger makes use of several new techniques. First, the debugger displays only a portion of the dataflow graph, called a constraint slice, that is directly related to an incorrect variable. This technique helps the debugger scale to a system containing thousands of constraints. Second, the debugger presents a visual representation of the solver's data structures and uses color encodings to highlight changes to the data structures. Finally, the debugger allows the user to point to a variable that has an unexpected value and ask the debugger to suggest reasons for the unexpected value. The debugger makes use of information gathered during the constraint satisfaction process to generate plausible suggestions. Informal testing has shown that the explanatory capability and the color coding of the constraint solver's data structures are particularly useful in locating bugs in constraint code.

Abstract: In this paper, we describe the YeTi information sharing system that has been designed to foster community building through informal digital content sharing. The YeTi system is a general information parsing, hosting and distribution infrastructure, with interfaces designed for individual and public content reading. In this paper we describe the YeTi public display interface, with a particular focus on tools we have designed to provide lightweight awareness of others' interactions with posted content. Our tools augment content with metadata that reflect people's reading of content - captured video clips of who's reading and interacting with content, tools to allow people to leave explicit freehand annotations about content, and a visualization of the content access history to show when content is interacted with. Results from an initial evaluation are presented and discussed.

Abstract: The last twenty years have seen enormous leaps forward in computers' abilities to generate sound and video. What happens when computers can produce scents on demand? In this talk, I present three approaches to this question. I first look at human olfactory processing: what is our olfactory bandwidth, and what are the limitations of our sense of smell? I then explore the use of scent to accompany other media, from historical examples like Sense-o-Rama and Aromarama, to more recent work including firefighter training systems, augmented gaming, and food and beverage applications. Finally, I look at the possibilities of olfactory output as an ambient display medium. I conclude with an overview of current computer-controlled olfactory output devices: off the shelf solutions for incorporating scent into user interface applications.

Abstract: It is a pleasure to welcome you to Santa Fe for UIST 2004, the Seventeenth Annual ACM Symposium on User Interface Software and Technology. This year's symposium continues the UIST tradition of being the premier forum for presentation of research innovations in the software and technology of human-computer interfaces. UIST brings together researchers and practitioners from diverse areas that include traditional graphical and web user interfaces, tangible and ubiquitous computing, virtual and augmented reality, multimedia, new input and output devices, and CSCW. The intimate size, the single track, and comfortable surroundings make this symposium an ideal opportunity to exchange research results and implementation experience. The call for papers attracted 118 full paper submissions and 52 TechNote submissions from Asia, Europe, North and South America, and the South Pacific. The program committee accepted 24 full papers (20%) and 12 TechNotes (23%) that cover a variety of exciting topics, including augmented reality, design tools, toolkits, pen and tactile input, and ambient interfaces. In addition, the program includes a keynote speech by Mary Czerwisnki on using theory as a basis for human-computer interaction research and two invited surveys by Saul Greenberg and Joseph Kaye on Physical User Interfaces and Olfactory Displays, respectively. Posters, demos, and our second annual doctoral symposium round out our program; UIST attendees receive a separate document that contains the accepted submissions for those categories. The UIST proceedings are in color for the first time ever and we hope that they will serve as a valuable reference for user interface researchers and practitioners. We have supplemented the proceedings with a DVD containing videos for almost all of the papers, TechNotes, posters, and demonstrations.

Abstract: How do we break away from existing tools and techniques in HCI and truly innovate in a way that benefits the next generation of computer users? Today, too many of our technological designs and inventions are "one off" point designs, not building on or contributing to a theoretical foundation of understanding around human perception, cognition, social behavior and physical movement. Of course, these point designs can be successful in and of themselves, so why bother with theory and models? In order to mature as a field in a way that benefits users, it can be argued that we need to work more closely together and with an awareness of multiple disciplines, including not just the computer science and engineering arenas, but also psychology, sociology, and any field of human behavior. Of course, this could be a daunting task-how do we know that important improvements in user interface design can be obtained?I will present a series of examples of what I consider to be significant contributions to the field of HCI, each based on a multidisciplinarian, theory-driven approach. I hope to challenge the audience to creatively consider ways that their own work could be more theoretically motivated, and what it might take for more of us to move forward in that direction.

Abstract: Physical user interfaces are special purpose devices that can be situated in a real-world setting. Unlike general purpose computers, they are typically designed for particular contexts and uses. In this survey, I present an introductory tour of this new interface genre. First, I will summarize what they are by describing several design niches for these devices: ubiquitous computing, tangible media, foreground and ambient devices, collaborative devices, roomware, and physical controls. Examples will be plentiful, and will range from the playful, to the artistic, and to the serious. Second, I will introduce technologies that are suitable for software professionals who wish to prototype these physical user interfaces. The commercially available Phidgets (www.phidgets.com) are used as a case study of what is available and what can be done with them.

Abstract: In this paper we present a system for providing tactile feedback for stylus-based touch-screen displays. The Haptic Pen is a simple low-cost device that provides individualized tactile feedback for multiple simultaneous users and can operate on large touch screens as well as ordinary surfaces. A pressure-sensitive stylus is combined with a small solenoid to generate a wide range of tactile sensations. The physical sensations generated by the Haptic pen can be used to enhance our existing interaction with graphical user interfaces as well as to help make modern computing systems more accessible to those with visual or motor impairments.

Abstract: Involving graphic designers in the large-scale development of user interfaces requires tools that provide more graphical flexibility and support efficient software processes. These requirements were analysed and used in the design of the TkZ-inc graphical library and the IntuiKit interface design environment. More flexibility is obtained through a wider palette of visual techniques and support for iterative construction of images, composition and parametric displays. More efficient processes are obtained with the use of the SVG standard to import graphics, support for linking graphics and behaviour, and a unifying model-driven architecture. We describe the corresponding features of our tools, and show their use in the development of an application for airports. Benefits include a wider access to high quality visual interfaces for specialised applications, and shorter prototyping and development cycles for multidisciplinary teams.

Abstract: We present SketchREAD, a multi-domain sketch recognition engine capable of recognizing freely hand-drawn diagrammatic sketches. Current computer sketch recognition systems are difficult to construct, and either are fragile or accomplish robustness by severely limiting the designer's drawing freedom. Our system can be applied to a variety of domains by providing structural descriptions of the shapes in that domain; no training data or programming is necessary. Robustness to the ambiguity and uncertainty inherent in complex, freely-drawn sketches is achieved through the use of context. The system uses context to guide the search for possible interpretations and uses a novel form of dynamically constructed Bayesian networks to evaluate these interpretations. This process allows the system to recover from low-level recognition errors (e.g., a line misclassified as an arc) that would otherwise result in domain level recognition errors. We evaluated Sketch-READ on real sketches in two domains--family trees and circuit diagrams--and found that in both domains the use of context to reclassify low-level shapes significantly reduced recognition error over a baseline system that did not reinterpret low-level classifications. We also discuss the system's potential role in sketch based user interfaces.

Abstract: Powerful mobile devices with minimal I/O capabilities increase the likelihood that we will want to annex these devices to I/O resources we encounter in the local environment. This opportunistic annexing will require authentication. We present a sensor-based authentication mechanism for mobile devices that relies on physical possession instead of knowledge to setup the initial connection to a public terminal. Our solution provides a simple mechanism for shaking a device to authenticate with the public infrastructure, making few assumptions about the surrounding infrastructure while also maintaining a reasonable level of security.

Abstract: Light Emitting Diodes (LEDs) offer long life, low cost, efficiency, brightness, and a full range of colors. Because of these properties, they are widely used for simple displays in electronic devices. A previously characterized, but little known property of LEDs allows them to be used as photo sensors. In this paper, we show how this capability can be used to turn unmodified, off the shelf, LED arrays into touch sensitive input devices (while still remaining capable of producing output). The technique is simple and requires little or no extra hardware - in some cases operating with the same micro-controller based circuitry normally used to produce output, requiring only software changes. We will describe a simple hybrid input/output device prototype implemented with this technique, and discuss the design opportunities that this type of device opens up.

Abstract: Ubiquitous and Wearable Computing both have the goal of pushing the computer into the background, supporting all kinds of human activities. Application areas include areas such as everyday environments (e.g. clothing, home, office), promoting new forms of creative learning via physical/virtual objects, and new tools for interactive design. In this paper, we thrust ubiquitous computing into the extremely hostile environment of the sparring ring of a martial art competition. Our system uses piezoelectric force sensors that transmit signals wirelessly to enable the detection of when a significant impact has been delivered to a competitor's body. The objective is to support the judges in scoring the sparring matches accurately, while preserving the goal of merging and blending into the background of the activity. The system therefore must take into account of the rules of the game, be responsive in real-time asynchronously, and often cope with untrained operators of the system. We present a pilot study of the finished prototype and detail our experience.

Abstract: We present content-aware free-space transparency, an approach to viewing and manipulating the otherwise hidden content of obscured windows through unimportant regions of overlapping windows. Traditional approaches to interacting with otherwise obscured content in a window system render an entire window uniformly transparent. In contrast, content-aware free-space transparency uses opaque-to-transparent gradients and image-processing filters to minimize the interference from overlapping material, based on properties of that material. By increasing the amount of simultaneously visible content and allowing basic interaction with otherwise obscured content, without modifying window geometry, we believe that free-space transparency has the potential to improve user productivity.

Abstract: To enable common mobile terminals to interact with contents shown on large screens, we propose "C-Blink", a new light signal marker method that uses the color liquid-crystal display of a mobile terminal as a visible light source. We overcome the performance limitations of such displays by developing a hue-difference-blink technique. In combination with a screen-side sensor, we describe a system that detects and receives light signal markers sent by cell phone displays. Evaluations of a prototype system confirm that C-Blink performs well under common indoor lighting. The C-Blink program can be installed in any mobile terminal that has a color display, and the installation costs are small. C-Blink is a very useful way of enabling ubiquitous large screens to become interfaces for mobile terminals.

Abstract: The difficulty of accessing information details while preserving context has generated many different focus-in-context techniques. A common limitation of focus-in-context techniques is their ability to work well at high magnification. We present a set of improvements that will make high magnification in context more feasible. We demonstrate new distortion functions that effectively integrate high magnification within its context. Finally, we show how lenses can be used on top of other lenses, effectively multiplying their magnification power in the same manner that a magnifying glass applied on top of another causes multiplicative magnification. The combined effect is to change feasible detail-in-context magnification factors from less than 8 to more than 40.

Abstract: Despite novel interaction techniques proposed for virtual desktops, common yet challenging tasks remain to be investigated. Dragging and dropping between overlapping windows is one of them. The fold-and-drop technique presented here offers a natural and efficient way of performing those tasks. We show how this technique successfully builds upon several interaction paradigms previously described, while shedding new light on them.

Abstract: Many applications provide a form-like interface for requesting information: the user fills in some fields, submits the form, and the application presents corresponding results. Such a procedure becomes burdensome if (1) the user must submit many different requests, for example in pursuing a trial-and-error search, (2) results from one application are to be used as inputs for another, requiring the user to transfer them by hand, or (3) the user wants to compare results, but only the results from one request can be seen at a time. We describe how users can reduce this burden by creating custom interfaces using three mechanisms: clipping of input and result elements from existing applications to form cells on a spreadsheet; connecting these cells using formulas, thus enabling result transfer between applications; and cloning cells so that multiple requests can be handled side by side. We demonstrate a prototype of these mechanisms, initially specialised for handling Web applications, and show how it lets users build new interfaces to suit their individual needs.