Abstract: Systems and methods for navigating hypermedia using multiple coordinated input/output device sets. Disclosed systems and methods allow a user and/or an author to control what resources are presented on which device sets (whether they are integrated or not), and provide for coordinating browsing activities to enable such a user interface to be employed across multiple independent systems. Disclosed systems and methods also support new and enriched aspects and applications of hypermedia browsing and related business activities.

Abstract: The development of Industry 4.0 will be accompanied by changing tasks and demands for the human in the factory. As the most flexible entity in cyber-physical production systems, workers will be faced with a large variety of jobs ranging from specification and monitoring to verification of production strategies. Through technological support it is guaranteed that workers can realize their full potential and adopt the role of strategic decision-makers and flexible problem-solvers. The use of established interaction technologies and metaphors from the consumer goods market seems to be promising. This paper demonstrates solutions for the technological assistance of workers, which implement the representation of a cyber-physical world and the therein occurring interactions in the form of intelligent user interfaces. Besides technological means, the paper points out the requirement for adequate qualification strategies, which will create the required, inter-disciplinary understanding for Industry 4.0.

Abstract: In 1965 Ivan E. Sutherland envisioned the Ultimate Display, a room in which a computer can directly control the existence of matter. This type of display would merge the digital and the physical world, dramatically changing how people interact with computers. This thesis explores flat displays, deformable displays, flexible materials, static, and mobile projection displays in dynamic environments. %Dynamic environments are inherent to human behavior, but pose big problems to Human-Computer Interaction since computing devices rely on many assumptions of the interaction. Two aspects of the dynamic environment are considered. One is mobile human nature -- a person moving through or inside an environment. The other is the change or movement of the environment itself. The initial study consisted of a mixed reality application, based on recent motor learning research. It tested if a performer's attentional focus on markers external to the body improves the accuracy and duration of acquiring a motor skill, as compared with the performer focusing on their own body accompanied by verbal instructions. This experiment showed the need for displays that resemble physical reality. Deformable displays and Organic User Interfaces (OUIs) leverage shape, material, and the inherent properties of matter in order to create natural, intuitive forms of interaction. We suggested designing OUIs employing depth sensors as 3D input, deformable displays as 3D output, and identifying attributes that couple matter to human perception and motor skills. Flexible materials were explored by developing a soft gripper able to hold everyday objects of various shapes and sizes. It did not use complex hardware or control algorithms, but rather combined sheets of flexible plastic materials and a single servo motor. The gripper showed how a simple design with a minimal control mechanism can solve a complex problem in a dynamic environment. It serves as an example application for merging the digital and the physical through flexible materials, embodied computation, and actuation. The next two experiments merge digital information with the physical dynamic environment by using mobile and static projectors. The mobile projector experiment consisted of GPS navigation using a bike-mounted projector, displaying a map on the pavement in front of the bike. We found out that if compared with a bike-mounted smartphone, the mobile projector yields a lower cognitive load for the map navigation task. A dynamic space emerges from the navigation task requirements, and the projected display becomes a part of the physical environment. In the final experiment, a person interacts with a changing, growing environment, on which digital information is projected from above using a static projector. The interactive space consists of cardboard building blocks, the arrangement of which are limited by the area of projection. The user adds cardboard blocks to the cluster based upon feedback projected from above. Concepts from artificial intelligence and architecture were applied for understanding the interaction between the environment, the user, the morphology, and the material of the physical building system.

Abstract: An electronic device with a display and a fingerprint sensor displays a fingerprint enrollment interface and detects, on the fingerprint sensor, a plurality of finger gestures performed with a finger. The device collects fingerprint information from the plurality of finger gestures performed with the finger. After collecting the fingerprint information, the device determines whether the collected fingerprint information is sufficient to enroll a fingerprint of the finger. When the collected fingerprint information for the finger is sufficient to enroll the fingerprint of the finger, the device enrolls the fingerprint of the finger with the device. When the collected fingerprint information for the finger is not sufficient to enroll the fingerprint of the finger, the device displays a message in the fingerprint enrollment interface prompting a user to perform one or more additional finger gestures on the fingerprint sensor with the finger.

Abstract: A thermostat for controlling an HVAC system is described, the thermostat having a user interface that is visually pleasing, approachable, and easy to use while also providing ready access to, and intuitive navigation within, a menuing system capable of receiving a variety of different types of user settings and/or control parameters. For some embodiments, the thermostat comprises a housing, a ring-shaped user-interface component configured to track a rotational input motion of a user, a processing system configured to identify a setpoint temperature value based on the tracked rotational input motion, and an electronic display coupled to the processing system. An interactive thermostat menuing system is accessible to the user by an inward pressing of the ring-shaped user interface component. User navigation within the interactive thermostat menuing system is achievable by virtue of respective rotational input motions and inward pressings of the ring-shaped user interface component.

Abstract: A system for managing edge devices, such as scanner devices, typically includes a management module having a processor. The processor is typically communicatively coupled to a user interface that includes a visual display and a plurality of edge devices. The processor may also be communicatively coupled to a plurality of intermediate devices. The management module enables a user of the user interface to manage various aspects of the system. The processor is typically configured for displaying with the visual display (i) a physical view of the connections between the management module, the intermediate devices, and the edge devices and (ii) a first user-defined view of the management module, the intermediate devices, and the scanner devices. The processor is typically also configured for managing operational information generated by connected edge devices, including for efficiently storing operational information and intelligently querying the operational information to assist the user in matching compatible plug-in applications with particular edge devices.

Abstract: We present an augmented reality system that supports human workers in a rapidly changing production environment. By providing spatially registered information on the task directly in the user's field of view the system can guide the user through unfamiliar tasks (e.g. assembly of new products) and visualize information directly in the spatial context were it is relevant. In the first version we present the user with picking and assembly instructions in an assembly application. In this paper we present the initial experience with this system, which has already been used successfully by several hundred users who had no previous experience in the assembly task.

Abstract: Android smartphones are becoming increasingly popular. The open nature of Android allows users to install miscellaneous applications, including the malicious ones, from third-party marketplaces without rigorous sanity checks. A large portion of existing malwares perform stealthy operations such as sending short messages, making phone calls and HTTP connections, and installing additional malicious components. In this paper, we propose a novel technique to detect such stealthy behavior. We model stealthy behavior as the program behavior that mismatches with user interface, which denotes the user's expectation of program behavior. We use static program analysis to attribute a top level function that is usually a user interaction function with the behavior it performs. Then we analyze the text extracted from the user interface component associated with the top level function. Semantic mismatch of the two indicates stealthy behavior. To evaluate AsDroid, we download a pool of 182 apps that are potentially problematic by looking at their permissions. Among the 182 apps, AsDroid reports stealthy behaviors in 113 apps, with 28 false positives and 11 false negatives.

Abstract: A handheld dimensioner with a user interface configured to present a quality indicator is disclosed. The handheld dimensioner is configured to capture three-dimensional (3D) data and assess the three-dimensional-data's quality. Based on this quality, a quality indicator may be generated and presented to a user via the user interface. This process may be repeated while the user repositions the handheld dimensioner. In this way, the user may use the quality indicators generated at different positions to find an optimal position for a particular dimension measurement.

Abstract: A smartwatch is provided with an embedded camera; barcode decoding software; a user interface system, the user interface system being provided with a visual display on the smartwatch; and a processor communicatively coupled to the camera, the user interface system, and the barcode decoding software. The processor is configured for a barcode scanning and decoding operation. The barcode scanning and decoding operation is configured for: triggering a view finder on the smartwatch, aligning a barcode in the view finder; capturing images of the barcode aligned in the view finder with the camera, feeding the barcode images to barcode decoding software; and sending the decoded barcode information to the visual display.

Abstract: Described are various embodiments of a user content sharing system and method with automated external content integration. In one embodiment, a system and method are provided in which a graphical user interface (GUI) is rendered on each system users' personal communication device. The GUI produces a content selection function selecting, under user operation, user content for sharing, and a sharing platform selection function selecting, under user operation, one or more sharing platforms. The system further comprise a digital content integrator communicatively linked to the user interface to gain access to the selected user content in response to the content selection function, the integrator having access to stored external content distinct from user content. The integrator integrates the external content with the selected user content to output integrated content. The system further comprises a communication interface communicatively linked to the integrator and selectively operable to interface with each of the distinct content sharing platforms, wherein the integrated content is concurrently relayed to the selected sharing platforms via the communication interface on behalf of system users as originating therefrom.

Abstract: The present disclosure relates to making payments with a mobile device. In one example process, the mobile device receives and stores information for one or more payment accounts on the mobile device. The mobile device is used to make payments using the payment accounts. In some examples, authorization to proceed with a payment is performed before each purchase made by the user. The authorization process can include receiving a verification of the user, such as a fingerprint scan or passcode. In some examples, a payment account is selected from among available payment accounts. In some examples, an indication is displayed of a digital item associated with a purchased item. In some examples, a payment transaction is initiated with participants of an ongoing communication. In some examples, an application of a retailer is invoked based on the availability of the application. In some examples, a purchase recommendation is provided.

Abstract: Various systems and methods are provided that display various interactive user interfaces containing interactive maps. The user may use the interactive map to perform searches for data objects, toggle different layers associated with different types of objects for display, generate heatmaps based upon a grid and/or data object shapes, copy data between different interactive maps, and/or define filters to filter the displayed data. In response to user interaction with the map at a client system, tile layers are generated using server-side components, assembled into map tiles, and transmitted to client-side components to be displayed to the user. The tile layers may include a base tile layer, vector tile layer, selection tile layer, and inactive tile layer.

Abstract: We discuss the architecture and design principles that underpin the latest version of SKIRT, a state-of-the-art open source code for simulating continuum radiation transfer in dusty astrophysical systems, such as spiral galaxies and accretion disks. SKIRT employs the Monte Carlo technique to emulate the relevant physical processes including scattering, absorption and emission by the dust. The code features a wealth of built-in geometries, radiation source spectra, dust characterizations, dust grids, and detectors, in addition to various mechanisms for importing snapshots generated by hydrodynamical simulations. The configuration for a particular simulation is defined at run-time through a user-friendly interface suitable for both occasional and power users. These capabilities are enabled by careful C++ code design. The programming interfaces between components are well defined and narrow. Adding a new feature is usually as simple as adding another class; the user interface automatically adjusts to allow configuring the new options. We argue that many scientific codes, like SKIRT, can benefit from careful object-oriented design and from a friendly user interface, even if it is not a graphical user interface.

Abstract: Current smartphones generally cannot continuously authenticate users during runtime. This poses severe security and privacy threats: A malicious user can manipulate the phone if bypassing the screen lock. To solve this problem, our work adopts a continuous and passive authentication mechanism based on a user’s touch operations on the touchscreen. Such a mechanism is suitable for smartphones, as it requires no extra hardware or intrusive user interface. We study how to model multiple types of touch data and perform continuous authentication accordingly. As a first attempt, we also investigate the fundamentals of touch operations as biometrics by justifying their distinctiveness and permanence. A onemonth experiment is conducted involving over 30 users. Our experiment results verify that touch biometrics can serve as a promising method for continuous and passive authentication.

Abstract: At least one aspect is directed to a wearable treatment device that includes a cardiac sensing electrode, a treatment electrode, a user interface, and a sensor. The cardiac sensing electrode detects cardiac information, and the treatment electrode applies treatment to the subject. The user interface receives quality of life information from the subject, and the sensor detects subject activity and wellness information. A controller coupled with the cardiac sensing electrode, the treatment electrode, the user interface, and the sensor receives the detected cardiac information, the quality of life information, and the detected subject activity and wellness information, and determines that treatment is to be applied to the body of the subject based upon the detected cardiac information. The controller can also adjust the treatment based on at least one of the detected subject activity and wellness information and the quality of life information, and an alarm module provides an alarm after the cardiac information is detected and before the treatment is applied to the body of the subject.

Abstract: Understanding the driver's cognitive load is important for evaluating in-vehicle user interfaces. This paper describes experiments to assess machine learning classification algorithms on their ability to automatically identify elevated cognitive workload levels in drivers, leading towards the development of robust tools for automobile user interface evaluation. We look at using both driver performance as well as physiological data. These measures can be collected in real-time and do not interfere with the primary task of driving the vehicle. We report classification accuracies of up to 90% for detecting elevated levels of cognitive load, and show that the inclusion of physiological data leads to higher classification accuracy than vehicle sensor data evaluated alone. Finally, we show results suggesting that models can be built to classify cognitive load across individuals, instead of building individual models for each per-son. By collecting data from drivers in two large field studies on the highway (20 drivers and 99 drivers), this work extends prior work and demonstrates feasibility and potential of such measures for HCI research in vehicles.

Abstract: A method and a system for connecting a service provider and a user at a remote location relative to the service provider, via a network based telecommunications device, are provided. The method includes utilizing a network operable terminal for transmitting communications between the service provider and the user, employing a display screen depicting a user selectable options menu, corresponding with service functions offered by the service provider, and enabling the user to choose an option from the user selectable options menu to initiate a corresponding communication to the service provider. The system includes a user operable terminal including a user interface display screen with user selectable menu options that are changeable in accordance with differing modes of operation, an internal processing unit configured for providing at least one selectable menu option; and a gateway service platform configured for transmitting an option selected from the at least one selectable menu option to and from one of the user or the provider.

Abstract: In this completely updated and revised edition of Designing with the Mind in Mind, Jeff Johnson provides you with just enough background in perceptual and cognitive psychology that user interface (UI) design guidelines make intuitive sense rather than being just a list or rules to follow. Early UI practitioners were trained in cognitive psychology, and developed UI design rules based on it. But as the field has evolved since the first edition of this book, designers enter the field from many disciplines. Practitioners today have enough experience in UI design that they have been exposed to design rules, but it is essential that they understand the psychology behind the rules in order to effectively apply them. In this new edition, you'll find new chapters on human choice and decision making, hand-eye coordination and attention, as well as new examples, figures, and explanations throughout. It provides an essential source for user interface design rules and how, when, and why to apply them. It arms designers with the science behind each design rule, allowing them to make informed decisions in projects, and to explain those decisions to others. It equips readers with the knowledge to make educated tradeoffs between competing rules, project deadlines, and budget pressures. It completely updated and revised, including additional coverage on human choice and decision making, hand-eye coordination and attention, and new mobile and touch-screen examples throughout.

Abstract: A portable device with a touch screen display detects a finger-in-contact event at a first position on the touch screen display and identifies a user interface object associated with the first position. The portable device determines whether the identified user interface object is moveable across the touch screen display. For an identified object that is determined to be moveable across the touch screen display, the portable device detects a sequence of one or more first finger-dragging, a first finger-out-of-contact, one or more second finger-dragging, and a first finger-out-of-range events and performs a first operation associated with the identified moveable user interface object at least in part in accordance with the first and second finger-dragging events. For an identified not moveable object, the portable device detects a second finger-out-of-contact event and performs a second operation, different from the first operation in response to a second finger-out-of-contact event.

Abstract: Plan recognition, activity recognition, and intent recognition together combine and unify techniques from user modeling, machine vision, intelligent user interfaces, human/computer interaction, autonomous and multi-agent systems, natural language understanding, and machine learning. Plan, Activity, and Intent Recognition explains the crucial role of these techniques in a wide variety of applications including: personal agent assistants, computer and network security, opponent modeling in games and simulation systems, coordination in robots and software agents, web e-commerce and collaborative filtering, dialog modeling, video surveillance, smart homes In this book, follow the history of this research area and witness exciting new developments in the field made possible by improved sensors, increased computational power, and new application areas. Combines basic theory on algorithms for plan/activity recognition along with results from recent workshops and seminars Explains how to interpret and recognize plans and activities from sensor data Provides valuable background knowledge and assembles key concepts into one guide for researchers or students studying these disciplines

Abstract: A method of profiling a physical environment via Internet of Things (IoT) devices connected via an IoT integration platform is disclosed. The method includes: receiving data streams from network-capable devices connected to the IoT integration platform; detecting presence of an entity in a physical environment surrounding or interacting with one or more of the network-capable devices from a subset of the data streams; storing an entity profile on a node graph representing the physical environment, wherein each entity profile is associated at least one of the network-capable device from which the presence is detected; and requesting a semantic label of the entity profile from a user of a first user account via one or more user interfaces.

Abstract: An interactive, graph-based user interaction data analysis system is disclosed. The system is configured to provide analysis and graphical visualizations of user interaction data to a system operator. In various embodiments, interactive visualizations and analyses provided by the system may be based on user interaction data aggregated across particular groups of users, across particular time frames, and/or from particular computer-based platforms and/or applications. According to various embodiments, the system may enable insights into, for example, user interaction patterns and/or ways to optimize for desired user interactions, among others. In an embodiment, the system allows an operator to analyze and investigate user interactions with content provided via one or more computer-based platforms, software applications, and/or software application editions.

Abstract: Assistive free-flyers (AFFs) are an emerging robotic platform with unparalleled flight capabilities that appear uniquely suited to exploration, surveillance, inspection, and telepresence tasks. However, unconstrained aerial movements may make it difficult for colocated operators, collaborators, and observers to understand AFF intentions, potentially leading to difficulties understanding whether operator instructions are being executed properly or to safety concerns if future AFF motions are unknown or difficult to predict. To increase AFF usability when working in close proximity to users, we explore the design of natural and intuitive flight motions that may improve AFF abilities to communicate intent while simultaneously accomplishing task goals. We propose a formalism for representing AFF flight paths as a series of motion primitives and present two studies examining the effects of modifying the trajectories and velocities of these flight primitives based on natural motion principles. Our first study found that modified flight motions might allow AFFs to more effectively communicate intent and, in our second study, participants preferred interacting with an AFF that used a manipulated flight path, rated modified flight motions as more natural, and felt safer around an AFF with modified motion. Our proposed formalism and findings highlight the importance of robot motion in achieving effective human-robot interactions. Categories and Subject Descriptors H.1.2 [Models and Principles]: User/Machine Systems— human factors, software psychology; H.5.2 [Information Interfaces and Presentation]: User Interfaces— evaluation/ methodology, user-centered design General Terms Design, Human Factors

Abstract: Adaptive user interfaces (UIs) were introduced to address some of the usability problems that plague many software applications. Model-driven engineering formed the basis for most of the systems targeting the development of such UIs. An overview of these systems is presented and a set of criteria is established to evaluate the strengths and shortcomings of the state of the art, which is categorized under architectures, techniques, and tools. A summary of the evaluation is presented in tables that visually illustrate the fulfillment of each criterion by each system. The evaluation identified several gaps in the existing art and highlighted the areas of promising improvement.

Abstract: Conversation user interfaces that are configured for virtual assistant interaction may include contextual interface items that are based on contextual information. The contextual information may relate to a current or previous conversation between a user and a virtual assistant and/or may relate to other types of information, such as a location of a user, an orientation of a device, missing information, and so on. The conversation user interfaces may additionally, or alternatively, control an input mode based on contextual information, such as an inferred input mode of a user or a location of a user. Further, the conversation user interfaces may tag conversation items by saving the conversation items to a tray and/or associating the conversation items with indicators.