Abstract: Photo-realistic rendering of virtual objects into real scenes is one of the most important research problems in computer graphics. Methods for capture and rendering of mixed reality scenes are driven by a large number of applications, ranging from augmented reality to visual effects and product visualization. Recent developments in computer graphics, computer vision, and imaging technology have enabled a wide range of new mixed reality techniques including methods for advanced image based lighting, capturing spatially varying lighting conditions, and algorithms for seamlessly rendering virtual objects directly into photographs without explicit measurements of the scene lighting. This report gives an overview of the state-of-the-art in this field, and presents a categorization and comparison of current methods. Our in-depth survey provides a tool for understanding the advantages and disadvantages of each method, and gives an overview of which technique is best suited to a specific problem.

Abstract: Remote rendering of three-dimensional images using virtual machines includes using a hypervisor executing on a physical computer to allocate exclusive and direct access to a graphics processing unit in the physical computer, to a first virtual machine. An agent executing on a second virtual machine intercepts three-dimensional draw commands generated by a three-dimensional application and forwards the intercepted draw commands to a rendering agent executing on the first virtual machine. The rendering agent then transmits the intercepted draw commands to the graphics processing unit for rendering upon which the graphics processing unit renders a three-dimensional image from the draw commands. The rendering agent obtains the rendered image from the graphics processing unit and forwards the image to the second virtual machine. Upon receiving the rendered image, the second virtual machine transmits the rendered image to another remote, physical computer where the rendered image is displayed to a user.

Abstract: Cloud gaming allows games to be rendered on the cloud server and allows the rendered videos to be encoded and streamed in real time to the player’s devices. Compared with other video streaming applications, cloud gaming offers a unique opportunity to enhance the video encoding process by exploiting rendering information. In this paper, we propose two techniques to improve cloud gaming video encoding, aiming at enhancing the perceived video quality and reducing the computational complexity, respectively. First, we develop a rendering-based prioritized encoding technique to improve the perceived game video quality according to network bandwidth constraints. We first propose a technique to generate a macroblock (MB)-level saliency map for every game video frame using rendering information. Furthermore, based on such a saliency map, a prioritized rate allocation scheme is proposed to dynamically adjust the value of quantization parameter of each MB. The experimental results indicate that the perceptual quality can be greatly improved using the proposed technique. We also develop a rendering-based encoding acceleration technique that utilizes rendering information to reduce the computational complexity of video encoding. This technique mainly consists of two parts. First, we propose a method to directly calculate the motion vectors (MVs) without employing the compute intensive motion search procedure. Second, based on the computed MVs, we propose a fast mode selection algorithm to reduce the number of candidate modes of each MB. The experimental results show that the proposed technique can achieve more than 42% saving in encoding time with very limited degradation in video quality.

Abstract: This article describes a Mixed-Reality (MR) application that superimposes lost buildings of a historical site onto real scenes virtualized using spherical aerial images. The proposed application is set at a UNESCO World Heritage site in Japan, and is based on a novel framework that supports the photorealistic superimposition of virtual objects onto virtualized real scenes. The proposed framework utilizes Image-Based Rendering (IBR), which enables users to freely change their viewpoint in a real-world virtualization constructed using precaptured images. This framework combines the offline rendering of virtual objects and IBR to take advantage of the higher quality of offline rendering without the additional computational cost of online processing; that is, it incurs only the cost of online lightweight IBR, which is simplified through the pregeneration of structured viewpoints (e.g., at grid points).

Abstract: We describe an image-based non-photorealistic rendering pipeline for creating portraits in two styles: The first is a somewhat "puppet" like rendering, that treats the face like a relatively uniform smooth surface, with the geometry being emphasised by shading. The second style is inspired by the artist Julian Opie, in which the human face is reduced to its essentials, i.e. homogeneous skin, thick black lines, and facial features such as eyes and the nose represented in a cartoon manner. Our method is able to automatically generate these stylisations without requiring the input images to be tightly cropped, direct frontal view, and moreover perform abstraction while maintaining the distinctiveness of the portraits (i.e. they should remain recognisable).

Abstract: Physically correct rendering of environment illumination has been a long-standing challenge in interactive graphics, since Monte-Carlo ray-tracing requires thousands of rays per pixel. We propose accurate filtering of a noisy Monte-Carlo image using Fourier analysis. Our novel analysis extends previous works by showing that the shape of illumination spectra is not always a line or wedge, as in previous approximations, but rather an ellipsoid. Our primary contribution is an axis-aligned filtering scheme that preserves the frequency content of the illumination. We also propose a novel application of our technique to mixed reality scenes, in which virtual objects are inserted into a real video stream so as to become indistinguishable from the real objects. The virtual objects must be shaded with the real lighting conditions, and the mutual illumination between real and virtual objects must also be determined. For this, we demonstrate a novel two-mode path tracing approach that allows ray-tracing a scene with image-based real geometry and mesh-based virtual geometry. Finally, we are able to de-noise a sparsely sampled image and render physically correct mixed reality scenes at over 5 fps on the GPU.

Abstract: BlenderVR is an open-source project framework for interactive and immersive applications based on an extension of the Blender Game Engine to Virtual Reality applications. BlenderVR is a generalization of the BlenderCAVE project, accounting for alternate platforms (e.g., HMD, video-walls). The goal is to provide a flexible and easy to use framework for the creation of VR applications for various platforms, making use of the existing power of the BGE's graphics rendering and physics engine. Compatible with 3 major Operating Systems, BlenderVR has been developed by VR researchers with support from the Blender Community. BlenderVR currently handles multi-screen/multi-user tracked stereoscopic rendering through efficient low-level master/slave synchronization process with multimodal interactions via OSC and VRPN protocols.

Abstract: Recent large-scale particle-based simulations are generating vast amounts of data posing a challenge to visualization algorithms. One possibility for addressing this challenge is to map particles into a regular grid for volume rendering, which carries the disadvantages of inefficient use of memory and undesired losses of dynamic range. As an alternative, we propose a method to efficiently visualize these massive particle datasets using point rendering techniques with neither loss of dynamic range nor memory overheads. In addition, a hierarchical reorganization of the data is desired to deliver meaningful visual representations of a large number of particles in a limited number of pixels, preserving point locality and also helping achieve interactive frame rates. In this paper, we present a framework for parallel rendering of large-scale particle data sets combining point sprites and z-ordering. The latter is used to create a multi level representation of the data which helps improving frame rates. Performance and scalability are evaluated on a GPU-based visualization cluster, scaling up to 128 GPUs. Results using particle datasets of up to 32 billion particles are shown.

Abstract: We present a fast reconstruction filtering method for images generated with Monte Carlo-based rendering techniques. Our approach specializes in reducing global illumination noise in the presence of depth-of-field effects at very low sampling rates and interactive frame rates. We employ edge-aware filtering in the sample space to locally improve outgoing radiance of each sample. The improved samples are then distributed in the image plane using a fast, linear manifold-based approach supporting very large circles of confusion. We evaluate our filter by applying it to several images containing noise caused by Monte Carlo-simulated global illumination, area light sources and depth of field. We show that our filter can efficiently denoise such images at interactive frame rates on current GPUs and with as few as 4-16 samples per pixel. Our method operates only on the colour and geometric sample information output of the initial rendering process. It does not make any assumptions on the underlying rendering technique and sampling strategy and can therefore be implemented completely as a post-process filter.

Abstract: We describe our successful initiative to accelerate Adobe Illustrator with the graphics hardware pipeline of modern GPUs. Relying on OpenGL 4.4 plus recent OpenGL extensions for advanced blend modes and first-class GPU-accelerated path rendering, we accelerate the Adobe Graphics Model (AGM) layer responsible for rendering sophisticated Illustrator scenes. Illustrator documents render in either an RGB or CMYK color mode. While GPUs are designed and optimized for RGB rendering, we orchestrate OpenGL rendering of vector content in the proper CMYK color space and accommodate the 5+ color components required. We support both non-isolated and isolated transparency groups, knockout, patterns, and arbitrary path clipping. We harness GPU tessellation to shade paths smoothly with gradient meshes. We do all this and render complex Illustrator scenes 2 to 6x faster than CPU rendering at Full HD resolutions; and 5 to 16x faster at Ultra HD resolutions.

Abstract: We present a web-based implementation of parallel coordinates suitable for big data visual analytics. While being easily accessible through web-browsers, the system supports advanced analytics on the server as well as density-based rendering on the client with support for hardware accelerated graphics. A prototype implementation is available at parallelcoordinates.de

Abstract: OSVR is a new open-source virtual reality platform that builds on and extends existing libraries. It provides a common ground for university and commercial research and development, enabling new development at any level to be rapidly tested and deployed in a full application stack. Devices are discovered and enumerated, with self-description data made available to applications for use in sensible auto-configuration. OSVR is agnostic about operating system, graphics language, and rendering framework — enabling rapid porting of applications to new systems.

Abstract: With the arrival of auto-stereoscopic 3D displays for mobile devices, and emergence of more 3D content, there is much anticipation for 3D mobile multimedia experiences, including 3D display gaming. Simultaneously, with the emergence of cloud computing, more mobile applications are being developed to take advantage of the elastic cloud resources. In this paper, we explore the possibility of developing Cloud Mobile 3D Display Gaming, where the 3D video rendering and encoding is performed on cloud servers, with the resulting 3D video streamed over wireless networks to mobile devices with 3D displays for a true 3D mobile gaming experience. However, with the significantly higher bitrate requirement for 3D video, ensuring user experience may be a challenge, both in terms of 3D video quality and network delay (response time), considering the bandwidth constraints and fluctuations of wireless networks. In this paper, we propose a new asymmetric graphics rendering approach which can significantly reduce the video encoding bitrate needed for a certain video quality, thereby making it easier to transmit the video over wireless network. However, since asymmetric graphics rendering may also impair the graphics quality, we need to be able to understand and measure its impact. We conduct subjective tests to study and model the impairments due to asymmetric graphics rendering and network delay, thereby developing a user experience model for cloud based mobile 3D display gaming. By conducting subsequent subjective tests, we prove the correctness of the impairment functions and the resulting user experience model. Furthermore, given any network condition, we propose to solve the problem of selecting the optimal graphics rendering factors for the left and right views so as to maximize user experience of cloud mobile 3D display gaming. In order to solve this problem, we first develop a model to estimate the resulting video bitrate of the rendered 3D video when certain graphics rendering factors are used. Next, we derive a model to predict the delay given the available network bandwidth and the video bitrate of the rendered 3D video. We use the above two models together with a branch and bound algorithm to solve the optimization problem and determine the optimal values for the left and right view rendering factors. Experiments conducted using real 4G-LTE network profiles on commercial cloud service demonstrate the feasibility of significant improvement in user experience when the proposed optimization algorithm is used to dynamically select optimal rendering factors according to changing network conditions.

Abstract: In order to achieve an immersive, natural 3D experience on a large screen, a 100-Mpixel multi-projection 3D display was developed. Ninety-six projectors were used to increase the number of rays emanating from each pixel in the horizontal direction to 96. Conventional algorithms use a large number of cameras or input images to process a large number of light rays. This creates difficulties in the design of both the large acquiring system and substantial memory storage. In this paper, we propose an efficient light-field rendering algorithm that utilizes only a few input colors and depth images. Using a depth map and estimated camera parameters, synthesized light-field images are directly generated. This algorithm requires a much lighter memory load than conventional light-field rendering algorithms. It is also much simpler than the image-based rendering algorithm because it does not require the generation of so many multiview images.

Abstract: This document introduces a new application for rendering massive LiDAR point cloud data sets of interior environments within highresolution immersive VR display systems. Overall contributions are: to create an application which is able to visualize large-scale point clouds at interactive rates in immersive display environments, to develop a flexible pipeline for processing LiDAR data sets that allows display of both minimally processed and more rigorously processed point clouds, and to provide visualization mechanisms that produce accurate rendering of interior environments to better understand physical aspects of interior spaces. The work introduces three problems with producing accurate immersive rendering of Li-DAR point cloud data sets of interiors and presents solutions to these problems. Rendering performance is compared between the developed application and a previous immersive LiDAR viewer.

Abstract: A device comprising a graphics processing unit (GPU) includes a memory and at least one processor. The at least one processor may be configured to: receive a GPU command packet that indicates the GPU may select between a direct rendering mode or a binning rendering mode for a portion of a frame to be rendered by the GPU, determine whether to use the direct rendering mode or the binning rendering mode for the portion of the frame to be rendered by the GPU based on at least one of: information in the received command packet or a state of the GPU, and render the portion of the frame using the determined direct rendering mode or the binning rendering mode.

Abstract: This paper features an advanced implementation of the X-ray rendering algorithm that harnesses the giant computing power of the current commodity graphics processors to accelerate the generation of high resolution digitally reconstructed radiographs (DRRs). The presented pipeline exploits the latest features of NVIDIA Graphics Processing Unit (GPU) architectures, mainly bindless texture objects and dynamic parallelism. The rendering throughput is substantially improved by exploiting the interoperability mechanisms between CUDA and OpenGL. The benchmarks of our optimized rendering pipeline reflect its capability of generating DRRs with resolutions of 20482 and 40962 at interactive and semi interactive frame-rates using an NVIDIA GeForce 970 GTX device.

Abstract: A cross-platform rendering engine. The cross-platform rendering engine serves as an intermediary between the application and the operating system for displaying application content on the screen allowing software developers to write platform-agnostic application code. The application sends content to the cross-platform rendering engine in the form of resource-efficient content descriptions describing the content to be displayed. In turn, cross-platform rendering engine stores the content descriptions and provides rasterized images generated from the content descriptions to the compositor as needed. In the event that a content description corresponding to the texture needed by the compositor is not available, the cross-platform rendering engine pulls the content description from the application. By producing rasterized images from a user interface thread on behalf of the application rather than waiting for the application thread, the cross-platform rendering engine improves the overall responsiveness of the application and contributes to a better user experience.

Abstract: We present a physically-based computational model of the light sheet fluorescence microscope (LSFM). Based on Monte Carlo ray tracing and geometric optics, our method simulates the operational aspects and image formation process of the LSFM. An extension for previous fluorescence models is developed to account for the intrinsic characteristics of fluorescent dyes in order to accurately simulate light interaction with fluorescent-tagged biological specimen. This extension was quantitatively validated against the fluorescence brightness equation and experimental spectra of different dyes. We demonstrate first results of our rendering pipeline to a simplified brain tissue model reconstructed from the somatosensory cortex of a young rat.

Abstract: In the area of computer graphics the design of hardware and software has primarily been driven by the need to achieve maximum performance. Energy efficiency was usually neglected, assuming that a stable always-on power source was available. However, the advent of the mobile era has brought into question these ideas and designs in computer graphics since mobile devices are both limited by their computational capabilities and their energy sources. Aligned to this emerging need in computer graphics for energy efficiency analysis we have setup a software framework to obtain power measurements from 3D scenes using off-the-shelf hardware that allows for sampling the energy consumption over the power rails of the CPU and GPU. Our experiments include geometric complexity, texture resolution and common CPU and GPU workloads. The goal of this work is to combine the knowledge obtained from these measurements into a prototype energy-aware balancer of processing resources. The balancer dynamically selects the rendering parameters and uses a simple framerate-based dynamic frequency scaling strategy. Our experimental results demonstrate that our power saving framework can achieve savings of approximately 40%.

Abstract: Three-dimension (3D) medical images on eHealth can facilitate early diagnosis and treatment planning for physicians and dentists anytime and anywhere. However, high-resolution 3D medical images created from most medical imaging devices are difficult to render interactively with high-quality in real-time on the client side. This paper tackles this problem. It presents a web-based client-server framework that provides apparent processing of high-quality volume rendering in real-time on personal computers and tablets. The framework employs the following technique. While zooming and rotating the volume rendered image, the graphic card on a client-side processes low-quality volume rendering. At the same time, images for performing high-quality rendering are being processed and streamed in from the server. When the interaction stops, the server completes the streaming process and the client can now display high-quality volume rendering. This would help visualizing 3D medical images in real-time and high-quality on web for fast diagnosis and treatment planning.

Abstract: Mobile devices are part of our everyday life and allow augmented reality (AR) with their integrated camera image. Recent research has shown that even photorealistic augmentations with consistent illumination are possible. A method, achieving this first, distributed lighting computations and the extraction of the important light sources. To reach real-time frame rates on a mobile device, the number of these extracted light sources must be low, limiting the scope of possible illumination scenarios and the quality of shadows. In this paper, we show how to reduce the computational cost per light using a combination of tile-based rendering and frustum culling techniques tailored for AR applications. Our approach runs entirely on the GPU and does not require any precomputation. Without reducing the displayed image quality, we achieve up to 2.2atimes; speedup for typical AR scenarios.

Abstract: We introduce an incremental rendering layer on top of standard graphics APIs such as OpenGL or DirectX in the form a virtual machine (VM), which efficiently maintains an optimized, compiled representation of arbitrary high-level scene representations at all times. This includes incremental processing of structural changes such as additions and removals of scene parts, as well as in-place updates of scene data. Our approach achieves a significant framerate increase for typical workloads and reasonable performance for high-frequency changes. Processing is performed in running time O(Δ), where Δ is proportional to the size of the change and the optimized representation has no runtime overhead with respect to the underlying graphics API. This is achieved by tracking and applying all changes as incremental updates to appropriate data structures and by adaptively synthesizing a program of abstract machine code. In a final step this abstract program is incrementally mapped to executable machine code --- comparable to what just-in-time compilers do. Our main contributions are (i) an abstract interface for rendering and visualization systems enabling incremental evaluation, (ii) adaptively optimized abstract machine code in the context of stateless graphics commands, and (iii) subsequent adaptive compilation to executable machine code including on-the-fly defragmentation.

Abstract: xviii

Abstract: We introduce the physically-based Material Definition Language (MDL). Based on the principle of strictly separating material definition and rendering algorithms, each MDL material is applicable across different rendering paradigms ranging from realtime over interactive solutions to advanced light transport simulation.

Abstract: GPU voxel-based surface ray-casting has positioned as an interesting alternative to rasterization-based rendering approaches, because it allows using many processing units simultaneously, can effectively exploit thread level parallelism, and enables fine-granularity occlusion culling on the pixel level. Yet voxel-based techniques face the problem that an extremely high resolution is necessary to avoid block artifacts at high zoom levels. In this work, we propose a novel improvement of voxel-based ray-casting to overcome this limitation. By using a hierarchical Vector-to-Closest-Point (VCP) representation, we can inherit the advantages of a voxel-based approach at a much smoother approximation of the surface. We demonstrate that, although the VCP grid consumes more memory per cell, it requires less memory overall, because it builds upon a significantly shallower tree hierarchy. In a number of examples we demonstrate the use of our approach for high-quality rendering of high resolution surface models.

Abstract: The interactive rendering of 3D computer graphics has approached the photorealistic quality, as evident by the vivid shading effects and lush outdoor scenes in recent game engines. Clearly, the traditional 3D graphics APIs are reaching their limits, and the need to switch to more extendable ray-tracing based techniques has arisen. In this work, we explore the fundamental differences between ray tracing based and rasterization based techniques, including how they are supported by the processors and by the programming platforms. We duplicate the typical shading effects in both ray tracing and rasterization, starting from the simple Phong lighting, to slightly more complex Whitted-style shadow and reflection. Although the rasterization-based techniques clearly outperform ray tracing in current generations of graphics processors, we show by more precise quantitative analysis that the performance gaps are not as wide as thought. And the gap may narrow further when the requirement of image quality increases in the future.

Abstract: A graphics processing unit (GPU) may perform three-dimensional (3D) graphics processing in accordance with a 3D graphics pipeline using a first plurality of graphics processing hardware units of the GPU. The GPU may further perform a two-dimensional (2D) graphics operation using a second plurality of graphics processing hardware units of the GPU not used in performing the 3D graphics processing and one or more graphics processing hardware units of the first plurality of graphics processing hardware units of the GPU.

Abstract: Technologies for web-based game execution include a computing device with a web rendering engine and a native game engine library. The web rendering engine establishes a a scripting environment that issues calls to a game engine interface established by the web rendering engine. The scripting environment may be a JavaScript engine. In response to calls to the game engine interface, the game engine interface issues calls to the native game engine library. The native game engine library issues native graphics commands to a graphics bridge of the computing device. The native graphics commands may be OpenGL calls. The graphics bridge translates the native graphics commands to a web graphics context, which renders graphical game content to a web content element of the web rendering engine. The web graphics context may be a WebGL context, and the web content element may be a canvas element. Other embodiments are described and claimed.