Accelerated 2D visualization using adaptive resolution scaling and temporal reconstruction

Question

1. What is the impact of rendering performance on 2D image synthesis in information visualizations?

2. What is checkerboard rendering?

3. How does the amortization algorithm improve response times?

4. How is amortization calculated?

Accepted Answer

Rendering performance has become an increasingly relevant issue for 2D image synthesis in typical information visualizations. As displays with resolutions above 1080p become more common, rendering times increase for large datasets or display resolutions, making interactivity hard to guarantee. In multi-view applications that combine 3D and 2D rendering, the 2D visualizations often become the limiting factor for overall performance and responsiveness of interactive brushing and linking. The sheer amount of vertices and generated fragments in 2D visualizations becomes a bottleneck for rendering, even when only simple geometric objects are used. To improve response times, different approaches can be employed, such as filtering and brushing of all individual data points, pre-rendering 2D visualizations into a high-resolution texture atlas, or lowering the rendering resolution and reconstructing a native-resolution image by reprojecting samples from previous frames. However, these approaches have their limitations and trade-offs, and achieving a balance between visual quality and performance is crucial for a good user experience.

Accepted Answer

Checkerboard rendering is a method used to increase performance in video games and interactive environments. It involves rendering at a lower resolution and utilizing several buffered frames to meet performance requirements. This approach was popularized with the Playstation 4 due to its hardware implementation. It differs from other methods as it focuses on reducing internal rendering resolution and relies on different strategies for image reconstruction, such as depth-based and velocity-based sample rejection. Checkerboard rendering is less aggressive in reducing image quality compared to other temporal coherence methods.

Accepted Answer

The amortization algorithm improves response times by reducing internal rendering resolution while still reconstructing a native-resolution output image each frame. It achieves this by combining low-resolution rendering with the native-resolution output of the previous frame, reprojecting it to compensate for camera movement. By jittering the camera position in each frame, the low-resolution rendering progressively samples all pixel positions of the native resolution, converging to the original quality over time. The sampling strategy for camera jitter is crucial, as it greatly impacts the speed of image quality convergence. The algorithm aims to enhance dynamic scenarios, including user interaction, rather than focusing solely on static cases where caching the rendering result would be a simpler solution. Additionally, the method is renderer-agnostic, making it easy to adopt to any visualization framework. The amortization pipeline consists of three steps: preparing a low-resolution framebuffer and calculating camera jitter, executing the target renderer, and reconstructing the native-resolution image by reprojecting the last image and integrating new samples from the current low-resolution rendering.

Accepted Answer

Amortization is calculated by dividing the width and height of the current native display resolution by the user-defined amortization level a. The internal framebuffer U is resized to the calculated resolution, if necessary, and set as the render target for the target renderer. The number of newly rendered pixels per frame is reduced to 1/a 2. Each pixel of the low-resolution image represents an a A a area of native-resolution pixels. To sample all pixel positions of the native-resolution image over time, the camera is moved in a pattern of sub-pixel offsets within the image plane, aligning pixel centers of U with the pixel centers of each block of a A a pixels of the native-resolution image. Camera offsets for each point of the pattern can be calculated using the equation EQUATION, with a being the level of amortization and indices i and j being grid positions within the a A a area. If the native display resolution is not divisible by a, the next multiple of a is assumed. Extra pixels rendered during jittering can be ignored during reconstruction. A pattern that subdivides each block into four quadrants and takes turns in picking a sample location from each quadrant is proposed. Quadrants are recursively subdivided to pick samples within the quadrants in each turn. Diagonal movement is preferred to maximize distances. This pattern is illustrated for a 1/4 2 and a 1/4 4 in Fig. 2 (left and middle). Independent scaling factors for width and height can also be used, allowing for a less aggressive reduction of image information. The 1D sampling pattern for distributing samples within blocks is computed similarly to the 2D blocks, recursively subdividing the blocks into half segments and taking turns in picking a sample location from each segment. The sampling pattern for 1 A 4 is illustrated in Fig. 2 (right).

Accepted Answer

The reconstruction process involves using two textures A and B at native resolution. A contains the reconstructed image of the previous frame, while B stores the current reconstruction. Additionally, position textures A p and B p hold the world-space positions of corresponding samples. After each reconstruction, A and B are swapped for the next frame. To obtain pixels in B, we distinguish two cases: direct copying from U for aligned pixels and finding the nearest sample within A and U for other pixels. Reprojection is used to project pixel positions from B to world space, considering camera changes and reading corresponding positions from A p. The process involves recursive sampling patterns and minimizing distances to find the nearest sample within U. The final pixel in B is calculated using the computed distances and comparing them. The reconstructed native-resolution image is displayed, and textures are swapped for the next frame. The algorithm converges to the ground truth image after two consecutive frames with unchanged camera and data.

Accepted Answer

Adaptive resolution scaling is a technique used in visualization techniques to dynamically adjust the amortization level based on the current rendering performance. This approach aims to minimize the distance between the current rendering performance and a user-defined performance target, typically measured in frames per second (fps). By automatically adjusting the amortization level, adaptive resolution scaling ensures that the rendering performance is optimized without compromising image quality. This technique is commonly employed in video games to meet performance targets and provide a highly reactive player feedback. In the context of scientific and information visualization, the refresh rates below 30 fps or even 20 fps are often still acceptable and considered interactive. Adaptive resolution scaling helps to maintain a balance between rendering performance and visual quality, allowing users to focus on data exploration tasks without constantly adjusting the amortization level manually. The algorithm for adaptive auto-scaling predicts frame time changes for increasing or decreasing the amortization level, using a prediction factor. It also considers the user-defined target frame time and the frame time average over the last frames to determine the appropriate amortization level for the next frame. By dynamically adjusting the amortization level, adaptive resolution scaling enhances the overall visualization experience and ensures optimal performance for various visualization scenarios.

Accepted Answer

Amortization affects line thickness and point size in OpenGL or DirectX by impacting operations that rely on absolute pixel size. In the worst case, the resulting image may differ. To preserve the original image, line thickness or point size would have to be set in the sub-pixel range, which is not guaranteed to be supported in the used graphics APIs. To counteract this, the size of employed render primitives cannot be expressed in absolute pixel sizes but should instead be given in world space. Both renderers show in the paper feature rendering modes that construct line and point geometry from triangles, avoiding issues with limited point size or using a line width other than 1.0 in an OpenGL core profile. Triangle-based rendering allows size and thickness to be varied and set to match desired pixel sizes in native resolution, without inherent loss of quality. However, vertex processing of triangle-based geometry is more costly compared to line and point primitives. Despite this, the additional cost is offset by the ability to use higher amortization levels without compromising visual quality. It is recommended to render UI elements either as an additional layer after amortization or using resolution-independent primitives and shading for UI rendering to avoid rendering artifacts caused by amortization.

Accepted Answer

The parameter study covered a wide range of rendering loads and hardware to understand how each parameter affects performance. Parameters used include GPU, target renderer, dataset, resolution, amortization mode, and amortization level. All possible combinations of parameter values were systematically run, with four full reconstruction cycles per configuration. Timings for 25 frames were measured for native rendering without amortization, and averaged for comparison. The benchmarks were performed on a PC with specific hardware configurations. The results were implemented within the MegaMol visualization framework using the C programming language. The parameter study helped in evaluating the performance of the method and understanding the impact of different parameters on acceleration and rendering times.

Accepted Answer

The trade-off in our method is between response time and convergence time. Response time affects interactivity and is determined by the time it takes to render a single frame. Convergence time refers to the time it takes to update every visible pixel. Each amortization level improves response time by reducing the amount of rendered pixels per frame, but requires a certain number of frames to converge with different modes. For the renderers used in this paper, convergence times of up to two seconds are acceptable, especially considering the rapid improvement in image quality during the first half (see Fig. 10).

Accepted Answer

In systematic evaluation, amortization level significantly impacts frame times and acceleration. As amortization levels increase, convergence times rise, leading to slower frame rates. However, frame rates can still be improved and reach above 30 fps for certain GPUs. The achieved acceleration also varies with amortization levels, with a consistent improvement across GPUs. However, for smaller datasets, the increase in acceleration begins to decline for higher amortization levels, converging towards a potential maximum acceleration value. This behavior is observed in all configurations, but often at amortization levels beyond 1/4 8, which are considered less relevant. The impact of amortization levels on frame times and acceleration is also influenced by the native display resolution and dataset size. Higher amortization levels become more favorable and acceleration increases noticeably with each additional level, especially at higher resolutions. The performance improvement varies across different GPUs and amortization modes, with the Isotropic mode generally performing better, but the Horizontal and Vertical modes showing similar performance for certain GPUs and resolutions.

Accepted Answer

Integrated graphics, such as iGPUs, offer less computing performance compared to dedicated GPUs. They are commonly used in laptops and mobile systems, which have strict power and thermal budgets. In the provided section, two mobile computers with different iGPUs were evaluated for visualization performance. The AMD Radeon RX Vega 7 graphics rendered at a resolution of 1920x1080, while the Intel Iris Xe Graphics G7 96EUs rendered at a resolution of 2880x1920. Both devices achieved barely interactive frame rates (around 5 fps) when rendering at native resolution. However, with the use of the researcher's method, performance improved significantly up to an amortization level of 1/4, reaching 30 fps. Beyond this level, both iGPUs converged towards a maximum acceleration, indicating limited improvement in performance.

Accepted Answer

The reconstruction stage adds a fixed resolution-dependent overhead to rendering, potentially affecting performance gains. It may not provide additional acceleration and can result in variable frame times due to costly image regions. This is particularly noticeable in small datasets and when heavy overdraw occurs. The method's amortized approach distributes structures both spatially and temporally, leading to inconsistent frame times. This limitation should be considered when evaluating the method's effectiveness and potential impact on overall rendering performance.

Accepted Answer

Amortization level affects static image quality by initially having lower Structural Similarity Index (SSIM) and requiring more time to converge to the ground truth image. Lower internal rendering resolution results in fewer pixels being updated per frame. However, all levels converge toward the ground truth image with an SSIM of more than 0.9999 after rendering two frames. Higher amortization levels initially have lower SSIM and require more time to converge to the ground truth image, as due to lower internal rendering resolution fewer pixels are updated per frame. All levels, however, converge toward the ground truth image with an SSIM of more than 0.9999 after a 2 frames. This information is supported by Figure 9, which shows frame times for selected GPUs and a specific scenario that causes inconsistent frame times. The frame times form a repeating pattern with a period of a 1/4 4 frames, as marked with cyan lines. Additionally, Figure 10 shows SSIM of each frame starting without any information from previous frames until convergence after a 2 frames have been rendered. This demonstrates the impact of amortization level on static image quality and convergence.

Accepted Answer

Camera translation affects image quality for dynamic content by allowing re-projection of known parts of the image that remain within the view frustum without loss of quality, similar to static image quality. However, areas that were not visible in the prior frames can become visible at the edge of the screen, resulting in a blocky appearance when an image region first enters the view. With each consecutive frame, these regions are progressively refined and converge to the original image. The overall shape of the image is mostly retained, allowing the user some level of orientation even with initially lower quality. Camera translation needs to be a multiple of the display pixel size in world space, which is usually the case as mouse input is given in pixels and the cursor position is fixed during interaction. Correcting errors in sampling positions and sampled area during zooming is left for future work.

Accepted Answer

AMD FidelityFX TM Super Resolution (FSR) improves rendering performance by reducing internal rendering resolution, similar to our method. However, unlike our method, FSR has no temporal component and uses edge-adaptive spatial upsampling and optional contrast-adaptive sharpening to reconstruct a native-resolution image from the low-resolution rendering. In a test scene using FSR with its Performance preset, which scales resolution by a factor of 2, the resulting image has an SSIM of 0.9668. Bi-linear upscaling results in an SSIM of 0.9626. Our method, with amortization level 1/4 2, starts off with slightly lower quality after a complete image refresh but exceeds both FSR Performance quality and bi-linear upscaling once a second frame has been rendered. Higher amortization levels require longer convergence time to exceed FSR Performance quality, e.g., approximately 500 ms for a 1/4 4. Apart from image quality, our method also improves average frame time, e.g., to 62 ms for a 1/4 4, compared to FSR's average frame time of 90 ms.

Accepted Answer

The proposed method improves rendering performance by accelerating GPU rendering of 2D information visualizations. It achieves this by rendering at a lower internal resolution and temporally reprojecting pixels to reconstruct a native-resolution image. This approach amortizes the rendering cost over multiple internal frames, resulting in reduced frame times and enhanced responsiveness of visualizations. Benchmarks conducted with different visualizations, dataset sizes, GPUs, and settings demonstrated significant performance improvements. However, the acceleration for high amortization levels may not always scale as well as convergence time, requiring a balance between interactive response time and convergence time in certain scenarios. Future work includes investigating continuous resolution scaling, integrating sophisticated reconstruction schemes, and conducting user studies to evaluate perceived quality for dynamic images.

Accelerated 2D visualization using adaptive resolution scaling and temporal reconstruction

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What is the impact of rendering performance on 2D image synthesis in information visualizations?

2. What is checkerboard rendering?

3. How does the amortization algorithm improve response times?

4. How is amortization calculated?

5. How is the reconstruction process achieved?

6. What is adaptive resolution scaling and how does it improve visualization performance?

7. How does amortization affect line thickness and point size in OpenGL or DirectX?

8. How did parameter study affect performance evaluation?

9. What is the trade-off in our method?

10. How does amortization level affect frame times and acceleration in systematic evaluation?

11. How do integrated graphics affect visualization performance?

12. What are the limitations of the reconstruction stage?

13. How does amortization level affect static image quality?

14. How does camera translation affect image quality for dynamic content?

15. How does FSR compare to our method?

16. How does the proposed method improve rendering performance?

References

Image quality assessment: from error visibility to structural similarity

Designing the user interface: strategies for the effective human-computer interaction

Study of visual acuity during the ocular pursuit of moving test objects. I. Introduction.

Neural supersampling for real-time rendering

Amortized supersampling

Related Papers (5)

GTVIS: Fast and Efficient Rendering System for Real-Time Terrain Visualization

Real-Time Recursive Specular Reflections on Planar and Curved Surfaces using Graphics Hardware.

GPU-Based Interactive Visualization Techniques

Modeling and visualization of knitwear

Video editing using figure tracking and image-based rendering