- Yunxiang Zhang*, New York University
- Bingxuan Li*, New York University
- Alexandr Kuznetsov, Advanced Micro Devices (AMD)
- Akshay Jindal, Intel Corporation
- Stavros Diolatzis, Intel Corporation
- Kenneth Chen, New York University
- Anton Sochenov, Intel Corporation
- Anton Kaplanyan, Intel Corporation
- Qi Sun, New York University
Content-adaptive image representation with Image-GS. Leveraging a tailored differentiable renderer, Image-GS adaptively distributes and progressively optimizes a set of 2D Gaussians to fit a target image. Image-GS shows high memory & computation e!iciency, supports fast random pixel access, and o!ers a natural level of detail. (a) shows the learned Gaussian position distribution (green dots); 20% of Gaussians are plotted for better visibility. (b) Compared to alternative methods, Image-GS's content-adaptive nature enables it to wisely allocate resources based on the local signal complexity and preserve fine image details with higher fidelity. The insets visualize the corresponding error images, with brighter colors indicating higher errors.
Abstract
Neural image representations have emerged as a promising approach for storing and rendering visual data. Combined with learning-based work-flows, these novel representations have demonstrated impressive balances between visual quality and memory footprint. Existing methods along this line, however, often rely on fixed data structures that suboptimally allocate memory budget or computation-intensive implicit neural models, limiting their adoption in real-time graphics applications. Inspired by recent advances in radiance field rendering, we introduce Image-GS, an efficient, flexible, and content-adaptive image representation based on anisotropic 2D Gaussians. Image-GS delivers remarkable visual quality and memory effciency while supporting fast random access and a natural level-of-detail stack. Leveraging a custom differentiable renderer imlemented via efficient CUDA kernels, Image-GS reconstructs target images by adaptively allocating and progressively optimizing a set of 2D Gaussians. Our method achieves superior visual fidelity over state-of-the-art neural image representations across diverse images and textures. Notably, Image-GS exhibits linear scaling in memory and computational requirements relative to the number of Gaussians, offering a flexible trade-off between fidelity and run-time efficiency, which we demonstrate in machine vision and image restoration tasks.