Li, Z. orcid.org/0000-0002-2066-8775, Lu, M. orcid.org/0000-0002-5044-8802, Zhang, X. orcid.org/0000-0002-1882-736X et al. (3 more authors) (2024) Efficient Visual Computing with Camera RAW Snapshots. IEEE Transactions on Pattern Analysis and Machine Intelligence. ISSN 0162-8828
Abstract
Conventional cameras capture image irradiance (RAW) on a sensor and convert it to RGB images using an image signal processor (ISP). The images can then be used for photography or visual computing tasks in a variety of applications, such as public safety surveillance and autonomous driving. One can argue that since RAW images contain all the captured information, the conversion of RAW to RGB using an ISP is not necessary for visual computing. In this paper, we propose a novel ρ-Vision framework to perform high-level semantic understanding and low-level compression using RAW images without the ISP subsystem used for decades. Considering the scarcity of available RAW image datasets, we first develop an unpaired CycleR2R network based on unsupervised CycleGAN to train modular unrolled ISP and inverse ISP (invISP) models using unpaired RAW and RGB images. We can then flexibly generate simulated RAW images (simRAW) using any existing RGB image dataset and finetune different models originally trained in the RGB domain to process real-world camera RAW images. We demonstrate object detection and image compression capabilities in RAW-domain using RAW-domain YOLOv3 and RAW image compressor (RIC) on camera snapshots. Quantitative results reveal that RAW-domain task inference provides better detection accuracy and compression efficiency compared to that in the RGB domain. Furthermore, the proposed ρ-Vision generalizes across various camera sensors and different task-specific models. An added benefit of employing the ρ-Vision is the elimination of the need for ISP, leading to potential reductions in computations and processing times.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. |
Keywords: | Camera RAW, RAW-domain Object Detection, RAW Image Compression |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > SWJTU Joint School (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 01 Feb 2024 15:34 |
Last Modified: | 01 Feb 2024 15:34 |
Published Version: | https://ieeexplore.ieee.org/document/10415533 |
Status: | Published online |
Publisher: | Institute of Electrical and Electronics Engineers (IEEE) |
Identification Number: | 10.1109/tpami.2024.3359326 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:208499 |