Ioannou, E. and Maddock, S. orcid.org/0000-0003-3179-0263 (2024) Towards real-time G-buffer-guided style transfer in computer games. IEEE Transactions on Games. ISSN 2475-1502
Abstract
Artistic Neural Style Transfer (NST) has achieved remarkable success for images. However, this is not the case for dynamic 3D environments, such as computer games, where temporal coherence remains a challenge. Our paper presents an approach that uses the G-buffer information available in a game pipeline to generate robust and temporally consistent in-game artistic stylizations based on a style reference image. We use a synthetic dataset created from open-source computer games and demonstrate that the utilization of depth, normals, and edge information enables the stylization process to be more aware of the geometric and semantic aspects of a game scene. The proposed approach builds on previous work by injecting style transfer in the rendering pipeline, while also utilizing G-buffer information during inference time to improve upon the stability of the stylizations, offering a controllable way to stylize computer games in terms of temporal coherence and content preservation. Qualitative and quantitative evaluations of our in-game stylization network demonstrate significantly higher temporal stability compared to existing style transfer approaches when stylizing 3D computer games.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2024 The Author(s). Except as otherwise noted, this author-accepted version of a journal article published in IEEE Transactions on Games is made available via the University of Sheffield Research Publications and Copyright Policy under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ |
Keywords: | Neural style transfer; computer games, G-buffer; convolutional neural network (CNN); graphics pipeline |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield) |
Funding Information: | Funder Grant number Engineering and Physical Sciences Research Council 2496728 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 12 Mar 2024 11:53 |
Last Modified: | 12 Mar 2024 14:17 |
Status: | Published online |
Publisher: | Institute of Electrical and Electronics Engineers |
Refereed: | Yes |
Identification Number: | 10.1109/TG.2024.3372829 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:209827 |
Download
Filename: G_buffer_Guided_NST_in_Games__IEEE_Transactions_on_Games_.pdf
Licence: CC-BY 4.0