Image Inpainting based on Dilated Neighborhood Attention
DOI:
https://doi.org/10.54691/sxsvpw35Keywords:
Network Image Inpainting; Generating Adversarial Networks; Attention Mechanism; Two-stage Framework.Abstract
In response to the phenomenon that existing image restoration algorithms have blurry edges, incoherent textures, and lack clarity and delicacy for large-area missing images, a two-stage generative adversarial image restoration algorithm based on dilated neighborhood attention is proposed. This algorithm decouples image restoration into edge structure restoration and texture structure restoration, introduces a dilated neighborhood attention mechanism, and enhances the generator's focus on important information and structures in the image by constructing a residual attention network, thereby improving the perception and utilization of texture details, resulting in more realistic image views and finer texture details. This paper introduces the Binary Cross-Entropy with Logits loss function in the discriminators of the two stages, which can help the discriminator learn how to more effectively identify real and generated images, thus improving the overall network performance. The Ranger21 optimizer is introduced to accelerate learning without affecting generalization, addressing the problem of traditional optimizers systematically staying in poor initial states. The datasets used in this paper are Paris Street View and CelebA-HQ. Comparative experiments with other image restoration methods show that both peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) have improved, and the larger the mask area, the more significant the improvement. Experiments prove that the images restored by the proposed algorithm have more reasonable structures and richer details, and the image restoration effect is superior.
Downloads
References
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in neural information processing systems, 2014, 27.
Efros A A, Leung T K. Texture synthesis by non-parametric sampling[C]//Proceedings of the seventh IEEE international conference on computer vision. IEEE, 1999, 2: 1033-1038.
Ashikhmin M. Synthesizing natural textures[C]//Proceedings of the 2001 symposium on Interactive 3D graphics. 2001: 217-226.
Wei L Y, Levoy M. Fast texture synthesis using tree-structured vector quantization[C]//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000: 479-488.
Drori I, Cohen-Or D, Yeshurun H. Fragment-based image completion[M]//ACM SIGGRAPH 2003 Papers. 2003: 303-312.
Zomet. Learning how to inpaint from global image statistics[C]//Proceedings Ninth IEEE international conference on computer vision. IEEE, 2003: 305-312 vol. 1.
Criminisi A, Pérez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on image processing, 2004, 13(9): 1200-1212.
ZHANG S H, WANG K G, ZHU X. Improved Criminisi algorithm constrained by local feature[J]. Computer Engineering and Applications, 2014, 50(8): 127-13.
Zhao N, WANG H Q, Wu M. Criminisi digital inpainting algorithm based on Markov random field matching criterion[J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(7): 1150-1158.
CROSS G R, JAIN A K. Markov random field texture models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1983(1): 25-39.
BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24.
Guillemot C, Le Meur O. Image inpainting: Overview and recent advances[J]. IEEE signal processing magazine, 2013, 31(1): 127-144.
Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C]//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000: 417-424.
Shen J, Chan T F. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3): 1019-1043.
Telea A. An image inpainting technique based on the fast marching method[J]. Journal of graphics tools, 2004, 9(1): 23-34.
PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting [C] // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 2536-2544.
S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics, 36(4), 2017.
CAO C, FU Y. Learning a sketch tensor space for image inpainting of man- made scenes [C]// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 14509- 14518.
YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the 2019 IEEE/ CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 3, 2019. Piscataway: IEEE, 2019: 4470-4479.
WANG Y, TAO X, QI X, et al. Image inpainting via generative multi-column convolutional neural networks[C]//Proceedings of the Annual Conference on Neural Information Processing Systems 2018, Montréal, Dec 3-8, 2018: 329-338.
LIU H, JIANG B, SONG Y, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations [C]//LNCS 12347: Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23- 28, 2020. Cham: Springer, 2020: 725-741.
SAGONG M, SHIN Y, KIM S, et al. PEPSI: fast image inpainting with parallel decoding network [C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Los Angeles, Jun 15-21, 2019. Piscataway: IEEE, 2019: 11360-11368.
GUO Z Y, CHEN Z B, YU T, et al. Progressive image inpainting with full-resolution residual network [C] // Proceedings of the 27th ACM International Conference on Multimedia, Nice, Oct 21-25, 2019. New York: ACM, 2019: 2496-2504.
Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks [C] //2017 IEEE Conference on Computer Vision and Pattern Recognition, July 21-26, 2017, Honolulu, HI, USA. New York: IEEE Press, 2017: 5967-5976.
Yeh R A, Chen C, Lim T Y, et al. Semantic image inpainting with deep generative models[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition, July 21-26, 2017, Honolulu, HI, USA. New York: IEEE Press, 2017: 6882-6890.
Nazeri K, Ng E, Joseph T, et al. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv 2019[J]. arXiv preprint arXiv:1901.00212, 2020.
Doersch C, Singh S, Gupta A, et al. What makes paris look like paris?[J]. Communications of the ACM, 2015, 58(12): 103-110.
Karras T, Aila T, Laine S, et al. Progressive growing of gans for improved quality, stability, and variation[J]. arXiv preprint arXiv:1710.10196, 2017.
Zhu M, He D, Li X, et al. Image inpainting by end-to-end cascaded refinement with mask awareness [J]. IEEE Trans on Image Processing, 2021, 30: 4855-4866.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
How to Cite
