More Than Lightening: A Self-Supervised Low-Light Image Enhancement Method Capable for Multiple Degradations

Han Xu; Jiayi Ma; Yixuan Yuan; Hao Zhang; Xin Tian; Xiaojie Guo

doi:10.1109/JAS.2024.124263

Volume 11 Issue 3

Mar. 2024

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 11.8, Top 4% (SCI Q1)

CiteScore: 17.6, Top 3% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2024 > 11(3): 622-637

H. Xu, J. Ma, Y. Yuan, H. Zhang, and X. Guo, “More than lightening: A self-supervised low-light image enhancement method capable for multiple degradations”, IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 622–637, Mar. 2024. doi: 10.1109/JAS.2024.124263

Citation:

H. Xu, J. Ma, Y. Yuan, H. Zhang, and X. Guo, “More than lightening: A self-supervised low-light image enhancement method capable for multiple degradations”, IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 622–637, Mar. 2024. doi: 10.1109/JAS.2024.124263

Citation:

PDF( 25490 KB)

More Than Lightening: A Self-Supervised Low-Light Image Enhancement Method Capable for Multiple Degradations

doi: 10.1109/JAS.2024.124263

Han Xu^{1
,
,},
Jiayi Ma^{1
,
,
,},
Yixuan Yuan^{2
,
,
,},
Hao Zhang^{1
,
,},
Xin Tian^{1
,
,},
Xiaojie Guo^{3
,
,}

1.
Electronic Information School, Wuhan University, Wuhan 430072, China
2.
Department of Electronic Engineering, Chinese University of Hong Kong, Hong Kong 999077, China
3.
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China

Funds:

the National Natural Science Foundation of China 62276192

More Information

Author Bio:
Han Xu received the B.S. degree in the Electronic Information School and the Ph.D. degree in the Multi-spectral Vision Processing Lab, Electronic Information School, Wuhan University, in 2018 and 2023. She has first-authored over 10 refereed journal and conference papers, including IEEE TPAMI/TIP, Information Fusion, CVPR, AAAI, IJCAI, etc. Her current research interests include computer vision and image processing. e-mail: xu_han@whu.edu.cn

Jiayi Ma (Senior Member, IEEE) received the B.S. degree in information and computing science and the Ph.D. degree in control science and engineering from the Huazhong University of Science and Technology, in 2008 and 2014, respectively. He is currently a Professor with the Electronic Information School, Wuhan University. He has authored or co-authored more than 300 refereed journal and conference papers, including IEEE TPAMI/TIP, IJCV, CVPR, ICCV, ECCV, etc. His research interests include computer vision, machine learning, and robotics. Dr. Ma has been identified in the 2019–2022 Highly Cited Researcher lists from the Web of Science Group. He is an Area Editor of Information Fusion, and an Associate Editor of IEEE/CAA Journal of Automatica Sinica. e-mail: jiayima@whu.edu.cn

Yixuan Yuan (Senior Member, IEEE) received the B.S. degree in automation from Northwestern Polytechnical University, in 2010, the Ph.D. degree in Electronic Engineering from the Chinese University of Hong Kong. From 2018–2022, she was an assistant professor in the Department of Electrical Engineering at the City University of Hong Kong. She is currently an assistant professor in the Department of Electronic Engineering at the Chinese University of Hong Kong. Her research interests include AI in healthcare to promote precision medicine, especially for medical image diagnosis, abnormality detection, and segmentation. She has authored or co-authored more than 100 refereed journal and conference papers. She received several premium awards including the CVPR 2022 Best Paper Finalist, MICCAI 2022 Young Scientist Award (supervisor), 2017 Young Scientist Award in Engineering by Hong Kong Institute of Science, Best Paper Award in ICMA 2013, etc. e-mail: yxyuan@ee.cuhk.edu.hk

Hao Zhang received the B.E. degree from the School of Mechanical Engineering and Electronic Information, China University of Geosciences, in 2019, and the M.S. degree from the Electronic Information School, Wuhan University, in 2021, where he is pursuing the Ph.D. degree. He has first-authored several refereed journal and conference papers, including IJCV, IEEE TCI, Information Fusion, ISPRS P & RS, AAAI, etc. His research interests include computer vision, machine learning, and pattern recognition. e-mail: zhanghao@whu.edu.cn

Xin Tian received the B.S. degree from the Department of Electronic Science and Technology, and the Ph.D. degree from the Institute for Pattern Recognition and Artificial Intelligence, Huazhong University of Science and Technology, in 2004 and 2010, respectively. He is currently a Professor with the School of Electronic Information, Wuhan University, China. He has authored or co-authored over 50 refereed journal and conference papers. His current research interests include computational imaging, image fusion, and sparse coding. e-mail: xin.tian@whu.edu.cn

Xiaojie Guo (Senior Member, IEEE) received the Ph.D. degree in computer science from the School of Computer Science and Technology, Tianjin University (TJU). He is currently an Associate Professor with tenure (Peiyang Young Scientist) with Tianjin University. Prior to joining TJU, he spent about four years with the Institute of Information Engineering, Chinese Academy of Sciences. He was a recipient of the Piero Zamperoni Best Student Paper Award in ICPR 2010, and the Best Student Paper Runner-up in ICME 2018. e-mail: Xiaojie.guo@tju.edu.cn
Corresponding author: Jiayi Ma, e-mail: jiayima@whu.edu.cn; Yixuan Yuan, e-mail: yxyuan@ee.cuhk.edu.hk
Received Date: 2023-11-06
Revised Date: 2024-01-05
Accepted Date: 2024-01-17

Abstract

Abstract

Low-light images suffer from low quality due to poor lighting conditions, noise pollution, and improper settings of cameras. To enhance low-light images, most existing methods rely on normal-light images for guidance but the collection of suitable normal-light images is difficult. In contrast, a self-supervised method breaks free from the reliance on normal-light data, resulting in more convenience and better generalization. Existing self-supervised methods primarily focus on illumination adjustment and design pixel-based adjustment methods, resulting in remnants of other degradations, uneven brightness and artifacts. In response, this paper proposes a self-supervised enhancement method, termed as SLIE. It can handle multiple degradations including illumination attenuation, noise pollution, and color shift, all in a self-supervised manner. Illumination attenuation is estimated based on physical principles and local neighborhood information. The removal and correction of noise and color shift removal are solely realized with noisy images and images with color shifts. Finally, the comprehensive and fully self-supervised approach can achieve better adaptability and generalization. It is applicable to various low light conditions, and can reproduce the original color of scenes in natural light. Extensive experiments conducted on four public datasets demonstrate the superiority of SLIE to thirteen state-of-the-art methods. Our code is available at
https://github.com/hanna-xu/SLIE
.
- Color correction,
- low-light image enhancement,
- self-supervised learning

FullText(HTML)

References(53)

References

[1]	M. Fan, W. Wang, W. Yang, and J. Liu, “Integrating semantic segmentation and retinex model for low-light image enhancement,” in Proc. ACM Int. Conf. Multimedia, 2020, pp. 2317–2325.
[2]	L. Tang, Y. Deng, Y. Ma, J. Huang, and J. Ma, “Superfusion: A versatile image registration and fusion network with semantic awareness,” EEE/CAA J. Autom. Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi: 10.1109/JAS.2022.106082
[3]	Y. Ma, X. Wang, W. Gao, Y. Du, J. Huang, and F. Fan, “Progressive fusion network based on infrared light field equipment for infrared image enhancement,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 9, pp. 1687–1690, 2022. doi: 10.1109/JAS.2022.105812
[4]	K. Wu, J. Huang, Y. Ma, F. Fan, and J. Ma, “Cycle-retinex: Unpaired low-light image enhancement via retinex-inline cyclegan,” IEEE Trans. Multimedia, vol. 26, pp. 1213–1228, 2023.
[5]	C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3291–3300.
[6]	J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Trans. Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018. doi: 10.1109/TIP.2018.2794218
[7]	C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” in Proc. British Machine Vision Conf. (BMVC), 2018, pp. 1–12.
[8]	Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in Proc. ACM Int. Conf. Multimedia, 2019, pp. 1632–1640.
[9]	Y. Zhang, X. Guo, J. Ma, W. Liu, and J. Zhang, “Beyond brightening low-light images,” Int. J. Computer Vision, vol. 129, no. 4, pp. 1013–1037, 2021. doi: 10.1007/s11263-020-01407-x
[10]	Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, “Enlightengan: Deep light enhancement without paired supervision,” IEEE Trans. Image Processing, vol. 30, pp. 2340–2349, 2021. doi: 10.1109/TIP.2021.3051462
[11]	Y. Qu, K. Chen, C. Liu, and Y. Ou, “Umle: Unsupervised multi-discriminator network for low light enhancement,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021, pp. 4318–4324.
[12]	C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1780–1789.
[13]	W. Wu, W. Wang, K. Jiang, X. Xu, and R. Hu, “Self-supervised learning on a lightweight low-light image enhancement model with curve refinement,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 1890–1894.
[14]	P. Garg, M. Mandal, and P. Narang, “Improving aerial instance segmentation in the dark with self-supervised low light enhancement,” arXiv preprint arXiv: 2102.05399, 2021.
[15]	J. Liang, Y. Xu, Y. Quan, B. Shi, and H. Ji, “Self-supervised low-light image enhancement using discrepant untrained network priors,” IEEE Trans. Circuits and Systems for Video Technology, vol. 32, no. 11, pp. 7332–7345, 2022. doi: 10.1109/TCSVT.2022.3181781
[16]	H.-D. Cheng and X. Shi, “A simple and effective histogram equalization approach to image enhancement,” Digital Signal Processing, vol. 14, no. 2, pp. 158–170, 2004. doi: 10.1016/j.dsp.2003.07.002
[17]	C. Lee, C. Lee, and C.-S. Kim, “Contrast enhancement based on layered difference representation of 2d histograms,” IEEE Trans. Image Processing, vol. 22, no. 12, pp. 5372–5384, 2013. doi: 10.1109/TIP.2013.2284059
[18]	K. Singh, R. Kapoor, and S. K. Sinha, “Enhancement of low exposure images via recursive histogram equalization algorithms,” Optik, vol. 126, no. 20, pp. 2619–2625, 2015. doi: 10.1016/j.ijleo.2015.06.060
[19]	D. J. Jobson, Z.-u. Rahman, and G. A. Woodell, “Properties and performance of a center/surround retinex,” IEEE Trans. Image Processing, vol. 6, no. 3, pp. 451–462, 1997. doi: 10.1109/83.557356
[20]	X. Fu, D. Zeng, Y. Huang, X.-P. Zhang, and X. Ding, “A weighted variational model for simultaneous reflectance and illumination estimation,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2782–2790.
[21]	B. Cai, X. Xu, K. Guo, K. Jia, B. Hu, and D. Tao, “A joint intrinsic-extrinsic prior model for retinex,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2017, pp. 4000–4009.
[22]	R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, and J. Jia, “Underexposed photo enhancement using deep illumination estimation,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6849–6857.
[23]	C. Li, J. Guo, F. Porikli, and Y. Pang, “Lightennet: A convolutional neural network for weakly illuminated image enhancement,” Pattern Recognition Letters, vol. 104, pp. 15–22, 2018. doi: 10.1016/j.patrec.2018.01.010
[24]	J. Wang, W. Tan, X. Niu, and B. Yan, “Rdgan: Retinex decomposition based adversarial learning for low-light enhancement,” in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), 2019, pp. 1186–1191.
[25]	W. Ren, S. Liu, L. Ma, Q. Xu, X. Xu, X. Cao, J. Du, and M.-H. Yang, “Low-light image enhancement via a deep hybrid network,” IEEE Trans. Image Processing, vol. 28, no. 9, pp. 4364–4375, 2019. doi: 10.1109/TIP.2019.2910412
[26]	Q. Zhao, X. Zhang, H. Tang, C. Gu, and S. Zhu, “Enlighten-anything: When segment anything model meets low-light image enhancement,” arXiv preprint arXiv: 2306.10286, 2023.
[27]	F. Cao and K. Li, “A new method for image super-resolution with multi-channel constraints,” Knowledge-Based Systems, vol. 146, pp. 118–128, 2018. doi: 10.1016/j.knosys.2018.01.034
[28]	K. Li and F. Cao, “Super-resolution using neighbourhood regression with local structure prior,” Signal Processing: Image Communication, vol. 72, pp. 58–68, 2019. doi: 10.1016/j.image.2018.12.006
[29]	E. H. Land, “The retinex theory of color vision,” Scientific American, vol. 237, no. 6, pp. 108–129, 1977. doi: 10.1038/scientificamerican1277-108
[30]	M. Afifi and M. S. Brown, “Deep white-balance editing,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1397–1406.
[31]	F. Kınlı, D. Yılmaz, B. Özcan, and F. Kıraç, “Modeling the lighting in scenes as style for auto white-balance correction,” in Proc. IEEE/CVF Winter Conf. Applications of Computer Vision (WACV), 2023, pp. 4903–4913.
[32]	M. Afifi, M. A. Brubaker, and M. S. Brown, “Auto white-balance correction for mixed-illuminant scenes,” in Proc. IEEE/CVF Winter Conf. Applications of Computer Vision (WACV), 2022, pp. 1210–1219.
[33]	F. Kınlı, D. Yılmaz, B. Özcan, and F. Kıraç, “Deterministic neural illumination mapping for efficient auto-white balance correction,” in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2023, pp. 1139–1147.
[34]	S. Zini, C. Rota, M. Buzzelli, S. Bianco, and R. Schettini, “Back to the future: a night photography rendering isp without deep learning,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1465–1473.
[35]	E. Ershov, A. Savchik, D. Shepelev, N. Banić, M. S. Brown, R. Timofte, K. Koščević, M. Freeman, V. Tesalin, D. Bocharov et al., “Ntire 2022 challenge on night photography rendering,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1287–1300.
[36]	A. Shutova, E. Ershov, G. Perevozchikov, I. Ermakov, N. Banić, R. Timofte, R. Collins, M. Efimova, A. Terekhin, S. Zini et al., “Ntire 2023 challenge on night photography rendering,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1981–1992.
[37]	Q. Yang, C. Jung, Q. Fu, and H. Song, “Low light image denoising based on poisson noise model and weighted tv regularization,” in Proc. IEEE Int. Conf. Image Processing (ICIP), 2018, pp. 3199–3203.
[38]	J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2noise: Learning image restoration without clean data,” in Proc. Int. Conf. Machine Learning (ICML), 2018, pp. 2965–2974.
[39]	O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015, pp. 234–241.
[40]	S. Bianco and C. Cusano, “Quasi-unsupervised color constancy,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 12 212–12 221.
[41]	X. Guo, Y. Li, and H. Ling, “Lime: Low-light image enhancement via illumination map estimation,” IEEE Trans. Image Processing, vol. 26, no. 2, pp. 982–993, 2016.
[42]	M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo, “Structure-revealing low-light image enhancement via robust retinex model,” IEEE Trans. Image Processing, vol. 27, no. 6, pp. 2828–2841, 2018. doi: 10.1109/TIP.2018.2810539
[43]	L. Ma, R. Liu, J. Zhang, X. Fan, and Z. Luo, “Learning deep context-sensitive decomposition for low-light image enhancement,” IEEE Trans. Neural Networks and Learning Systems, vol. 33, no. 10, pp. 5666–5680, 2022. doi: 10.1109/TNNLS.2021.3071245
[44]	R. Liu, L. Ma, J. Zhang, X. Fan, and Z. Luo, “Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2021, pp. 10 561–10 570.
[45]	X. Yi, H. Xu, H. Zhang, L. Tang, and J. Ma, “Diff-retinex: Rethinking low-light image enhancement with a generative diffusion model,” in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2023, pp. 12 302–12 311.
[46]	Z. Liang, C. Li, S. Zhou, R. Feng, and C. C. Loy, “Iterative prompt learning for unsupervised backlit image enhancement,” in Proc. IEEE/CVF Int. Conf. Computer Vision (CVPR), 2023, pp. 8094–8103.
[47]	S. Yang, M. Ding, Y. Wu, Z. Li, and J. Zhang, “Implicit neural representation for cooperative low-light image enhancement,” in Proc. IEEE/CVF Int. Conf. Computer Vision (CVPR), 2023, pp. 12 918–12 927.
[48]	F. Lv, Y. Li, and F. Lu, “Attention guided low-light image enhancement with a large scale low-light simulation dataset,” Int. J. Computer Vision, vol. 129, no. 7, pp. 2175–2193, 2021. doi: 10.1007/s11263-021-01466-8
[49]	K. Ma, K. Zeng, and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” IEEE Trans. Image Processing, vol. 24, no. 11, pp. 3345–3356, 2015. doi: 10.1109/TIP.2015.2442920
[50]	J. Hai, Z. Xuan, R. Yang, Y. Hao, F. Zou, F. Lin, and S. Han, “R2rnet: Low-light image enhancement via real-low to real-normal network,” J. Visual Communication and Image Representation, vol. 90, p. 103712, 2023. doi: 10.1016/j.jvcir.2022.103712
[51]	Y. Han, Y. Cai, Y. Cao, and X. Xu, “A new image fusion performance metric based on visual information fidelity,” Information Fusion, vol. 14, no. 2, pp. 127–135, 2013. doi: 10.1016/j.inffus.2011.08.002
[52]	S. Wang, J. Zheng, H.-M. Hu, and B. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” IEEE Trans. Image Processing, vol. 22, no. 9, pp. 3538–3548, 2013. doi: 10.1109/TIP.2013.2261309
[53]	A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2012.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(20) / Tables(5)

Get Citation

PDF

XML

Article Metrics

Article views (239) PDF downloads(44)

Highlights

Our method addresses multiple degradations, including illumination, noise, and color shift
A network is designed to adjust low illumination in a self-supervised manner
A color correction block is designed, breaking free from reliance on white balanced images
The self-supervised manner improves the generalization for various low-light conditions
It achieves a balance between parameters and performance

More Than Lightening: A Self-Supervised Low-Light Image Enhancement Method Capable for Multiple Degradations

doi: 10.1109/JAS.2024.124263

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content