A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 11 Issue 3
Mar.  2024

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2024  >  11(3): 708-722
H. Liu, Q. Zhang, Y. Hu, H. Zeng, and  B. Fan,  “Unsupervised multi-expert learning model for underwater image enhancement,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 708–722, Mar. 2024. doi: 10.1109/JAS.2023.123771
Citation: H. Liu, Q. Zhang, Y. Hu, H. Zeng, and  B. Fan,  “Unsupervised multi-expert learning model for underwater image enhancement,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 708–722, Mar. 2024. doi: 10.1109/JAS.2023.123771
shu

Unsupervised Multi-Expert Learning Model for Underwater Image Enhancement

doi: 10.1109/JAS.2023.123771
Funds:  This work was supported in part by the National Key Research and Development Program of China (2020YFB1313002), the National Natural Science Foundation of China (62276023, U22B2055, 62222302, U2013202), the Fundamental Research Funds for the Central Universities (FRF-TP-22-003C1), and the Postgraduate Education Reform Project of Henan Province (2021SJGLX260Y)
More Information
    Abstract

  • Underwater image enhancement aims to restore a clean appearance and thus improves the quality of underwater degraded images. Current methods feed the whole image directly into the model for enhancement. However, they ignored that the R, G and B channels of underwater degraded images present varied degrees of degradation, due to the selective absorption for the light. To address this issue, we propose an unsupervised multi-expert learning model by considering the enhancement of each color channel. Specifically, an unsupervised architecture based on generative adversarial network is employed to alleviate the need for paired underwater images. Based on this, we design a generator, including a multi-expert encoder, a feature fusion module and a feature fusion-guided decoder, to generate the clear underwater image. Accordingly, a multi-expert discriminator is proposed to verify the authenticity of the R, G and B channels, respectively. In addition, content perceptual loss and edge loss are introduced into the loss function to further improve the content and details of the enhanced images. Extensive experiments on public datasets demonstrate that our method achieves more pleasing results in vision quality. Various metrics (PSNR, SSIM, UIQM and UCIQE) evaluated on our enhanced images have been improved obviously.

     

    • FullText(HTML)
  • loading
    • References(59)
  • [1]
    Z. Zhou, J. Liu, and J. Yu, “A survey of underwater multi-robot systems,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 1–18, 2022. doi: 10.1109/JAS.2021.1004269
    [2]
    H. Wei and Y. Shi, “MPC-based motion planning and control enables smarter and safer autonomous marine vehicles: Perspectives and a tutorial survey,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 1, pp. 8–24, 2023. doi: 10.1109/JAS.2022.106016
    [3]
    M. Johnson-Roberson, M. Bryson, A. Friedman, O. Pizarro, G. Troni, P. Ozog, and J. C. Henderson, “High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology,” J. Field Robot., vol. 34, no. 4, pp. 625–643, 2017. doi: 10.1002/rob.21658
    [4]
    D. Akkaynak, T. Treibitz, T. Shlesinger, Y. Loya, R. Tamir, and D. Iluz, “What is the space of attenuation coefficients in underwater computer vision?” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 568–577.
    [5]
    T. P. Marques and A. B. Albu, “L2UWE: A framework for the efficient enhancement of low-light underwater images using local contrast and multi-scale fusion,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshops, 2020, pp. 538–539.
    [6]
    M. Moniruzzaman, S. M. S. Islam, M. Bennamoun, and P. Lavery, “Deep learning on underwater marine object detection: A survey,” in Lect. Notes Comput. Sci. Springer, 2017, pp. 150–160.
    [7]
    M. J. Islam, C. Edge, Y. Xiao, P. Luo, M. Mehtaz, C. Morse, S. S. Enan, and J. Sattar, “Semantic segmentation of underwater imagery: Dataset and benchmark,” in Proc. IEEE Int. Conf. Intell. Rob. Syst., 2020, pp. 1769–1776.
    [8]
    P. Drews, E. Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2013, pp. 825–830.
    [9]
    A. Galdran, D. Pardo, A. Picón, and A. Alvarez-Gila, “Automatic red-channel underwater image restoration,” J. Vis. Commun. Image Represent., vol. 26, pp. 132–145, 2015. doi: 10.1016/j.jvcir.2014.11.006
    [10]
    Y.-T. Peng and P. C. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE Trans. Image Process., vol. 26, no. 4, pp. 1579–1594, 2017. doi: 10.1109/TIP.2017.2663846
    [11]
    D. Akkaynak and T. Treibitz, “Sea-thru: A method for removing water from underwater images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 1682–1691.
    [12]
    N. Carlevaris-Bianco, A. Mohan, and R. M. Eustice, “Initial results in underwater single image dehazing,” in MTS/IEEE Seattle, Oceans., 2010, pp. 1–8.
    [13]
    K. Iqbal, R. A. Salam, A. Osman, and A. Z. Talib, “Underwater image enhancement using an integrated colour model,” IAENG Int. J. Comput. Sci., vol. 34, no. 2, 2007.
    [14]
    A. Ghani and N. Isa, “Underwater image quality enhancement through composition of dual-intensity images and rayleigh-stretching,” SpringerPlus, vol. 3, no. 1, pp. 1–14, 2014. doi: 10.1186/2193-1801-3-1
    [15]
    C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 81–88.
    [16]
    X. Fu, P. Zhuang, Y. Huang, Y. Liao, X.-P. Zhang, and X. Ding, “A retinex-based enhancing approach for single underwater image,” in Proc. IEEE Int. Conf. Image Process., 2014, pp. 4572–4576.
    [17]
    K. Liu, Z. Ye, H. Guo, D. Cao, L. Chen, and F. Wang, “FISS GAN: A generative adversarial network for foggy image semantic segmentation,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 8, pp. 1428–1439, 2021. doi: 10.1109/JAS.2021.1004057
    [18]
    X. Wang, J. Ma, and J. Jiang, “Contrastive learning for blind super-resolution via a distortion-specific network,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 1, pp. 78–89, 2023. doi: 10.1109/JAS.2022.105914
    [19]
    X. Tian, K. Xu, X. Yang, L. Du, B. Yin, and R. W. Lau, “Bi-directional object-context prioritization learning for saliency ranking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 5882–5891.
    [20]
    H. Liu, F. Jin, H. Zeng, H. Pu, and B. Fan, “Image enhancement guided object detection in visually degraded scenes,” IEEE Trans. Neural Netw. Learn. Syst., pp. 1–14, 2023.
    [21]
    C. Li, C. Guo, W. Ren, R. Cong, J. Hou, S. Kwong, and D. Tao, “An underwater image enhancement benchmark dataset and beyond,” IEEE Trans. Image Process., vol. 29, pp. 4376–4389, 2020. doi: 10.1109/TIP.2019.2955241
    [22]
    C. Li, S. Anwar, and F. Porikli, “Underwater scene prior inspired deep underwater image and video enhancement,” Pattern Recognit., vol. 98, p. 107038, 2020. doi: 10.1016/j.patcog.2019.107038
    [23]
    A. Naik, A. Swarnakar, and K. Mittal, “Shallow-UWNet: Compressed model for underwater image enhancement,” in Proc. AAAI Conf. Artif. Intell., vol. 35, no. 18, 2021, pp. 15853–15854.
    [24]
    C. Fabbri, M. J. Islam, and J. Sattar, “Enhancing underwater imagery using generative adversarial networks,” in Proc. IEEE Int. Conf. Rob. Autom., 2018, pp. 7159–7165.
    [25]
    X. Xue, Z. Li, L. Ma, Q. Jia, R. Liu, and X. Fan, “Investigating intrinsic degradation factors by multi-branch aggregation for real-world underwater image enhancement,” Pattern Recognit., vol. 133, p. 109041, 2023. doi: 10.1016/j.patcog.2022.109041
    [26]
    J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2242–2251.
    [27]
    C. Li, J. Guo, and C. Guo, “Emerging from water: Underwater image color correction based on weakly supervised color transfer,” IEEE Signal Process. Lett., vol. 25, no. 3, pp. 323–327, 2018. doi: 10.1109/LSP.2018.2792050
    [28]
    M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement for improved visual perception,” IEEE Robot. Autom. Lett., vol. 5, no. 2, pp. 3227–3234, 2020. doi: 10.1109/LRA.2020.2974710
    [29]
    L. Hong, X. Wang, Z. Xiao, G. Zhang, and J. Liu, “WSUIE: Weakly supervised underwater image enhancement for improved visual perception,” IEEE Robot. Autom. Lett., vol. 6, no. 4, pp. 8237–8244, 2021. doi: 10.1109/LRA.2021.3105144
    [30]
    J. S. Jaffe, “Underwater optical imaging: The past, the present, and the prospects,” IEEE J. Ocean. Eng., vol. 40, no. 3, pp. 683–700, 2015. doi: 10.1109/JOE.2014.2350751
    [31]
    R. Liu, X. Fan, M. Zhu, M. Hou, and Z. Luo, “Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 12, pp. 4861–4875, 2020. doi: 10.1109/TCSVT.2019.2963772
    [32]
    H. Li and P. Zhuang, “DewaterNet: A fusion adversarial real underwater image enhancement network,” Signal Process.-Image Commun., vol. 95, p. 116248, 2021. doi: 10.1016/j.image.2021.116248
    [33]
    J. Jaffe, “Computer modeling and the design of optimal underwater imaging systems,” IEEE J. Ocean. Eng., vol. 15, no. 2, pp. 101–111, 1990. doi: 10.1109/48.50695
    [34]
    J. Han, M. Shoeiby, T. Malthus, E. Botha, J. Anstee, S. Anwar, R. Wei, M. A. Armin, H. Li, and L. Petersson, “Underwater image restoration via contrastive learning and a real-world dataset,” Remote Sens., vol. 14, no. 17, 2022.
    [35]
    W. Wang and X. Yuan, “Recent advances in image dehazing,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 3, pp. 410–436, 2017. doi: 10.1109/JAS.2017.7510532
    [36]
    H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, and L. Ma, “Contrastive learning for compact single image dehazing,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2021, pp. 10551–10560.
    [37]
    O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “DeblurGAN: Blind motion deblurring using conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 8183–8192.
    [38]
    P. Liang, J. Jiang, X. Liu, and J. Ma, “BaMBNet: A blur-aware multi-branch network for dual-pixel defocus deblurring,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 5, pp. 878–892, 2022. doi: 10.1109/JAS.2022.105563
    [39]
    W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 1874–1883.
    [40]
    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Adv. Neural Inf. Process. Syst., 2014.
    [41]
    P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 1125–1134.
    [42]
    T. Wang, M. Liu, J. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 8798–8807.
    [43]
    W. Tang, T. Li, F. Nian, and M. Wang, “MSCGAN: Multi-scale conditional generative adversarial networks for person image generation,” in Proc. IEEE Chin. Control Decis. Conf., 2020, pp. 1440–1445.
    [44]
    R. Chen, W. Huang, B. Huang, F. Sun, and B. Fang, “Reusing discriminators for encoding: Towards unsupervised image-to-image translation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2020, pp. 8168–8177.
    [45]
    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. Int. Conf. Learn. Represent., 2015, pp. 1–14.
    [46]
    Y. Lin, J. Zhou, W. Ren, and W. Zhang, “Autonomous underwater robot for underwater image enhancement via multi-scale deformable convolution network with attention mechanism,” Comput. Electron. Agric., vol. 191, p. 106497, 2021. doi: 10.1016/j.compag.2021.106497
    [47]
    X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, and Z. Fan, “Unpaired deep image deraining using dual contrastive learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 2007–2016.
    [48]
    K. de Langis, M. Fulton, and J. Sattar, “An analysis of deep object detectors for diver detection,” arXiv preprint arXiv: 2012.05701, 2020.
    [49]
    M. J. Islam, R. Wang, and J. Sattar, “SVAM: Saliency-guided visual attention modeling by autonomous underwater robots,” in Robot. Sci. Syst., 2022.
    [50]
    I. Avcibas, B. Sankur, and K. Sayood, “Statistical evaluation of image quality measures,” J. Electron. Imaging, vol. 11, no. 2, pp. 206–223, 2002. doi: 10.1117/1.1455011
    [51]
    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004. doi: 10.1109/TIP.2003.819861
    [52]
    M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 6062–6071, 2015. doi: 10.1109/TIP.2015.2491020
    [53]
    K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” IEEE J. Ocean. Eng., vol. 41, no. 3, pp. 541–551, 2016. doi: 10.1109/JOE.2015.2469915
    [54]
    J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv: 1804.02767, 2018.
    [55]
    Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2017, pp. 1302–1310.
    [56]
    X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “BASNet: Boundary-aware salient object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 7471–7481.
    [57]
    J. Ye, S. Liu, and X. Wang, “Partial network cloning,” arXiv preprint arXiv: 2303.10597, 2023.
    [58]
    X. Yang, D. Zhou, S. Liu, J. Ye, and X. Wang, “Deep model reassembly,” Adv. Neural Inf. Process. Syst., vol. 35, pp. 25739–25753, 2022.
    [59]
    X. Yang, J. Ye, and X. Wang, “Factorizing knowledge in neural networks,” in Proc. Eur. Conf. Comput. Vis., 2022, pp. 73–91.
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(14)  / Tables(5)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (307) PDF downloads(57) Cited by()

    Highlights

    • An unsupervised framework is used to obviate the need for paired underwater images
    • Design a multi-expert model by considering channel differences of underwater images
    • Content perceptual loss and edge loss are introduced to preserve image details
    • The enhanced images improve the performance of underwater visual tasks significantly

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return