A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 11 Issue 5
May  2024

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
Q. Hu, J. Ma, Y. Gao, J. Jiang, and Y. Yuan, "MAUN: Memoryaugmented deep unfolding network for hyperspectral image reconstruction", IEEE/CAA J. Autom. Sinica, vol. 11, no. 5 pp. 1139–1150. May 2024. doi: 10.1109/JAS.2024.124362
Citation: Q. Hu, J. Ma, Y. Gao, J. Jiang, and Y. Yuan, "MAUN: Memoryaugmented deep unfolding network for hyperspectral image reconstruction", IEEE/CAA J. Autom. Sinica, vol. 11, no. 5 pp. 1139–1150. May 2024. doi: 10.1109/JAS.2024.124362

MAUN: Memory-Augmented Deep Unfolding Network for Hyperspectral Image Reconstruction

doi: 10.1109/JAS.2024.124362
Funds:

the National Natural Science Foundation of China 62276192

More Information
  • Spectral compressive imaging has emerged as a powerful technique to collect the 3D spectral information as 2D measurements. The algorithm for restoring the original 3D hyperspectral images (HSIs) from compressive measurements is pivotal in the imaging process. Early approaches painstakingly designed networks to directly map compressive measurements to HSIs, resulting in the lack of interpretability without exploiting the imaging priors. While some recent works have introduced the deep unfolding framework for explainable reconstruction, the performance of these methods is still limited by the weak information transmission between iterative stages. In this paper, we propose a Memory-Augmented deep Unfolding Network, termed MAUN, for explainable and accurate HSI reconstruction. Specifically, MAUN implements a novel CNN scheme to facilitate a better extrapolation step of the fast iterative shrinkage-thresholding algorithm, introducing an extra momentum incorporation step for each iteration to alleviate the information loss. Moreover, to exploit the high correlation of intermediate images from neighboring iterations, we customize a cross-stage transformer (CSFormer) as the deep denoiser to simultaneously capture self-similarity from both in-stage and cross-stage features, which is the first attempt to model the long-distance dependencies between iteration stages. Extensive experiments demonstrate that the proposed MAUN is superior to other state-of-the-art methods both visually and metrically. Our code is publicly available at

    https://github.com/HuQ1an/MAUN

    .

     

  • loading
  • Recommended by Associate Editor Hui Yu.
  • [1]
    R. O. Green, M. L. Eastwood, C. M. Sarture, T. G. Chrien, M. Aronsson, B. J. Chippendale, J. A. Faust, B. E. Pavri, C. J. Chovit, M. Solis et al. , "Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (aviris), " Remote Sensing of Environment, vol. 65, no. 3, pp. 227–248, 1998. doi: 10.1016/S0034-4257(98)00064-9
    [2]
    G. Channing, "Spectral defocuscam: Compressive hyperspectral imaging from defocus measurements, " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 11, 2022, pp. 13 128–13 129.
    [3]
    X. Wang, Q. Hu, Y. Cheng, and J. Ma, "Hyperspectral image superresolution meets deep learning: A survey and perspective, " IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 8, pp. 1664–1687, 2023.
    [4]
    F. F. Sabins, "Remote sensing for mineral exploration, " Ore Geology Reviews, vol. 14, no. 3-4, pp. 157–183, 1999. doi: 10.1016/S0169-1368(99)00007-4
    [5]
    H. Akbari, Y. Kosugi, K. Kojima, and N. Tanaka, "Detection and analysis of the intestinal ischemia using visible and invisible hyperspectral imaging, " IEEE Transactions on Biomedical Engineering, vol. 57, no. 8, pp. 2011–2017, 2010. doi: 10.1109/TBME.2010.2049110
    [6]
    A. Lowe, N. Harrison, and A. P. French, "Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress, " Plant Methods, vol. 13, no. 1, p. 80, 2017. doi: 10.1186/s13007-017-0233-z
    [7]
    M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, "High-fidelity video and stillimage communication based on spectral information: Natural vision system and its applications, " in Spectral Imaging: Eighth International Symposium on Multispectral Color Science, vol. 6062, 2006, pp. 129– 140.
    [8]
    M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, "Single-shot compressive spectral imaging with a dual-disperser architecture, " Optics Express, vol. 15, no. 21, pp. 14 013–14 027, 2007. doi: 10.1364/OE.15.014013
    [9]
    A. Wagadarikar, R. John, R. Willett, and D. Brady, "Single disperser design for coded aperture snapshot spectral imaging, " Applied Optics, vol. 47, no. 10, pp. B44–B51, 2008. doi: 10.1364/AO.47.000B44
    [10]
    A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager, " Optics Express, vol. 17, no. 8, pp. 6368–6388, 2009. doi: 10.1364/OE.17.006368
    [11]
    M. A. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems, " IEEE Journal of Selected Topics in Signal Processing, vol. 1, no. 4, pp. 586–597, 2007. doi: 10.1109/JSTSP.2007.910281
    [12]
    J. M. Bioucas-Dias and M. A. Figueiredo, "A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration, " IEEE Transactions on Image Processing, vol. 16, no. 12, pp. 2992–3004, 2007. doi: 10.1109/TIP.2007.909319
    [13]
    L. Wang, Z. Xiong, D. Gao, G. Shi, and F. Wu, "Dual-camera design for coded aperture snapshot spectral imaging, " Applied Optics, vol. 54, no. 4, pp. 848–858, 2015. doi: 10.1364/AO.54.000848
    [14]
    X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing, " in Proceedings of the IEEE International Conference on Image Processing, 2016, pp. 2539–2543.
    [15]
    L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, "Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 10, pp. 2104–2111, 2017. doi: 10.1109/TPAMI.2016.2621050
    [16]
    Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, "Rank minimization for snapshot compressive imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 12, pp. 2990–3006, 2019. doi: 10.1109/TPAMI.2018.2873587
    [17]
    X. Miao, X. Yuan, Y. Pu, and V. Athitsos, "l-net: Reconstruct hyperspectral images from a snapshot measurement, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4059–4069.
    [18]
    Z. Meng, J. Ma, and X. Yuan, "End-to-end low cost compressive spectral imaging with spatial-spectral self-attention, " in Proceedings of the European Conference on Computer Vision, 2020, pp. 187–204.
    [19]
    Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool, "Coarse-to-fine sparse transformer for hyperspectral image reconstruction, " in Proceedings of the European Conference on Computer Vision, 2022, pp. 686–704.
    [20]
    X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Hdnet: High-resolution dual-domain learning for spectral compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 542–17 551.
    [21]
    Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 502–17 511.
    [22]
    Z. Cheng, B. Chen, R. Lu, Z. Wang, H. Zhang, Z. Meng, and X. Yuan, "Recurrent neural networks for snapshot compressive imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2264–2281, 2023. doi: 10.1109/TPAMI.2022.3161934
    [23]
    J. Ma, X. -Y. Liu, Z. Shou, and X. Yuan, "Deep tensor admm-net for snapshot compressive imaging, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 10 223–10 232.
    [24]
    L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, "Hyperspectral image reconstruction using a deep spatial-spectral prior, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8032–8041.
    [25]
    L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, "Dnu: Deep nonlocal unrolling for computational spectral imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1661–1671.
    [26]
    Z. Meng, S. Jalali, and X. Yuan, "Gap-net for snapshot compressive imaging, " arXiv preprint arXiv: 2012.08364, 2020.
    [27]
    T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, "Deep gaussian scale mixture prior for spectral compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 216–16 225.
    [28]
    Y. Cai, J. Lin, H. Wang, X. Yuan, H. Ding, Y. Zhang, R. Timofte, and L. V. Gool, "Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging, " Advances in Neural Information Processing Systems, vol. 35, pp. 37 749–37 761, 2022.
    [29]
    Z. Meng, X. Yuan, and S. Jalali, "Deep unfolding for snapshot compressive imaging, " International Journal of Computer Vision, vol. 131, pp. 2933–2958, 2023. doi: 10.1007/s11263-023-01844-4
    [30]
    M. Li, Y. Fu, J. Liu, and Y. Zhang, "Pixel adaptive deep unfolding transformer for hyperspectral image reconstruction, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 959–12 968.
    [31]
    P. Xu, L. Liu, H. Zheng, X. Yuan, C. Xu, and L. Xue, "Degradationaware dynamic fourier-based network for spectral compressive imaging, " IEEE Transactions on Multimedia, vol. 26, pp. 2838–2850, 2024. doi: 10.1109/TMM.2023.3304450
    [32]
    Y. Dong, D. Gao, T. Qiu, Y. Li, M. Yang, and G. Shi, "Residual degradation learning unfolding framework with mixing priors across spectral and spatial for compressive spectral imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 262–22 271.
    [33]
    X. Zhang, Y. Zhang, R. Xiong, Q. Sun, and J. Zhang, "Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 532–17 541.
    [34]
    A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems, " SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 183–202, 2009. doi: 10.1137/080716542
    [35]
    M. Fornasier and H. Rauhut, "Compressive sensing. " Handbook of Mathematical Methods in Imaging, vol. 1, pp. 187–229, 2015.
    [36]
    B. Shi and K. Liu, "Regularization by multiple dual frames for compressed sensing magnetic resonance imaging with convergence analysis, " IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 11, pp. 2136–2153, 2023. doi: 10.1109/JAS.2023.123543
    [37]
    T. Zhang, W. Cui, C. Hui, and F. Jiang, "Hierarchical interactive reconstruction network for video compressive sensing, " in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, pp. 1–5.
    [38]
    L. Zhuang, L. Shen, Z. Wang, and Y. Li, "Ucsnet: Priors guided adaptive compressive sensing framework for underwater images, " IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 10, pp. 5587–5604, 2023. doi: 10.1109/TCSVT.2023.3261542
    [39]
    D. Geman and C. Yang, "Nonlinear image recovery with half-quadratic regularization, " IEEE Transactions on Image Processing, vol. 4, no. 7, pp. 932–946, 1995. doi: 10.1109/83.392335
    [40]
    J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Nonlocal sparse models for image restoration, " in Proceedings of the IEEE International Conference on Computer Vision, 2009, pp. 2272–2279.
    [41]
    D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, "Non-local recurrent network for image restoration, " Advances in Neural Information Processing Systems, vol. 31, 2018.
    [42]
    Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, "Residual non-local attention networks for image restoration, " arXiv preprint arXiv: 1903.10082, 2019.
    [43]
    J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks, " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.
    [44]
    M. Shen, H. Gan, C. Ning, Y. Hua, and T. Zhang, "Transcs: A transformer-based hybrid architecture for image compressed sensing, " IEEE Transactions on Image Processing, vol. 31, pp. 6991–7005, 2022. doi: 10.1109/TIP.2022.3217365
    [45]
    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows, " in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
    [46]
    J. -I. Park, M. -H. Lee, M. D. Grossberg, and S. K. Nayar, "Multispectral imaging using multiplexed illumination, " in Proceedings of the IEEE International Conference on Computer Vision, 2007, pp. 1–8.
    [47]
    I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, "High-quality hyperspectral reconstruction using a spectral prior, " ACM Transactions on Graphics, vol. 36, no. 6, pp. 1–13, 2017.

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)  / Tables(6)

    Article Metrics

    Article views (137) PDF downloads(29) Cited by()

    Highlights

    • We develop MAUN for interpretable and accurate hyperspectral image reconstruction
    • Our method enhances the information transmission of deep unfolding network
    • A momentum incorporation step is proposed to propagate historical information
    • A cross-stage transformer is designed to capture in- and cross-stage self-similarity

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return