MAUN: Memory-Augmented Deep Unfolding Network for Hyperspectral Image Reconstruction

Qian Hu; Jiayi Ma; Yuan Gao; Junjun Jiang; Yixuan Yuan

doi:10.1109/JAS.2024.124362

Volume 11 Issue 5

May 2024

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 11.8, Top 4% (SCI Q1)

CiteScore: 17.6, Top 3% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2024 > 11(5): 1139-1150

Q. Hu, J. Ma, Y. Gao, J. Jiang, and Y. Yuan, "MAUN: Memoryaugmented deep unfolding network for hyperspectral image reconstruction", IEEE/CAA J. Autom. Sinica, vol. 11, no. 5 pp. 1139–1150. May 2024. doi: 10.1109/JAS.2024.124362

Citation:

Q. Hu, J. Ma, Y. Gao, J. Jiang, and Y. Yuan, "MAUN: Memoryaugmented deep unfolding network for hyperspectral image reconstruction", IEEE/CAA J. Autom. Sinica, vol. 11, no. 5 pp. 1139–1150. May 2024. doi: 10.1109/JAS.2024.124362

Citation:

PDF( 10937 KB)

MAUN: Memory-Augmented Deep Unfolding Network for Hyperspectral Image Reconstruction

doi: 10.1109/JAS.2024.124362

Qian Hu^{1
,
,},
Jiayi Ma^{1
,
,
,},
Yuan Gao^{1
,
,},
Junjun Jiang^{2
,
,},
Yixuan Yuan^{3
,
,}

1.
Electronic Information School, Wuhan University, Wuhan 430072, China
2.
School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China
3.
Department of Electronic Engineering, Chinese University of Hong Kong, Hong Kong 999077, China

Funds:

the National Natural Science Foundation of China 62276192

More Information

Author Bio:
Qian Hu received the B.S. degree in Electronic Information Engineering from Wuhan University, Wuhan, China, in 2022. He is currently pursuing a Master's degree in Information and Communication Engineering from Electronic Information School, Wuhan University. His current research interests include computer vision and image processing. e-mail: huq1an@whu.edu.cn

Jiayi Ma (Senior Member, IEEE) received the B.S. degree in Information and Computing Science and the Ph.D. degree in Control Science and Engineering from the Huazhong University of Science and Technology, in 2008 and 2014, respectively. He is currently a Professor with the Electronic Information School, Wuhan University. He has authored or co-authored more than 300 refereed journal and conference papers, including IEEE TPAMI/TIP, IJCV, CVPR, ICCV, ECCV, etc. His research interests include computer vision, machine learning, and robotics. Dr. Ma has been identified in the 2019—2022 Highly Cited Researcher lists from the Web of Science Group. He is an Area Editor of Information Fusion, and an Associate Editor of IEEE/CAA Journal of Automatica Sinica. e-mail: jyma2010@gmail.com

Yuan Gao(Member, IEEE) received the B.S. degree in Biomedical Engineering, the M.S. degree in Pattern Recognition and Intelligent Systems from the Huazhong University of Science and Technology, and the Ph.D. degree in Electronic Engineering from City University of Hong Kong, in 2009, 2012, and 2016, respectively. He was a visiting graduate researcher with the Department of Statistics, University of California, Los Angeles in 2015. Currently, he is an Associate Professor with the Electronic Information School, Wuhan University. His research interests include computer vision, machine learning, and pattern recognition. e-mail: yuangaoeis@whu.edu.cn

Junjun Jiang(Senior Member, IEEE) received the B.S. degree in Information and Computing Science from Huaqiao University, Quanzhou, China, in 2009, and the Ph.D. degree in Computer Science and Technology from Wuhan University, Wuhan, China, in 2014. He is currently a Professor with the School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China. He received the 2016 China Computer Federation (CCF) Outstanding Doctoral Dissertation Award and 2015 ACM Wuhan Doctoral Dissertation Award. His research interests include image processing and computer vision. e-mail: jiangjunjun@hit.edu.cn

Yixuan Yuan(Senior Member, IEEE) received the B.S. degree in Automation from Northwestern Polytechnical University, in 2010, the Ph.D. degree in Electronic Engineering from the Chinese University of Hong Kong. From 2018–2022, she was an Assistant Professor in the Department of Electrical Engineering at the City University of Hong Kong. She is currently an Assistant Professor in the Department of Electronic Engineering at the Chinese University of Hong Kong. Her research interests include AI in healthcare to promote precision medicine, especially for medical image diagnosis, abnormality detection, and segmentation. She has authored or co-authored more than 100 refereed journal and conference papers. She received several premium awards including the CVPR 2022 Best Paper Finalist, MICCAI 2022 Young Scientist Award (supervisor), 2017 Young Scientist Award in Engineering by Hong Kong Institute of Science, Best Paper Award in ICMA 2013, etc. e-mail: yxyuan@ee.cuhk.edu.hk
Corresponding author: Jiayi Ma, e-mail: jyma2010@gmail.com
Recommended by Associate Editor Hui Yu.
Received Date: 2023-12-14
Revised Date: 2024-02-08
Accepted Date: 2024-02-29

Available Online: 2024-04-18

Abstract

Abstract

Spectral compressive imaging has emerged as a powerful technique to collect the 3D spectral information as 2D measurements. The algorithm for restoring the original 3D hyperspectral images (HSIs) from compressive measurements is pivotal in the imaging process. Early approaches painstakingly designed networks to directly map compressive measurements to HSIs, resulting in the lack of interpretability without exploiting the imaging priors. While some recent works have introduced the deep unfolding framework for explainable reconstruction, the performance of these methods is still limited by the weak information transmission between iterative stages. In this paper, we propose a Memory-Augmented deep Unfolding Network, termed MAUN, for explainable and accurate HSI reconstruction. Specifically, MAUN implements a novel CNN scheme to facilitate a better extrapolation step of the fast iterative shrinkage-thresholding algorithm, introducing an extra momentum incorporation step for each iteration to alleviate the information loss. Moreover, to exploit the high correlation of intermediate images from neighboring iterations, we customize a cross-stage transformer (CSFormer) as the deep denoiser to simultaneously capture self-similarity from both in-stage and cross-stage features, which is the first attempt to model the long-distance dependencies between iteration stages. Extensive experiments demonstrate that the proposed MAUN is superior to other state-of-the-art methods both visually and metrically. Our code is publicly available at
https://github.com/HuQ1an/MAUN
.
- Compressive imaging,
- deep unfolding network hyperspectral image

FullText(HTML)

Recommended by Associate Editor Hui Yu.

References(47)

References

[1]	R. O. Green, M. L. Eastwood, C. M. Sarture, T. G. Chrien, M. Aronsson, B. J. Chippendale, J. A. Faust, B. E. Pavri, C. J. Chovit, M. Solis et al. , "Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (aviris), " Remote Sensing of Environment, vol. 65, no. 3, pp. 227–248, 1998. doi: 10.1016/S0034-4257(98)00064-9
[2]	G. Channing, "Spectral defocuscam: Compressive hyperspectral imaging from defocus measurements, " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 11, 2022, pp. 13 128–13 129.
[3]	X. Wang, Q. Hu, Y. Cheng, and J. Ma, "Hyperspectral image superresolution meets deep learning: A survey and perspective, " IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 8, pp. 1664–1687, 2023.
[4]	F. F. Sabins, "Remote sensing for mineral exploration, " Ore Geology Reviews, vol. 14, no. 3-4, pp. 157–183, 1999. doi: 10.1016/S0169-1368(99)00007-4
[5]	H. Akbari, Y. Kosugi, K. Kojima, and N. Tanaka, "Detection and analysis of the intestinal ischemia using visible and invisible hyperspectral imaging, " IEEE Transactions on Biomedical Engineering, vol. 57, no. 8, pp. 2011–2017, 2010. doi: 10.1109/TBME.2010.2049110
[6]	A. Lowe, N. Harrison, and A. P. French, "Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress, " Plant Methods, vol. 13, no. 1, p. 80, 2017. doi: 10.1186/s13007-017-0233-z
[7]	M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, "High-fidelity video and stillimage communication based on spectral information: Natural vision system and its applications, " in Spectral Imaging: Eighth International Symposium on Multispectral Color Science, vol. 6062, 2006, pp. 129– 140.
[8]	M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, "Single-shot compressive spectral imaging with a dual-disperser architecture, " Optics Express, vol. 15, no. 21, pp. 14 013–14 027, 2007. doi: 10.1364/OE.15.014013
[9]	A. Wagadarikar, R. John, R. Willett, and D. Brady, "Single disperser design for coded aperture snapshot spectral imaging, " Applied Optics, vol. 47, no. 10, pp. B44–B51, 2008. doi: 10.1364/AO.47.000B44
[10]	A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager, " Optics Express, vol. 17, no. 8, pp. 6368–6388, 2009. doi: 10.1364/OE.17.006368
[11]	M. A. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems, " IEEE Journal of Selected Topics in Signal Processing, vol. 1, no. 4, pp. 586–597, 2007. doi: 10.1109/JSTSP.2007.910281
[12]	J. M. Bioucas-Dias and M. A. Figueiredo, "A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration, " IEEE Transactions on Image Processing, vol. 16, no. 12, pp. 2992–3004, 2007. doi: 10.1109/TIP.2007.909319
[13]	L. Wang, Z. Xiong, D. Gao, G. Shi, and F. Wu, "Dual-camera design for coded aperture snapshot spectral imaging, " Applied Optics, vol. 54, no. 4, pp. 848–858, 2015. doi: 10.1364/AO.54.000848
[14]	X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing, " in Proceedings of the IEEE International Conference on Image Processing, 2016, pp. 2539–2543.
[15]	L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, "Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 10, pp. 2104–2111, 2017. doi: 10.1109/TPAMI.2016.2621050
[16]	Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, "Rank minimization for snapshot compressive imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 12, pp. 2990–3006, 2019. doi: 10.1109/TPAMI.2018.2873587
[17]	X. Miao, X. Yuan, Y. Pu, and V. Athitsos, "l-net: Reconstruct hyperspectral images from a snapshot measurement, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4059–4069.
[18]	Z. Meng, J. Ma, and X. Yuan, "End-to-end low cost compressive spectral imaging with spatial-spectral self-attention, " in Proceedings of the European Conference on Computer Vision, 2020, pp. 187–204.
[19]	Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool, "Coarse-to-fine sparse transformer for hyperspectral image reconstruction, " in Proceedings of the European Conference on Computer Vision, 2022, pp. 686–704.
[20]	X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Hdnet: High-resolution dual-domain learning for spectral compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 542–17 551.
[21]	Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 502–17 511.
[22]	Z. Cheng, B. Chen, R. Lu, Z. Wang, H. Zhang, Z. Meng, and X. Yuan, "Recurrent neural networks for snapshot compressive imaging, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2264–2281, 2023. doi: 10.1109/TPAMI.2022.3161934
[23]	J. Ma, X. -Y. Liu, Z. Shou, and X. Yuan, "Deep tensor admm-net for snapshot compressive imaging, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 10 223–10 232.
[24]	L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, "Hyperspectral image reconstruction using a deep spatial-spectral prior, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8032–8041.
[25]	L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, "Dnu: Deep nonlocal unrolling for computational spectral imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1661–1671.
[26]	Z. Meng, S. Jalali, and X. Yuan, "Gap-net for snapshot compressive imaging, " arXiv preprint arXiv: 2012.08364, 2020.
[27]	T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, "Deep gaussian scale mixture prior for spectral compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 216–16 225.
[28]	Y. Cai, J. Lin, H. Wang, X. Yuan, H. Ding, Y. Zhang, R. Timofte, and L. V. Gool, "Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging, " Advances in Neural Information Processing Systems, vol. 35, pp. 37 749–37 761, 2022.
[29]	Z. Meng, X. Yuan, and S. Jalali, "Deep unfolding for snapshot compressive imaging, " International Journal of Computer Vision, vol. 131, pp. 2933–2958, 2023. doi: 10.1007/s11263-023-01844-4
[30]	M. Li, Y. Fu, J. Liu, and Y. Zhang, "Pixel adaptive deep unfolding transformer for hyperspectral image reconstruction, " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 959–12 968.
[31]	P. Xu, L. Liu, H. Zheng, X. Yuan, C. Xu, and L. Xue, "Degradationaware dynamic fourier-based network for spectral compressive imaging, " IEEE Transactions on Multimedia, vol. 26, pp. 2838–2850, 2024. doi: 10.1109/TMM.2023.3304450
[32]	Y. Dong, D. Gao, T. Qiu, Y. Li, M. Yang, and G. Shi, "Residual degradation learning unfolding framework with mixing priors across spectral and spatial for compressive spectral imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 262–22 271.
[33]	X. Zhang, Y. Zhang, R. Xiong, Q. Sun, and J. Zhang, "Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging, " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 532–17 541.
[34]	A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems, " SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 183–202, 2009. doi: 10.1137/080716542
[35]	M. Fornasier and H. Rauhut, "Compressive sensing. " Handbook of Mathematical Methods in Imaging, vol. 1, pp. 187–229, 2015.
[36]	B. Shi and K. Liu, "Regularization by multiple dual frames for compressed sensing magnetic resonance imaging with convergence analysis, " IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 11, pp. 2136–2153, 2023. doi: 10.1109/JAS.2023.123543
[37]	T. Zhang, W. Cui, C. Hui, and F. Jiang, "Hierarchical interactive reconstruction network for video compressive sensing, " in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, pp. 1–5.
[38]	L. Zhuang, L. Shen, Z. Wang, and Y. Li, "Ucsnet: Priors guided adaptive compressive sensing framework for underwater images, " IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 10, pp. 5587–5604, 2023. doi: 10.1109/TCSVT.2023.3261542
[39]	D. Geman and C. Yang, "Nonlinear image recovery with half-quadratic regularization, " IEEE Transactions on Image Processing, vol. 4, no. 7, pp. 932–946, 1995. doi: 10.1109/83.392335
[40]	J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Nonlocal sparse models for image restoration, " in Proceedings of the IEEE International Conference on Computer Vision, 2009, pp. 2272–2279.
[41]	D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, "Non-local recurrent network for image restoration, " Advances in Neural Information Processing Systems, vol. 31, 2018.
[42]	Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, "Residual non-local attention networks for image restoration, " arXiv preprint arXiv: 1903.10082, 2019.
[43]	J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks, " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.
[44]	M. Shen, H. Gan, C. Ning, Y. Hua, and T. Zhang, "Transcs: A transformer-based hybrid architecture for image compressed sensing, " IEEE Transactions on Image Processing, vol. 31, pp. 6991–7005, 2022. doi: 10.1109/TIP.2022.3217365
[45]	Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows, " in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
[46]	J. -I. Park, M. -H. Lee, M. D. Grossberg, and S. K. Nayar, "Multispectral imaging using multiplexed illumination, " in Proceedings of the IEEE International Conference on Computer Vision, 2007, pp. 1–8.
[47]	I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, "High-quality hyperspectral reconstruction using a spectral prior, " ACM Transactions on Graphics, vol. 36, no. 6, pp. 1–13, 2017.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(10) / Tables(6)

Get Citation

PDF

XML

Article Metrics

Article views (72) PDF downloads(17)

Highlights

We develop MAUN for interpretable and accurate hyperspectral image reconstruction
Our method enhances the information transmission of deep unfolding network
A momentum incorporation step is proposed to propagate historical information
A cross-stage transformer is designed to capture in- and cross-stage self-similarity

MAUN: Memory-Augmented Deep Unfolding Network for Hyperspectral Image Reconstruction

doi: 10.1109/JAS.2024.124362

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content