A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 11 Issue 2
Feb.  2024

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 15.3, Top 1 (SCI Q1)
    CiteScore: 23.5, Top 2% (Q1)
    Google Scholar h5-index: 77， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2024  >  11(2): 391-404
Y. Jia, Q. Hu, R. Dian, J. Ma, and  X. Guo,  “PAPS: Progressive attention-based pan-sharpening,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 2, pp. 391–404, Feb. 2024. doi: 10.1109/JAS.2023.123987
Citation: Y. Jia, Q. Hu, R. Dian, J. Ma, and  X. Guo,  “PAPS: Progressive attention-based pan-sharpening,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 2, pp. 391–404, Feb. 2024. doi: 10.1109/JAS.2023.123987
shu

PAPS: Progressive Attention-Based Pan-sharpening

doi: 10.1109/JAS.2023.123987
Funds:  This work was partially supported by the National Natural Science Foundation of China (62372251)
More Information
    Abstract

  • Pan-sharpening aims to seek high-resolution multispectral (HRMS) images from paired multispectral images of low resolution (LRMS) and panchromatic (PAN) images, the key to which is how to maximally integrate spatial and spectral information from PAN and LRMS images. Following the principle of gradual advance, this paper designs a novel network that contains two main logical functions, i.e., detail enhancement and progressive fusion, to solve the problem. More specifically, the detail enhancement module attempts to produce enhanced MS results with the same spatial sizes as corresponding PAN images, which are of higher quality than directly up-sampling LRMS images. Having a better MS base (enhanced MS) and its PAN, we progressively extract information from the PAN and enhanced MS images, expecting to capture pivotal and complementary information of the two modalities for the purpose of constructing the desired HRMS. Extensive experiments together with ablation studies on widely-used datasets are provided to verify the efficacy of our design, and demonstrate its superiority over other state-of-the-art methods both quantitatively and qualitatively. Our code has been released at

    https://github.com/JiaYN1/PAPS

    .

     

    • FullText(HTML)
  • loading
    • 1 http://www.rscloudmart.com/dataProduct/sample
    2 http://www.digitalglobe.com/samples
    References(60)
  • [1]
    G. Cheng and J. Han, “A survey on object detection in optical remote sensing images,” ISPRS J. Photogramm. Remote Sens., vol. 117, pp. 11–28, 2016. doi: 10.1016/j.isprsjprs.2016.03.014
    [2]
    G. M. Foody, “Remote sensing of tropical forest environments: Towards the monitoring of environmental resources for sustainable development,” Int. J. Remote Sens., vol. 24, no. 20, pp. 4035–4046, 2003. doi: 10.1080/0143116031000103853
    [3]
    X.-B. Jin, Z.-Y. Wang, J.-L. Kong, Y.-T. Bai, T.-L. Su, H.-J. Ma, and P. Chakrabarti, “Deep spatio-temporal graph network with self-optimization for air quality prediction,” Entropy, vol. 25, no. 2, p. 247, 2023. doi: 10.3390/e25020247
    [4]
    M. A. Mulders, “Advances in the application of remote sensing and gis for surveying mountainous land,” INT. J. Appl. Earth Obs. Geoinformation, vol. 3, no. 1, pp. 3–10, 2001. doi: 10.1016/S0303-2434(01)85015-7
    [5]
    K. Nogueira, O. A. Penatti, and J. A. Dos Santos, “Towards better exploiting convolutional neural networks for remote sensing scene classification,” Pattern Recognit., vol. 61, pp. 539–556, 2017. doi: 10.1016/j.patcog.2016.07.001
    [6]
    F. D. Javan, F. Samadzadegan, S. Mehravar, A. Toosi, R. Khatami, and A. Stein, “A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery,” ISPRS J. Photogramm. Remote Sens., vol. 171, pp. 101–117, 2021. doi: 10.1016/j.isprsjprs.2020.11.001
    [7]
    J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y. Ma, “SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 7, pp. 1200–1217, 2022. doi: 10.1109/JAS.2022.105686
    [8]
    L. Tang, Y. Deng, Y. Ma, J. Huang, and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 12, pp. 2121–2137, 2022. doi: 10.1109/JAS.2022.106082
    [9]
    H. Zhang, H. Xu, Y. Xiao, X. Guo, and J. Ma, “Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity,” in Proc.AAAI Conf. Artificial Intelligence, vol. 34, no. 7, 2020, pp. 12797–12804.
    [10]
    G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, “Pansharpening by convolutional neural networks,” Remote Sens., vol. 8, no. 7, p. 594, 2016. doi: 10.3390/rs8070594
    [11]
    Y. Wei, Q. Yuan, H. Shen, and L. Zhang, “Boosting the accuracy of multispectral image pansharpening by learning a deep residual network,” IEEE Geosci. Remote Sens. Lett., vol. 14, no. 10, pp. 1795–1799, 2017. doi: 10.1109/LGRS.2017.2736020
    [12]
    J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding, and J. Paisley, “PanNet: A deep network architecture for pan-sharpening,” in Proc. IEEE Int. Conf. Computer Vision, 2017, pp. 5449–5457.
    [13]
    D. Wang, Y. Li, L. Ma, Z. Bai, and C.-W. J. Chan, “Going deeper with densely connected convolutional neural networks for multispectral pansharpening,” Remote Sens., vol. 11, no. 22, p. 2608, 2019. doi: 10.3390/rs11222608
    [14]
    D. Wang, Y. Bai, C. Wu, Y. Li, C. Shang, and Q. Shen, “Convolutional lstm-based hierarchical feature fusion for multispectral pan-sharpening,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–16, 2021.
    [15]
    X. Liu, Q. Liu, and Y. Wang, “Remote sensing image fusion based on two-stream fusion network,” Inf. Fusion, vol. 55, pp. 1–15, 2020. doi: 10.1016/j.inffus.2019.07.010
    [16]
    J. Wang, Z. Shao, X. Huang, T. Lu, and R. Zhang, “A dual-path fusion network for pan-sharpening,” IEEE Trans. Geosci. and Remote Sens., vol. 60, pp. 1–14, 2021.
    [17]
    G. Cheng, Z. Shao, J. Wang, X. Huang, and C. Dang, “Dual-branch multi-level feature aggregation network for pansharpening,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 11, pp. 2023–2026, 2022. doi: 10.1109/JAS.2022.105956
    [18]
    H. Xu, J. Ma, Z. Shao, H. Zhang, J. Jiang, and X. Guo, “SDPNet: A deep network for pan-sharpening with enhanced information representation,” IEEE Trans. Geosci. and Remote Sens., vol. 59, no. 5, pp. 4120–4134, 2020.
    [19]
    Y. Wang, L. Deng, T. Zhang, and X. Wu, “SSconv: Explicit spectral-to-spatial convolution for pansharpening,” in Proc. ACM Int. Conf. Multimedia, 2021, pp. 4472–4480.
    [20]
    G. Yang, M. Zhou, K. Yan, A. Liu, X. Fu, and F. Wang, “Memory-augmented deep conditional unfolding network for pan-sharpening,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2022, pp. 1788–1797.
    [21]
    T.-M. Tu, S.-C. Su, H.-C. Shyu, and P. S. Huang, “A new look at IHS-like image fusion methods,” Inf. Fusion, vol. 2, no. 3, pp. 177–186, 2001. doi: 10.1016/S1566-2535(01)00036-7
    [22]
    C. A. Laben and B. V. Brower, “Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening,” January 2000, uS Patent 6, 011, 875.
    [23]
    M. Ghadjati, A. Moussaoui, and A. Boukharouba, “A novel iterative PCA-based pansharpening method,” IEEE Geosci. Remote Sens. Lett., vol. 10, no. 3, pp. 264–273, 2019. doi: 10.1080/2150704X.2018.1547443
    [24]
    R. A. Schowengerdt, “Reconstruction of multispatial, multispectral image data using spatial frequency content,” Photogramm. Eng. Remote Sens., vol. 46, no. 10, pp. 1325–1334, 1980.
    [25]
    J. Liu, “Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details,” Int. J. Remote Sens., vol. 21, no. 18, pp. 3461–3472, 2000. doi: 10.1080/014311600750037499
    [26]
    C. Ballester, V. Caselles, L. Igual, J. Verdera, and B. Rougé, “A variational model for P + XS image fusion,” Int. J. Comput. Vis., vol. 69, no. 1, pp. 43–58, 2006. doi: 10.1007/s11263-006-6852-x
    [27]
    G. Vivone, M. Simões, M. Dalla Mura, R. Restaino, J. M. Bioucas-Dias, G. A. Licciardi, and J. Chanussot, “Pansharpening based on semiblind deconvolution,” IEEE Trans. Geosci. Remote Sens., vol. 53, no. 4, pp. 1997–2010, 2014.
    [28]
    P. Zhuang, “Pan-sharpening with a gradient domain guided image filtering prior,” in Proc. IEEE Int. Conf. Signal and Image Processing, 2019, pp. 1031–1036.
    [29]
    X. Tian, Y. Chen, C. Yang, and J. Ma, “Variational pansharpening by exploiting cartoon-texture similarities,” IEEE Trans. Geosci. and Remote Sens., vol. 60, pp. 1–16, 2021.
    [30]
    Q. Liu, H. Zhou, Q. Xu, X. Liu, and Y. Wang, “PSGAN: A generative adversarial network for remote sensing image pan-sharpening,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 12, pp. 10227–10242, 2020.
    [31]
    W. Xie, Y. Cui, Y. Li, J. Lei, Q. Du, and J. Li, “HPGAN: Hyperspectral pansharpening using 3-d generative adversarial networks,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 1, pp. 463–477, 2020.
    [32]
    J. Ma, W. Yu, C. Chen, P. Liang, X. Guo, and J. Jiang, “Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion,” Inf. Fusion, vol. 62, pp. 110–120, 2020. doi: 10.1016/j.inffus.2020.04.006
    [33]
    K. Zhang, “On mode collapse in generative adversarial networks,” in Proc. Int. Conf. Artificial Neural Networks, 2021, pp. 563–574.
    [34]
    V. Nagarajan and J. Z. Kolter, “Gradient descent GAN optimization is locally stable,” in Proc. Int. Conf. Neural Information Processing Systems, 2017, pp. 5591–5600.
    [35]
    S. Xu, J. Zhang, Z. Zhao, K. Sun, J. Liu, and C. Zhang, “Deep gradient projection networks for pan-sharpening,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2021, pp. 1366–1375.
    [36]
    H. Zhang and J. Ma, “GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening,” ISPRS J. Photogramm. Remote Sens., vol. 172, pp. 223–239, 2021. doi: 10.1016/j.isprsjprs.2020.12.014
    [37]
    H. Xu, Z. Le, J. Huang, and J. Ma, “A cross-direction and progressive network for pan-sharpening,” Remote Sens., vol. 13, no. 15, p. 3045, 2021. doi: 10.3390/rs13153045
    [38]
    Y. Zhang, C. Liu, M. Sun, and Y. Ou, “Pan-sharpening using an efficient bidirectional pyramid network,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 8, pp. 5549–5563, 2019. doi: 10.1109/TGRS.2019.2900419
    [39]
    M. Zhou, K. Yan, J. Huang, Z. Yang, X. Fu, and F. Zhao, “Mutual information-driven pan-sharpening,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2022, pp. 1798–1808.
    [40]
    C. Jin, L. Deng, T. Huang, and G. Vivone, “Laplacian pyramid networks: A new approach for multispectral pansharpening,” Inf. Fusion, vol. 78, pp. 158–170, 2022. doi: 10.1016/j.inffus.2021.09.002
    [41]
    Z. Jin, Y. Zhuo, T. Zhang, X. Jin, S. Jing, and L. Deng, “Remote sensing pansharpening by full-depth feature fusion,” Remote Sens., vol. 14, no. 3, p. 466, 2022. doi: 10.3390/rs14030466
    [42]
    J. Cai and B. Huang, “Super-resolution-guided progressive pansharpening based on a deep convolutional neural network,” IEEE Trans. Geosci. and Remote Sens., vol. 59, no. 6, pp. 5206–5220, 2020.
    [43]
    M. Zhou, J. Huang, X. Fu, F. Zhao, and D. Hong, “Effective pan-sharpening by multiscale invertible neural network and heterogeneous task distilling,” IEEE Trans. Geosci. and Remote Sens., vol. 60, pp. 1–14, 2022.
    [44]
    S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2021, pp. 14821–14831.
    [45]
    S. Rahmani, M. Strait, D. Merkurjev, M. Moeller, and T. Wittman, “An adaptive IHS pan-sharpening method,” IEEE Geosci. Remote Sens. Lett., vol. 7, no. 4, pp. 746–750, 2010. doi: 10.1109/LGRS.2010.2046715
    [46]
    N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp. 528–537, 2011.
    [47]
    G. Vivone, R. Restaino, and J. Chanussot, “Full scale regression-based injection coefficients for panchromatic sharpening,” IEEE Trans. on Image Process., vol. 27, no. 7, pp. 3418–3431, 2018. doi: 10.1109/TIP.2018.2819501
    [48]
    Z. Yang, X. Fu, A. Liu, and Z.-J. Zha, “Progressive pan-sharpening via cross-scale collaboration networks,” IEEE Geosci. Remote Sens. Lett., vol. 19, pp. 1–5, 2022.
    [49]
    W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016, pp. 1874–1883.
    [50]
    S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in Proc. European Conf. Computer Vision, 2018, pp. 3–19.
    [51]
    L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images,” Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp. 691–699, 1997.
    [52]
    X. Meng, Y. Xiong, F. Shao, H. Shen, W. Sun, G. Yang, Q. Yuan, R. Fu, and H. Zhang, “A large-scale benchmark data set for evaluating pansharpening performance: Overview and implementation,” IEEE Geosci. Remote Sens. Mag., vol. 9, no. 1, pp. 18–52, 2020.
    [53]
    A. Hore and D. Ziou, “Image quality metrics: PSNR vs. SSIM,” in Proc. Int. Conf. Pattern Recognition, 2010, pp. 2366–2369.
    [54]
    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. on Image Process., vol. 13, no. 4, pp. 600–612, 2004. doi: 10.1109/TIP.2003.819861
    [55]
    S. Kaneko, Y. Satoh, and S. Igarashi, “Using selective correlation coefficient for robust image registration,” Pattern Recognit., vol. 36, no. 5, pp. 1165–1173, 2003. doi: 10.1016/S0031-3203(02)00081-X
    [56]
    L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce, “Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest,” IEEE Trans. Geosci. and Remote Sens., vol. 45, no. 10, pp. 3012–3021, 2007. doi: 10.1109/TGRS.2007.904923
    [57]
    R. H. Yuhas, A. F. Goetz, and J. W. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm,” in Proc. 3rd Annu. JPL Airborne Geoscience Workshop, 1992, pp. 147–149.
    [58]
    L. Alparone, B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva, “Multispectral and panchromatic data fusion assessment without reference,” Photogramm. Eng. Remote Sens., vol. 74, no. 2, pp. 193–200, 2008. doi: 10.14358/PERS.74.2.193
    [59]
    N. Ketkar, “Stochastic gradient descent,” in Deep learning with Python, 2017, pp. 113–132.
    [60]
    Z. Jin, L. Deng, T. Zhang, and X. Jin, “BAM: Bilateral activation mechanism for image fusion,” in Proc. ACM Int. Conf. Multimedia, 2021, pp. 4315–4323.
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)  / Tables(6)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (305) PDF downloads(33) Cited by()

    Highlights

    • A progressive attention-based network for pan-sharpening is designed
    • The detail enhancement module is introduced to provide better multispectral references for fusion
    • The progressive fusion module is proposed to take full advantage of spectral and spatial information
    • The proposed method can flexibly reduce the parameters and produce appealing results

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return