A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 8 Issue 7
Jul.  2021

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan and Xiaonan Luo, "Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network," IEEE/CAA J. Autom. Sinica, vol. 8, no. 7, pp. 1271-1280, July 2021. doi: 10.1109/JAS.2021.1004009
Citation: Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan and Xiaonan Luo, "Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network," IEEE/CAA J. Autom. Sinica, vol. 8, no. 7, pp. 1271-1280, July 2021. doi: 10.1109/JAS.2021.1004009

Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network

doi: 10.1109/JAS.2021.1004009
Funds:  This work was supported in part by the National Natural Science Foundation of China (61772149, 61866009, 61762028, U1701267, 61702169), Guangxi Science and Technology Project (2019GXNSFFA245014, ZY20198016, AD18281079, AD18216004), the Natural Science Foundation of Hunan Province (2020JJ3014), and Guangxi Colleges and Universities Key Laboratory of Intelligent Processing of Computer Images and Graphics (GIIP202001)
More Information
  • The tradeoff between efficiency and model size of the convolutional neural network (CNN) is an essential issue for applications of CNN-based algorithms to diverse real-world tasks. Although deep learning-based methods have achieved significant improvements in image super-resolution (SR), current CNN-based techniques mainly contain massive parameters and a high computational complexity, limiting their practical applications. In this paper, we present a fast and lightweight framework, named weighted multi-scale residual network (WMRN), for a better tradeoff between SR performance and computational efficiency. With the modified residual structure, depthwise separable convolutions (DS Convs) are employed to improve convolutional operations’ efficiency. Furthermore, several weighted multi-scale residual blocks (WMRBs) are stacked to enhance the multi-scale representation capability. In the reconstruction subnetwork, a group of Conv layers are introduced to filter feature maps to reconstruct the final high-quality image. Extensive experiments were conducted to evaluate the proposed model, and the comparative results with several state-of-the-art algorithms demonstrate the effectiveness of WMRN.

     

  • loading
  • [1]
    B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy, and P. Milanfar, “Handheld multi-frame superresolution,” ACM Trans. Graph., vol. 38, no. 4, pp. 28:1–28:18, July. 2019. doi: 10.1145/3306346.3323024
    [2]
    Z. Ren, K. Qian, Z. Zhang, V. Pandit, A. Baird, and B. Schuller, “Deep scalogram representations for acoustic scene classification,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 3, pp. 662–669, 2018. doi: 10.1109/JAS.2018.7511066
    [3]
    A. A. M. Muzahid, W. Wan, F. Sohel, L. Wu, and L. Hou, “Curvenet: Curvature-based multitask learning deep networks for 3D object recognition,” IEEE/CAA Journal of Automatica Sinica, vol. 8, no. 6, pp. 1194–1204, 2020.
    [4]
    R. Lan, Y. Zhou, Z. Liu, and X. Luo, “Prior knowledge-based probabilistic collaborative representation for visual recognition,” IEEE Trans. Cybernetics, vol. 50, no. 4, pp. 1498–1508, 2020. doi: 10.1109/TCYB.2018.2880290
    [5]
    X. Wang, K. C. Chan, K. Yu, C. Dong, and C. C. Loy, “Edvr: Video restoration with enhanced deformable convolutional networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 2019, pp. 1954–1963.
    [6]
    R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, “Video super-resolution via deep draft-ensemble learning,” in Proc. IEEE Int. Conf. Computer Vision, Santiago, Chile, 2015, pp. 531–539.
    [7]
    R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Proc. Asian Conf. Computer Vision, Singapore, 2014, pp. 111–126.
    [8]
    B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 2017, pp. 1132–1140.
    [9]
    C. Wang, Z. Li, and J. Shi, “Lightweight image super-resolution with adaptive weighted learning network,” arXiv preprint arXiv:1904.02358, 2019.
    [10]
    R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Transactions on Acoustics,Speech,and Signal Processing, vol. 29, no. 6, pp. 1153–1160, 1981. doi: 10.1109/TASSP.1981.1163711
    [11]
    R. Timofte, R. Rothe, and L. Van Gool, “Seven ways to improve example-based single image super resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1865–1873.
    [12]
    J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010. doi: 10.1109/TIP.2010.2050625
    [13]
    J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 5197–5206.
    [14]
    S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang, “Toward convolutional blind denoising of real photographs,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 1712–1722.
    [15]
    P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level waveletcnn for image restoration,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 2018, pp. 886–88609.
    [16]
    X. Xu, D. Sun, J. Pan, Y. Zhang, H. Pfister, and M.-H. Yang, “Learning to super-resolve blurry face and text images,” in Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 251–260.
    [17]
    D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng, “Progressive image deraining networks: A better and simpler baseline,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 3932–3941.
    [18]
    B. Li, F. Zhao, Z. Su, X. Liang, Y.-K. Lai, and R. L. Rosin, “Example based image colorization using locality consistent sparse representation,” IEEE Trans. Image Processing, vol. 26, no. 11, pp. 5188–5202, 2017. doi: 10.1109/TIP.2017.2732239
    [19]
    C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Proc. European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 184–199.
    [20]
    J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2016.
    [21]
    J. Kim, J. Kwon Lee, and K. Mu Lee, “Deeply-recursive convolutional network for image superresolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp.1637–1645.
    [22]
    Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 2790–2798.
    [23]
    W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1874–1883.
    [24]
    R. Lan, L. Sun, Z. Liu, H. Lu, Z. Su, C. Pang, and X. Luo, “Cascading and enhanced residual networks for accurate single-image superresolution,” IEEE Trans. Cybernetics, vol. 51, no. 1, pp. 115–125, Jan. 2021. doi: 10.1109/TCYB.2019.2952710
    [25]
    R. Lan, L. Sun, Z. Liu, H. Lu, C. Pang, and X. Luo, “Madnet: A fast and lightweight network for single-image super resolution,” IEEE Trans. Cybernetics, vol. 51, no. 3, pp. 1443–1453, Mar. 2021. doi: 10.1109/tcyb.2020.2970104
    [26]
    N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 252–268.
    [27]
    T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 11057–11066.
    [28]
    G. Gao, D. Zhu, M. Yang, H. Lu, W. Yang, and H. Gao, “Face image super-resolution with pose via nuclear norm regularized structural orthogonal procrustes regression,” Neural Computing and Applications, vol. 32, no. 9, pp. 4361–4371, 2020. doi: 10.1007/s00521-018-3826-1
    [29]
    K. Zhang, S. Gu, R. Timofte, Z. Hui, X. Wang, X. Gao, D. Xiong, S. Liu, R. Gang, and N. Nan, “Aim 2019 challenge on constrained super-resolution: Methods and results,” in Proc. IEEE Int. Conf. Computer Vision Workshops, Seoul, Korea (South), 2019, pp. 3565–3574.
    [30]
    C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. doi: 10.1109/TPAMI.2015.2439281
    [31]
    W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 5835–5843.
    [32]
    Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 4549–4557.
    [33]
    C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in Proc. European Conf. Computer Vision, Amsterdam, Netherlands, 2016, pp. 391–407.
    [34]
    M. D. Zeiler, G. W. Taylor, and R. Fergus, “Adaptive deconvolutional networks for mid and high level feature learning,” in Proc. IEEE Int. Conf. Computer Vision, Barcelona, Spain, 2011, pp. 2018–2025.
    [35]
    W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 11, pp. 2599–2613, 2018.
    [36]
    Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image superresolution using very deep residual channel attention networks,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 286–301.
    [37]
    Z. Hui, X. Wang, and X. Gao, “Fast and accurate single image superresolution via information distillation network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 723–731.
    [38]
    Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image super-resolution: A survey,” CoRR, vol. abs/1902.06068, 2019. [Online]. Available: http://arxiv.org/abs/1902.06068
    [39]
    C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 105–114.
    [40]
    J. Johnson, A. Alahi, and F. Li, “Perceptual losses for real-time style transfer and super-resolution,” CoRR, vol. abs/1603.08155, 2016. [Online]. Available: http://arxiv.org/abs/1603.08155
    [41]
    X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in Proc. European Conf. Computer Vision Workshops, Munich, Germany, 2018, pp. 63–79.
    [42]
    A. Marquina and S. J. Osher, “Image super-resolution by tvregularization and bregman iteration,” Journal of Scientific Computing, vol. 37, no. 3, pp. 367–382, 2008. doi: 10.1007/s10915-008-9214-8
    [43]
    C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2818–2826.
    [44]
    C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Proc. AAAI Conf. Artificial Intelligence, San Francisco, California, USA, 2017.
    [45]
    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition), Salt Lake City, UT, USA, 2018, pp. 4510–4520.
    [46]
    K. Hung, Z. Zhang, and J. Jiang, “Real-time image super-resolution using recursive depthwise separable convolution network,” IEEE Access, vol. 7, pp. 99 804–99 816, 2019. doi: 10.1109/ACCESS.2019.2929223
    [47]
    S. Liu, D. Huang, and Y. Wang, “Receptive field block net for accurate and fast object detection,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 404–419.
    [48]
    S. Gao, M. Cheng, K. Zhao, X. Zhang, M. Yang, and P. H. S. Torr, “Res2net: A new multi-scale backbone architecture,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 2, pp. 652–662, Feb. 2021. doi: 10.1109/TPAMI.2019.2938758
    [49]
    F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. Int. Conf. Learn. Representations, San Juan, Puerto Rico, 2016.
    [50]
    W. Ren, S. Liu, L. Ma, Q. Xu, X. Xu, X. Cao, J. Du, and M. Yang, “Lowlight image enhancement via a deep hybrid network,” IEEE Trans. Image Processing, vol. 28, no. 9, pp. 4364–4375, Sep. 2019. doi: 10.1109/TIP.2019.2910412
    [51]
    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proc. Int. Conf. Learn. Representations, San Diego, CA, USA, 2015.
    [52]
    R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 2017, pp. 1110–1121.
    [53]
    M. Bevilacqua, A. Roumy, C. Guillemot, and M. line Alberi Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in Proc. British Machine Vision Conf. BMVA Press, Surrey, Australia, 2012, pp. 135.1–135.10.
    [54]
    R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Curves and Surfaces, J.-D. Boissonnat, P. Chenin, A. Cohen, C. Gout, T. Lyche, M.-L. Mazure, and L. Schumaker, Eds. Berlin Heidelberg, Germany: Springer, 2012, pp. 711–730.
    [55]
    P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, May 2011. doi: 10.1109/TPAMI.2010.161
    [56]
    Y. Matsui, K. Ito, Y. Aramaki, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” CoRR, vol. abs/1510.04389, 2015. [Online]. Available: http://arxiv.org/abs/1510.04389
    [57]
    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600–612, April. 2004. doi: 10.1109/TIP.2003.819861
    [58]
    K. Zhang, W. Zuo, and L. Zhang, “Learning a single convolutional superresolution network for multiple degradations,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 3262–3271.

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(6)  / Tables(3)

    Article Metrics

    Article views (1766) PDF downloads(125) Cited by()

    Highlights

    • A novel weighted multi-scale residual block (WMRB) is proposed, which can not only effectively exploit multi-scale features but also dramatically reduce the computational burden.
    • A global residual shortcut is deployed, which adds high frequency features to generate more clear details and promote gradient information propagation.
    • Extensive experiments show that the WMRN model utilizes only a modest number of parameters and operations to achieve competitive SR performance on different benchmarks with different upscaling factors.

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return