A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 10 Issue 6
Jun.  2023

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2023  >  10(6): 1407-1420
D. Zhang, Q. S. Lian, Y. M. Su, and  T. F. Ren,  “Dual-prior integrated image reconstruction for quanta image sensors using multi-agent consensus equilibrium,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 6, pp. 1407–1420, Jun. 2023. doi: 10.1109/JAS.2023.123390
Citation: D. Zhang, Q. S. Lian, Y. M. Su, and  T. F. Ren,  “Dual-prior integrated image reconstruction for quanta image sensors using multi-agent consensus equilibrium,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 6, pp. 1407–1420, Jun. 2023. doi: 10.1109/JAS.2023.123390
shu

Dual-Prior Integrated Image Reconstruction for Quanta Image Sensors Using Multi-Agent Consensus Equilibrium

doi: 10.1109/JAS.2023.123390
Funds:  This work was supported by Hebei Natural Science Foundation (F2022203030) and the National Natural Science Foundation of China (61471313)
More Information
    Abstract

  • Quanta image sensors (QIS) are a new type of single-photon imaging device that can oversample the light field to generate binary bit-streams. The reconstruction for QIS refers to the recovery of original scenes from these binary measurements. Conventional reconstruction algorithms for QIS generally depend solely on one instantiated prior and are certainly insufficient for capturing the statistical properties over high-dimensional space. On the other hand, deep learning-based methods have shown promising performance, due to their excellent ability to learn feature representations from relevant databases. However, most deep models only focus on exploring local features while generally overlooking long-range similarity. In view of this, a dual-prior integrated reconstruction algorithm for QIS (DPI-QIS) is proposed, which combines a deep prior with a non-local self-similarity one using the multi-agent consensus equilibrium (MACE) framework. In comparison to the approaches that utilize a single prior, DPI-QIS fits the reconstruction model sufficiently by leveraging the respective merits of both priors. An effective yet flexible MACE framework is employed to integrate the physical forward model allying with the two prior-based models to achieve an overall better result. Extensive experiments demonstrate that the proposed algorithm achieves state-of-the-art performance in terms of objective and visual perception at multiple oversampling factors, while having stronger robustness to noise.

     

    • FullText(HTML)
  • loading
    • References(54)
  • [1]
    E. R. Fossum, J. J. Ma, S. Masoodian, L. Anzagira, and R. Zizza, “The quanta image sensor: Every photon counts,” Sensors, vol. 16, no. 8, p. 1260, Aug. 2016. doi: 10.3390/s16081260
    [2]
    L. Liang, H. Y. Shen, P. De Camilli, and J. S. Duncan, “A novel multiple hypothesis based particle tracking method for clathrin mediated endocytosis analysis using fluorescence microscopy,” IEEE Trans. Image Process., vol. 23, no. 4, pp. 1844–1857, Apr. 2014. doi: 10.1109/TIP.2014.2303633
    [3]
    S. P. Poland, N. Krstajić, J. Monypenny, S. Coelho, D. Tyndall, R. J. Walker, V. Devauges, J. Richardson, N. Dutton, P. Barber, D. D. U. Li, K. Suhling, T. Ng, R. K. Henderson, and S. M. Ameer-Beg, “A high speed multifocal multiphoton fluorescence lifetime imaging microscope for live-cell fret imaging,” Biomed. Opt. Express, vol. 6, no. 2, pp. 277–296, Jan. 2015. doi: 10.1364/BOE.6.000277
    [4]
    L. H. C. Braga, L. Gasparini, L. Grant, R. K. Henderson, N. Massari, M. Perenzoni, D. Stoppa, and R. Walker, “A fully digital 8×16 SiPM array for PET applications with per-pixel TDCs and real-time energy output,” IEEE J. Solid-State Circuits, vol. 49, no. 1, pp. 301–314, Jan. 2014. doi: 10.1109/JSSC.2013.2284351
    [5]
    P. Seitz and A. J. P. Theuwissen, Single-Photon Imaging. Berlin, Germany: Springer, 2011.
    [6]
    L. J. Meng, “An intensified EMCCD camera for low energy gamma ray imaging applications,” IEEE Trans. Nucl. Sci., vol. 53, no. 4, pp. 2376–2384, Aug. 2006. doi: 10.1109/TNS.2006.878574
    [7]
    S. A. Kay, S. Julier, and V. M. Pawar, “Semantically informed next best view planning for autonomous aerial 3D reconstruction,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Prague, Czech Republic, 2021, pp. 3125–3130.
    [8]
    Y. Y. Zhou, Y. Takeda, M. Tomizuka, and W. Zhan, “Automatic construction of lane-level HD maps for urban scenes,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Prague, Czech Republic, 2021, pp. 6649–6656.
    [9]
    F. Yang, Y. M. Lu, L. Sbaiz, and M. Vetterli, “Bits from photons: Oversampled image acquisition using binary Poisson statistics,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 1421–1436, Apr. 2012. doi: 10.1109/TIP.2011.2179306
    [10]
    F. Yang, L. Sbaiz, E. Charbon, S. Süsstrunk, and M. Vetterli, “Image reconstruction in the gigavision camera,” in Proc. IEEE 12th Int. Conf. Computer Vision Workshops, Kyoto, Japan, 2009, pp. 2212– 2219.
    [11]
    S. H. Chan and Y. M. Lu, “Efficient image reconstruction for gigapixel quantum image sensors,” in Proc. IEEE Global Conf. Signal Information Processing, Atlanta, USA, 2014, pp. 312–316.
    [12]
    S. H. Chan, X. R. Wang, and O. A. Elgendy, “Plug-and-play ADMM for image restoration: Fixed-point convergence and applications,” IEEE Trans. Comput. Imaging, vol. 3, no. 1, pp. 84–98, Mar. 2017. doi: 10.1109/TCI.2016.2629286
    [13]
    E. K. Ryu, J. L. Liu, S. C. Wang, X. H. Chen, Z. Y. Wang, and W. T. Yin, “Plug-and-play methods provably converge with properly trained denoisers,” in Proc. 36th Int. Conf. Machine Learning, Long Beach, USA, 2019, pp. 5546–5557.
    [14]
    K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080–2095, Aug. 2007. doi: 10.1109/TIP.2007.901238
    [15]
    A. Buades, B. Coll, and J. M. Morel, “A non-local algorithm for image denoising,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Diego, USA, 2005, pp. 60–65.
    [16]
    C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. 6th Int. Conf. Computer Vision, Bombay, India, 1998, pp. 839–846.
    [17]
    K. Zhang, W. M. Zuo, Y. J. Chen, D. Y. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017. doi: 10.1109/TIP.2017.2662206
    [18]
    S. H. Chan, O. A. Elgendy, and X. R. Wang, “Images from bits: Non-iterative image reconstruction for quanta image sensors,” Sensors, vol. 16, no. 11, p. 1961, Nov. 2016. doi: 10.3390/s16111961
    [19]
    R. A. Rojas, W. Y. Luo, V. Murray, and Y. M. Lu, “Learning optimal parameters for binary sensing image reconstruction algorithms,” in Proc. IEEE Int. Conf. Image Processing, Beijing, China, 2017, pp. 2791–2795.
    [20]
    J. H. Choi, O. A. Elgendy, and S. H. Chan, “Image reconstruction for quanta image sensors using deep neural networks,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Calgary, Canada, 2018, pp. 6543–6547.
    [21]
    G. T. Buzzard, S. H. Chan, S. Sreehari, and C. A. Bouman, “Plug-and-play unplugged: Optimization-free reconstruction using consensus equilibrium,” SIAM J. Imaging Sci., vol. 11, no. 3, pp. 2001–2020, Jan. 2018. doi: 10.1137/17M1122451
    [22]
    A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci., vol. 2, no. 1, pp. 183–202, Jan. 2009. doi: 10.1137/080716542
    [23]
    S. Ono, “Primal-dual plug-and-play image restoration,” IEEE Signal Process. Lett., vol. 24, no. 8, pp. 1108–1112, Aug. 2017. doi: 10.1109/LSP.2017.2710233
    [24]
    C. A. Metzler, A. Maleki, and R. G. Baraniuk, “BM3D-PRGAMP: Compressive phase retrieval based on BM3D denoising,” in Proc. IEEE Int. Conf. Multimedia and Expo Workshops, Seattle, USA, 2016, pp. 2504–2508.
    [25]
    Q. B. Wu, W. Q. Ren, and X. C. Cao, “Learning interleaved cascade of shrinkage fields for joint image dehazing and denoising,” IEEE Trans. Image Process., vol. 29, pp. 1788–1801, Jan. 2020. doi: 10.1109/TIP.2019.2942504
    [26]
    K. Zhang, W. M. Zuo, and L. Zhang, “FFDNet: Toward a fast and flexible solution for CNN-based image denoising,” IEEE Trans. Image Process., vol. 27, no. 9, pp. 4608–4622, Sept. 2018. doi: 10.1109/TIP.2018.2839891
    [27]
    K. Zhang, Y. W. Li, W. M. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 6360–6376, Oct. 2022. doi: 10.1109/TPAMI.2021.3088914
    [28]
    S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in Proc. IEEE Global Conf. Signal and Information Processing, Austin, USA, 2013, pp. 945–948.
    [29]
    Z. Y. Zha, B. H. Wen, X. Yuan, J. T. Zhou, J. T. Zhou, and C. Zhu, “Triply complementary priors for image restoration,” IEEE Trans. Image Process., vol. 30, pp. 5819–5834, Jun. 2021. doi: 10.1109/TIP.2021.3086049
    [30]
    H. H. Bauschke and J. M. Borwein, “On projection algorithms for solving convex feasibility problems,” SIAM Rev., vol. 38, no. 3, pp. 367–426, Sept. 1996. doi: 10.1137/S0036144593251710
    [31]
    J. A. Tropp, I. S. Dhillon, R. W. Heath, and T. Strohmer, “Designing structured tight frames via an alternating projection method,” IEEE Trans. Inf. Theory, vol. 51, no. 1, pp. 188–209, Jan. 2005. doi: 10.1109/TIT.2004.839492
    [32]
    B. Li, Y. Shen, and J. Li, “Dictionaries construction using alternating projection method in compressive sensing,” IEEE Signal Process. Lett., vol. 18, no. 11, pp. 663–666, Nov. 2011. doi: 10.1109/LSP.2011.2168517
    [33]
    Y. M. Lu, M. Karzand, and M. Vetterli, “Demosaicking by alternating projections: Theory and fast one-step implementation,” IEEE Trans. Image Process., vol. 19, no. 8, pp. 2085–2098, Aug. 2010. doi: 10.1109/TIP.2010.2045710
    [34]
    Y. Yang, R. S. Blum, Z. S. He, and D. R. Fuhrmann, “MIMO radar waveform design via alternating projection,” IEEE Trans. Image Process., vol. 58, no. 3, pp. 1440–1445, Mar. 2010. doi: 10.1109/TSP.2009.2033314
    [35]
    S. Majee, T. Balke, C. A. J. Kemp, G. T. Buzzard, and C. A. Bouman, “Multi-slice fusion for sparse-view and limited-angle 4D CT reconstruction,” IEEE Trans. Comput. Imaging, vol. 7, pp. 448–462, Apr. 2021. doi: 10.1109/TCI.2021.3074881
    [36]
    N. Parikh and S. Boyd, “Proximal algorithms,” Found. Trends.® Optim., vol. 1, no. 3, pp. 127–239, Jan. 2014.
    [37]
    L. Q. Yan, D. F. Liu, Y. X. Song, and C. B. Yu, “Multimodal aggregation approach for memory vision-voice indoor navigation with meta-learning,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Las Vegas, USA, 2020, pp. 5847–5854.
    [38]
    V. Sridhar, X. Wang, G. T. Buzzard, and C. A. Bouman, “Distributed iterative CT reconstruction using multi-agent consensus equilibrium,” IEEE Trans. Comput. Imaging, vol. 6, pp. 1153–1166, Jul. 2020. doi: 10.1109/TCI.2020.3008782
    [39]
    M. U. Ghani and W. C. Karl, “Data and image prior integration for image reconstruction using consensus equilibrium,” IEEE Trans. Comput. Imaging, vol. 7, pp. 297–308, Mar. 2021. doi: 10.1109/TCI.2021.3062986
    [40]
    R. Byers, “A bisection method for measuring the distance of a stable matrix to the unstable matrices,” SIAM J. Sci. Stat. Comput., vol. 9, no. 5, pp. 875–881, Sept. 1988. doi: 10.1137/0909059
    [41]
    K. Zhang, W. M. Zuo, S. H. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 3929–3938.
    [42]
    S. Hurault, A. Leclaire, and N. Papadakis, “Proximal denoiser for convergent plug-and-play optimization with nonconvex regularization,” in Proc. Int. Conf. Machine Learning, Baltimore, USA, 2022, pp. 9483–9505.
    [43]
    C. Cruz, A. Foi, V. Katkovnik, and K. Egiazarian, “Nonlocality-reinforced convolutional neural networks for image denoising,” IEEE Signal Process. Lett., vol. 25, no. 8, pp. 1216–1220, Aug. 2018. doi: 10.1109/LSP.2018.2850222
    [44]
    Y. Guo, A. Davy, G. Facciolo, J. M. Morel, and Q. Y. Jin, “Fast, nonlocal and neural: A lightweight high quality solution to image denoising,” IEEE Signal Process. Lett., vol. 28, pp. 1515–1519, Jul. 2021. doi: 10.1109/LSP.2021.3099963
    [45]
    C. Mou, J. Zhang, and Z. Y. Wu, “Dynamic attentive graph learning for image restoration,” in Proc. IEEE/CVF Int. Conf. Computer Vision, Montréal, Canada, 2021, pp. 4328–4337.
    [46]
    Y. L. Zhang, K. P. Li, K. Li, B. N. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” in Proc. 7th Int. Conf. Learning Representations, New Orleans, USA, 2019.
    [47]
    D. Liu, B. H. Wen, Y. C. Fan, C. C. Loy, and T. S. Huang, “Non-local recurrent network for image restoration,” in Proc. 32nd Int. Conf. Neural Information Processing Systems, Montréal, Canada, 2018, pp. 1680–1689.
    [48]
    T. Plötz and S. Roth, “Neural nearest neighbors networks,” in Proc. 32nd Int. Conf. Neural Information Processing Systems, Montréal, Canada, 2018, pp. 1095–1106.
    [49]
    K. D. Ma, Z. Duanmu, Q. B. Wu, Z. Wang, H. W. Yong, H. L. Li, and L. Zhang, “Waterloo exploration database: New challenges for image quality assessment models,” IEEE Trans. Image Process., vol. 26, no. 2, pp. 1004–1016, Feb. 2017. doi: 10.1109/TIP.2016.2631888
    [50]
    D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int. Conf. Computer Vision, Vancouver, Canada, 2001, pp. 416–423.
    [51]
    E. R. Fossum, “Modeling the performance of single-bit and multi-bit quanta image sensors,” IEEE J. Electron Devices Soc., vol. 1, no. 9, pp. 166–174, Sept. 2013. doi: 10.1109/JEDS.2013.2284054
    [52]
    Y. H. Chi, A. Gnanasambandam, V. Koltun, and S. H. Chan, “Dynamic low-light imaging with quanta image sensors,” in Proc. 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 122–138.
    [53]
    A. Gnanasambandam and S. H. Chan, “Image classification in the dark using quanta image sensors,” in Proc. 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 484–501.
    [54]
    X. R. Wang, J. Juang, and S. H. Chan, “Automatic foreground extraction from imperfect backgrounds using multi-agent consensus equilibrium,” J. Vis. Commun. Image Represent., vol. 72, p. 102907, Oct. 2020. doi: 10.1016/j.jvcir.2020.102907
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(11)  / Tables(8)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (337) PDF downloads(70) Cited by()

    Highlights

    • A dual-prior integrated reconstruction algorithm for Quanta image sensors using multi-agent consensus equilibrium framework is proposed
    • Two priors (i.e., the deep prior and the non-local self-similarity prior) with complementary properties are embedded into the multi-agent consensus equilibrium framework in a plug-and-play manner
    • From the view of projection, a clear exposition of the working mechanism for multi-agent consensus equilibrium framework is provided

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return