A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 11 Issue 1
Jan.  2024

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2024  >  11(1): 113-130
Y. Lu, J. Ma, X. Mei, J. Huang, and X.-P. Zhang, “Feature matching via topology-aware graph interaction model,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 1, pp. 113–130, Jan. 2024. doi: 10.1109/JAS.2023.123774
Citation: Y. Lu, J. Ma, X. Mei, J. Huang, and X.-P. Zhang, “Feature matching via topology-aware graph interaction model,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 1, pp. 113–130, Jan. 2024. doi: 10.1109/JAS.2023.123774
shu

Feature Matching via Topology-Aware Graph Interaction Model

doi: 10.1109/JAS.2023.123774
Funds:  This work was supported by the National Natural Science Foundation of China (62276192)
More Information
    Abstract
  • Feature matching plays a key role in computer vision. However, due to the limitations of the descriptors, the putative matches are inevitably contaminated by massive outliers. This paper attempts to tackle the outlier filtering problem from two aspects. First, a robust and efficient graph interaction model, is proposed, with the assumption that matches are correlated with each other rather than independently distributed. To this end, we construct a graph based on the local relationships of matches and formulate the outlier filtering task as a binary labeling energy minimization problem, where the pairwise term encodes the interaction between matches. We further show that this formulation can be solved globally by graph cut algorithm. Our new formulation always improves the performance of previous locality-based method without noticeable deterioration in processing time, adding a few milliseconds. Second, to construct a better graph structure, a robust and geometrically meaningful topology-aware relationship is developed to capture the topology relationship between matches. The two components in sum lead to topology interaction matching (TIM), an effective and efficient method for outlier filtering. Extensive experiments on several large and diverse datasets for multiple vision tasks including general feature matching, as well as relative pose estimation, homography and fundamental matrix estimation, loop-closure detection, and multi-modal image matching, demonstrate that our TIM is more competitive than current state-of-the-art methods, in terms of generality, efficiency, and effectiveness. The source code is publicly available at http://github.com/YifanLu2000/TIM.

     

    • FullText(HTML)
  • loading
    • 1 https://github.com/JiawangBian/FM-Bench
    References(71)
  • [1]
    J. Ma, X. Jiang, A. Fan, J. Jiang, and J. Yan, “Image matching from handcrafted to deep features: A survey,” Int. J. Comput. Vision, vol. 129, no. 1, pp. 23–79, Jan. 2021. doi: 10.1007/s11263-020-01359-2
    [2]
    C. Leng, H. Zhang, G. Cai, Z. Chen, and A. Basu, “Total variation constrained non-negative matrix factorization for medical image registration,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 5, pp. 1025–1037, May 2021. doi: 10.1109/JAS.2021.1003979
    [3]
    D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, Nov. 2004. doi: 10.1023/B:VISI.0000029664.99615.94
    [4]
    D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperPoint: Self-supervised interest point detection and description,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018, pp. 337-33712.
    [5]
    M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981. doi: 10.1145/358669.358692
    [6]
    K. M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua, “Learning to find good correspondences,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 2666–2674.
    [7]
    J. Yang, Z. Huang, S. Quan, Z. Cao, and Y. Zhang, “RANSACs for 3D rigid registration: A comparative evaluation,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 10, pp. 1861–1878, Oct. 2022. doi: 10.1109/JAS.2022.105500
    [8]
    J. Ma, J. Zhao, J. Jiang, H. Zhou, and X. Guo, “Locality preserving matching,” Int. J. Comput. Vision, vol. 127, no. 5, pp. 512–531, May 2019. doi: 10.1007/s11263-018-1117-z
    [9]
    X. Jiang, J. Jiang, A. Fan, Z. Wang, and J. Ma, “Multiscale locality and rank preservation for robust feature matching of remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 9, pp. 6462–6472, Sept. 2019. doi: 10.1109/TGRS.2019.2906183
    [10]
    X. Jiang, Y. Xia, X.-P. Zhang, and J. Ma, “Robust image matching via local graph structure consensus,” Pattern Recognit., vol. 126, p. 108588, Jun. 2022. doi: 10.1016/j.patcog.2022.108588
    [11]
    J. Ma, K. Zhang, and J. Jiang, “Loop closure detection via locality preserving matching with global consensus,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 2, pp. 411–426, Feb. 2023. doi: 10.1109/JAS.2022.105926
    [12]
    P. H. S. Torr and A. Zisserman, “MLESAC: A new robust estimator with application to estimating image geometry,” Comput. Vision Image Understanding, vol. 78, no. 1, pp. 138–156, Apr. 2000. doi: 10.1006/cviu.1999.0832
    [13]
    O. Chum and J. Matas, “Matching with PROSAC-progressive sample consensus,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Diego, USA, 2005, pp. 220–226.
    [14]
    O. Chum, J. Matas, and J. Kittler, “Locally optimized RANSAC,” in Proc. 25th Joint Pattern Recognition Symp., Magdeburg, Germany, 2003, pp. 236–243.
    [15]
    O. Chum, T. Werner, and J. Matas, “Two-view geometry estimation unaffected by a dominant plane,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Diego, USA, 2005, pp. 772–779.
    [16]
    R. Raguram, O. Chum, M. Pollefeys, J. Matas, and J.-M. Frahm, “USAC: A universal framework for random sample consensus,” IEEE Trans. Pattern Analy. Mach. Intell., vol. 35, no. 8, pp. 2022–2038, Aug. 2013. doi: 10.1109/TPAMI.2012.257
    [17]
    D. Barath and J. Matas, “Graph-cut RANSAC,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 6733–6741.
    [18]
    D. Barath, J. Matas, and J. Noskova, “MAGSAC: Marginalizing sample consensus,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 10189–10197.
    [19]
    D. Baráth, J. Noskova, M. Ivashechkin, and J. Matas, “MAGSAC++, a fast, reliable and accurate robust estimator,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 1301–1309.
    [20]
    X. Li and Z. Hu, “Rejecting mismatches by correspondence function,” Int. J. Comput. Vision, vol. 89, no. 1, pp. 1–17, Aug. 2010. doi: 10.1007/s11263-010-0318-x
    [21]
    J. Ma, J. Zhao, J. Tian, A. L. Yuille, and Z. Tu, “Robust point matching via vector field consensus,” IEEE Trans. Image Process., vol. 23, no. 4, pp. 1706–1721, Apr. 2014. doi: 10.1109/TIP.2014.2307478
    [22]
    W.-Y. Lin, F. Wang, M.-M. Cheng, S.-K. Yeung, P. H. S. Torr, M. N. Do, and J. Lu, “CODE: Coherence based decision boundaries for feature correspondence,” IEEE Trans. Pattern Analy. Mach. Intell., vol. 40, no. 1, pp. 34–47, Jan. 2018. doi: 10.1109/TPAMI.2017.2652468
    [23]
    J. Ma, J. Zhao, J. Tian, X. Bai, and Z. Tu, “Regularized vector field learning with sparse approximation for mismatch removal,” Pattern Recognit., vol. 46, no. 12, pp. 3519–3532, Dec. 2013. doi: 10.1016/j.patcog.2013.05.017
    [24]
    A. Fan, X. Jiang, Y. Ma, X. Mei, and J. Ma, “Smoothness-driven consensus based on compact representation for robust feature matching,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 8, pp. 4460–4472, Aug. 2023. doi: 10.1109/TNNLS.2021.3118409
    [25]
    Y. Lu, J. Ma, L. Fang, X. Tian, and J. Jiang, “Robust and scalable Gaussian process regression and its applications,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Vancouver, Canada, 2023, pp. 21950–21959.
    [26]
    E. M. Loiola, N. M. M. de Abreu, P. O. Boaventura-Netto, P. Hahn, and T. Querido, “A survey for the quadratic assignment problem,” Eur. J. Oper. Res., vol. 176, no. 2, pp. 657–690, Jan. 2007. doi: 10.1016/j.ejor.2005.09.032
    [27]
    B. Jiang, J. Tang, C. Ding, and B. Luo, “Binary constraint preserving graph matching,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 550–557.
    [28]
    S. Lee, J. Lim, and I. H. Suh, “Progressive feature matching: Incremental graph construction and optimization,” IEEE Trans. Image Process., vol. 29, pp. 6992–7005, May 2022.
    [29]
    J. Bian, W.-Y. Lin, Y. Matsushita, S.-K. Yeung, T.-D. Nguyen, and M.-M. Cheng, “GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 2828–2837.
    [30]
    L. Cavalli, V. Larsson, M. R. Oswald, T. Sattler, and M. Pollefeys, “Handcrafted outlier detection revisited,” in Proc. 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 770–787.
    [31]
    R. Q. Charles, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 77–85.
    [32]
    J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, Y. Chen, H. Liao, and L. Quan, “Learning two-view correspondences and geometry using order-aware network,” in Proc. IEEE/CVF Int. Conf. Computer Vision, Seoul, Korea, 2019, pp. 5844–5853.
    [33]
    C. Zhao, Y. Ge, F. Zhu, R. Zhao, H. Li, and M. Salzmann, “Progressive correspondence pruning by consensus learning,” in Proc. IEEE/CVF Int. Conf. Computer Vision, Montreal, Canada, 2021, pp. 6444–6453.
    [34]
    W. Sun, W. Jiang, E. Trulls, A. Tagliasacchi, and K. M. Yi, “ACNe: Attentive context normalization for robust permutation-equivariant learning,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 11283–11292.
    [35]
    C. Zhao, Z. Cao, C. Li, X. Li, and J. Yang, “NM-net: Mining reliable neighbors for robust feature correspondences,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp. 215–224.
    [36]
    P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learning feature matching with graph neural networks,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 4937–4946.
    [37]
    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. 31st Int. Conf. Neural Information Processing Systems, Long Beach, USA, 2017, pp. 6000–6010.
    [38]
    C. Wang, N. Komodakis, and N. Paragios, “Markov random field modeling, inference and learning in computer vision and image understanding: A survey,” Comput. Vision Image Understanding, vol. 117, no. 11, pp. 1610–1627, Nov. 2013. doi: 10.1016/j.cviu.2013.07.004
    [39]
    A. Shekhovtsov, I. Kovtun, and V. Hlaváç, “Efficient MRF deformation model for non-rigid image matching,” Comput. Vision Image Understanding, vol. 112, no. 1, pp. 91–99, Oct. 2008. doi: 10.1016/j.cviu.2008.06.006
    [40]
    V. Kolmogorov and R. Zabin, “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Analy. Mach. Intell., vol. 26, no. 2, pp. 147–159, Feb. 2004. doi: 10.1109/TPAMI.2004.1262177
    [41]
    L. Torresani, V. Kolmogorov, and C. Rother, “Feature correspondence via graph matching: Models and global optimization,” in Proc. 10th European Conf. Computer Vision, Marseille, France, 2008, pp. 596–609.
    [42]
    Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Analy. Mach. Intell., vol. 23, no. 11, pp. 1222–1239, Nov. 2001. doi: 10.1109/34.969114
    [43]
    K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool, “A comparison of affine region detectors,” Int. J. Comput. Vision, vol. 65, no. 1, pp. 43–72, Oct. 2005.
    [44]
    J.-M. Morel and G. Yu, “ASIFT: A new framework for fully affine invariant image comparison,” SIAM J. Imag. Sci., vol. 2, no. 2, pp. 438–469, Apr. 2009. doi: 10.1137/080732730
    [45]
    J. L. Bentley, “Multidimensional binary search trees used for associative searching,” Commun. ACM, vol. 18, no. 9, pp. 509–517, Sept. 1975. doi: 10.1145/361002.361007
    [46]
    Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Analy. Mach. Intell., vol. 26, no. 9, pp. 1124–1137, Sept. 2004. doi: 10.1109/TPAMI.2004.60
    [47]
    J. Ma, J. Jiang, H. Zhou, J. Zhao, and X. Guo, “Guided locality preserving feature matching for remote sensing image registration,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 8, pp. 4435–4447, Aug. 2018. doi: 10.1109/TGRS.2018.2820040
    [48]
    H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl, “Large-scale data for multiple-view stereopsis,” Int. J. Comput. Vision, vol. 120, no. 2, pp. 153–168, Apr. 2016. doi: 10.1007/s11263-016-0902-9
    [49]
    H. S. Wong, T.-J. Chin, J. Yu, and D. Suter, “Dynamic and hierarchical multi-structure geometric model fitting,” in Proc. Int. Conf. Computer Vision, Barcelona, Spain, 2011, pp. 1044–1051.
    [50]
    L. Liang, W. Zhao, X. Hao, Y. Yang, K. Yang, L. Liang, and Q. Yang, “Image registration using two-layer cascade reciprocal pipeline and context-aware dissimilarity measure,” Neurocomputing, vol. 371, pp. 1–14, Jan. 2020. doi: 10.1016/j.neucom.2019.06.101
    [51]
    M. Horst and R. Möller, “Visual place recognition for autonomous mobile robots,” Robotics, vol. 6, no. 2, p. 9, Apr. 2017. doi: 10.3390/robotics6020009
    [52]
    J. Ma, A. Fan, X. Jiang, and G. Xiao, “Feature matching via motion-consistency driven probabilistic graphical model,” Int. J. Comput. Vision, vol. 130, no. 9, pp. 2249–2264, Jul. 2022. doi: 10.1007/s11263-022-01644-2
    [53]
    G. Wang and Y. Chen, “Robust feature matching using guided local outlier factor,” Pattern Recognit., vol. 117, p. 107986, Sept. 2021. doi: 10.1016/j.patcog.2021.107986
    [54]
    L. Zheng, G. Xiao, Z. Shi, S. Wang, and J. Ma, “MSA-net: Establishing reliable correspondences by multiscale attention network,” IEEE Trans. Image Process., vol. 31, pp. 4598–4608, Jul. 2022. doi: 10.1109/TIP.2022.3186535
    [55]
    B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “YFCC100M: The new data in multimedia research,” Commun. ACM, vol. 59, no. 2, pp. 64–73, Feb. 2016. doi: 10.1145/2812802
    [56]
    J. Xiao, A. Owens, and A. Torralba, “SUN3D: A database of big spaces reconstructed using SFM and object labels,” in Proc. IEEE Int. Conf. Computer Vision, Sydney, Australia, 2013, pp. 1625–1632.
    [57]
    J. Heinly, J. L. Schönberger, E. Dunn, and J.-M. Frahm, “Reconstructing the world in six days,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, USA, 2015, pp. 3287–3295.
    [58]
    D. Mishkin, J. Matas, and M. Perdoch, “MODS: Fast and robust method for two-view matching,” Comput. Vision Image Understanding, vol. 141, pp. 81–93, Dec. 2015. doi: 10.1016/j.cviu.2015.08.005
    [59]
    V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk, “HPatches: A benchmark and evaluation of handcrafted and learned local descriptors,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 3852–3861.
    [60]
    J.-W. Bian, Y.-H. Wu, J. Zhao, Y. Liu, L. Zhang, M.-M. Cheng, and I. Reid, “An evaluation of feature matchers for fundamental matrix estimation,” in Proc. 30th British Machine Vision Conf., Cardiff, UK, 2019, pp. 25.
    [61]
    A. Mishchuk, D. Mishkin, F. Radenović, and J. Matas, “Working hard to know your neighbor’s margins: Local descriptor learning loss,” in Proc. 31st Int. Conf. Neural Information Processing Systems, Long Beach, USA, 2017, pp. 4829–4840.
    [62]
    J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Vilamoura-Algarve, Portugal, 2012, pp. 573–580.
    [63]
    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Providence, USA, 2012, pp. 3354–3361.
    [64]
    A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and temples: Benchmarking large-scale scene reconstruction,” ACM Trans. Graphics, vol. 36, no. 4, p. 78, Jul. 2017.
    [65]
    K. Wilson and N. Snavely, “Robust global translations with 1DSfM,” in Proc. 13th European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 61–75.
    [66]
    R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval. New York, USA: ACM Press, 1999.
    [67]
    M. Cummins and P. Newman, “FAB-MAP: Probabilistic localization and mapping in the space of appearance,” Int. J. Rob. Res., vol. 27, no. 6, pp. 647–665, Jun. 2008. doi: 10.1177/0278364908090961
    [68]
    X. Jiang, J. Ma, G. Xiao, Z. Shao, and X. Guo, “A review of multimodal image matching: Methods and applications,” Inf. Fusion, vol. 73, pp. 22–71, Sept. 2021. doi: 10.1016/j.inffus.2021.02.012
    [69]
    L. Tang, Y. Deng, Y. Ma, J. Huang, and J. Ma, “SuperFusion: A versatile image registration and fusion network with semantic awareness,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 12, pp. 2121–2137, Dec. 2022. doi: 10.1109/JAS.2022.106082
    [70]
    E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,” in Proc. 9th European Conf. Computer Vision, Graz, Austria, 2006, pp. 430–443.
    [71]
    J. Li, Q. Hu, and M. Ai, “RIFT: Multi-modal image matching based on radiation-variation insensitive feature transform,” IEEE Trans. Image Process., vol. 29, pp. 3296–3310, Jan. 2020. doi: 10.1109/TIP.2019.2959244
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(15)  / Tables(4)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (478) PDF downloads(94) Cited by()

    Highlights

    • A graph interaction model is proposed for feature matching problem
    • We show the proposed model can be solved globally using graph-cut technique
    • A topology-aware relationship is designed to capture the geometric relationship
    • We apply our method to several vision tasks and demonstrate its superiority

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return