A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 11 Issue 10
Oct.  2024

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 15.3, Top 1 (SCI Q1)
    CiteScore: 23.5, Top 2% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
X. Li, S. Yu, Y. Lei, N. Li, and  B. Yang,  “Dynamic vision-based machinery fault diagnosis with cross-modality feature alignment,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 10, pp. 2068–2081, Oct. 2024. doi: 10.1109/JAS.2024.124470
Citation: X. Li, S. Yu, Y. Lei, N. Li, and  B. Yang,  “Dynamic vision-based machinery fault diagnosis with cross-modality feature alignment,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 10, pp. 2068–2081, Oct. 2024. doi: 10.1109/JAS.2024.124470

Dynamic Vision-Based Machinery Fault Diagnosis With Cross-Modality Feature Alignment

doi: 10.1109/JAS.2024.124470
Funds:  This work was supported by the National Science Fund for Distinguished Young Scholars of China (52025056), the China Postdoctoral Science Foundation (2023M732789), the China Postdoctoral Innovative Talents Support Program (BX20230290), and the Fundamental Research Funds for the Central Universities (xzy012022062)
More Information
  • Intelligent machinery fault diagnosis methods have been popularly and successfully developed in the past decades, and the vibration acceleration data collected by contact accelerometers have been widely investigated. In many industrial scenarios, contactless sensors are more preferred. The event camera is an emerging bio-inspired technology for vision sensing, which asynchronously records per-pixel brightness change polarity with high temporal resolution and low latency. It offers a promising tool for contactless machine vibration sensing and fault diagnosis. However, the dynamic vision-based methods suffer from variations of practical factors such as camera position, machine operating condition, etc. Furthermore, as a new sensing technology, the labeled dynamic vision data are limited, which generally cannot cover a wide range of machine fault modes. Aiming at these challenges, a novel dynamic vision-based machinery fault diagnosis method is proposed in this paper. It is motivated to explore the abundant vibration acceleration data for enhancing the dynamic vision-based model performance. A cross-modality feature alignment method is thus proposed with deep adversarial neural networks to achieve fault diagnosis knowledge transfer. An event erasing method is further proposed for improving model robustness against variations. The proposed method can effectively identify unseen fault mode with dynamic vision data. Experiments on two rotating machine monitoring datasets are carried out for validations, and the results suggest the proposed method is promising for generalized contactless machinery fault diagnosis.

     

  • loading
  • [1]
    Y. Lei, N. Li, and X. Li, Big Data-Driven Intelligent Fault Diagnosis and Prognosis for Mechanical Systems. Springer, 2022.
    [2]
    J. Huang, Z. Li, and Z. Zhou, “A simple framework to generalized zero-shot learning for fault diagnosis of industrial processes,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 6, pp. 1504–1506, 2023. doi: 10.1109/JAS.2023.123426
    [3]
    X. Gu, Y. Shang, Y. Kang, J. Li, Z. Mao, and C. Zhang, “An early minor-fault diagnosis method for lithiumion battery packs based on unsupervised learning,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 3, pp. 810–812, 2023. doi: 10.1109/JAS.2023.123099
    [4]
    W. Zhang and X. Li, “Data privacy preserving federated transfer learning in machinery fault diagnostics using prior distributions,” Structural Health Monitoring, vol. 21, no. 4, pp. 1329–1344, 2021.
    [5]
    Z. Zhang, C. Guan, H. Chen, X. Yang, W. Gong, and A. Yang, “Adaptive privacy-preserving federated learning for fault diagnosis in internet of ships,” IEEE Internet of Things J., vol. 9, no. 9, pp. 6844–6854, 2022. doi: 10.1109/JIOT.2021.3115817
    [6]
    B. Yang, Y. Lei, X. Li, and N. Li, “Targeted transfer learning through distribution barycenter medium for intelligent fault diagnosis of machines with data decentralization,” Expert Systems With Applic ations, vol. 244, p. 122997, 2024. doi: 10.1016/j.eswa.2023.122997
    [7]
    M. Xia, H. Shao, D. Williams, S. Lu, L. Shu, and C. W. de Silva, “Intelligent fault diagnosis of machinery using digital twin-assisted deep transfer learning,” Reliability Engineering and System Safety, vol. 215, p. 107938, 2021.
    [8]
    A. Glover, A. Dinale, L. D. S. Rosa, S. Bamford, and C. Bartolozzi, “luvHarris: A practical corner detector for event-cameras,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10087–10098, 2022. doi: 10.1109/TPAMI.2021.3135635
    [9]
    L. Wang, T. K. Kim, and K. J. Yoon, “Joint framework for single image reconstruction and super-resolution with an event camera,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7657–7673, 2022. doi: 10.1109/TPAMI.2021.3113352
    [10]
    L. Yu, X. Zhang, W. Liao, W. Yang, and G. S. Xia, “Learning to see through with events,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8660–8678, 2023.
    [11]
    Y. Li, Y. Chen, K. Zhu, C. Bai, and J. Zhang, “An effective federated learning verification strategy and its applications for fault diagnosis in industrial IoT systems,” IEEE Internet of Things J., vol. 9, no. 18, pp. 16835–16849, 2022. doi: 10.1109/JIOT.2022.3153343
    [12]
    X. Li, W. Zhang, X. Li, and H. Hao, “Partial domain adaptation in remaining useful life prediction with incomplete target data,” IEEE/ASME Trans. Mechatronics, vol. 29, no. 3, pp. 1903–1913, 2024.
    [13]
    B. Yang, Y. Lei, X. Li, N. Li, and A. K. Nandi, “Label recovery and trajectory designable network for transfer fault diagnosis of machines with incorrect annotation,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 4, pp. 932–945, 2024. doi: 10.1109/JAS.2024.124386
    [14]
    X. Li, C. Zhang, X. Li, and W. Zhang, “Federated transfer learning in fault diagnosis under data privacy with target self-adaptation,” J. Manufacturing Systems, vol. 68, pp. 523–535, 2023. doi: 10.1016/j.jmsy.2023.05.006
    [15]
    W. Zhang, M. Xu, H. Yang, X. Wang, S. Zheng, and X. Li, “Data-driven deep learning approach for thrust prediction of solid rocket motors,” Measurement, vol. 225, p. 114051, 2024. doi: 10.1016/j.measurement.2023.114051
    [16]
    T. Zhou, T. Han, and E. L. Droguett, “Towards trustworthy machine fault diagnosis: A probabilistic bayesian deep learning framework,” Reliability Engineering and System Safety, vol. 224, p. 108525, 2022.
    [17]
    J. Liu, C. Zhang, and X. Jiang, “Imbalanced fault diagnosis of rolling bearing using improved MsR-GAN and feature enhancement-driven CapsNet,” Mechanical Systems and Signal Processing, vol. 168, p. 108664, 2022. doi: 10.1016/j.ymssp.2021.108664
    [18]
    B. Yang, Y. Lei, X. Li, and C. Roberts, “Deep targeted transfer learning along designable adaptation trajectory for fault diagnosis across different machines,” IEEE Trans. Industrial Electronics, vol. 70, no. 9, pp. 9463–9473, 2023. doi: 10.1109/TIE.2022.3212415
    [19]
    T. Han, W. Xie, and Z. Pei, “Semi-supervised adversarial discriminative learning approach for intelligent fault diagnosis of wind turbine,” Information Sciences, vol. 648, p. 119496, 2023. doi: 10.1016/j.ins.2023.119496
    [20]
    T. Han and Y. Li, “Out-of-distribution detection-assisted trustworthy machinery fault diagnosis approach with uncertainty-aware deep ensembles,” Reliability Engineering System Safety, vol. 226, p. 108648, 2022. doi: 10.1016/j.ress.2022.108648
    [21]
    Y. Chen, M. Rao, K. Feng, and M. J. Zuo, “Physics-informed LSTM hyperparameters selection for gearbox fault detection,” Mechanical Systems and Signal Processing, vol. 171, p. 108907, 2022. doi: 10.1016/j.ymssp.2022.108907
    [22]
    Y. Chen, M. Rao, K. Feng, and G. Niu, “Modified varying index coefficient autoregression model for representation of the nonstationary vibration from a planetary gearbox,” IEEE Trans. Instrumentation and Measurement, vol. 72, pp. 1–12, 2023.
    [23]
    X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Intelligent machinery fault diagnosis with event-based camera,” IEEE Trans. Industrial Informatics, vol. 20, no. 1, pp. 380–389, 2024. doi: 10.1109/TII.2023.3262854
    [24]
    S. Guo and T. Delbruck, “Low cost and latency event camera background activity denoising,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 785–795, 2023. doi: 10.1109/TPAMI.2022.3152999
    [25]
    X. Chen, X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Dynamic vision enabled contactless cross-domain machine fault diagnosis with neuromorphic computing,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 788–790, 2024. doi: 10.1109/JAS.2023.124107
    [26]
    Y. Zhou, G. Gallego, X. Lu, S. Liu, and S. Shen, “Event-based motion segmentation with spatio-temporal graph cuts,” IEEE Trans. Neural Networks and Learning Systems, vol. 34, no. 8, pp. 4868–4880, 2023.
    [27]
    P. R. G. Cadena, Y. Qian, C. Wang, and M. Yang, “SPADE-E2VID: Spatially-adaptive denormalization for event-based video reconstruction,” IEEE Trans. Image Processing, vol. 30, pp. 2488–2500, 2021. doi: 10.1109/TIP.2021.3052070
    [28]
    C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-time face and eye tracking and blink detection using event cameras,” Neural Networks, vol. 141, pp. 87–97, 2021. doi: 10.1016/j.neunet.2021.03.019
    [29]
    Z. Chen, J. Wu, J. Hou, L. Li, W. Dong, and G. Shi, “ECSNet: Spatio-temporal feature learning for event camera,” IEEE Trans. Circuits and Systems for Video Technology, vol. 33, no. 2, pp. 701–712, 2023.
    [30]
    A. Rahate, R. Walambe, S. Ramanna, and K. Kotecha, “Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions,” Information Fusion, vol. 81, pp. 203–239, 2022. doi: 10.1016/j.inffus.2021.12.003
    [31]
    B. Goyal, A. Dogra, D. C. Lepcha, D. Koundal, A. Alhudhaif, F. Alenezi, and S. A. Althubiti, “Multi-modality image fusion for medical assistive technology management based on hybrid domain filtering,” Expert Systems With Applications, vol. 209, p. 118283, 2022. doi: 10.1016/j.eswa.2022.118283
    [32]
    L. A. Passos, J. P. Papa, J. D. Ser, A. Hussain, and A. Adeel, “Multimodal audio-visual information fusion using canonical-correlated graph neural network for energy-efficient speech enhancement,” Information Fusion, vol. 90, pp. 1–11, 2023. doi: 10.1016/j.inffus.2022.09.006
    [33]
    S. Qiu, H. Zhao, N. Jiang, et al., “Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges,” Information Fusion, vol. 80, pp. 241–265, 2022. doi: 10.1016/j.inffus.2021.11.006
    [34]
    J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multimodality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, pp. 5802–5811, 2022.
    [35]
    W. Li, R. Huang, J. Li, Y. Liao, Z. Chen, G. He, R. Yan, and K. Gryllias, “A perspective survey on deep transfer learning for fault diagnosis in industrial scenarios: Theories, applications and challenges,” Mechanical Systems and Signal Processing, vol. 167, p. 108487, 2022. doi: 10.1016/j.ymssp.2021.108487
    [36]
    G. Chakrapani and V. Sugumaran, “Transfer learning based fault diagnosis of automobile dry clutch system,” Engineering Applications of Artificial Intelligence, vol. 117, p. 105522, 2023. doi: 10.1016/j.engappai.2022.105522
    [37]
    W. Zhang, X. Li, H. Ma, Z. Luo, and X. Li, “Universal domain adaptation in fault diagnostics with hybrid weighted deep adversarial learning,” IEEE Trans. Industrial Informatics, vol. 17, no. 12, pp. 7957–7967, 2021.
    [38]
    S. Asutkar and S. Tallur, “Deep transfer learning strategy for efficient domain generalisation in machine fault diagnosis,” Scientific Reports, vol. 13, no. 1, p. 6607, 2023. doi: 10.1038/s41598-023-33887-5
    [39]
    S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked autoencoders,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, pp. 16133–16142, 2023.
    [40]
    W. Lu, B. Liang, Y. Cheng, D. Meng, J. Yang, and T. Zhang, “Deep model based domain adaptation for fault diagnosis,” IEEE Trans. Industrial Electronics, vol. 64, no. 3, pp. 2296–2305, 2017. doi: 10.1109/TIE.2016.2627020

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(13)  / Tables(8)

    Article Metrics

    Article views (225) PDF downloads(73) Cited by()

    Highlights

    • A contactless machine fault diagnosis method is proposed with dynamic vision
    • A cross-modality alignment method is proposed for vision and accelerometer data
    • An event erasing method is proposed to enhance model robustness
    • The proposed method can diagnose unseen machine fault with dynamic vision

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return