A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 8 Issue 2
Feb.  2021

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
Qiyue Wang, Wenhua Jiao, Peng Wang and YuMing Zhang, "Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 334-343, Feb. 2021. doi: 10.1109/JAS.2020.1003518
Citation: Qiyue Wang, Wenhua Jiao, Peng Wang and YuMing Zhang, "Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 334-343, Feb. 2021. doi: 10.1109/JAS.2020.1003518

Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis

doi: 10.1109/JAS.2020.1003518
More Information
  • This paper presents an innovative investigation on prototyping a digital twin (DT) as the platform for human-robot interactive welding and welder behavior analysis. This human-robot interaction (HRI) working style helps to enhance human users’ operational productivity and comfort; while data-driven welder behavior analysis benefits to further novice welder training. This HRI system includes three modules: 1) a human user who demonstrates the welding operations offsite with her/his operations recorded by the motion-tracked handles; 2) a robot that executes the demonstrated welding operations to complete the physical welding tasks onsite; 3) a DT system that is developed based on virtual reality (VR) as a digital replica of the physical human-robot interactive welding environment. The DT system bridges a human user and robot through a bi-directional information flow: a) transmitting demonstrated welding operations in VR to the robot in the physical environment; b) displaying the physical welding scenes to human users in VR. Compared to existing DT systems reported in the literatures, the developed one provides better capability in engaging human users in interacting with welding scenes, through an augmented VR. To verify the effectiveness, six welders, skilled with certain manual welding training and unskilled without any training, tested the system by completing the same welding job; three skilled welders produce satisfied welded workpieces, while the other three unskilled do not. A data-driven approach as a combination of fast Fourier transform (FFT), principal component analysis (PCA), and support vector machine (SVM) is developed to analyze their behaviors. Given an operation sequence, i.e., motion speed sequence of the welding torch, frequency features are firstly extracted by FFT and then reduced in dimension through PCA, which are finally routed into SVM for classification. The trained model demonstrates a 94.44% classification accuracy in the testing dataset. The successful pattern recognition in skilled welder operations should benefit to accelerate novice welder training.


  • loading
  • [1]
    H. S. Kang, J. Y. Lee, S. Choi, H. Kim, J. H. Park, J. Y. Son, B. H. Kim, and S. D. Noh, “Smart manufacturing: Past research, present findings, and future directions,” Int. J. Precis. Eng. Manuf. - Green Technol., vol. 3, no. 1, pp. 111–128, Jan. 2016. doi: 10.1007/s40684-016-0015-5
    N. Q. Wu, Z. W. Li, K. Barkaoui, X. O. Li, T. Murata, and M. C. Zhou, “IoT-based smart and complex systems: A guest editorial report,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 69–73, Jan. 2018. doi: 10.1109/JAS.2017.7510748
    A. Kusiak, “Smart manufacturing must embrace big data,” Nature, vol. 544, no. 7648, pp. 23–25, Apr. 2017. doi: 10.1038/544023a
    M. Ghahramani, Y. Qiao, M. C. Zhou, A. O. Hagan, and J. Sweeney, “AI-based modeling and data-driven evaluation for smart manufacturing processes,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 4, pp. 1026–1037, Jul. 2020. doi: 10.1109/JAS.2020.1003114
    Q. L. Qi and F. Tao, “Digital twin and big data towards smart manufacturing and industry 4.0: 360 degree comparison,” IEEE Access, vol. 6, pp. 3585–3593, Jan. 2018. doi: 10.1109/ACCESS.2018.2793265
    F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, “Digital twin in industry: State-of-the-art,” IEEE Trans. Ind. Inform., vol. 15, no. 4, pp. 2405–2415, Apr. 2019. doi: 10.1109/TII.2018.2873186
    S. Haag and R. Anderl, “Digital twin – Proof of concept,” Manuf. Lett., vol. 15, pp. 64–66, Jan. 2018. doi: 10.1016/j.mfglet.2018.02.006
    T. DebRoy, W. Zhang, J. Turner, and S. S. Babu, “Building digital twins of 3D printing machines,” Scr. Mater., vol. 135, pp. 119–124, Jul. 2017. doi: 10.1016/j.scriptamat.2016.12.005
    T. Mukherjee and T. DebRoy, “A digital twin for rapid qualification of 3D printed metallic components,” Appl. Mater. Today, vol. 14, pp. 59–65, Mar. 2019. doi: 10.1016/j.apmt.2018.11.003
    R. Soderberg, K. Warmefjord, J. S. Carlson, and L. Lindkvist, “Toward a digital twin for real-time geometry assurance in individualized production,” CIRP Ann., vol. 66, no. 1, pp. 137–140, 2017. doi: 10.1016/j.cirp.2017.04.038
    R. Cupek, M. Drewniak, A. Ziebinski, and M. Fojcik, ““Digital twins” for highly customized electronic devices – Case study on a rework operation,” IEEE Access, vol. 7, pp. 164127–164143, Nov. 2019. doi: 10.1109/ACCESS.2019.2950955
    H. Zhang, Q. Liu, X. Chen, D. Zhang, and J. W. Leng, “A digital twin-based approach for designing and multi-objective optimization of hollow glass production line,” IEEE Access, vol. 5, pp. 26901–26911, Oct. 2017. doi: 10.1109/ACCESS.2017.2766453
    R. Dong, C. Y. She, W. Hardjawana, Y. H. Li, and B. Vucetic, “Deep learning for hybrid 5G services in mobile edge computing systems: Learn from a digital twin,” IEEE Trans. Wirel. Commun., vol. 18, no. 10, pp. 4692–4707, Oct. 2019. doi: 10.1109/TWC.2019.2927312
    R. H. Guerra, R. Quiza, A. Villalonga, J. Arenas, and F. Castano, “Digital twin-based optimization for ultraprecision motion systems with backlash and friction,” IEEE Access, vol. 7, pp. 93462–93472, Jul. 2019. doi: 10.1109/ACCESS.2019.2928141
    R. L. Zhao, D. X. Yan, Q. Liu, J. W. Leng, J. F. Wan, X. Chen, and X. F. Zhang, “Digital twin-driven cyber-physical system for autonomously controlling of micro punching system,” IEEE Access, vol. 7, pp. 9459–9469, Jan. 2019. doi: 10.1109/ACCESS.2019.2891060
    Y. L. Fang, C. Peng, P. Lou, Z. D. Zhou, J. M. Hu, and J. W. Yan, “Digital-twin-based job shop scheduling toward smart manufacturing,” IEEE Trans. Ind. Inform., vol. 15, no. 12, pp. 6425–6435, Dec. 2019. doi: 10.1109/TII.2019.2938572
    B. Akteke-Ozturk, G. W. Weber, and G. Koksal, “Optimization of generalized desirability functions under model uncertainty,” Optimization, vol. 66, no. 12, pp. 2157–2169, Sept. 2017. doi: 10.1080/02331934.2017.1371167
    A. Ozmen, E. Kropat, and G. W. Weber, “Robust optimization in spline regression models for multi-model regulatory networks under polyhedral uncertainty,” Optimization, vol. 66, no. 12, pp. 2135–2155, Dec. 2017. doi: 10.1080/02331934.2016.1209672
    B. Schleich, N. Anwer, L. Mathieu, and S. Wartzack, “Shaping the digital twin for design and production engineering,” CIRP Ann., vol. 66, no. 1, pp. 141–144, 2017. doi: 10.1016/j.cirp.2017.04.040
    J. L. Wang, L. K. Ye, R. X. Gao, C. Li, and L. B. Zhang, “Digital twin for rotating machinery fault diagnosis in smart manufacturing,” Int. J. Prod. Res., vol. 57, no. 12, pp. 3920–3934, 2019. doi: 10.1080/00207543.2018.1552032
    P. Jain, J. Poon, J. P. Singh, C. Spanos, S. R. Sanders, and S. K. Panda, “A digital twin approach for fault diagnosis in distributed photovoltaic systems,” IEEE Trans. Power Electron., vol. 35, no. 1, pp. 940–956, Jan. 2020. doi: 10.1109/TPEL.2019.2911594
    Y. Liu, L. Zhang, Y. Yang, L. F. Zhou, L. Ren, F. Wang, R. Liu, Z. B. Pang, and M. J. Deen, “A novel cloud-based framework for the elderly healthcare services using digital twin,” IEEE Access, vol. 7, pp. 49088–49101, Apr. 2019. doi: 10.1109/ACCESS.2019.2909828
    F. Tao, M. Zhang, Y. S. Liu, and A. Y. C. Nee, “Digital twin driven prognostics and health management for complex equipment,” CIRP Ann., vol. 67, no. 1, pp. 169–172, 2018. doi: 10.1016/j.cirp.2018.04.055
    M. K. Zhou, J. F. Yan, and D. H. Feng, “Digital twin framework and its application to power grid online analysis,” CSEE J. Power Energy Syst., vol. 5, no. 3, pp. 391–398, Sept. 2019.
    S. M. M. Rahman, “Cyber-physical-social system between a humanoid robot and a virtual human through a shared platform for adaptive agent ecology,” IEEE/CAA J. Autom. Sinina, vol. 5, no. 1, pp. 190–203, Jan. 2018. doi: 10.1109/JAS.2017.7510760
    H. H. Zhang and S. K. Agrawal, “An active neck brace controlled by a joystick to assist head motion,” IEEE Robot. Autom. Lett., vol. 3, no. 1, pp. 37–43, Jan. 2018. doi: 10.1109/LRA.2017.2728858
    Z. Ma and P. Ben-Tzvi, “RML glove – An exoskeleton glove mechanism with haptics feedback,” IEEE/ASME Trans. Mechatronics, vol. 20, no. 2, pp. 641–652, Apr. 2015. doi: 10.1109/TMECH.2014.2305842
    G. L. Du, P. Zhang, and X. Liu, “Markerless human-manipulator interface using leap motion with interval Kalman filter and improved particle filter,” IEEE Trans. Ind. Inform., vol. 12, no. 2, pp. 694–704, Apr. 2016. doi: 10.1109/TII.2016.2526674
    K. Zinchenko, C. Y. Wu, and K. T. Song, “A study on speech recognition control for a surgical robot,” IEEE Trans. Ind. Inform., vol. 13, no. 2, pp. 607–615, Apr. 2017. doi: 10.1109/TII.2016.2625818
    J. I. Lipton, A. J. Fay, and D. Rus, “Baxter’s homunculus: Virtual reality spaces for teleoperation in manufacturing,” IEEE Robot. Autom. Lett., vol. 3, no. 1, pp. 179–186, Jan. 2018. doi: 10.1109/LRA.2017.2737046
    Q. Y. Wang, W. H. Jiao, R. Yu, M. T. Johnson, and Y. M. Zhang, “Virtual reality robot-assisted welding based on human intention recognition,” IEEE Trans. Autom. Sci. Eng., vol. 17, no. 2, pp. 799–808, Apr. 2020. doi: 10.1109/TASE.2019.2945607
    M. Borges, A. Symington, B. Coltin, T. Smith, and R. Ventura, “HTC vive: Analysis and accuracy improvement, ” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Madrid, Spain, 2018, pp. 2610–2615.
    D. Whitney, E. Rosen, D. Ullman, E. Phillips, and S. Tellex, “ROS reality: A virtual reality framework using consumer-grade hardware for ROS-enabled robots, ” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Madrid, Spain, 2018, pp. 1–9.
    L. K. Shi, C. D. Tong, T. Lan, and X. H. Shi, “Statistical process monitoring based on ensemble structure analysis,” IEEE/CAA J. Autom. Sinica, 2018. DOI: 10.1109/JAS.2017.7510877
    P. Y. Zhang, S. Shu, and M. C. Zhou, “An online fault detection model and strategies based on SVM-grid in clouds,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 2, pp. 445–456, Mar. 2018. doi: 10.1109/JAS.2017.7510817
    B. Eygi Erdogan, S. Özöğür-Akyüz, and P. Karadayı Ataş, “A novel approach for panel data: An ensemble of weighted functional margin SVM models,” Inf. Sci, 2019. DOI: 10.1016/j.ins.2019.02.045
    M. S. Erden and T. Tomiyama, “Identifying welding skills for training and assistance with robot,” Sci. Technol. Weld. Join., vol. 14, no. 6, pp. 523–532, 2009. doi: 10.1179/136217109X437150
    Y. K. Liu, Y. M. Zhang, and L. Kvidahl, “Skilled human welder intelligence modeling and control: Part I – Modeling,” Weld. J., vol. 93, no. 2, pp. 46-s–52-s, 2014.
    Y. K. Liu, Y. M. Zhang, and L. Kvidahl, “Skilled human welder intelligence modeling and control: Part Ⅱ – Analysis and control applications,” Weld. J., vol. 93, no. 5, pp. 162-s–170-s, 2014.
    A. J. Smola and B. Scholkopf, “A tutorial on support vector regression,” Stat. Comput., vol. 14, no. 3, pp. 199–222, Aug. 2004. doi: 10.1023/B:STCO.0000035301.49549.88
    T. P. Minka, “Automatic choice of dimensionality for PCA,” in Proc. 13th Int. Conf. Neural Information Processing Systems, Cambridge USA, 2000, pp. 577–583.
    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, no. 85, pp. 2825–2830, 2011.


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(11)  / Tables(3)

    Article Metrics

    Article views (8607) PDF downloads(148) Cited by()


    • A digital twin framework is developed for human-robot interactive welding.
    • Virtual reality enhances the interactive ability of the digital twins with users.
    • A data-driven method is developed for welder behaving pattern recognition.


    DownLoad:  Full-Size Img  PowerPoint