A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 8 Issue 2
Feb.  2021

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77, TOP 5
Turn off MathJax
Article Contents
Yuzhen Liu, Ziyang Meng, Yao Zou and Ming Cao, "Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 344-360, Feb. 2021. doi: 10.1109/JAS.2020.1003530
Citation: Yuzhen Liu, Ziyang Meng, Yao Zou and Ming Cao, "Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 344-360, Feb. 2021. doi: 10.1109/JAS.2020.1003530

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments

doi: 10.1109/JAS.2020.1003530
Funds:  This work was supported in part by the Institute for Guo Qiang of Tsinghua University (2019GQG1023), in part by Graduate Education and Teaching Reform Project of Tsinghua University (202007J007), in part by National Natural Science Foundation of China (U19B2029, 62073028, 61803222), and in part by the Independent Research Program of Tsinghua University (2018Z05JDX002)
More Information
  • There are two main trends in the development of unmanned aerial vehicle (UAV) technologies: miniaturization and intellectualization, in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems. In this paper, we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor. A lightweight visual module is integrated to enable object tracking capabilities, and a micro positioning deck is mounted to provide accurate pose estimation. In order to be robust against object appearance variations, a novel object tracking algorithm, denoted by RMCTer, is proposed, which integrates a powerful short-term tracking module and an efficient long-term processing module. In particular, the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner. Furthermore, a position-based visual servoing control method is proposed for the quadrotor, where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques. Stable and accurate object tracking is achieved even under disturbances. Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.


  • loading
  • 1 https://www.dji.com/cn
    2 https://wiki.bitcraze.io/
    3 In Algorithm 1, we test the sub profiles.
    4 http://www.optitrack.com/
  • [1]
    K. Fregene, “Unmanned aerial vehicles and control: Lockheed martin advanced technology laboratories,” IEEE Control Syst. Mag., vol. 32, no. 5, pp. 32–34, Oct. 2012. doi: 10.1109/MCS.2012.2205474
    X. Zhang, B. Xian, B. Zhao, and Y. Zhang, “Autonomous flight control of a nano quadrotor helicopter in a GPS-denied environment using on-board vision,” IEEE Trans. Ind. Electron., vol. 62, no. 10, pp. 6392–6403, Oct. 2015. doi: 10.1109/TIE.2015.2420036
    G. W. Cai, J. Dias, and L. Seneviratne, “A survey of small-scale unmanned aerial vehicles: Recent advances and future development trend,” Unmanned Syst., vol. 2, no. 2, pp. 175–199, Apr. 2014. doi: 10.1142/S2301385014300017
    V. Srisamosorn, N. Kuwahara, A. Yamashita, T. Ogata, and J. Ota, “Human-tracking system using quadrotors and multiple environmental cameras for face-tracking application,” Int. J. Adv. Robot. Syst., vol. 14, no. 5, pp. 1–18, Oct. 2017.
    A. Briod, J. C. Zufferey, and D. Floreano, “Optic-flow based control of a 46 g quadrotor,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Tokyo, Japan, 2013, pp. 149–158.
    D. Palossi, J. Singh, M. Magno, and L. Benini, “Target following on nano-scale unmanned aerial vehicles,” in Proc. 7th IEEE Int. Workshop on Advances in Sensors and Interfaces, Vieste, Italy, 2017, pp. 170–175.
    D. Floreano and R. J. Wood, “Science, technology and the future of small autonomous drones,” Nature, vol. 521, no. 7553, pp. 460–466, May 2015. doi: 10.1038/nature14542
    P. Serra, R. Cunha, T. Hamel, D. Cabecinhas, and C. Silvestre, “Landing of a quadrotor on a moving target using dynamic image-based visual servo control,” IEEE Trans. Robot., vol. 32, no. 6, pp. 1524–1535, Dec. 2016. doi: 10.1109/TRO.2016.2604495
    H. Cheng, L. S. Lin, Z. Q. Zheng, Y. W. Guan, and Z. C. Liu, “An autonomous vision-based target tracking system for rotorcraft unmanned aerial vehicles,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Vancouver, Canada, 2017, pp. 1732–1738.
    A. Rodriguez-Ramos, A. Alvarez-Fernandez, H. Bavle, P. Campoy, and J. P. How, “Vision-based multirotor following using synthetic learning techniques,” Sensors, vol. 19, no. 21, pp. 4794, Nov. 2019. doi: 10.3390/s19214794
    Y. J. Yin, X. G. Wang, D. Xu, F. F. Liu, Y. L. Wang, and W. Q. Wu, “Robust visual detection-learning-tracking framework for autonomous aerial refueling of UAVs,” IEEE Trans. Instrum. Meas., vol. 65, no. 3, pp. 510–521, Mar. 2016. doi: 10.1109/TIM.2015.2509318
    P. Campoy, J. F. Correa, I. Mondragón, I. Mondragón, C. Martínez, M. Olivares, L. Mejías, and J. Artieda, “Computer vision onboard UAVs for civilian tasks,” J. Intell. Robot. Syst., vol. 54, no. 1–3, pp. 105–134, 2009. doi: 10.1007/s10846-008-9256-z
    J. Pestana, J. L. Sanchez-Lopez, S. Saripalli, and P. Campoy, “Computer vision based general object following for GPS-denied multirotor unmanned vehicles,” in Proc. American Control Conf., Portland, USA, 2014, pp. 1886–1891.
    R. Li, M. Pang, C. Zhao, G. Y. Zhou, and L. Fang, “Monocular long-term target following on UAVs,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Las Vegas, USA, 2016, pp. 29–37.
    Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1409–1422, Jul. 2012. doi: 10.1109/TPAMI.2011.239
    C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in Proc. 13th European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 391–405.
    B. Babenko, M. H. Yang, and S. Belongie, “Robust object tracking with online multiple instance learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1619–1632, Aug. 2011. doi: 10.1109/TPAMI.2010.226
    S. Hare, S. Golodetz, A. Saffari, V. Vineet, M. M. Cheng, S. L. Hicks, and P. H. S. Torr, “Struck: Structured output tracking with kernels,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 10, pp. 2096–2109, Oct. 2016. doi: 10.1109/TPAMI.2015.2509974
    D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Francisco, USA, 2010, pp. 2544–2550.
    J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 3, pp. 583–596, Mar. 2015. doi: 10.1109/TPAMI.2014.2345390
    J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the circulant structure of tracking-by-detection with kernels,” in Proc. 12th European Conf. Computer Vision, Florence, Italy, 2012, pp. 702–715.
    M. Danelljan, G. Häger, F. S. Khan, and M. Felsberg, “Discriminative scale space tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 8, pp. 1561–1575, Aug. 2017. doi: 10.1109/TPAMI.2016.2609928
    J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv: 1804.02767, 2018.
    K. M. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 2, pp. 386–397, Feb. 2020. doi: 10.1109/TPAMI.2018.2844175
    D. L. Zheng, H. S. Wang, W. D. Chen, and Y. Wang, “Planning and tracking in image space for image-based visual servoing of a quadrotor,” IEEE Trans. Ind. Electron., vol. 65, no. 4, pp. 3376–3385, Apr. 2018. doi: 10.1109/TIE.2017.2752124
    F. Chaumette and S. Hutchinson, “Visual servo control. I. Basic approaches,” IEEE Robot. &Automat. Mag., vol. 13, no. 4, pp. 82–90, Dec. 2006.
    N. Guenard, T. Hamel, and R. Mahony, “A practical visual servo control for an unmanned aerial vehicle,” IEEE Trans. Robot., vol. 24, no. 2, pp. 331–340, Apr. 2008. doi: 10.1109/TRO.2008.916666
    M. G. Popova and H. H. Liu, “Position-based visual servoing for target tracking by a quadrotor UAV,” in Proc. AIAA Guidance, Navigation, and Control Conf., San Diego, USA, 2016, pp. 2092–2103.
    W. B. Zhao, H. Liu, F. L. Lewis, K. P. Valavanis, and X. L. Wang, “Robust visual servoing control for ground target tracking of quadrotors,” IEEE Trans. Control Syst. Technol., vol. 28, no. 5, pp. 1980–1987, Sept. 2020. doi: 10.1109/TCST.2019.2922159
    F. Rinaldi, S. Chiesa, and F. Quagliotti, “Linear quadratic control for quadrotors UAVs dynamics and formation flight,” J. Intell. &Robot. Syst., vol. 70, no. 1–4, pp. 203–220, Apr. 2013.
    D. Mellinger, M. Shomin, and V. Kumar, “Control of quadrotors for robust perching and landing,” in Proc. Int. Powered Lift Conf., Philadelphia, USA, 2010, pp. 205–225.
    Y. Z. Liu and Z. Y. Meng, “Visual object tracking for a nano-scale quadrotor,” in Proc. 15th Int. Conf. Control, Automation, Robotics and Vision, Singapore, 2018, pp. 843–847.
    T. Yang, N. Sun, H. Chen, and Y. C. Fang, “Neural network-based adaptive antiswing control of an underactuated ship-mounted crane with roll motions and input dead zones,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 3, pp. 901–914, Mar. 2020. doi: 10.1109/TNNLS.2019.2910580
    N. Sun, D. K. Liang, Y. M. Wu, Y. H. Chen, Y. D. Qin, and Y. C. Fang, “Adaptive control for pneumatic artificial muscle systems with parametric uncertainties and unidirectional input constraints,” IEEE Trans. Ind. Informatics, vol. 16, no. 2, pp. 969–979, Feb. 2020. doi: 10.1109/TII.2019.2923715
    B. Zhao, B. Xian, Y. Zhang, and X. Zhang, “Nonlinear robust adaptive tracking control of a quadrotor UAV via immersion and invariance methodology,” IEEE Trans. Ind. Electron., vol. 62, no. 5, pp. 2891–2902, May 2015. doi: 10.1109/TIE.2014.2364982
    Y. Zou and W. Huo, “Adaptive tracking control for a model helicopter with disturbances,” in Proc. American Control Conf, Chicago, USA, 2015, pp. 3824–3829.
    P. Marantos, C. P. Bechlioulis, and K. J. Kyriakopoulos, “Robust trajectory tracking control for small-scale unmanned helicopters with model uncertainties,” IEEE Trans. Control Syst. Technol., vol. 25, no. 6, pp. 2010–2021, Nov. 2017. doi: 10.1109/TCST.2016.2642160
    M. W. Mueller, M. Hamer, and R. D’Andrea, “Fusing ultra-wideband range measurements with accelerometers and rate gyroscopes for quadrocopter state estimation,” in Proc. IEEE Int. Conf. Robotics & Automation, Seattle, USA, 2015, pp. 1730–1736.
    M. Greiff, “Modelling and control of the crazyflie quadrotor for aggressive and autonomous flight by optical flow driven state estimation,” M.S. thesis, Lund University, Sweden, 2017.
    Y. J. Li, “The design and implementation of four rotor UAV fixed landing system based on machine vision,” M.S. thesis, South China University of Technology, Guangzhou, China, 2015.
    R. Mahony, V. Kumar, and P. Corke, “Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor,” IEEE Robot. &Autom. Mag., vol. 19, no. 3, pp. 20–32, Sept. 2012.
    Y. Zou, “Trajectory tracking controller for quadrotors without velocity and angular velocity measurements,” IET Control Theory &Appl., vol. 11, no. 1, pp. 101–109, Jan. 2017.
    M. Huang, B. Xian, C. Diao, K. Y. Yang, and Y. Feng, “Adaptive tracking control of underactuated quadrotor unmanned aerial vehicles via backstepping,” in Proc. American Control Conf., Baltimore, USA, 2010, pp. 2076–2081.
    B. Zhu and W. Huo, “Robust nonlinear control for a model-scaled helicopter with parameter uncertainties,” Nonlinear Dyn., vol. 73, no. 1–2, pp. 1139–1154, Jul. 2013. doi: 10.1007/s11071-013-0858-z
    M. Krstić, I. Kanellakopoulos, and P. Kokotović, Nonlinear and Adaptive Control Design. New York, UK: Wiley, 1995.
    A. Levant, “Higher-order sliding modes, differentiation and output-feedback control,” Int. J. Control, vol. 76, no. 9–10, pp. 924–941, 2003. doi: 10.1080/0020717031000099029
    Y. Wu, J. Lim, and M. H. Yang, “Object tracking benchmark,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1834–1848, Sept. 2015. doi: 10.1109/TPAMI.2014.2388226


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(14)  / Tables(1)

    Article Metrics

    Article views (2231) PDF downloads(111) Cited by()


    • This paper proposes a complete visual object tracking and servoing control system using a tailor-made 38 g nano-scale quadrotor platform. This tracking system is composed of a versatile and robust visual object tracking module, and an efficient PBVS control module. Due to the limited payload, a lightweight monocular visual module is integrated to equip the quadrotor with the capability of object tracking. Additionally, we present a micro positioning deck to provide more stable and accurate pose estimation for the quadrotor.
    • This paper proposes a novel object tracking algorithm, i.e., RMCTer, where a two-stage short-term tracking module and an efficient long-term processing module are tightly integrated to collaboratively process the input frames. Compared with the tracking algorithms such as STUCK, DSST and KCF, the proposed tracker is more applicable in the presence of the variations of object appearance and can effectively compensate the visual tracking errors thanks to the adequate model modification provided by the long-term processing module.
    • This paper proposes an adaptive PBVS control algorithm by leveraging the backstepping and adaption techniques. The proposed controller is robust against the uncertain model parameters and the existence of external disturbances, and their exact model information is not needed in the design of the controller.


    DownLoad:  Full-Size Img  PowerPoint