A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 18.3, Top 1 (SCI Q1)
    CiteScore: 28.2, Top 1% (Q1)
    Google Scholar h5-index: 95， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2026  > In Press, Accepted Manuscript
L. Zhang, J. Fan, T. Chai, and F. L. Lewis, “Optimality-guaranteed model-free output feedback control using reinforcement learning,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2026.126134
Citation: L. Zhang, J. Fan, T. Chai, and F. L. Lewis, “Optimality-guaranteed model-free output feedback control using reinforcement learning,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2026.126134
shu

Optimality-Guaranteed Model-Free Output Feedback Control Using Reinforcement Learning

doi: 10.1109/JAS.2026.126134
Funds:  This work was supported in part by the Research Program of the Liaoning Liaohe Laboratory (LLL23ZZ-05-01), the National Natural Science Foundation of China (61991404), the Key Research and Development Program of Liaoning Province (2023JH26/10200011), China Academy of Engineering institute of Land Cooperation Consulting Project (2023-DFZD-60-02), the National Key Research and Development Program of China (2024YFB3309701), and the Science and Technology Major Project of Liaoning Province (2024JH1/11700048)
More Information
    Abstract
  • This paper investigates a data-driven, model-free optimal output feedback (OPFB) control approach for linear time-invariant (LTI) continuous-time (CT) systems. Existing data-driven OPFB control methods are hindered by the challenge that the optimality of the control policy cannot be guaranteed within the OPFB framework. To overcome this limitation, the paper introduces two novel algorithms aimed at ensuring the optimal OPFB control solution. First, by redefining the traditional performance index, an offline learning algorithm based on an augmented Hamiltonian function (AHF) is introduced. This algorithm ensures the optimality of the OPFB control solution while eliminating the need for an initial control gain. Second, to extend the algorithm’s applicability to systems with unknown dynamics, a data-driven, model-free approach leveraging off-policy integral reinforcement learning (IRL) is developed. This method operates solely on system data, demonstrating robustness against probing noise and ensuring unbiased learning outcomes. The proposed algorithms are validated through numerical simulation experiments using an F-16 aircraft system, confirming their effectiveness in practical scenarios.

     

    • FullText(HTML)
  • loading
    • References(59)
  • [1]
    B. Dong, T. An, F. Zhou, K. Liu, W. Yu, and Y. Li, “Actor-critic-identifier structure-based decentralized neuro-optimal control of modular robot manipulators with environmental collisions,” IEEE Access, vol. 7, pp. 96148–96165, Jul. 2019. doi: 10.1109/ACCESS.2019.2927511
    [2]
    X. Wang, W. Liu, Q. Wu, and S. Li, “A modular optimal formation control scheme of multiagent systems with application to multiple mobile robots,” IEEE Trans. Ind. Electron., vol. 69, no. 9, pp. 9331–9341, Sep. 2022. doi: 10.1109/TIE.2021.3114732
    [3]
    F. Lin and R. D. Brandt, “An optimal control approach to robust control of robot manipulators,” IEEE Trans. Rob. Autom., vol. 14, no. 1, pp. 69–77, Feb. 1998. doi: 10.1109/70.660845
    [4]
    W. Dai, T. Li, L. Zhang, Y. Jia, and H. Yan, “Multi-rate layered operational optimal control for large-scale industrial processes,” IEEE Trans. Ind. Inf., vol. 18, no. 7, pp. 4749–4761, Jul. 2022. doi: 10.1109/TII.2021.3105487
    [5]
    T. Bian, Y. Jiang, and Z.-P. Jiang, “Decentralized adaptive optimal control of large-scale systems with application to power systems,” IEEE Trans. Ind. Electron., vol. 62, no. 4, pp. 2439–2447, Apr. 2015. doi: 10.1109/TIE.2014.2345343
    [6]
    G. P. Padilla, S. Weiland, and M. C. F. Donkers, “A global optimal solution to the eco-driving problem,” IEEE Control Syst. Lett., vol. 2, no. 4, pp. 599–604, Oct. 2018. doi: 10.1109/LCSYS.2018.2846182
    [7]
    D. Wang, D. Liu, H. Li, B. Luo, and H. Ma, “An approximate optimal control approach for robust stabilization of a class of discrete-time nonlinear systems with uncertainties,” IEEE Trans. Syst. Man Cybern. Syst., vol. 46, no. 5, pp. 713–717, May 2016. doi: 10.1109/TSMC.2015.2466191
    [8]
    D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, “Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems,” IEEE Trans. Cybern., vol. 44, no. 12, pp. 2834–2847, Dec. 2014. doi: 10.1109/TCYB.2014.2357896
    [9]
    Y. Peng, Q. Chen, and W. Sun, “Reinforcement Q-learning algorithm for H∞ tracking control of unknown discrete-time linear systems,” IEEE Trans. Syst. Man Cybern. Syst., vol. 50, no. 11, pp. 4109–4122, Nov. 2020. doi: 10.1109/TSMC.2019.2957000
    [10]
    Q. Wei and D. Liu, “Optimal learning control for discrete-time nonlinear systems using generalized policy iteration based adaptive dynamic programming,” in Proc. 11th World Congr. Intelligent Control and Automation, Shenyang, China, 2014, pp. 1781−1786.
    [11]
    Y. Luo and G. Xiao, “ADP-based optimal control for a class of nonlinear discrete-time systems with inequality constraints,” in Proc. IEEE Symp. Adaptive Dynamic Programming and Reinforcement Learning, Orlando, USA, 2014, pp. 1−5.
    [12]
    Q. Wei, D. Liu, and H. Lin, “Value iteration adaptive dynamic programming for optimal control of discrete-time nonlinear systems,” IEEE Trans. Cybern., vol. 46, no. 3, pp. 840–853, Mar. 2016. doi: 10.1109/TCYB.2015.2492242
    [13]
    D. Liu and Q. Wei, “Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems,” IEEE Trans. Neural Networks Learn. Syst., vol. 25, no. 3, pp. 621–634, Mar. 2014. doi: 10.1109/TNNLS.2013.2281663
    [14]
    S. A. A. Rizvi and Z. Lin, “Output feedback optimal tracking control using reinforcement Q-learning,” in Proc. Ann. American Control Conf., Milwaukee, USA, 2018, pp. 3423−3428.
    [15]
    P. Tooranjipour and B. Kiumarsi, “Output feedback H∞ control of unknown discrete-time linear systems: Off-policy reinforcement learning,” in Proc. 60th IEEE Conf. Decision and Control, Austin, USA, 2021, pp. 2264−2269.
    [16]
    Y. Jiang, J. Fan, T. Chai, J. Li, and F. L. Lewis, “Data-driven flotation industrial process operational optimal control based on reinforcement learning,” IEEE Trans. Ind. Inf., vol. 14, no. 5, pp. 1974–1989, May 2018. doi: 10.1109/TII.2017.2761852
    [17]
    P. Shih, B. C. Kaul, S. Jagannathan, and J. A. Drallmeier, “Reinforcement-learning-based output-feedback control of nonstrict nonlinear discrete-time systems with application to engine emission control,” IEEE Trans. Syst. Man Cybern. B Cybern., vol. 39, no. 5, pp. 1162–1179, Oct. 2009. doi: 10.1109/TSMCB.2009.2013272
    [18]
    L. Zhang, J. Fan, W. Xue, V. G. Lopez, J. Li, T. Chai, and F. L. Lewis, “Data-driven H∞ optimal output feedback control for linear discrete-time systems based on off-policy Q-learning,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 7, pp. 3553–3567, Jul. 2023. doi: 10.1109/TNNLS.2021.3112457
    [19]
    S. J. Bradtke, B. E. Ydstie, and A. G. Barto, “Adaptive linear quadratic control using policy iteration,” in Proc. American Control Conf., Baltimore, USA, 1994, pp. 3475−3479.
    [20]
    Z. Ni, H. He, and J. Wen, “Adaptive learning in tracking control based on the dual critic network design,” IEEE Trans. Neural Networks Learn. Syst., vol. 24, no. 6, pp. 913–928, Jul. 2013. doi: 10.1109/TNNLS.2013.2247627
    [21]
    P. J. Werbos, “Neural networks for control and system identification,” in Proc. 28th IEEE Conf. Decision and Control, Tampa, USA, 1989, pp. 260−265.
    [22]
    J. Li, D. Yuan, and Z. Ding, “Optimal tracking control for discrete-time systems by model-free off-policy Q-learning approach,” in Proc. 11th Asian Control Conf., Gold Coast, Australia, 2017, pp. 7−12.
    [23]
    L. Li, L. Wang, Y. Yang, J. Dong, Y. Yin, and S. Cheng, “Model-free solution to the discrete-time coupled Riccati equation using off-policy reinforcement learning,” in Proc. Chinese Control Conf., Guangzhou, China, 2019, pp. 6813−6818.
    [24]
    B. Pang, T. Bian, and Z.-P. Jiang, “Data-driven finite-horizon optimal control for linear time-varying discrete-time systems,” in Proc. IEEE Conf. Decision and Control, Miami, USA, 2018, pp. 861−866.
    [25]
    S. A. A. Rizvi and Z. Lin, “Output feedback reinforcement learning control for the continuous-time linear quadratic regulator problem,” in Proc. Ann. American Control Conf., Milwaukee, USA, 2018, pp. 3417−3422.
    [26]
    D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based on policy iteration,” Automatica, vol. 45, no. 2, pp. 477–484, Feb. 2009. doi: 10.1016/j.automatica.2008.08.017
    [27]
    S. K. Jha and S. Bhasin, “Adaptive linear quadratic regulator for continuous-time systems with uncertain dynamics,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 3, pp. 833–841, May 2020. doi: 10.1109/jas.2019.1911438
    [28]
    Y. Jiang and Z.-P. Jiang, “Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics,” Automatica, vol. 48, no. 10, pp. 2699–2704, Oct. 2012. doi: 10.1016/j.automatica.2012.06.096
    [29]
    J. Y. Lee, J. B. Park, and Y. H. Choi, “Model-free approximate dynamic programming for continuous-time linear systems,” in Proc. 48th IEEE Conf. Decision and Control Held Jointly with 28th Chinese Control Conf., Shanghai, China, 2009, pp. 5009−5014.
    [30]
    F. L. Lewis and V. L. Syrmos, Optimal Control. New York, USA: Wiley, 1995.
    [31]
    K. G. Vamvoudakis, “Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach,” Syst. Control Lett., vol. 100, pp. 14–20, Feb. 2017. doi: 10.1016/j.sysconle.2016.12.003
    [32]
    Z. Shi and Z. Wang, “Adaptive output-feedback optimal control for continuous-time linear systems based on adaptive dynamic programming approach,” Neurocomputing, vol. 438, pp. 334–344, May 2021. doi: 10.1016/j.neucom.2021.01.070
    [33]
    B. Gravell, P. M. Esfahani, and T. Summers, “Learning optimal controllers for linear systems with multiplicative noise via policy gradient,” IEEE Trans. Autom. Control, vol. 66, no. 11, pp. 5283–5298, Nov. 2021. doi: 10.1109/TAC.2020.3037046
    [34]
    Y. Jiang and Z.-P. Jiang, “Approximate dynamic programming for output feedback control,” in Proc. 29th Chinese Control Conf., Beijing, China, 2010, pp. 5815−5820.
    [35]
    Z.-P. Jiang and Y. Jiang, “Robust adaptive dynamic programming for linear and nonlinear systems: An overview,” Eur. J. Control, vol. 19, no. 5, pp. 417–425, Sep. 2013. doi: 10.1016/j.ejcon.2013.05.017
    [36]
    D. Kleinman, “On an iterative technique for Riccati equation computations,” IEEE Trans. Autom. Control, vol. 13, no. 1, pp. 114–115, Feb. 1968. doi: 10.1109/tac.1968.1098829
    [37]
    H. Shi, J. Ma, Q. Liu, J. Li, X. Jiang, and P. Li, “Model-free output feedback optimal tracking control for two-dimensional batch processes,” Eng. Appl. Artif. Intell., vol. 143, Art. no. 109989, Mar. 2025. doi: 10.1016/j.engappai.2024.109989
    [38]
    Y.-S. Ma, J. Sun, Y. Xu, and S.-S. Cui, “Data-driven optimal output feedback control of unknown system model via adaptive dynamic programming,” IEEE Trans. Autom. Sci. Eng., vol. 22, pp. 19187–19196, Jul. 2025. doi: 10.1109/TASE.2025.3593481
    [39]
    S. Ohtake and M. Yamakita, “Adaptive output optimal control algorithm for unknown system dynamics based on policy iteration,” in Proc. American Control Conf., Baltimore, USA, 2010, pp. 1671−1676.
    [40]
    D. Moerder and A. Calise, “Convergence of a numerical algorithm for calculating optimal output feedback gains,” IEEE Trans. Autom. Control, vol. 30, no. 9, pp. 900–903, Sep. 1985. doi: 10.1109/TAC.1985.1104073
    [41]
    Y. G. Srinivasa and T. Rajagopalan, “Algorithms for the computation of optimal output feedback gains,” in Proc. 18th IEEE Conf. Decision and Control Including the Symp. Adaptive Processes, Fort Lauderdale, USA, 1979, pp. 576−579.
    [42]
    S. Choi and H. Sirisena, “Computation of optimal output feedback gains for linear multivariable systems,” IEEE Trans. Autom. Control, vol. 19, no. 3, pp. 257–258, Jun. 1974. doi: 10.1109/TAC.1974.1100534
    [43]
    L. M. Zhu, H. Modares, G. O. Peen, F. L. Lewis, and B. Yue, “Adaptive suboptimal output-feedback control for linear systems using integral reinforcement learning,” IEEE Trans. Control Syst. Technol., vol. 23, no. 1, pp. 264–273, Jan. 2015. doi: 10.1109/TCST.2014.2322778
    [44]
    G. Gin and P. Misra, “Disturbance attenuation and H∞ optimization with linear output feedback control,” in Proc. 30th IEEE Conf. Decision and Control, Brighton, UK, 1991, pp. 2705−2709.
    [45]
    R. Moghadam and F. L. Lewis, “Output-feedback H∞ quadratic tracking control of linear systems using reinforcement learning,” Int. J. Adapt. Control Signal Process., vol. 33, no. 2, pp. 300–314, Feb. 2019. doi: 10.1002/acs.2830
    [46]
    W. Xue, B. Lian, J. Fan, T. Chai, and F. L. Lewis, “Inverse reinforcement learning for trajectory imitation using static output feedback control,” IEEE Trans. Cybern., vol. 54, no. 3, pp. 1695–1707, Mar. 2024. doi: 10.1109/TCYB.2023.3241015
    [47]
    W. Levine and M. Athans, “On the determination of the optimal constant output feedback gains for linear multivariable systems,” IEEE Trans. Autom. Control, vol. 15, no. 1, pp. 44–48, Feb. 1970. doi: 10.1109/TAC.1970.1099363
    [48]
    J. Brewer, “Kronecker products and matrix calculus in system theory,” IEEE Trans. Circuits Syst., vol. 25, no. 9, pp. 772–781, Sep. 1978. doi: 10.1109/TCS.1978.1084534
    [49]
    X. Yang, H. He, and Q. Wei, “An adaptive critic approach to event-triggered robust control of nonlinear systems with unmatched uncertainties,” Int. J. Robust Nonlinear Control, vol. 28, no. 10, pp. 3501–3519, Jul. 2018. doi: 10.1002/rnc.4096
    [50]
    B. L. Stevens and F. L. Lewis, "Aircraft Control and Simulation," Aircraft Engineering & Aerospace Technology, Leeds, UK: Emerald Publishing, 2004.
    [51]
    J. Zhao, Y. Lv, Z. Wang, and Z. Zhao, “Adaptive Q-learning based model-free H∞ control of continuous-time nonlinear systems: Theory and application,” IEEE Trans. Emerg. Top. Comput. Intell., vol. 9, no. 2, pp. 1143–1152, Apr. 2025. doi: 10.1109/TETCI.2024.3449870
    [52]
    J. Liang, X. Bu, L. Cui, and Z. Hou, “Event-triggered asymmetric bipartite consensus tracking for nonlinear multi-agent systems based on model-free adaptive control,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 3, pp. 662–672, Mar. 2023. doi: 10.1109/JAS.2022.106070
    [53]
    C. Qin, X. Qiao, J. Wang, D. Zhang, Y. Hou, and S. Hu, “Barrier-critic adaptive robust control of nonzero-sum differential games for uncertain nonlinear systems with state constraints,” IEEE Trans. Syst. Man Cybern. Syst., vol. 54, no. 1, pp. 50–63, Jan. 2024. doi: 10.1109/TSMC.2023.3302656
    [54]
    C. Qin, H. Zhang, and Y. Luo, “Model-free H∞ control design for unknown continuous-time linear system using adaptive dynamic programming,” Asian J. Control, vol. 18, no. 2, pp. 609–618, Mar. 2016. doi: 10.1002/asjc.1102
    [55]
    C. Qin, S. Hou, M. Pang, Z. Wang, and D. Zhang, “Reinforcement learning-based secure tracking control for nonlinear interconnected systems: An event-triggered solution approach,” Eng. Appl. Artif. Intell., vol. 161, Art. no. 112243, Dec. 2025. doi: 10.1016/j.engappai.2025.112243
    [56]
    C. Qin, M. Pang, Z. Wang, S. Hou, and D. Zhang, “Observer based fault tolerant control design for saturated nonlinear systems with full state constraints via a novel event-triggered mechanism,” Eng. Appl. Artif. Intell., vol. 161, Art. no. 112221, Dec. 2025. doi: 10.1016/j.engappai.2025.112221
    [57]
    C. Qin, K. Jiang, Y. Wang, T. Zhu, Y. Wu, and D. Zhang, “Event-triggered H∞ control for unknown constrained nonlinear systems with application to robot arm,” Appl. Math. Modell., vol. 144, Art. no. 116089, Aug. 2025. doi: 10.1016/j.apm.2025.116089
    [58]
    L. Chu and Y. Liu, “Adaptive event-triggered control of time-varying nonlinear systems: A tight and powerful strategy,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 11, pp. 2194–2206, Nov. 2025. doi: 10.1109/JAS.2025.125786
    [59]
    K. G. Vamvoudakis, “Event-triggered optimal adaptive control algorithm for continuous-time nonlinear systems,” IEEE/CAA J. Autom. Sinica, vol. 1, no. 3, pp. 282–293, Jul. 2014.
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(5)  / Tables(1)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (299) PDF downloads(93) Cited by()

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return