A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 18.3, Top 1 (SCI Q1)
    CiteScore: 28.2, Top 1% (Q1)
    Google Scholar h5-index: 95， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2026  > In Press, Accepted Manuscript
S. Yang, M. Ruan, Y. Yang, T. Zhang, and M. Fu, “Learnable reeds-shepp cost map-guided dual-critic reinforcement learning for generalizable autonomous parking,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2026.126437
Citation: S. Yang, M. Ruan, Y. Yang, T. Zhang, and M. Fu, “Learnable reeds-shepp cost map-guided dual-critic reinforcement learning for generalizable autonomous parking,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2026.126437
shu

Learnable Reeds-Shepp Cost Map-Guided Dual-Critic Reinforcement Learning for Generalizable Autonomous Parking

doi: 10.1109/JAS.2026.126437
Funds:  This work was supported by the National Natural Science Foundation of China (NSFC 62503049, 62233002), the Postdoctoral Fellowship Program of the China Postdoctoral Science Foundation (GZC20252478), and the China Postdoctoral Science Foundation (2025M774202)
More Information
    Abstract
  • Autonomous parking in diverse and constrained environments remains a key challenge for intelligent vehicles. Existing learning-based approaches often suffer from poor generalizability and rely heavily on handcrafted, distance-driven rewards, leading to inefficient and unnatural behaviors in complex spaces. This work proposes a Reeds-Shepp (RS) cost map-guided dual-critic reinforcement learning (RL) framework to address these issues. First, an attention-based learnable RS cost map is constructed to encode environment-dependent pose feasibility for motion planning. Second, to improve the effectiveness of the RL method, a dual-critic architecture is introduced, decoupling Euclidean distance-based rewards from heuristic-guided feasibility-aware evaluations. The distance-based reward facilitates efficient task completion, while the heuristic-guided reward incorporates global maneuverability cost, guiding the agent to select feasible yet potentially non-intuitive maneuvers. Furthermore, a curriculum learning strategy is employed to accelerate convergence and enhance parking precision, particularly in confined parallel slots. Extensive experiments across multiple parking slot types, including parallel, vertical, and slant, demonstrate that the proposed method achieves a 97.3% average success rate, outperforming the baselines by up to 24.5%. Notably, for the most challenging parallel parking scenario, our method achieves a 92.9% success rate, which still represents an 8.3% improvement over the best baseline. The results in both simulation and real-vehicle experiments highlight the superior generalizability, safety, and convergence efficiency of the proposed method.

     

    • FullText(HTML)
  • loading
    • References(43)
  • [1]
    Y. Yang, M. Pan, S. Jiang, J. Wang, W. Wang, J. Wang, and M. Wang, “Towards autonomous parking using vision-only sensors,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Prague, Czech Republic, 2021, pp. 2038−2044.
    [2]
    G. Liu, K. Huang, X. Lv, Y. Sun, H. Li, X. Lei, Q. Yuan, and L. Shu, “Innovations and refinements in LiDAR odometry and mapping: A comprehensive review,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 6, pp. 1072−1094, Jun. 2025. doi: 10.1109/JAS.2025.125198
    [3]
    L. E. Dubins, “On curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents,” Am. J. Math., vol. 79, no. 3, pp. 497−516, Jul. 1957.
    [4]
    J. A. Reeds and L. A. Shepp, “Optimal paths for a car that goes both forwards and backwards,” Pac. J. Math., vol. 145, no. 2, pp. 367−393, Oct. 1990. doi: 10.2140/pjm.1990.145.367
    [5]
    D. Dolgov, S. Thrun, M. Montemerlo, and J. Diebel, “Path planning for autonomous vehicles in unknown semi-structured environments,” Int. J. Rob. Res., vol. 29, no. 5, pp. 485−501, Apr. 2010. doi: 10.1177/0278364909359210
    [6]
    S. M. LaValle, “Rapidly-exploring random trees: A new tool for path planning,” Comput. Sci. Dept., Iowa State Univ., Tech. Rep. TR 98-11, 1998.
    [7]
    Y. Shi, P. Wang, and X. Wang, “An autonomous valet parking algorithm for path planning and tracking,” in Proc. IEEE 96th Vehicular Technology Conf., London, United Kingdom, 2022, pp. 1−7.
    [8]
    M. Cai, W. Wu, and X. Zhou, “Trajectory tracking control for autonomous parking based on adaptive reduced-horizon model predictive control,” in Proc. IEEE Int. Conf. Networking, Sensing and Control, Shanghai, China, 2022, pp. 1−6.
    [9]
    Y. Wu, S. Li, Q. Zhang, K. Sun-Woo, and L. Yan, “Route planning and tracking control of an intelligent automatic unmanned transportation system based on dynamic nonlinear model predictive control,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 9, pp. 16576−16589, Sep. 2022. doi: 10.1109/TITS.2022.3141214
    [10]
    L. D’Alfonso, F. Giannini, G. Franzè, G. Fedele, F. Pupo, and G. Fortino, “Autonomous vehicle platoons in urban road networks: A joint distributed reinforcement learning and model predictive control approach,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 1, pp. 141−156, Jan. 2024. doi: 10.1109/JAS.2023.123705
    [11]
    C. Li, Z. Ji, Z. Chen, T. Qin, and M. Yang, “ParkingE2E: Camera-based end-to-end parking network, from images to planning,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Abu Dhabi, United Arab Emirates, 2024, pp. 13206−13212.
    [12]
    Y. Yang, D. Chen, T. Qin, X. Mu, C. Xu, and M. Yang, “E2E parking: Autonomous parking by the end-to-end neural network on the CARLA simulator,” in Proc. IEEE Intelligent Vehicles Symp., Jeju Island, Korea, Republic of, 2024, pp. 2375−2382.
    [13]
    P. Zhang, L. Xiong, Z. Yu, P. Fang, S. Yan, J. Yao, and Y. Zhou, “Reinforcement learning-based end-to-end parking for automatic parking system,” Sensors, vol. 19, no. 18, Art. no. 3996, Sep. 2019. doi: 10.3390/s19183996
    [14]
    S. Song, H. Chen, H. Sun, and M. Liu, “Data efficient reinforcement learning for integrated lateral planning and control in automated parking system,” Sensors, vol. 20, no. 24, Art. no. 7297, Dec. 2020. doi: 10.3390/s20247297
    [15]
    X. Tang, Y. Yang, T. Liu, X. Lin, K. Yang, and S. Li, “Path planning and tracking control for parking via soft actor-critic under non-ideal scenarios,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 1, pp. 181−195, Jan. 2024. doi: 10.1109/JAS.2023.123975
    [16]
    M. Jiang, Y. Li, S. Zhang, S. Chen, C. Wang, and M. Yang, “HOPE: A reinforcement learning-based hybrid policy path planner for diverse parking scenarios,” IEEE Trans. Intell. Transp. Syst., vol. 26, no. 5, pp. 6130−6141, May 2025. doi: 10.1109/TITS.2025.3550417
    [17]
    A. Suleman, M. U. Khan, Z. Kaleem, A. H. Alenezi, I. Shabbir, S. Coleri, and C. Yuen, “Reward-augmented reinforcement learning for continuous control in precision autonomous parking via policy optimization methods,” IEEE Trans. Veh. Technol., vol. 75, no. 8, pp. 15849−15861, Aug. 2026. doi: 10.1109/TVT.2026.3676611
    [18]
    Y. Zhong, S. Gan, B. Fan, J. Cheng, T. Chen, and Q. Shi, “Advancing narrow space parking with latent memory-based reinforcement learning,” in Proc. IEEE Intelligent Vehicles Symp., Cluj-Napoca, Romania, 2025, pp. 2273−2280.
    [19]
    X. Xu, Y. Xie, R. Li, Y. Zhao, R. Song, and W. Zhang, “Hierarchical reinforcement learning for autonomous parking based on kinematic constraints,” in Proc. IEEE Int. Conf. Robotics and Biomimetics, Bangkok, Thailand, 2024, pp. 274−279.
    [20]
    F. Tao, L. Paparusso, C. Gu, R. M. Koehler, C. Wu, X. Huang, C. Juette, D. Paz, and R. Liu, “Adapting reinforcement learning for path planning in constrained parking scenarios,” in Proc. 14th Int. Conf. Learning Representations, Rio de Janeiro, Brazil, 2026.
    [21]
    Y. Zhai, M. S. Selezneva, and H. Zhu, “Automatic parking trajectory optimization: A hybrid genetic algorithm with Q learning,” in Proc. Int. Russian Automation Conf., Sochi, Russian Federation, 2025, pp. 12−17.
    [22]
    S. Yang, M. Ruan, Q. Huang, Y. Yang, T. Zhang, and M. Fu, “Tackling narrow-space parallel parking: Reeds–Shepp-integrated reinforcement learning with learnable cost heuristic,” in Proc. IEEE Intelligent Vehicles Symp., Plymouth, USA, 2026, pp. 545−552.
    [23]
    Y. Liu and D. Li, “Hybrid reinforcement learning-based path planner for narrow vertical parking scenarios,” in Proc. 10th Int. Conf. Intelligent Informatics and Biomedical Sciences, Okinawa, Japan, 2025, pp. 377−383.
    [24]
    X. Shen, Y. Choi, A. Wong, F. Borrelli, S. Moura, and S. Woo, “Parking of connected automated vehicles: Vehicle control, parking assignment, and multi-agent simulation,” Int. J. Urban Sci., 2024, DOI: 10.1080/12265934.2026.2715995.
    [25]
    X. Wang and M. Althoff, “Safe reinforcement learning for automated vehicles via online reachability analysis,” IEEE Trans. Intell. Veh., vol. 9, no. 9, pp. 5490−5505, Sep. 2024. doi: 10.1109/TIV.2023.3344226
    [26]
    J. Jia, Y. Shu, Z. Gan, and W. Ding, “Learning occlusion-aware decision-making from agent interaction via active perception,” in Proc. IEEE Intelligent Vehicles Symp., Cluj-Napoca, Romania, 2025, pp. 1201−1208.
    [27]
    M. Mazouchi, S. Nageshrao, and H. Modares, “Conflict-aware safe reinforcement learning: A meta-cognitive learning framework,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 3, pp. 466−481, Mar. 2022. doi: 10.1109/JAS.2021.1004353
    [28]
    A. Abouelazm, J. Michel, and J. M. Zöllner, “A review of reward functions for reinforcement learning in the context of autonomous driving,” in Proc. IEEE Intelligent Vehicles Symp., Jeju Island, Korea, Republic of, 2024, pp. 156−163.
    [29]
    X. Chen, B. Xu, M. Hu, Y. Bian, Y. Li, and X. Xu, “Safe efficient policy optimization algorithm for unsignalized intersection navigation,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 9, pp. 2011−2026, Sep. 2024. doi: 10.1109/JAS.2024.124287
    [30]
    L. Chen, P. Deng, L. Li, and X. Hu, “Mixed motivation driven social multi-agent reinforcement learning for autonomous driving,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 6, pp. 1272−1282, Jun. 2025. doi: 10.1109/JAS.2025.125201
    [31]
    A. Sharif and D. Marijan, “Evaluating the robustness of deep reinforcement learning for autonomous policies in a multi-agent urban driving environment,” in Proc. IEEE 22nd Int. Conf. Software Quality, Reliability and Security, Guangzhou, China, 2022, pp. 785−796.
    [32]
    M. Zhu, Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke, “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving,” Transp. Res. Part C Emerging Technol., vol. 117, Art. no. 102662, Aug. 2020. doi: 10.1016/j.trc.2020.102662
    [33]
    R. Paleja, L. Chen, Y. Niu, A. Silva, Z. Li, S. Zhang, et al., “Interpretable reinforcement learning for robotics and continuous control,” arXiv preprint arXiv: 2311.10041, 2023.
    [34]
    B. Leng, R. Yu, W. Han, L. Xiong, Z. Li, and H. Huang, “Risk-aware reinforcement learning for autonomous driving: Improving safety when driving through intersection,” arXiv preprint arXiv: 2503.19690, 2025.
    [35]
    J. Yin, Z. Jiang, Q. Liang, W. Li, Z. Pan, H. Li, and J. Liu, “Efficient-enhanced reinforcement learning for autonomous driving in urban traffic scenarios,” in Proc. IEEE 26th Int. Conf. Intelligent Transportation Systems, Bilbao, Spain, 2023, pp. 887−893.
    [36]
    P. Polack, F. Altché, B. d’Andréa-Novel, and A. de La Fortelle, “The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?,” in Proc. IEEE Intelligent Vehicles Symp., Los Angeles, USA, 2017, pp. 812−818.
    [37]
    A. Yin, Y.-J. Yan, H.-L. Wang, S.-L. Tao, Y. Wang, and D.-W. Pi, “Research on automatic parking motion control based on a variable radius Reeds-Shepp method,” in Proc. 8th CAA Int. Conf. Vehicular Control and Intelligence, Chongqing, China, 2024, pp. 1−6.
    [38]
    Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. 26th Annu. Int. Conf. Machine Learning, Montreal, Canada, 2009, pp. 41−48.
    [39]
    H.-C. Wang, S.-C. Huang, P.-J. Huang, K.-L. Wang, Y.-C. Teng, Y.-T. Ko, D. Jeon, and I.-C. Wu, “Curriculum reinforcement learning from avoiding collisions to navigating among movable obstacles in diverse environments,” IEEE Robot. Autom. Lett., vol. 8, no. 5, pp. 2740−2747, May 2023. doi: 10.1109/LRA.2023.3251193
    [40]
    D. Hoeller, N. Rudin, D. Sako, and M. Hutter, “ANYmal parkour: Learning agile navigation for quadrupedal robots,” Sci. Robot., vol. 9, no. 88, Art. no. eadi7566, Mar. 2024. doi: 10.1126/scirobotics.adi7566
    [41]
    J. P. Carvalho and A. P. Aguiar, “Deep reinforcement learning for zero-shot coverage path planning with mobile robots,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 8, pp. 1594−1609, Aug. 2025. doi: 10.1109/JAS.2024.125064
    [42]
    J. Kang, J. Chen, M. Xu, Z. Xiong, Y. Jiao, L. Han, D. Niyato, Y. Tong, and S. Xie, “UAV-assisted dynamic avatar task migration for vehicular Metaverse services: A multi-agent deep reinforcement learning approach,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 2, pp. 430−445, Feb. 2024. doi: 10.1109/JAS.2023.123993
    [43]
    E. Leurent, “An environment for autonomous driving decision-making,” 2018. [Online]. Available: https://github.com/eleurent/highway-env
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(11)  / Tables(7)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (16) PDF downloads(8) Cited by()

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return