Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

Yuqiu Liu; Huanqian Yan; Xiaopei Zhu; Xiaolin Hu; Liang Tang; Hang Su; Chen Lv

doi:10.1109/JAS.2025.125438

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 19.2, Top 1 (SCI Q1)

CiteScore: 28.2, Top 1% (Q1)
Google Scholar h5-index: 95， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2026 > In Press, Accepted Manuscript

Y. Liu, H. Yan, X. Zhu, X. L. Hu, L. Tang, H. Su, and C. Lv, “Crafting physical adversarial examples by combining differentiable and physically based renders,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2025.125438

Citation:

Y. Liu, H. Yan, X. Zhu, X. L. Hu, L. Tang, H. Su, and C. Lv, “Crafting physical adversarial examples by combining differentiable and physically based renders,” IEEE/CAA J. Autom. Sinica, early access, 2026. doi: 10.1109/JAS.2025.125438

Citation:

PDF( 49007 KB)

Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

doi: 10.1109/JAS.2025.125438

Funds: This work was supported by the National Natural Science Foundation of China (U2341228)

More Information

Author Bio:
Yuqiu Liu received the M.S. degree in mechanical engineering from Beijing Forestry University, in 2024. She is currently a Ph.D. degree candidate at Simon Fraser University, Canada. Her research interests include physical adversarial attacks and 3D fluid reconstruction

Huanqian Yan received the Ph.D. degree in computer application and technology from Beihang University in 0000. He is now a Researcher at Qiyuan Lab, Beijing, China. His current research interests include object detection, adversarial examples, and clustering analysis, etc

Xiaopei Zhu received the Ph.D. degree in Tsinghua University in 2024. His current research interests include computer vision, adversarial example, generative model, 3D modeling, AI4sicence, etc. He has published several papers in CVPR, AAAI, ACM MM, etc

Xiaolin Hu (Senior Member, IEEE) received the B.E. and M.E. degrees in automotive engineering from the Wuhan University of Technology, in 2001 and 2004, respectively, and the Ph.D. degree in automation and computer-aided engineering from the Chinese University of Hong Kong, Hong Kong, China in 2007. He is currently an Associate Professor with the Department of Computer Science and Technology, Tsinghua University. His research interests include deep learning and computational neuroscience. Currently, he is an Associate Editor of IEEE Transactions on Pattern Analysis and Machine Intelligence, Cognitive Neurodynamics, CAAI Transactions on Intelligent Systems. Previouslyhe was an Associate Editor of IEEE Transactions on Image Processing and, IEEE Transactions on Neural Networks and Learning Systems

Liang Tang received the Ph.D. degree in mechanical engineering from the Department of Automotive Engineering, Tsinghua University in 2010. From 2015 to 2016, and 2018, she was a Visiting Scholar in University of Michigan, USA. Since 2014, she was a Professor in Beijing Forestry University. Her research interests mainly include vehicle physical adversarial attack, injury biomechanics in motor-vehicle crashes by a multidisciplinary approach. Her research has been funded by National Science Foundation, Beijing Science Foundation and Auto Industry

Hang Su (IEEE member) is an Associated Professor in the Department of Computer Science and Technology, Tsinghua University. His research interests include the adversarial machine learning and robust computer vision. He has published more than 50 papers including CVPR, ECCV, TMI, etc. He has served as Area Chair in NeurIPS and the workshop Co-Chair in AAAI2022. He received “Young Investigator Award” from MICCAI2012, the “Best Paper Award” in AVSS2012, and “Platinum Best Paper Award” in ICME2018

Chen Lv is an Associate Professor of the School of Mechanical and Aerospace Engineering, and the Cluster Director in Future Mobility Solutions, Nanyang Technological University, Singapore. He joined NTU and founded the Automated Driving and Human-Machine System (AutoMan) Research Lab since June 2018. His research focuses on intelligent vehicles, automated driving, and human-machine systems. He has published 4 books, over 100 papers, and obtained 12 granted patents. He serves as Associate Editor for IEEE T-ITS, IEEE TVT, and IEEE T-IV. He received many awards and honors, selectively including the IEEE IV Best Workshop/Special Session Paper Award in 2018, Automotive Innovation Best Paper Award in 2020, the winner of Waymo Open Dataset Challenges at CVPR 2021 and 2022, Machines Young Investigator Award, Nanyang Research Award (Young Investigator), SAE Ralph R. Teetor Educational Award, etc
Corresponding author: Liang Tang, e-mail: happyliang@bjfu.edu.cn; Hang Su, e-mail: suhangss@mail.tsinghua.edu.cn
¹ The dataset will be released after the paper is accepted.
Received Date: 2024-12-09
Revised Date: 2025-02-06
Accepted Date: 2025-03-30

Available Online: 2026-02-28

Abstract

Abstract

Recently we have witnessed progress in hiding road vehicles against object detectors through adversarial camouflage in the digital world. The extension of this technique to the physical world is crucial for testing the robustness of autonomous driving systems. However, existing methods do not show good performances when applied to the physical world. This is partly due to insufficient photorealism in training examples, and lack of proper physical realization methods for camouflage. To generate a robust adversarial camouflage suitable for real vehicles, we propose a novel method called PAV-Camou. We propose to adjust the mapping from the coordinates in the 2D map to those of corresponding 3D model. This process is critical for mitigating texture distortion and ensuring the camouflage’s effectiveness when applied in the real world. Then we combine two renderers with different characteristics to obtain adversarial examples that are photorealistic that closely mimic real-world lighting and texture properties. The method ensures that the generated textures remain effective under diverse environmental conditions. Our adversarial camouflage can be optimized and printed in the form of 2D patterns, allowing for direct application on real vehicles. Extensive experiments demonstrated that our proposed method achieved good performance in both the digital world and the physical world.
- Adversarial camouflage,
- neural rendering,
- object detection,
- physical adversarial attacks

FullText(HTML)

¹ The dataset will be released after the paper is accepted.

References(51)

References

[1]	I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proc. 3rd Int. Conf. Learning Representations, San Diego, USA, 2015, pp. 1−11.
[2]	N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Proc. IEEE Symp. Security and Privacy, San Jose, USA, 2017, pp. 39−57.
[3]	A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in Proc. 6th Int. Conf. Learning Representations, Vancouver, Canada, 2018, pp. 1−23.
[4]	Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 9185−9193.
[5]	D. Zhou, N. Wang, B. Han, and T. Liu, “Modeling adversarial noise for adversarial training,” in Proc. 39th Int. Conf. Machine Learning, Baltimore, USA, 2022, pp. 27353−27366.
[6]	L. Huang, C. Gao, Y. Zhou, C. Xie, A. L. Yuille, C. Zou, and N. Liu, “Universal physical camouflage attacks on object detectors,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 717−726.
[7]	K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P.-Y. Chen, Y. Wang, and X. Lin, “Adversarial T-shirt! Evading person detectors in a physical world,” in Proc. 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 665−681.
[8]	Z. Hu, S. Huang, X. Zhu, F. Sun, B. Zhang, and X. Hu, “Adversarial texture for fooling person detectors in the physical world,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, USA, 2022, pp. 13297−13306.
[9]	S. Thys, W. Van Ranst, and T. Goedemé, “Fooling automated surveillance cameras: Adversarial patches to attack person detection,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops, Long Beach, USA, 2019, pp. 49−55.
[10]	D. Wang, W. Yao, T. Jiang, W. Zhou, C. Li, and X. Chen, “Naturalistic-aware adversarial texture black-box attack for 3D objects against object detection in the physical world,” SSRN preprint SSRN:4545326, 2023.
[11]	J. Sun, W. Yao, T. Jiang, D. Wang, and X. Chen, “Differential evolution based dual adversarial camouflage: Fooling human eyes and object detectors,” Neural Netw., vol. 163, pp. 256–271, Jun. 2023. doi: 10.1016/j.neunet.2023.03.041
[12]	X. Zhu, Z. Hu, S. Huang, J. Li, and X. Hu, “Infrared invisible clothing: Hiding from infrared detectors at multiple angles in real world,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, USA, 2022, pp. 13307−13316.
[13]	X. Zhu, X. Li, J. Li, Z. Wang, and X. Hu, “Fooling thermal infrared pedestrian detectors in real world using small bulbs,” in Proc. 35th AAAI Conf. Artificial Intelligence, 2021, pp. 3616−3624.
[14]	H. Wei, Z. Wang, X. Jia, Y. Zheng, H. Tang, S. Satoh, and Z. Wang, “HOTCOLD block: Fooling thermal infrared detectors with a novel wearable design,” in Proc. 37th AAAI Conf. Artificial Intelligence, Washington, USA, 2023, pp. 15233−15241.
[15]	H. Wei, Z. Wang, K. Zhang, J. Hou, Y. Liu, H. Tang, and Z. Wang, “Revisiting adversarial patches for designing camera-agnostic attacks against person detection,” in Proc. 38th Int. Conf. Neural Information Processing Systems, Vancouver, Canada, 2024, pp. 259.
[16]	T. Wu, X. Ning, W. Li, R. Huang, H. Yang, and Y. Wang, “Physical adversarial attack on vehicle detector in the Carla simulator,” arXiv preprint arXiv: 2007.16118, 2020.
[17]	Z. Hu, W. Chu, X. Zhu, H. Zhang, B. Zhang, and X. Hu, “Physically realizable natural-looking clothing textures evade person detectors via 3D modeling,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Vancouver, Canada, 2023, pp. 16975−16984.
[18]	J. Wang, A. Liu, Z. Yin, S. Liu, S. Tang, and X. Liu, “Dual attention suppression attack: Generate adversarial camouflage in physical world,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, USA, 2021, pp. 8561−8570.
[19]	D. Wang, T. Jiang, J. Sun, W. Zhou, Z. Gong, X. Zhang, W. Yao, and X. Chen, “FCA: Learning a 3D full-coverage vehicle camouflage for multi-view physical adversarial attack,” in Proc. 36th AAAI Conf. Artificial Intelligence, 2022, pp. 2414−2422.
[20]	Y. Zhang, Z. Gong, Y. Zhang, K. Bin, Y. Li, J. Qi, H. Wen, and P. Zhong, “Boosting transferability of physical attack against detectors by redistributing separable attention,” Pattern Recognit., vol. 138, p. 109435, Jun. 2023. doi: 10.1016/j.patcog.2023.109435
[21]	Y. Duan, J. Chen, X. Zhou, J. Zou, Z. He, J. Zhang, W. Zhang, and Z. Pan, “Learning coated adversarial camouflages for object detectors,” in Proc. 31st Int. Joint Conf. Artificial Intelligence, Vienna, Austria, 2022, pp. 891−897.
[22]	N. Suryanto, Y. Kim, H. Kang, H. T. Larasati, Y. Yun, T.-T.-H. Le, H. Yang, S.-Y. Oh, and H. Kim, “DTA: Physical camouflage attacks using differentiable transformation network,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, USA, 2022, pp. 15284−15293.
[23]	N. Suryanto, Y. Kim, H. T. Larasati, H. Kang, T.-T.-H. Le, Y. Hong, H. Yang, S.-Y. Oh, and H. Kim, “ACTIVE: Towards highly transferable 3D physical camouflage for universal and robust vehicle evasion,” in Proc. IEEE/CVF Int. Conf. Computer Vision, Paris, France, 2023, pp. 4282−4291.
[24]	C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in Proc. 2nd Int. Conf. Learning Representations, 2013, pp. 1−10.
[25]	S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, USA, 2017, pp. 86−94.
[26]	S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “DeepFool: A simple and accurate method to fool deep neural networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, pp. 2574−2582.
[27]	J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” IEEE Trans. EComput., vol. 23, no. 5, pp. 828–841, Oct. 2019. doi: 10.1109/tevc.2019.2890858
[28]	A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing robust adversarial examples,” in Proc. 35th Int. Conf. Machine Learning, Stockholm, Sweden, 2018, pp. 284−293.
[29]	K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, “Robust physical-world attacks on deep learning visual classification,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 1625−1634.
[30]	J. Li, F. Schmidt, and Z. Kolter, “Adversarial camera stickers: A physical camera-based attack on deep learning systems,” in Proc. 36th Int. Conf. Machine Learning, Long Beach, USA, 2019, pp. 3896−3904.
[31]	R. Duan, X. Ma, Y. Wang, J. Bailey, A. K. Qin, and Y. Yang, “Adversarial camouflage: Hiding physical-world attacks with natural styles,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2020, pp. 997−1005.
[32]	S.-T. Chen, C. Cornelius, J. Martin, and D. H. Chau, “ShapeShifter: Robust physical adversarial attack on Faster R-CNN object detector,” in Proc. European Conf. Machine Learning and Knowledge Discovery in Databases, Dublin, Ireland, 2019, pp. 52−68.
[33]	Y. Zhang, H. Foroosh, P. David, and B. Gong, “CAMOU: Learning a vehicle camouflage for physical adversarial attack on object detectors in the wild,” in Proc. 7th Int. Conf. Learning Representations, New Orleans, LA, USA, 2019, pp. 1−20.
[34]	A. Sanders, An Introduction to Unreal Engine 4. New York, USA: A K Peters/CRC Press, 2016.
[35]	H. Kato, Y. Ushiku, and T. Harada, “Neural 3D mesh renderer,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 3907−3916.
[36]	J. Johnson, N. Ravi, J. Reizenstein, D. Novotny, S. Tulsiani, C. Lassner, and S. Branson, “Accelerating 3D deep learning with PyTorch3D,” in Proc. SIGGRAPH Asia Courses, 2020, pp. 10.
[37]	E. Meloni, M. Tiezzi, L. Pasqualini, M. Gori, and S. Melacci, “Messing up 3D virtual environments: Transferable adversarial 3D objects,” in Proc. 20th IEEE Int. Conf. Machine Learning and Applications, Pasadena, USA, 2021, pp. 1−8.
[38]	D. Derakhshani, Introducing Autodesk MAYA 2013. Indianapolis, USA: John Wiley & Sons, 2012.
[39]	M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition,” in Proc. ACM SIGSAC Conf. Computer and Communications Security, 2016, pp. 1528−1540.
[40]	T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. 13th European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 740−755.
[41]	J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Miami, USA, 2009, pp. 248−255.
[42]	A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Proc. 1st Annu. Conf. Robot Learning, Mountain View, USA, 2017, pp. 1−16.
[43]	S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Proc. 29th Int. Conf. Neural Information Processing Systems, Montreal, Canada, 2015, pp. 91−99.
[44]	J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv: 1804.02767, 2018.
[45]	W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot MultiBox detector,” in Proc. 14th European Conf. Computer Vision, Amsterdam, The Netherlands, 2016, pp. 21−37.
[46]	N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in Proc. 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 213−229.
[47]	H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum, “DINO: DETR with improved DeNoising anchor boxes for end-to-end object detection,” in Proc. 11th Int. Conf. Learning Representations, Kigali, Rwanda, 2023, pp. 1−19.
[48]	S. Zhang, X. Wang, J. Wang, J. Pang, C. Lyu, W. Zhang, P. Luo, and K. Chen, “Dense distinct query for end-to-end object detection,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Vancouver, Canada, 2023, pp. 7329−7338.
[49]	W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” in Proc. Network and Distributed Systems Security Symp., San Diego, USA, 2018, pp. 1−15.
[50]	C. Yu, J. Chen, Y. Xue, Y. Liu, W. Wan, J. Bao, and H. Ma, “Defending against universal adversarial patches by clipping feature norms,” in Proc. IEEE/CVF Int. Conf. Computer Vision, Montreal, QC, Canada, 2021, pp. 16414−16422.
[51]	T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” arXiv preprint arXiv: 1708.04552, 2017.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(15) / Tables(6)

Get Citation

PDF

XML

Article Metrics

Article views (59) PDF downloads(2)

Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

doi: 10.1109/JAS.2025.125438

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Export File

Citation

Format

Content