Actor complexification in TD3 and curriculum learning with structural composition for drone countering
https://doi.org/10.21122/2309-4923-2025-4-41-48
Abstract
The work suggests complexifying actors within the framework of TD3 which involves the usage of different state vectors for actors and critics in order to assure convergence of the algorithm. It also describes a process of aggregating models, separately trained on datasets or in simulation on tasks with increasing difficulty, stitching everything together step by step into a single end-to-end system. It allows utilizing existing algorithms, such as YOLO, in reinforcement learning systems, performing sensor fusion and gradually adding functionality without losing convergence. Assistance providing allows training systems in simulation from hardcoded algorithms that use simplified states. These techniques are demonstrated on a particular task of building an anti-drone system for armored vehicles.
About the Author
E. V. Rulko
Military Academy of the Republic of Belarus
Belarus
Belarus
Eugene V. Rulko – PhD of Engineering Sciences. Associate Professor,
Minsk
E-mail: eugeni1533@gmail.com
References
1. NVIDIA Omniverse. Available at: https://www.nvidia.com/en-us/omniverse (accessed: 08 August 2025).
2. NVIDIA Isaac Sim. Available at: https://developer.nvidia.com/isaac/sim (accessed: 08 August 2025).
3. Narrowing the Sim2Real Gap with NVIDIA Isaac Sim. Nvidia. Available at: https://www.youtube.com/watch?v=VW-dOMBFj7o (accessed: 08 August 2025).
4. Unity. Available at: https://unity.com (accessed: 08 August 2025).
5. Unreal engine. Available at: https://www.unrealengine.com/en-US (accessed: 08 August 2025).
6. Loitering Munition Lancet Drone 3D Model. Renderhub. Available at: https://www.renderhub.com/sergeydzyuba/loitering-munition-lancet-drone (accessed: 08 August 2025).
7. Unity ML-Agents Toolkit. GitHub. Available at: https://github.com/Unity-Technologies/ml-agents (accessed: 08 August 2025).
8. Grid Sensors for Unity ML-Agents. GitHub. Available at: https://github.com/mbaske/grid-sensor (accessed: 08 August 2025).
9. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. 2018. arXiv: 1801.01290. https://doi.org/10.48550/arXiv.1801.01290
10. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707.06347. https://doi.org/10.48550/arXiv.1707.06347
11. Lillicrap T.P., Hunt J.J, Pritzel A., Heess N., Erez T., Tassa Y., et al. Continuous control with deep reinforcement learning. 2019. arXiv: 1509.02971. https://doi.org/10.48550/arXiv.1509.02971
12. Fujimoto S., Herke van Hoof, Meger D. Addressing Function Approximation Error in Actor-Critic Methods. 2018. arXiv: 1802.09477. https://doi.org/10.48550/arXiv.1802.09477
13. Rulko E.V. Complexification through gradual involvement and reward providing in deep reinforcement learning. System analysis and applied information science. 2024;4:13-20. https://doi.org/10.21122/2309-4923-2024-4-13-20
14. ESC-50: Dataset for Environmental Sound Classification. GitHub. Available at: https://github.com/karolpiczak/ESC-50/tree/master (accessed: 08 August 2025).
15. DroneAudioDataset. GitHub. Available at: https://github.com/saraalemadi/DroneAudioDataset (accessed: 08 August 2025).
16. Drone Audio Detection Samples. Hugging Face. Available at: https://huggingface.co/datasets/geronimobasso/drone-audio-detection-samples (accessed: 08 August 2025).
17. Vit_b_16. PyTorch documentation. Available at: https://docs.pytorch.org/vision/main/models/generated/torchvision.models.vit_b_16.html (accessed: 08 August 2025).
18. Smart Bee Colony Monitor: Clips of Beehive Sounds. Kaggle. Available at: https://www.kaggle.com/datasets/annajyang/beehive-sounds (accessed: 08 August 2025).
19. Time series classification. Dataset: MosquitoSound. Time series classification website. Available at: https://www.timeseriesclassification.com/description.php?Dataset=MosquitoSound (accessed: 08 August 2025).
20. Audio Dataset of Low-Flying Aircraft: AeroSonicDB. Kaggle. Available at: https://www.kaggle.com/datasets/gray8ed/audio-dataset-of-low-flying-aircraft-aerosonicdb (accessed: 08 August 2025).
21. YOLO Drone Detection Dataset. Kaggle. Available at: https://www.kaggle.com/datasets/muki2003/yolo-drone-detection-dataset (accessed: 08 August 2025).
22. Caltech-UCSD Birds-200-2011. Kaggle. Available at: https://www.kaggle.com/datasets/veeralakrishna/200-bird-species-with-11788-images/data (accessed: 08 August 2025).
23. Perception Package. Unity documentation. Available at: https://docs.unity3d.com/Packages/com.unity.perception@1.0/manual/index.html (accessed: 08 August 2025).
24. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, et al. Dota 2 with Large Scale Deep Reinforcement Learning. 2019. arXiv: 1912.06680. https://doi.org/10.48550/arXiv.1912.06680
25. Articulation Body component reference. Unity documentation. Available at: https://docs.unity3d.com/6000.1/Documentation/Manual/class-ArticulationBody.html (accessed: 08 August 2025).
26. Lowe R., Wu Y., Tamar A., Harb J., Abbeel P., Mordatch I. Multi-Agent Actor-Critic for Mixed CooperativeCompetitive Environments. 2020. arXiv: 1706.02275. https://doi.org/10.48550/arXiv.1706.02275
27. Projects motion of pixels to a voxel. GitHub. Available at: https://github.com/ConsistentlyInconsistentYT/Pixeltovoxelprojector (accessed: 08 August 2025).
28. Tracking faint objects with cheap cameras. Consistently inconsistent. Available at: https://www.youtube.com/watch?v=m-b51C82-UE&t=98s (accessed: 08 August 2025).
29. Friston K. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience. 2010;11:127–138. https://doi.org/10.1038/nrn2787
Review
For citations:
Rulko E.V. Actor complexification in TD3 and curriculum learning with structural composition for drone countering. «System analysis and applied information science». 2025;(4):41-48. https://doi.org/10.21122/2309-4923-2025-4-41-48
Views:
326
JATS XML
Email this article 