Skip to main content
SpringerLink
Log in

An Empirical Study of Reward Structures for Actor-Critic Reinforcement Learning in Air Combat Manoeuvring Simulation

  • Conference paper
  • First Online:
AI 2019: Advances in Artificial Intelligence (AI 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11919))

Included in the following conference series:

Abstract

Reinforcement learning techniques for solving complex problems are resource-intensive and take a long time to converge, prompting a need for methods that encourage faster learning. In this paper we show our successful application of actor-critic reinforcement learning to the air combat simulation domain and how reward structures affect the learning speed to find effective air combat tactics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alford, R., Borck, H., Karneeb, J., Aha, D.: Active behavior recognition in beyond visual range air combat. In: Proceedings of the Third Annual Conference on Advances in Cognitive Systems (2015)

    Google Scholar 

  2. Barto, A., Sutton, R.S., Anderson, C.W.: Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 9, 833–836 (1983)

    Google Scholar 

  3. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculumm learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)

    Google Scholar 

  4. Fang, J., Yan, W.J., Fang, W.: Air combat strategies of CGF based on Q-learning and behavior tree. DEStech Trans. Eng. Technol. Res. (2017)

    Google Scholar 

  5. Floyd, M.W., Karneeb, J., Moore, P., Aha, D.W.: A goal reasoning agent for controlling UAVs in beyond-visual-range air combat. In: Proceedings 26th International Joint Conference on Artificial Intelligence (2017)

    Google Scholar 

  6. Grześ, M., Kudenko, D.: Improving optimistic exploration in model-free reinforcement learning. In: Kolehmainen, M., Toivanen, P., Beliczynski, B. (eds.) ICANNGA 2009. LNCS, vol. 5495, pp. 360–369. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04921-7_37

    Chapter  Google Scholar 

  7. Heinze, C., Papasimeon, M., Goss, S., Cross, M., Connell, R.: Simulating fighter pilots. In: Defence Industry Applications of Autonomous Agents and Multi-Agent Systems, pp. 113–130 (2007)

    Google Scholar 

  8. Lee, D., Bang, H.: Planar evasive aircrafts maneuvers using reinforcement learning. In: Lee, S., Cho, H., Yoon, K.J., Lee, J. (eds.) Intelligent Autonomous Systems. AISC, vol. 193, pp. 533–542. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-33926-4_49

    Chapter  Google Scholar 

  9. Liu, P., Ma, Y.: A deep reinforcement learning based intelligent decision method for UCAV air combat. In: Mohamed Ali, M.S., Wahid, H., Mohd Subha, N.A., Sahlan, S., Md. Yunus, M.A., Wahap, A.R. (eds.) AsiaSim 2017. CCIS, vol. 751, pp. 274–286. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-6463-0_24

    Chapter  Google Scholar 

  10. Masek, M., Lam, C.P., Benke, L., Kelly, L., Papasimeon, M.: Discovering emergent agent behaviour with evolutionary finite state machines. In: Miller, T., Oren, N., Sakurai, Y., Noda, I., Savarimuthu, B.T.R., Cao Son, T. (eds.) PRIMA 2018. LNCS, vol. 11224, pp. 19–34. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03098-8_2

    Chapter  Google Scholar 

  11. McGrew, J., How, J.P., Williams, B., Roy, N.: Air-combat strategy using approximate dynamic programming. J. Guidance Control Dyn. 33, 1641–1654 (2010)

    Article  Google Scholar 

  12. Mizokami, K.: This chart explains how crazy-expensive fighter jets Have Gotten. Popular Mechanics, March 2017

    Google Scholar 

  13. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 29–33 (2015)

    Article  Google Scholar 

  14. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings 33rd International Conference on Machine Learning, vol. 48, pp. 1928–1937 (2016)

    Google Scholar 

  15. Mouton, H., Roodt, J., le Roux, H.: Applying reinforcement learning to the weapon assignment problem in air defence. Scientia Militaria S. Afr. J. Mil. Stud. 123–140 (2011)

    Google Scholar 

  16. Park, H., Lee, B., Takh, M.: Differential game based air combat maneuver generation using scoring function matrix. Int. J. Aeronaut. Space Sci. 17(2), 204–213 (2016)

    Article  Google Scholar 

  17. Ramirez, M., et al.: Integrated hybrid planning and programmed control for real time UAV maneuvering. In: Proceedings 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1318–1326 (2018)

    Google Scholar 

  18. Rodin, E.Y., Amin, S.M.: Maneuver prediction in air combat via artificial neural networks. Comput. Math. Appl. 24(3), 95–112 (1982)

    Article  Google Scholar 

  19. Roijers, D.M., Vamplew, P., Whiteson, S., Dazeley, R.: A survey of multi-objective sequential decision-making. J. Artif. Intell. Res. 48, 67–113 (2013)

    Article  MathSciNet  Google Scholar 

  20. Shalev, S., Shammah, S., Shashua, A.: Safe, multi-agent, reinforcement learning for autonomous driving: CoRR, vol abs/1610.03295 (2016)

    Google Scholar 

  21. Shaw, R.L.: Fighter Combat: Tactics and Maneuvering. Naval Institute Press, Annapolis (1985)

    Google Scholar 

  22. Sherstov, A.A., Stone, P.: Three automated stock-trading agents: a comparative study. In: Faratin, P., Rodríguez-Aguilar, J.A. (eds.) AMEC 2004. LNCS (LNAI), vol. 3435, pp. 173–187. Springer, Heidelberg (2006). https://doi.org/10.1007/11575726_13

    Chapter  Google Scholar 

  23. Silver, D., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529, 484 (2016)

    Article  Google Scholar 

  24. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  25. Sutton, R., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems, pp. 1057–1063 (2000)

    Google Scholar 

  26. Teng, T.H., Tan, A.H., Tan, Y.S., Yeo, A.: Self-organizing neural networks for learning air combat maneuvers: In: IEEE World Congress on Computational Intelligence, Brisbane, Australia (2012)

    Google Scholar 

  27. Vamplew, P., Dazeley, R., Berry, A., Issabekov, R., Dekker, E.: Empirical evaluation methods for multiobjective reinforcement learning algorithms. Mach. Learn. 84, 51 (2011)

    Article  MathSciNet  Google Scholar 

  28. Vinberg, D.: Guided reinforcement learning applied to air-combat simulation. Master ’s thesis. Royal Institute of Technology, Sweden (2010)

    Google Scholar 

  29. Wharington, J.: Autonomous control of soaring aircraft by reinforcement learning. Doctorate thesis at Royal Melbourne Institute of Technology (1998)

    Google Scholar 

Download references

Acknowledgements

This research is supported by the Defence Science and Technology Group, Australia; the Defence Science Institute, Australia; and an Australian Government Research Training Program Fee-offset scholarship. Associate Professor Joarder Kamruzzaman of the Centre for Multimedia Computing, Communications, and Artificial Intelligence Research (MCCAIR) at Federation University contributed some of the computing resources for this project.

Author information

Authors and Affiliations

  1. Federation University, Mount Helen, VIC, 3350, Australia

    Budi Kurniawan, Peter Vamplew & Cameron Foale

  2. Defence Science and Technology Group, Fishermans Bend, VIC, 3207, Australia

    Michael Papasimeon

  3. School of Information Technology, Deakin University, Geelong, VIC, 3220, Australia

    Richard Dazeley

Authors
  1. Budi Kurniawan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Vamplew
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Papasimeon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard Dazeley
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cameron Foale
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Budi Kurniawan .

Editor information

Editors and Affiliations

  1. University of South Australia, Adelaide, SA, Australia

    Jixue Liu

  2. The University of Melbourne, Melbourne, VIC, Australia

    James Bailey

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kurniawan, B., Vamplew, P., Papasimeon, M., Dazeley, R., Foale, C. (2019). An Empirical Study of Reward Structures for Actor-Critic Reinforcement Learning in Air Combat Manoeuvring Simulation. In: Liu, J., Bailey, J. (eds) AI 2019: Advances in Artificial Intelligence. AI 2019. Lecture Notes in Computer Science(), vol 11919. Springer, Cham. https://doi.org/10.1007/978-3-030-35288-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35288-2_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35287-5

  • Online ISBN: 978-3-030-35288-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation