Skip to main content
Log in

Large-scale network intrusion detection based on distributed learning algorithm

  • Regular Contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

As network traffic bandwidth is increasing at an exponential rate, it’s impossible to keep up with the speed of networks by just increasing the speed of processors. Besides, increasingly complex intrusion detection methods only add further to the pressure on network intrusion detection (NIDS) platforms, so the continuous increasing speed and throughput of network poses new challenges to NIDS. To make NIDS usable in Gigabit Ethernet, the ideal policy is using a load balancer to split the traffic data and forward those to different detection sensors, which can analyze the splitting data in parallel. In order to make each slice contains all the evidence necessary to detect a specific attack, the load balancer design must be complicated and it becomes a new bottleneck of NIDS. To simplify the load balancer this paper put forward a distributed neural network learning algorithm (DNNL). Using DNNL a large data set can be split randomly and each slice of data is presented to an independent neural network; these networks can be trained in distribution and each one in parallel. Completeness analysis shows that DNNL’s learning algorithm is equivalent to training by one neural network which uses the technique of regularization. The experiments to check the completeness and efficiency of DNNL are performed on the KDD’99 Data Set which is a standard intrusion detection benchmark. Compared with other approaches on the same benchmark, DNNL achieves a high detection rate and low false alarm rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Song, H.Y., Lockwood, J.W.: Efficient packet classification for network intrusion detection using FPGA. In: Proceedings of the 13th International Symposium on Field-programmable Gate Arrays, pp. 238–245. Monterey (2005)

  2. Yang W., Fang B.X., Liu B., Zhang H.L.: Intrusion detection system for high-speed network. J. Comput. Commun. 27, 1288–1294 (2004)

    Article  Google Scholar 

  3. Baker, Z.K., Prasanna, V.K.: Automatic synthesis of efficient intrusion detection systems on FPGAs. In: Proceedings of the 14th Field Programmable Logic and Application, pp. 311–321. Leuven, Belgium (2004)

  4. Baker, Z.K., Prasanna, V.K.: A methodology for synthesis of efficient intrusion detection systems on FPGAs. In: Proceedings of the 12th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM’04), pp. 135–144. Napa (2004)

  5. McAlerney, J., Coit, C., Staniford, S.: Towards faster string matching for intrusion detection or exceeding the speed of snort. In: Proceedings of DARPA Information Survivability Conference and Exposition, pp. 367–373. Anaheim (2001)

  6. Tuck, N., Sherwood, T., Calder, B., Varghese, G.: Deterministic memory-efficient string matching algorithms for intrusion detection. In: Proceedings of the 23rd Conference of the IEEE Communications Society, pp. 2628–2639. Hong Kong (2004)

  7. Tan, L., Sherwood, T.: A high throughput string matching architecture for intrusion detection and prevention. In: Proceedings of the 32nd International Symposium on Computer Architecture, pp. 112–122. Madison, Wisconsin (2005)

  8. Aggarwal C., Yu S.: An effective and efficient algorithm for high-dimensional outlier detection. J. Int. J. Very Large Data Bases 14, 211–221 (2005)

    Article  Google Scholar 

  9. Rawat S., Pujari A.K., Gulati V.P.: On the use of singular value decomposition for a fast intrusion detection system. J. Electronic Notes Theor. Comput. Sci. 142, 215–228 (2006)

    Article  Google Scholar 

  10. Kruegel, C., Valeur, F., Vigna, G., Kemmerer, R.: Stateful intrusion detection for high-speed networks. In: Proceedings of the IEEE Symposium on Security and Privacy, pp. 285–294. California (2002)

  11. Lai, H.G., Cai, S.W., Huang, H., Xie, J.Y., Li, H.: A parallel intrusion detection system for high-speed networks. In: Proceedings of Applied Cryptography and Network Security: Second International Conference, pp.439–451. ACNS 2004, Yellow Mountain (2004)

  12. Jiang W.B., Song H., Dai Y.Q.: Real-time intrusion detection for high-speed networks. J. Comput. Secur. 24, 287–294 (2005)

    Article  Google Scholar 

  13. Xinidis K., Charitakis I., Antonatos S., Anagnostakis K.G., Markatos E.P.: An active splitter architecture for intrusion detection and prevention. J. IEEE Trans. Dependable. Secure Comput. 3, 31–44 (2006)

    Article  Google Scholar 

  14. Schaelicke, L., Wheeler, K., Freeland, C.: SPANIDS: a scalable network intrusion detection loadbalancer. In: Proceedings of the 2nd Conference on Computing Frontiers, pp. 315–322. Ischia (2005)

  15. Szalay A., Gray J.: The world-wide telescope. Science 293, 2037–2040 (2001)

    Article  Google Scholar 

  16. Martone M.E., Gupta A., Ellisman M.H.: E-neuroscience: challenges and triumphs in integrating distributed data from molecules to brains. Nature Neurosci. 7, 467–472 (2004)

    Article  Google Scholar 

  17. Wroe C., Goble C., Greenwood M., Lord P., Miles S., Papay J., Payne T., Moreau L.: Automating experiments using semantic data on a bioinformatics grid. IEEE Intell. Syst. 19, 48–55 (2004)

    Article  Google Scholar 

  18. Wang Y.X., Behera S.R., Wong J., Helmer G., Honavar V., Miller L., Lutz R., Slagell M.: Towards the automatic generation of mobile agents for distributed intrusion detection system. J. Syst. Softw. 79, 1–14 (2006)

    Article  Google Scholar 

  19. Bala, J., Weng, Y., Williams, A., Gogia, B.K., Lesser, H.K.: Applications of Distributed Mining Techniques For Knowledge Discovery in Dispersed Sensory Data. In: Proceedings of the 7th Joint Conference on Information Sciences, pp. 1–4. Cary (2003)

  20. Kourai, K., Chiba, S.: HyperSpector virtual distributed monitoring environments for secure intrusion detection. In: Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments, pp. 197–207. Chicago (2005)

  21. Folino, G., Pizzuti, C., Spezzano, G.: GP ensemble for distributed intrusion detection systems. In: Proceedings of the 3rd International Conference on Advanced in Pattern Recognition, pp. 54–62. Bath, UK (2005)

  22. Geman S., Bienenstock E., Doursat R.: Neural networks and the bias/variance dilema. Neural Comput. 4, 1–58 (1992)

    Article  Google Scholar 

  23. Kuo R.J., An Y.L., Wang H.S., Chung W.J.: Integration of self-organizing feature maps neural network and genetic K-means algorithm for market segmentation. J. Expert Syst. Appl. 30, 313–324 (2006)

    Article  Google Scholar 

  24. Carpenter G.A., Milenova B.L., Noeske B.W.: Distributed ARTMAP: a neural network for fast distributed supervised learning. J. Neural Networks 11, 793–813 (1998)

    Article  Google Scholar 

  25. Nair T.M., Zheng C.L., Fink J.L., Stuart R.O., Gribskov M.: Rival penalized competitive learning (RPCL): a topology-determining algorithm for analyzing gene expression data. J. Comput. Biol. Chem. 27, 565–574 (2003)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yanheng Liu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tian, D., Liu, Y. & Xiang, Y. Large-scale network intrusion detection based on distributed learning algorithm. Int. J. Inf. Secur. 8, 25–35 (2009). https://doi.org/10.1007/s10207-008-0061-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-008-0061-2

Keywords

Navigation