Skip to main content
SpringerLink
Log in

Structure learning of Bayesian Networks using global optimization with applications in data classification

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

Bayesian Networks are increasingly popular methods of modeling uncertainty in artificial intelligence and machine learning. A Bayesian Network consists of a directed acyclic graph in which each node represents a variable and each arc represents probabilistic dependency between two variables. Constructing a Bayesian Network from data is a learning process that consists of two steps: learning structure and learning parameter. Learning a network structure from data is the most difficult task in this process. This paper presents a new algorithm for constructing an optimal structure for Bayesian Networks based on optimization. The algorithm has two major parts. First, we define an optimization model to find the better network graphs. Then, we apply an optimization approach for removing possible cycles from the directed graphs obtained in the first part which is the first of its kind in the literature. The main advantage of the proposed method is that the maximal number of parents for variables is not fixed a priory and it is defined during the optimization procedure. It also considers all networks including cyclic ones and then choose a best structure by applying a global optimization method. To show the efficiency of the algorithm, several closely related algorithms including unrestricted dependency Bayesian Network algorithm, as well as, benchmarks algorithms SVM and C4.5 are employed for comparison. We apply these algorithms on data classification; data sets are taken from the UCI machine learning repository and the LIBSVM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Asuncion, A., Newman, D.: UCI machine learning repository. School of Information and Computer Science, University of California. http://www.ics.uci.edu/mlearn/MLRepository.html (2007)

  2. Bender, M.A., Fineman, J.T. Gilbert, S.: A new approach to incremental topological ordering. In: Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms Society for Industrial and Applied Mathematics. Philadelphia (2009)

  3. Campos, L., Fernandez-Luna, M., Gamez, A., Puerta, M.: Ant colony optimization for learning Bayesian networks. Int. J. Approx. Reason. 31(3), 291–311 (2002)

  4. Castillo, E., Gutierrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models. Springer, New York (1997)

    Book  Google Scholar 

  5. Chang, C., Lin, C.: LIBSVM: a library for support vector machines, 2001a. http://www.csie.ntu.edu.tw/cjlin/libsvm (2001)

  6. Chickering, D.M.: Learning Bayesian networks is NP-complete. In: Fisher, D., Lenz, H.-J. (eds.) Learning from Data: Artificial Intelligence and Statistics, pp. 121–130. Springer-Verlag (1996)

  7. Daly, R., Shen, Q.: Learning Bayesian network equivalence classes with ant colony optimization. J. Artif. Intell. Res. 35, 391–447 (2009)

    MATH  MathSciNet  Google Scholar 

  8. Domingos, P., Pazzani, M.: On the optimality of the simple Bayesian classier under zero-one loss. Mach. Learn. 29, 103–130 (1997)

    Article  MATH  Google Scholar 

  9. Fayyad, U.M., Irani, K.B.: On the handling of continuous-valued attributes in decision tree generation. Mach. Learn. 8, 87–102 (1993)

    Google Scholar 

  10. Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29, 131–163 (1997)

    Article  MATH  Google Scholar 

  11. Haeupler, B., Kavitha, T., Mathew, R., Sen, S., Tarjan, R.E.: Incremental cycle detection, topological ordering, and strong component maintenance. In: 35th International Colloquium on Automata, Languages, and Programming (ICALP). Reykjavik, Iceland (2008)

  12. Heckerman, D., Geiger, D., Chickering, D.M.: Learning Bayesian Networks: the combination of knowledge and statistical data. Mach. Learn. 20, 197–243 (1995)

    MATH  Google Scholar 

  13. Heckerman, D., Chickering, D., Meek, C.: Large-sample learning of Bayesian Networks is NP-hard. Mach. Learn. 5, 1287–1330 (2004)

    MATH  MathSciNet  Google Scholar 

  14. Janzura, M., Nielsen, J.: A simulated annealing-based method for learning Bayesian Networks from statistical data. Int. J. Intell. Syst. 21, 335–348 (2006)

    Article  MATH  Google Scholar 

  15. Jensen, F.: An Introduction to Bayesian Networks. Springer, New York (1996)

    Google Scholar 

  16. Ji, Z., Zhong, H., Hu, R., Liu, C.: A Bayesian Network learning algorithm based on independence test and ant colony optimization. Acta Autom. Sin. 35(3), 281–288 (2009)

  17. Kabli, R., Herrmann, F., McCall, J.: A chain-model genetic algorithm for Bayesian Network structure learning. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation. ACM, New York (2007)

  18. Kolda, T. G., Lewis, R. M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

  19. Kouhbor, S., Ugon, J., Rubinov, A., Kruger, A., Mammadov, M.: Coverage in WLAN with minimum number of access points. In: Vehicular Technology Conference, VTC Spring, pp. 1166–1170 (2006)

  20. Langley, P., Iba, W., Thompson, K.: An analysis of Bayesian classiers. In: 10th International Conference Artificial Intelligence, pp. 223–228. AAAI Press (1992)

  21. Larranaga, P., Murga, R., Poza, M., Kuijpers, C.: Structure Learning of Bayesian Networks by Hybrid Genetic Algorithms. Preliminary papers 5th international workshop artificial intelligence and statistics, 310–316 (1995)

  22. Larranaga, P., Poza, M., Yurramendi, Y., Murga, H., Kuijpers, C.: Structure learning of Bayesian networks by genetic algorithms: a performance analysis of control parameters. In: IEEE Transactions on Pattern Analysis and Machine Intelligence archive (1996)

  23. Larranaga, P., Sierra, B., Gallego, J., Michelena, J., Picaza. M.: Learning Bayesian Networks by genetic algorithms: A case study in the prediction of survival in malignant skin melanoma. Artif. Intell. Med. 261–272 (1997)

  24. Mammadov, M.A., Rubinov, A.M., Sniedovich, M.: A new global optimization algorithm based on dynamical systems approach. In: 6th International Conference on Optimization: Techniques and Applications. Ballarat, Australia (2004)

  25. Mammadov, M.A., Rubinov, A.M., Yearwood, J.: Dynamical systems described by relational elasticities with applications to global optimization. In: Jeyakumar, V., Rubinov, A. (eds.) Continuous Optimisation: Current Trends and Modern Applications, pp. 365–387. Springer (2005)

  26. Mammadov, M.A., Orsi, R.: H_ infinity systhesis via a nonsmooth, nonconvex optimization approach. Pac. J. Optim. 1(2), 405–420 (2005)

  27. Marinescu, R., Dechter, R.: AND/OR branch-and-bound search for combinatorial optimization in graphical models. Artif. Intell. 173, 1457–1491 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  28. Maroosi, A., Amiri, B.: A new clustering algorithm based on hybrid global optimization based on a dynamical systems approach algorithm. Expert Syst. Appl. 37(8), 5645–5652 (2010)

  29. Park, H., Cho, S.: An Effcient Attribute Ordering Optimization in Bayesian Networks for Prognostic Modeling of the Metabolic Syndrome, pp. 381–391. Springer, Berlin (2006)

    Google Scholar 

  30. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo (1988)

    Google Scholar 

  31. Richter, F., Fettweis, G.: Base Station Placement Based on Force Fields. IEEE VTC-Spring, Yokohama (2012)

    Book  Google Scholar 

  32. Robinson, R.W.: Counting Unlabeled Acyclic Diagraphs, pp. 28–43. Springer, New York (1997)

    Google Scholar 

  33. Sahami, M.: Learning limited dependence Bayesian classiers. In: The 2nd International Conference. Knowledge Discovery and Data mining (KKD), pp. 335–338 (1996)

  34. Sahin, F., Yavuz, M.C., Arnavut, Z., Uluyol, O.: Fault diagnosis for airplane engines using Bayesian networks and distributed particle swarm optimization. Parallel Comput. 33(2), 124–143 (2007)

  35. Schleip, C., Rais, A., Menzel, A.: Bayesian analysis of temperature sensitivity of plant phenology in Germany. Agric. For. Meteorol. 149, 1699–1708 (2009)

    Article  Google Scholar 

  36. Sedgewick, R: Algorithms. Addison-Wesley Publishing Company. (1983)

  37. Shafer, G., Pearl, J.: Readings in Uncertain Reasoning. Morgan Kaufmann, San Mateo (1990)

    MATH  Google Scholar 

  38. Sun, W., Yuan, Y.X.: Optimization Theory and Methods, Nonlinear Programming. Springer Optimization and its Applications. vol. 1 (2006)

  39. Taheri, S., Mammadov, M.: Solving systems of nonlinear equations using a globally convergent optimization algorithm. Glob. J. Technol. Optim. vol. 3, 132–138 (2012)

  40. Taheri, S., Mammadov, M. Seifollahi, S.: Globally convergent optimization methods for unconstrained problems. Optimization: A journal of mathematical programming and operations research. pp. 124–143 (2012)

  41. Taheri, S., Mammadov, M.: Structure learning of Bayesian networks using a new unrestricted dependency algorithm. In: Second International Conference on Social Eco-Informatics. Venice, Italy (2012)

  42. Tilakaratne, C.D., Mammadov, M., Morris, S.A.: Modied neural network algorithms for predicting trading signals of stock market indices. J. Appl. Math. Decis. Sci. (2009)

  43. Tucker, A.: Covering Circuits and Graph Coloring, Applied Combinatorics, 5th edn. John Wiley and sons, Hoboken (2006)

    Google Scholar 

  44. Yatsko, A., Bagirov, A., Stranieri, A.: On the discretization of continuous features for classication. In: The Proceedings of Ninth Australasian Data Mining Conference (AusDM 2011). Ballarat, Australia (2011)

  45. Zhao, J., Sun, J., Xu, W., Zhou, D.: Structure learning of Bayesian networks based on discrete binary quantum-behaved particle swarm optimization algorithm. In: Proceedings of the Fifth International Conference on Natural Computation. IEEE Computer Society, Washington (2009)

Download references

Author information

Authors and Affiliations

  1. Faculty of Science and Technology, Federation University, Ballarat, VIC, 3353, Australia

    Sona Taheri & Musa Mammadov

  2. National ICT Australia, Melbourne, VIC, 3010, Australia

    Musa Mammadov

Authors
  1. Sona Taheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Musa Mammadov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sona Taheri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taheri, S., Mammadov, M. Structure learning of Bayesian Networks using global optimization with applications in data classification. Optim Lett 9, 931–948 (2015). https://doi.org/10.1007/s11590-014-0803-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-014-0803-1

Keywords

Search

Navigation