Skip to main content

Advertisement

SpringerLink
Log in

Affective evolutionary music composition with MetaCompose

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

This paper describes the MetaCompose music generator, a compositional, extensible framework for affective music composition. In this context ‘affective’ refers to the music generator’s ability to express emotional information. The main purpose of MetaCompose is to create music in real-time that can express different mood-states, which we achieve through a unique combination of a graph traversal-based chord sequence generator, a search-based melody generator, a pattern-based accompaniment generator, and a theory for mood expression. Melody generation uses a novel evolutionary technique combining FI-2POP with multi-objective optimization. This allows us to explore a Pareto front of diverse solutions that are creatively equivalent under the terms of a multi-criteria objective function. Two quantitative user studies were performed to evaluate the system: one focusing on the music generation technique, and the other that explores valence expression, via the introduction of dissonances. The results of these studies demonstrate (i) that each part of the generation system improves the perceived quality of the music produced, and (ii) how valence expression via dissonance produces the perceived affective state. This system, which can reliably generate affect-expressive music, can subsequently be integrated in any kind of interactive application (e.g., games) to create an adaptive and dynamic soundtrack.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.allmusic.com.

  2. http://www.cs.cinvestav.mx/~constraint/papers/.

  3. https://www.oracle.com/java/index.html.

  4. http://explodingart.com/jmusic/.

  5. http://www.beadsproject.net/.

  6. This method is inspired by the “ablation studies” performed by Stanley [87].

References

  1. S. Abrams, D.V. Oppenheim, D. Pazel, J. Wright, et al. Higher-level composition control in music sketcher: modifiers and smart harmony, in Proceedings of the ICMC. Citeseer (1999)

  2. A. Alpern, Techniques for algorithmic composition of music (1995), http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23.9364&rep=rep1&type=pdf

  3. D. Arsenault, Guitar hero:” not like playing guitar at all. J. Can. Game Stud. Assoc. 1(2), 1–7 (2008)

    Google Scholar 

  4. J.J. Aucouturier, F. Pachet, M. Sandler, the way it sounds: timbre models for analysis and retrieval of music signals. IEEE Trans. Multimed. 7(6), 1028–1035 (2005)

    Article  Google Scholar 

  5. C.P.E. Bach, W.J. Mitchell, W. John, Essay on the True Art of Playing Keyboard Instruments (WW Norton, New York, 1949)

    Google Scholar 

  6. C.D. Batson, L.L. Shaw, K.C. Oleson, Differentiating affect, mood, and emotion: toward functionally based conceptual distinctions, in Emotion (Sage Publications, Inc., Thousand Oaks, CA, 1992), pp. 294–326

  7. C. Beedie, P. Terry, A. Lane, Distinctions between emotion and mood. Cognit. Emot. 19(6), 847–878 (2005)

    Article  Google Scholar 

  8. J. Biles, Genjam: a genetic algorithm for generating jazz solos, in Proceedings of the International Computer Music Conference (International Computer Music Association, 1994), pp. 131–131

  9. D. Birchfield, Generative model for the creation of musical emotion, meaning, and form, in Proceedings of the 2003 ACM SIGMM Workshop on Experiential Telepresence (2003), pp. 99–104

  10. O. Bown, Experiments in modular design for the creative composition of live algorithms. Comput. Music J. 35(3), 73–85 (2011)

    Article  Google Scholar 

  11. C.R. Brewin, Cognitive change processes in psychotherapy. Psychol. Rev. 96(3), 379 (1989)

    Article  Google Scholar 

  12. D. Brown, Mezzo: an adaptive, real-time composition program for game soundtracks, in Proceedings of the AIIDE 2012 Workshop on Musical Metacreation (2012), pp. 68–72

  13. G.C. Bruner, Music, mood, and marketing. J. Mark. 1, 94–104 (1990)

    Google Scholar 

  14. D. Butler, An historical investigation and bibliography of nineteenth century music psychology literature. Ph.D. thesis, Ohio State University (1973)

  15. T. Byron, C. Stevens, Steps and leaps in human memory for melodies: the effect of pitch interval magnitude in a melodic contour discrimination task, in 9th International Conference on Music Perception and Cognition (ICMPC9), Bologna, Italy (Citeseer, 2006)

  16. D. Chafekar, J. Xuan, K. Rasheed, Constrained multi-objective optimization using steady state genetic algorithms, in Genetic and Evolutionary Computation GECCO (Springer, 2003), pp. 813–824

  17. H. Chan, D.A. Ventura, Automatic composition of themed mood pieces, in Proceedings of the International Joint Workshop on Computational Creativity (2008), pp. 19–115

  18. K. Collins, An introduction to procedural music in video games. Contemp. Music Rev. 28(1), 5–15 (2009). doi:10.1080/07494460802663983

    Article  MathSciNet  Google Scholar 

  19. D. Cope, Algorithmic music composition, in Patterns of Intuition, ed. by G. Nierhaus (Springer Netherlands, 2015), pp. 405–416. doi:10.1007/978-94-017-9561-6_19

  20. P. Dahlstedt, Autonomous evolution of complete piano pieces and performances, in Proceedings of Music AL Workshop (Citeseer, 2007)

  21. B. De Haas, R.C. Veltkamp, F. Wiering, Tonal pitch step distance: a similarity measure for chord progressions, in ISMIR (2008), pp. 51–56

  22. K. Deb, Multi-objective Optimization Using Evolutionary Algorithms, vol. 16 (Wiley, Hoboken, 2001)

    MATH  Google Scholar 

  23. K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: Nsga-II. IEEE Trans. Evolut. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  24. K. Deb, A. Pratap, T. Meyarivan, Constrained test problems for multi-objective evolutionary optimization, in Evolutionary Multi-Criterion Optimization (Springer, 2001), pp. 284–298

  25. P. Doornbusch, A brief survey of mapping in algorithmic composition, in Proceedings of the International Computer Music Conference (2002), http://www.academia.edu/download/33447946/A_Brief_Survey_of_Mapping_in_Algorithmic_Composition.pdf

  26. M. Edwards, Algorithmic composition: computational thinking in music. Commun. ACM 54(7), 58–67 (2011). doi:10.1145/1965724.1965742

    Article  Google Scholar 

  27. A.E. Eiben, J. Smith, From evolutionary computation to the evolution of things. Nature 521(7553), 476–482 (2015)

    Article  Google Scholar 

  28. P. Ekman, Are there basic emotions? Psychol. Rev. 99(3), 550–553 (1992). doi:10.1037/0033-295X.99.3.550

    Article  Google Scholar 

  29. P. Ekman, An argument for basic emotions. Cognit. Emot. 6(3–4), 169–200 (1992)

    Article  Google Scholar 

  30. M. Eladhari, R. Nieuwdorp, M. Fridenfalk, The soundtrack of your mind: mind music-adaptive audio for game characters, in Proceedings of Advances in Computer Entertainment Technology (2006)

  31. B. Eno, The ship (2016), http://www.brian-eno.net/

  32. P.R. Farnsworth, The Social Psychology of Music (Dryden, Oxford, 2003), p. 304

  33. A. Gabrielsson, P.N. Juslin, Emotional Expression in Music (Oxford University Press, Oxford, 2003)

    Google Scholar 

  34. J.M. Grey, J.W. Gordon, Perceptual effects of spectral modifications on musical timbres. J. Acoust. Soc. Am. 63(5), 1493–1500 (1978)

    Article  Google Scholar 

  35. R.H. Gundlach, Factors determining the characterization of musical phrases. Am. J. Psychol. 47(4), 624–643 (1935)

    Article  Google Scholar 

  36. K. Hevner, The affective character of the major and minor modes in music. Am. J. Psychol. 47(1), 103–118 (1935)

    Article  Google Scholar 

  37. K. Hevner, Experimental studies of the elements of expression in music. Am. J. Psychol. 48(2), 246–268 (1936)

    Article  Google Scholar 

  38. K. Hevner, The affective value of pitch and tempo in music. Am. J. Psychol. 49(4), 621–630 (1937)

    Article  Google Scholar 

  39. G. Husain, W.F. Thompson, E.G. Schellenberg, Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. Interdiscip. J. 20(2), 151–171 (2002)

    Article  Google Scholar 

  40. G. Ilie, W.F. Thompson, A comparison of acoustic cues in music and speech for three dimensions of affect. Music Percept. Interdiscip. J. 23(4), 319–330 (2006)

    Article  Google Scholar 

  41. A. Isaacs, T. Ray, W. Smith, Blessings of maintaining infeasible solutions for constrained multi-objective optimization problems, in IEEE Congress on Evolutionary Computation (IEEE, 2008), pp. 2780–2787

  42. F. Jimenez, A.F. Gómez-Skarmeta, G. Sánchez, K. Deb, An evolutionary algorithm for constrained multi-objective optimization. In: Proceedings of the Congress on Evolutionary Computation, pp. 1133–1138. IEEE (2002)

  43. P.N. Juslin, S. Liljeström, D. Västfjäll, G. Barradas, A. Silva, An experience sampling study of emotional reactions to music: listener, music, and situation. Emotion 8(5), 668 (2008)

    Article  Google Scholar 

  44. H. Katayose, M. Imai, S. Inokuchi, Sentiment extraction in music, in Proceedings of the 9th International Conference on Pattern Recognition (1988), pp. 1083–1087

  45. S.O. Kimbrough, G.J. Koehler, M. Lu, D.H. Wood, On a feasible-infeasible two-population (fi-2pop) genetic algorithm for constrained optimization: distance tracing and no free lunch. Eur. J. Oper. Res. 190(2), 310–327 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  46. A. Kirke, E.R. Miranda, A survey of computer systems for expressive music performance. ACM Comput. Surv. 42(1), 3:1–3:41 (2009). doi:10.1145/1592451.1592454

    Article  Google Scholar 

  47. V.J. Konečni, Does music induce emotion? A theoretical and methodological analysis. Psychol. Aesthet. Creat. Arts 2(2), 115 (2008)

    Article  Google Scholar 

  48. A.E. Krause, A.C. North, L.Y. Hewitt, Music-listening in everyday life: devices and choice. Psychol. Music 43(2), 155–170 (2015)

    Article  Google Scholar 

  49. G. Kreutz, U. Ott, D. Teichmann, P. Osawa, D. Vaitl, Using music to induce emotions: influences of musical preference and absorption. Psychol. Music 36(1), 101–126 (2008)

    Article  Google Scholar 

  50. C.L. Krumhansl, An exploratory study of musical emotions and psychophysiology. Can. J. Exp. Psychol. Revue canadienne de psychologie expérimentale 51(4), 336 (1997)

    Article  Google Scholar 

  51. C.G. Lange, W. James, The Emotions (Williams & Wilkins, Baltimore, 1922)

    Book  Google Scholar 

  52. T. Langlois, G. Marques, A music classification method based on timbral features, in ISMIR (2009), pp. 81–86

  53. R.S. Lazarus, Emotion and Adaptation (Oxford University Press, Oxford, 1991)

    Google Scholar 

  54. F. Lerdahl, Tonal pitch space. Music Percept. 5, 315–349 (1988)

    Article  Google Scholar 

  55. J.S. Lerner, D. Keltner, Beyond valence: toward a model of emotion-specific influences on judgement and choice. Cognit. Emot. 14(4), 473–493 (2000)

    Article  Google Scholar 

  56. E. Lindström, P.N. Juslin, R. Bresin, A. Williamon, Expressivity comes from within your soul: a questionnaire study of music students’ perspectives on expressivity. Res. Stud. Music Educ. 20(1), 23–47 (2003)

    Article  Google Scholar 

  57. D. Liu, L. Lu, H.J. Zhang, Automatic mood detection from acoustic music data, in Proceedings of the International Symposium on Music Information Retrieval (2003), pp. 81–87

  58. S.R. Livingstone, A.R. Brown, Dynamic response: real-time adaptation for music emotion, in Proceedings of the 2nd Australasian Conference on Interactive Entertainment (2005), pp. 105–111

  59. R. Loughran, J. McDermott, M. O’Neill, Tonality driven piano compositions with grammatical evolution, in IEEE Congress on Evolutionary Computation (CEC) (IEEE, 2015), pp. 2168–2175

  60. B.A. Martin, The influence of gender on mood effects in advertising. Psychol. Mark. 20(3), 249–273 (2003)

    Article  Google Scholar 

  61. H.P. Martinez, G.N. Yannakakis, J. Hallam, Don’t classify ratings of affect; rank them!. IEEE Trans. Affect. Comput. 5(3), 314–326 (2014)

    Article  Google Scholar 

  62. S.K. Meier, J.L. Briggs, System for real-time music composition and synthesis. US Patent 5,496,962 (1996)

  63. L.B. Meyer, Emotion and Meaning in Music (University of Chicago Press, Chicago, 2008)

    Google Scholar 

  64. K. Miller, Schizophonic performance: guitar hero, rock band, and virtual virtuosity. J. Soc. Am. Music 3(04), 395–429 (2009)

    Article  Google Scholar 

  65. E.R. Miranda, Readings in Music and Artificial Intelligence, vol. 20 (Routledge, London, 2013)

    Google Scholar 

  66. E.R. Miranda, A. Biles, Evolutionary Computer Music (Springer, Berlin, 2007)

    Book  Google Scholar 

  67. K. Monteith, T. Martinez, D. Ventura, Automatic generation of music for inducing emotive response, in Proceedings of the International Conference on Computational Creativity (Citeseer, 2010), pp. 140–149

  68. S. Mugglin, Chord charts and maps, http://mugglinworks.com/chordmaps/chartmaps.htm. Accessed 14 Sept 2015

  69. A.C. North, D.J. Hargreaves, J.J. Hargreaves, Uses of music in everyday life. Music Percept. Interdiscip. J. 22(1), 41–77 (2004)

    Article  Google Scholar 

  70. G. Papadopoulos, G. Wiggins, AI methods for algorithmic composition: a survey, a critical view and future prospects, in AISB Symposium on Musical Creativity, Edinburgh, UK (1999), pp. 110–117

  71. G. Perle, Serial Composition and Atonality: An Introduction to the Music of Schoenberg, Berg, and Webern (Univ of California Press, Berkeley, 1972)

    Google Scholar 

  72. J. Posner, J.A. Russell, B.S. Peterson, The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(03), 715–734 (2005)

    Article  Google Scholar 

  73. M. Puckette, et al. Pure data: another integrated computer music environment, in Proceedings of the Second Intercollege Computer Music Concerts (1996), pp. 37–41

  74. A.P. Rigopulos, E.B. Egozy, Real-time music creation system. US Patent 5,627,335 (1997)

  75. J. Robertson, A. de Quincey, T. Stapleford, G. Wiggins, Real-time music generation for a virtual environment, in Proceedings of ECAI-98 Workshop on AI/Alife and Entertainment (Citeseer, 1998)

  76. R. Rosenthal, D.B. Rubin, A simple, general purpose display of magnitude of experimental effect. J. Educ. Psychol. 74(2), 166 (1982)

    Article  Google Scholar 

  77. J.A. Russell, A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)

    Article  Google Scholar 

  78. E.G. Schellenberg, A.M. Krysciak, R.J. Campbell, Perceiving emotion in melody: interactive effects of pitch and rhythm. Music Percept. Interdiscip. J. 18(2), 155–171 (2000)

    Article  Google Scholar 

  79. K.R. Scherer, A. Schorr, T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research (Oxford University Press, Oxford, 2001)

    Google Scholar 

  80. H. Schlosberg, Three dimensions of emotion. Psychol. Rev. 61(2), 81 (1954)

    Article  Google Scholar 

  81. M. Scirea, Mood dependent music generator, in Proceedings of Advances in Computer Entertainment (2013), pp. 626–629

  82. M. Scirea, G.A. Barros, N. Shaker, J. Togelius, Smug: scientific music generator, in Proceedings of the Sixth International Conference on Computational Creativity (2015), p. 204

  83. M. Scirea, M.J. Nelson, J. Togelius, Moody music generator: characterising control parameters using crowdsourcing, in Evolutionary and Biologically Inspired Music, Sound, Art and Design (Springer, 2015), pp. 200–211

  84. M. Scirea, J. Togelius, P. Eklund, S. Risi, Metacompose: a compositional evolutionary music composer, in International Conference on Evolutionary and Biologically Inspired Music and Art (Springer, 2016), pp. 202–217

  85. J.A. Sloboda, S.A. O’Neill, Emotions in everyday listening to music, in Music and Emotion: Theory and Research (Oxford University Press, New York, NY, 2001), pp. 415–429

  86. A. Smaill, G. Wiggins, M. Harris, Hierarchical music representation for composition and analysis. Comput. Humanit. 27(1), 7–17 (1993)

    Article  Google Scholar 

  87. K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  88. R.E. Thayer, The Biopsychology of Mood and Arousal (Oxford University Press, Oxford, 1989)

    Google Scholar 

  89. S.S. Tomkins, Affect Imagery Consciousness: Volume I: The Positive Affects, vol. 1 (Springer, Berlin, 1962)

    Google Scholar 

  90. G.T. Toussaint, et al. The Euclidean algorithm generates traditional musical rhythms, in Proceedings of BRIDGES: Mathematical Connections in Art, Music and Science (2005), pp. 47–56

  91. L.J. Trainor, B.M. Heinmiller, The development of evaluative responses to music: infants prefer to listen to consonance over dissonance. Infant Behav. Dev. 21(1), 77–88 (1998)

    Article  Google Scholar 

  92. D. Watson, A. Tellegen, Toward a consensual structure of mood. Psychol. Bull. 98(2), 219 (1985)

    Article  Google Scholar 

  93. G. Wiggins, M. Harris, A. Smaill, Representing music for analysis and composition. University of Edinburgh, Department of Artificial Intelligence (1990)

  94. R. Wooller, A.R. Brown, E. Miranda, J. Diederich, R. Berry, A framework for comparison of process in algorithmic music systems, in Generative Arts Practice 2005—A Creativity & Cognition Symposium (2005)

  95. W. Wundt, Outlines of Psychology (Springer, Berlin, 1980)

    Book  Google Scholar 

  96. G.N. Yannakakis, J. Togelius, Experience-driven procedural content generation. IEEE Trans. Affect. Comput. 2(3), 147–161 (2011)

    Article  Google Scholar 

  97. E. Zitzler, K. Deb, L. Thiele, Comparison of multiobjective evolutionary algorithms: empirical results. Evol. Comput. 8(2), 173–195 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IT University of Copenhagen, Copenhagen, Denmark

    Marco Scirea, Peter Eklund & Sebastian Risi

  2. New York University, New York, NY, USA

    Julian Togelius

Authors
  1. Marco Scirea
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Julian Togelius
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Eklund
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sebastian Risi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Scirea.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Scirea, M., Togelius, J., Eklund, P. et al. Affective evolutionary music composition with MetaCompose. Genet Program Evolvable Mach 18, 433–465 (2017). https://doi.org/10.1007/s10710-017-9307-y

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-017-9307-y

Keywords

Search

Navigation