skip to main content
research-article

Complex activity recognition using context-driven activity theory and activity signatures

Published:01 December 2013Publication History
Skip Abstract Section

Abstract

In pervasive and ubiquitous computing systems, human activity recognition has immense potential in a large number of application domains. Current activity recognition techniques (i) do not handle variations in sequence, concurrency and interleaving of complex activities; (ii) do not incorporate context; and (iii) require large amounts of training data. There is a lack of a unifying theoretical framework which exploits both domain knowledge and data-driven observations to infer complex activities. In this article, we propose, develop and validate a novel Context-Driven Activity Theory (CDAT) for recognizing complex activities. We develop a mechanism using probabilistic and Markov chain analysis to discover complex activity signatures and generate complex activity definitions. We also develop a Complex Activity Recognition (CAR) algorithm. It achieves an overall accuracy of 95.73% using extensive experimentation with real-life test data. CDAT utilizes context and links complex activities to situations, which reduces inference time by 32.5% and also reduces training data by 66%.

References

  1. Robert P. Abelson. 1981. Psychological status of the script concept. American Psychologist 36, 7 (1981), 715--729.Google ScholarGoogle ScholarCross RefCross Ref
  2. Gregory D. Abowd and Elizabeth D. Mynatt. 2000. Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput.-Hum. Interact. 7, 1 (March 2000), 29--58. DOI: http://dx.doi.org/10.1145/344949.344988. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Fahd Albinali, Nigel Davies, and Adrian Friday. 2007. Structural learning of activities from sparse datasets. In Proceedings of the 5th International Conference on Pervasive Computing and Communications (PerCom'07). 221--228. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Oliver Amft and Gerhard Troster. 2008. Recognition of dietary activity events using on-body sensors. Artif. Intell. Med. 42, 2 (2008), 121--136. 0933-3657 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Android-Platform. 2010. Android Platform. Retrieved May 17, 2010 from http://www.android.com/.Google ScholarGoogle Scholar
  6. Ling Bao. 2003. Physical Activity Recognition from Acceleration Data under Semi-Naturalistic Conditions. Ph.D. dissertation. Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science.Google ScholarGoogle Scholar
  7. L. Bao and S. Intille. 2004. Activity Recognition from User-Annotated Acceleration data. In Proc. Pervasive. 1--17.Google ScholarGoogle Scholar
  8. Roland Bennett. 2007. Time Tracker: PC Activity Monitor for Windows. Retrieved May 17, 2010 from http://ttracker.sourceforge.net/.Google ScholarGoogle Scholar
  9. O. Brdiczka, J. Crowley, and P. Reignier. 2007. Learning situation models for providing context-aware services. In Universal Access in Human-Computer Interaction. Ambient Interaction, Constantine Stephanidis (Ed.). Lecture Notes in Computer Science, Vol. 4555. Springer, Berlin, 23--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Oliver Brdiczka, Norman Makoto Su, and James Bo Begole. 2010. Temporal task footprinting: Identifying routine tasks by their temporal patterns. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI'10). ACM, New York, NY, 281--284. DOI: http://dx.doi.org/10.1145/1719970.1720011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Liming Chen, C. D. Nugent, and Hui Wang. 2012. A knowledge-driven approach to activity recognition in smart homes. IEEE Transactions on Knowledge and Data Engineering 24, 6 (June 2012), 961--974. 1041-4347 DOI: http://dx.doi.org/10.1109/TKDE.2011.51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. T. Choudhury, S. Consolvo, B. Harrison, J. Hightower, A. LaMarca, L. LeGrand, A. Rahimi, A. Rea, G. Bordello, B. Hemingway, P. Klasnja, K. Koscher, J. A. Landay, J. Lester, D. Wyatt, and D. Haehnel. 2008. The mobile sensing platform: An embedded activity recognition system. Pervasive Computing, IEEE 7, 2 (2008), 32--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sunny Consolvo, David W. McDonald, Tammy Toscos, Mike Y. Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, Ian Smith, and James A. Landay. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems (CHI'08). ACM, New York, NY, 1797--1806. DOI: http://dx.doi.org/10.1145/1357054.1357335. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Nigel Davies, Daniel P. Siewiorek, and Rahul Sukthankar. 2008. Activity-based computing. Pervasive Computing, IEEE 7, 2 (2008), 20--21. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. W. Deng and H. T. Tsui. 2000. An HMM-based approach for gesture segmentation and recognition. In Proceedings of the 15th International Conference on Pattern Recognition, Vol. 3. 679--682. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Anind Kumar Dey. 2000. Providing Architectural Support for Building Context-Aware Applications. Ph.D. Dissertation. Georgia Institute of Technology.Google ScholarGoogle Scholar
  17. GreenerBuildings. 2010. GreenerBuildings. Retrieved May 17, 2010 from http://www.greenerbuildings. eu/sites/greenerbuildings.eu/files/GreenerBuildingsFiche.pdf.Google ScholarGoogle Scholar
  18. G. Guerra-Filho and Y. Aloimonos. 2007. A language for human action. Computer 40, 5 (2007), 42--51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. R. Hamid, S. Maddi, A. Bobick, and M. Essa. 2007. Structure from statistics—unsupervised activity analysis using suffix trees. In Proceedings of the 11th International Conference on Computer Vision (ICCV'07). 1--8.Google ScholarGoogle Scholar
  20. Rim Helaoui, Mathias Niepert, and Heiner Stuckenschmidt. 2011a. Recognizing interleaved and concurrent activities: A statistical-relational approach. In Proceedings of the 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom'11). IEEE Computer Society, 1--9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Rim Helaoui, Mathias Niepert, and Heiner Stuckenschmidt. 2011b. Recognizing interleaved and concurrent activities using qualitative and quantitative temporal relationships. Pervasive and Mobile Computing 7, 6 (2011), 660--670. DOI: http://dx.doi.org/10.1016/j.pmcj.2011.08.004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. I. Homed, A. Misra, M. Ebling, and W. Jerome. 2008. HARMONI: Context-aware filtering of sensor data for continuous remote health monitoring. In Proceedings of the 6th International Conference on Pervasive Computing and Communications (PerCom'08). 248--251. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Xin Hong, Chris Nugent, Maurice Mulvenna, Sally McClean, Bryan Scotney, and Steven Devlin. 2009. Evidential fusion of sensor data for activity recognition in smart homes. Pervasive and Mobile Computing 5, 3 (2009), 236--252. DOI: http://dx.doi.org/10.1016/j.pmcj.2008.05.002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Derek Hao Hu and Qiang Yang. 2008. CIGAR: Concurrent and interleaving goal and activity recognition. In Proceedings of the 23rd National Conference on Artificial Intelligence, Volume 3. AAAI Press, Chicago, Illinois. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. T. Huynh and B. Schiele. 2006. Unsupervised discovery of structure in activity data using multiple eigenspaces. In Proceedings of the International Conference on Location- and Context-Awareness (LoCA'06). 151--167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. E. Jovanov, A. O'Donnell Lords, D. Raskovic, P. G. Cox, R. Adhami, and F. Andrasik. 2003. Stress monitoring using a distributed wireless intelligent sensor system. IEEE Engineering in Medicine and Biology Magazine 22, 3 (2003), 49--55.Google ScholarGoogle ScholarCross RefCross Ref
  27. V. Kaptelinin, BA Nardi, and C Macaulay. 1999. Methods and tools: The activity checklist: A tool for representing the space of context. interactions 6, 4 (1999), 27--39. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. Kasteren, G. Englebienne, and B. Krose. 2010. Transferring knowledge of activity recognition across sensor networks. In Pervasive Computing, Vol. 6030. Springer Berlin, 283--300. DOI: http://dx.doi.org/10.1007/978-3-642-12654-3_17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Tim van Kasteren, Athanasios Noulas, Gwenn Englebienne, and Ben Krose. 2008. Accurate activity recognition in a home setting. In Proceedings of the 10th International Conference on Ubiquitous Computing (Ubicomp'08). ACM, Seoul, Korea. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Young-Seol Lee and Sung-Bae Cho. 2011. Activity recognition using hierarchical hidden Markov models on a smartphone with 3D accelerometer. In Hybrid Artificial Intelligent Systems. Lecture Notes in Computer Science, Vol. 6678. Springer, Berlin, 460--467. DOI: http://dx.doi.org/10.1007/978-3-642-21219-2_58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Ian Li, Anind K. Dey, and Jodi Forlizzi. 2012. Using context to reveal factors that affect physical activity. ACM Trans. Comput.-Hum. Interact. 19, 1, Article 7 (May 2012), 21 pages. DOI: http://dx.doi.org/ 10.1145/2147783.2147790. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. L. Liao, T. Choudhury, D. Fox, and H. Kautz. 2007. Training conditional random fields using virtual evidence boosting. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'07). 2530--2535. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Paul Lukowicz, Jamie A. Ward, Holger Junker, Mathias Stäger, Gerhard Tröster, Amin Atrash, and Thad Starner. 2004. Recognizing workshop activity using body worn microphones and accelerometers. In Pervasive Computing. 18--32.Google ScholarGoogle Scholar
  34. Sven Meyer and Andry Rakotonirainy. 2003. A survey of research on context-aware homes. In Proceedings of the Australasian Information Security Workshop Conference on ACSW Frontiers 2003, Volume 21. 159--168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. S. P. Meyn and R. L. Tweedie. 1993. Markov Chains and Stochastic Stability. Springer-Verlag.Google ScholarGoogle Scholar
  36. D. Minnen, T. Starner, M. Essa, and C. Isbell. 2006. Discovering characteristic actions from on-body sensor data. In Proceedings of the 10th IEEE International Symposium on Wearable Computers (ISWC'06). 11--18.Google ScholarGoogle Scholar
  37. Mulle-v3. 2010. Mulle v3 Sensor Platform. Retrieved May 17, 2010 from http://www.eistec.se/.Google ScholarGoogle Scholar
  38. D. Ocone. 2010. Lecture Notes in Discrete and Probabilistic Models in Biology: Markov Chain Models. Retrieved May 17, 2010 from http://www.math.rutgers.edu/courses/338/coursenotes/chapter5.pdf.Google ScholarGoogle Scholar
  39. Nuria Oliver, Eric Horvitz, and Ashutosh Garg. 2002. Layered representations for human activity recognition. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (ICMI'02). IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Amir Padovitz, Seng Wai Loke, and Arkady Zaslavsky. 2004. Towards a theory of context spaces. In Proceedings of the 2nd IEEE Annual Conference on Pervasive Computing and Communications Workshops (PerCom'04). Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Amir Padovitz, Seng Wai Loke, and Arkady B. Zaslavsky. 2008. Multiple-agent perspectives in reasoning about situations for context-aware pervasive computing systems. IEEE Transactions on Systems, Man, and Cybernetics, Part A 38, 4 (2008), 729--742. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Etienne Pardoux. 2008. Markov Processes and Applications: Algorithms, Networks, Genome and Finance. John Wiley and Sons Ltd.Google ScholarGoogle Scholar
  43. M. Philipose, K. P. Fishkin, M. Perkowitz, D. J. Patterson, D. Fox, H. Kautz, and D. Hahnel. 2004. Inferring activities from interactions with objects. Pervasive Computing, IEEE 3, 4 (2004), 50--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. P. Rashidi, D. Cook, L. Holder, and M. Schmitter-Edgecombe. 2011. Discovering activities to recognize and track in a smart environment. IEEE Transactions on Knowledge and Data Engineering 23, 4 (2011), 527--539. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Parisa Rashidi and Diane J. Cook. 2011. Activity knowledge transfer in smart environments. Pervasive and Mobile Computing 7, 3 (2011), 331--343. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Daniele Riboni and Claudio Bettini. 2009. Context-aware activity recognition through a combination of ontological and statistical reasoning. In Proceedings of Conference on Ubiquitous Intelligence and Computing (UIC'09). 39--53. http://dx.doi.org/10.1007/978-3-642-02830-4_5. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Daniele Riboni and Claudio Bettini. 2011. COSAR: Hybrid reasoning for context-aware activity recognition. Personal Ubiquitous Comput. 15, 3 (March 2011), 271--289. DOI: http://dx.doi.org/10.1007/ s00779-010-0331-7. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. M. S. Ryoo and J. K. Aggarwal. 2006. Recognition of composite human activities through context-free grammar based representation. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2. 1709--1718. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Saguna, Arkady Zaslavsky, and Dipanjan Chakraborty. 2011. Complex activity recognition using context driven activity theory in home environments. In Smart Spaces and Next Generation Wired/Wireless Networking, Sergey Balandin, Yevgeni Koucheryavy, and Honglin Hu (Eds.). Lecture Notes in Computer Science, Vol. 6869. Springer, Berlin, 38--50. DOI: http://dx.doi.org/10.1007/978-3-642-22875-9_4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Roger C. Schank and Robert P. Abelson. 1975. Scripts, plans, and knowledge. In Proceedings of the 4th International Joint Conference on Artificial Intelligence, Volume 1(IJCAI'75). Morgan Kaufmann, San Francisco, CA, 151--157. http://dl.acm.org/citation.cfm?id=1624626.1624649. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Thomas Stiefmeier. 2008. Real-Time Spotting of Human Activities in Industrial Environments. Ph.D. Dissertation. ETH, Zürich.Google ScholarGoogle Scholar
  52. Thomas Stiefmeier, Daniel Roggen, and Gerhard Troster. 2007. Gestures are strings: Efficient online gesture spotting and classification using string matching. In Proceedings of the 2nd International Conference on Body Area Networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. T. Stiefmeier, D. Roggen, G. Troster, G. Ogris, and P. Lukowicz. 2008. Wearable activity tracking in car manufacturing. Pervasive Computing, IEEE 7, 2 (2008), 42--50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. M. Stikic, T. Huynh, K. Van Laerhoven, and B. Schiele. 2008. ADL recognition based on the combination of RFID and accelerometer sensing. In Proceedings of the 2nd International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth'08). 258--263.Google ScholarGoogle Scholar
  55. Holger Storf, Thomas Kleinberger, Martin Becker, Mario Schmitt, Frank Bomarius, and Stephan Prueckner. 2009. An Event-Driven Approach to Activity Recognition in Ambient Assisted Living Ambient Intelligence. In Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 123--132. http://dx.doi.org/10.1007/978-3-642-05408-2_16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Gu Tao, Wu Zhanqing, Tao Xianping, Pung Hung Keng, and Lu Jian. 2009. epSICAR: An Emerging Patterns based approach to sequential, interleaved and Concurrent Activity Recognition. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom'09). 1--9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Emmanuel Munguia Tapia, Stephen S. Intille, and Kent Larson. 2004. Activity recognition in the home using simple and ubiquitous sensors. In Proceedings of the 2nd International Conference on Pervasive. 158--175.Google ScholarGoogle ScholarCross RefCross Ref
  58. M. Tentori and J. Favela. 2008. Activity-aware computing for healthcare. Pervasive Computing, IEEE 7, 2 (2008), 51--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Huawei Tu, Xiangshi Ren, and Shumin Zhai. 2012. A comparative evaluation of finger and pen stroke gestures. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems (CHI'12). ACM, New York, NY, 1287--1296. DOI: http://dx.doi.org/10.1145/2207676.2208584. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Kristof Van Laerhoven. 2001. Combining the self-organizing map and K-means clustering for on-line classification of sensor data. In Proceedings of the International Conference on Artificial Neural Networks. 464--469--. http://dx.doi.org/10.1007/3-540-44668-0_65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. K. Van Laerhoven and O. Cakmakci. 2000. What shall we teach our pants?. In Proceedings of the 4th International Symposium on Wearable Computers. 77--83. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Shiaokai Wang, William Pentney, Ana-Maria Popescu, Tanzeem Choudhury, and Matthai Philipose. 2007. Common sense based joint training of human activity recognizers. In Proceedings of the 20th International Joint Conference on Artificial Intelligence. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. J. A. Ward, P. Lukowicz, G. Troster, and T. E. Starner. 2006. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 10 (2006), 1553--1567. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Tsu-Yu Wu, Chia-Chun Lian, and Jane Hsu. 2007. Joint Recognition of multiple concurrent activities using factorial conditional random fields. In Proceedings of the 2007 AAAI Workshop on Plan, Activity, and Intent Recognition. AAAI Press, Menlo Park.Google ScholarGoogle Scholar
  65. Guang-Zhong Yang, Benny Lo, and Surapa Thiemjarus. 2006. Body Sensor Networks. Springer, London.Google ScholarGoogle Scholar
  66. Juan Ye, L. Coyle, S. Dobson, and P. Nixon. 2007. Using situation lattices to model and reason about context. In Proceedings of the 4th International Workshop on Modeling and Reasoning in Context (MRC'07). 1--12.Google ScholarGoogle Scholar
  67. Juan Ye, L. Coyle, S. Dobson, and P. Nixon. 2009. Using situation lattices in sensor analysis. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom'09). 1 --11. DOI: http://dx.doi.org/10.1109/PERCOM.2009.4912762. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Shumin Zhai and Victoria Bellotti. 2005. Introduction to sensing-based interaction. ACM Trans. Comput.-Hum. Interact. 12, 1 (March 2005), 1--2. DOI: http://dx.doi.org/10.1145/1057237.1057238. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Complex activity recognition using context-driven activity theory and activity signatures

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Computer-Human Interaction
      ACM Transactions on Computer-Human Interaction  Volume 20, Issue 6
      December 2013
      155 pages
      ISSN:1073-0516
      EISSN:1557-7325
      DOI:10.1145/2562181
      Issue’s Table of Contents

      Copyright © 2013 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 1 December 2013
      • Accepted: 1 May 2013
      • Revised: 1 October 2012
      • Received: 1 June 2012
      Published in tochi Volume 20, Issue 6

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader