Abstract
We propose a new approach to the 3D layout problems based on the integration of constraint programming and virtual reality interaction techniques. Our method uses an open-source constraint solver integrated in a popular 3D game engine. We designed multimodal interaction techniques for the system, based on gesture and voice input. We conducted a user study with an interactive task of laying out room furniture to compare and evaluate the mono- and multimodal interaction techniques. Results showed that voice command provided the best performance and was most preferred by participants, based on the analysis of both objective and subjective data. Results also revealed that there was no significant difference between the voice and multimodal input (voice and gesture). Our original approach opens the way to multidisciplinary theoretical work and promotes the development of high-level applications for the VR applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Azenkot S, Lee NB (2013) Exploring the use of speech input by blind people on mobile devices. In: Proceedings of the 15th international ACM SIGACCESS conference on computers and accessibility. ACM, pp 1–8
Billinghurst M (1998) Put that where? Voice and gesture at the graphics interface. SIGGRAPH Comput. Graph. 32(4):60–63
Bolt RA, Herranz E (1992) Two-handed gesture in multi-modal natural dialog. In: 5th Annual ACM symposium on user interface software and technology, pp 7–14
Cal C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, Magistretti PJ (2016) Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J Comp Neurol 524(1):23–38
Calderon C, Cavazza M, Diaz D (2003) A new approach to the interactive resolution of configuration problems in virtual environments. Lect Notes Comput Sci 2733:112–122
Chun LM, Arshad H, Piumsomboon T, Billinghurst M (2015) A combination of static and stroke gesture with speech for multimodal interaction in a virtual environment. In: 2015 International conference on electrical engineering and informatics (ICEEI). IEEE, pp 59–64
Fages F, Soliman S, Coolenand R (2004) CLPGUI: a generic graphical user interface for constraint logic programming. Constraints 9:241–262
Fernando T, Murray N, Tan K, Wimalaratne P (1999) Software architecture for a constraint-based virtual environment. In: Proceedings of the ACM symposium on virtual reality software and technology, pp 147–154
Gao S, Wan H, Peng Q (2000) An approach to solid modeling in a semi-immersive virtual environment. Comput Graph 24:191–202
Garcia VM, Ruíz MP, Pérez JR (2010) Voice interactive classroom, a service-oriented software architecture for speech-enabled learning. J Netw Comput Appl 33:603–610
Goel V, Slusky M, van Hoeve WJ, Furman KC, Shao Y (2015) Constraint programming for LNG ship scheduling and inventory management. Eur J Oper Res 241(3):662–673
Hart S, Staveland L (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research, pp 139–183
Honda K, Mizoguchi F (1995) Constraint-based approach for automatic spatial layout planning. Conference on artificial intelligence for applications. IEEE Press, Ne York
ISO/IEC (2011) 25010:2011 Systems and software engineering—systems and software quality requirements and evaluation (SQuaRE)—system and software quality models, ISO
Jacob R, Sibert L, MacFarlane D, Mullen P Jr (1994) Integrality and separability of input devices. Comput Hum Interact 1:3–26
Jacquenot G (2009) Méthode générique pour l’optimisation d’agencement géometrique et fonctionnel. Thése de Doctorat, Ecole Centrale de Nantes
Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1):116–134
Kefi M, Barichard V, Richard P (2012) A constraint-solver based tool for user-assisted interactive 3D layout. In: ICTAI, pp 199–206
Ku P-Y et al (2013) Ergonomics design on expert convenience of voice-based interface for vehicles AV systems. In: Kurosu M (ed) Human–computer interaction. Applications and services: 15th international conference, HCI international 2013, Las Vegas, NV, USA, July 21–26, 2013, Proceedings, Part II. Springer, Berlin, pp 606–611
Kulyukin VA (2004) Human–robot interaction through gesture-free spoken dialogue
Laviola JJ Jr (1999) Whole-hand and speech input in virtual environments (Doctoral dissertation, Brown University)
Medjdoub B (2004) Constraint-based adaptation for complex space configuration in building services. J Inf Technol Constr 243:627–636
Patrick E et al (2000) Using a large projection screen as an alternative to head-mounted displays for virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM
Pfefferkorn C (1975) A heuristic problem solving design system for equipment or furniture layouts. Commun ACM 18(5):286–297
Pires JN (2006) Industrial robots programming: building applications for the factories of the future. Springer, Berlin
Régin JC (2004) Modlisation et Contraintes Globales en Programmation par Contraintes. Habilitation diriger des recherches, Universit de Nice
Rogalski T, Wielgat R (2010) A concept of voice guided general aviation aircraft. Aerosp Sci Technol 14:321–328
Rogowski A (2012a) A industrially oriented voice control system. Robot Comput Integr Manuf 28:303–315
Rogowski A (2012b) Web-based remote voice control of robotized cells. Robot Comput Integr Manuf 4:77–89
Rossi F, Van Beek P, Walsh T (eds) (2006) Handbook of constraint programming. Elsevier, Amsterdam
Sanchez S, Le Roux O, Inglese F, Luga H, Gaildart V (2003) Constraint-based 3D-object layout using a genetic algorithm. 3IA
Schulte C, Tack G, Lagerkvist M (2013) Modeling with Gecode
Seth A, Vance JM, Oliver JH (2011) Virtual reality for assembly methods prototyping: a review. Virtual Real 15(1):5–20
Tim T, Rafael B, Ruben MS, Klaas JK (2009) Rule-based layout solving and its application to procedural interior generation. In: Proceedings of the CASA workshop on 3D advanced media in gaming and simulation (3AMIGAS), pp 212–227
Turk M (2014) Multimodal interaction: a review. Pattern Recogn Lett 36:189–195
Unity 3D Game Engine (2013) http://Unity3d.com
Vacher M, Lecouteux B, Istrate D, Joubert T, Portet F et al (2013) Experimental evaluation of speech recognition technologies for voice-based home automation control in a smart home. In: 4th Workshop on speech and language processing for assistive technologies, Grenoble, France, pp 99–105
Weyrich M, Drews P (1999) An interactive environment for virtual manufacturing: the virtual workbench. Comput Ind 38:5–15
Xu K, Stewart J, Fiume E (2002) Constraint-based automatic placement for scene composition. In: Graphics interface proceedings, University of Calgary, pp 25–34
Zhao W, Madhavan V (2005) Integration of voice commands into a virtual reality environment for assembly design. In: Proceedings of the 10th annual international conference on industrial engineering theory, applications & practice, Clearwater Beach, FL, USA
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kefi, M., Hoang, T.N., Richard, P. et al. An evaluation of multimodal interaction techniques for 3D layout constraint solver in a desktop-based virtual environment. Virtual Reality 22, 339–351 (2018). https://doi.org/10.1007/s10055-018-0337-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-018-0337-4