Lecture Notes in Artificial Intelligence 2155 Subseries of Lecture Notes in Computer Science Edited by J. G. Carbonell and J. Siekmann Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen
3 Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo
Harry Bunt Robbert-Jan Beun (Eds.) Cooperative Multimodal Communication Second International Conference, CMC 98 Tilburg, The Netherlands, January 28-30, 1998 Selected Papers 13
Series Editors Jaime G. Carbonell, Carnegie Mellon University, Pittsburgh, PA, USA Jörg Siekmann, University of Saarland, Saarbrücken, Germany Volume Editors Harry Bunt Tilburg University, Computational Linguistics and AI Group P.O. Box 90153, 5000 LE Tilburg, The Netherlands E-mail: Harry.Bunt@kub.nl Robbert-Jan Beun Utrecht University, Department of Information and Computing Science P.O. Box 80.089, 3508 TB Utrecht, The Netherlands E-mail: rj@cs.uu.nl Cataloging-in-Publication Data applied for Die Deutsche Bibliothek- CIP-Einheitsaufnahme Cooperative multimodal communication : second international conference ; revised papers / CMC 98, Tilburg, The Netherlands, Januar 28-30, 1998. Harry Bunt ; Robbert-Jan Beun (ed.). - Berlin ; Heidelberg ; New York; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Tokyo : Springer, 2001 (Lecture notes in computer science ; Vol. 2155 : Lecture notes in artificial intelligence) ISBN 3-540-42806-2 CR Subject Classification (1998): I.2, H.5.3, H.5, D.2, I.5, K.4 ISBN 3-540-42806-2 Springer-Verlag Berlin Heidelberg New York This workis subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de Springer-Verlag Berlin Heidelberg 2001 Printed in Germany Typesetting: Camera-ready by author, data conversion by Christian Grosche, Hamburg Printed on acid-free paper SPIN: 10845541 06/3142 543210
Preface The chapters in this book are revised, updated, and edited versions of 13 selected papers from the Second International Conference on Cooperative Multimodal Communication (CMC 98), held in Tilburg, The Netherlands, in 1998. This was the second conference in a series, of which the first one was held in Eindhoven, The Netherlands, in 1995. Three of these papers were presented by invited speakers; those by Donia Scott (co-authored with Richard Power), Steven Feiner (co-authored with Michele Zhou), and Oliviero Stock (co-authored with Carlo Strapparava and Massimo Zancanaro). The other ten were among the submitted papers that were accepted by the CMC 98 program committee. The editors contributed an introductory chapter to set the stage for the rest of the book. We thank the program committee for their excellent and timely feedback to the authors of the submitted papers, and at a later stage for advising on the contents of this volume and for providing additional suggestions for improving the selected contributions. The program committee consisted of Nicholas Asher, Normann Badler, Don Bouwhuis, Harry Bunt, Walther von Hahn, Dieter Huber, Hans Kamp, John Lee, Joseph Mariani, Jean-Claude Martin, Mark Maybury, Paul Mc Kevitt, Rob Nederpelt, Kees van Overveld, Ray Perrault, Donia Scott, Jan Treur, Wolfgang Wahlster, Bonnie Webber, Kent Wittenburg, and Henk Zeevat. We thank the Royal Dutch Academy of Sciences (KNAW)and the Organization for Cooperation among Universities in Brabant (SOBU)for their grants that supported the conference. July 2001 Robbert-Jan Beun Harry Bunt
Table of Contents Multimodal Cooperative Communication 1 Robbert-Jan Beun and Harry Bunt Part 1: Multimodal Generation Generating Textual Diagrams and Diagrammatic Texts 13 Donia Scott and Richard Power Pedro: Assessing Presentation Decodability on the Basis of Empirically Validated Models 30 Susanne van Mulken Improvise: Automated Generation ofanimated Graphics for Coordinated Multimedia Presentations 43 Michelle X. Zhou and Steven K. Feiner Multimodal Reference to Objects: An Empirical Approach 64 Robbert-Jan Beun and Anita Cremers Part 2: Multimodal Cooperation Augmenting and Executing SharedPlans for Multimodal Communication 89 Oliviero Stock, Carlo Strapparava, and Massimo Zancanaro Cooperation and Flexibility in Multimodal Communication 113 Jens Allwood Communication and Manipulation Acts in a Collaborative Dialogue Model 125 Martine Hurault-Plantet and Cecile Balkanski Relating Imperatives to Action 140 Paul Piwek
VIII Table of Contents Part 3: Multimodal Interpretation Interpretation ofgestures and Speech: A Practical Approach to Multimodal Communication 159 Xavier Pouteau Why Are Multimodal Systems so Difficult to Build? About the Difference between Deictic Gestures and Direct Manipulation 176 Michael Streit Multimodal Cooperative Resolution ofreferential Expressions in the DenK System 197 Leen Kievit, Paul Piwek, Robbert-Jan Beun, and Harry Bunt Part 4: Multimedia Platforms and Test Environments The IntelliMedia WorkBench An Environment for Building Multimodal Systems 217 Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen, Michael Manthey, Paul Mc Kevitt, Thomas B. Moeslund, and Kristian G. Olesen A Unified Framework for Constructing Multimodal Experiments and Applications 234 Adam Cheyer, Luc Julia, and Jean-Claude Martin Index 243 Author Index 251