The language of an application is often not the same as the language of its users. Users who want to straighten a photo have to know to reach for the ruler tool and users who want to work with 3D objects have to learn about transforming and scaling. This language gap is especially problematic for novices but remains an issue for expert users who are learning how to do something new. In the context of creativity support tools where users are often working with visuals, this language gap is exacerbated by the language of images and graphics is not easily discoverable. In this talk, I will cover recent projects that try to make it easier for users to work with creative tools through natural language interfaces. I will cover how we can automatically learn mappings between natural language words and application tool names, how we can combine multiple modalities to make it easier to do creative tasks, and how we can address the inherent ambiguity in user natural language requests.
Mira Dontcheva is a Senior Research Scientist at Adobe. Her research focuses on creativity support tools, natural language interaction, and information visualization. Since joining Adobe, she has worked on making Adobe’s products easier to learn and use by leveraging the vast amount of help content on the Web and building novel forms of instructional content including interactive tutorials and games. Before joining Adobe in 2008, she completed her PhD in Computer Science at the University of Washington.