A grand challenge in human-computer interaction (HCI) is constructing user interfaces (UIs) that make computers useful for all users across all contexts. UIs today are static, manually-constructed artifacts that limit how users and external software can interact with them. In this talk, I describe two types of machine learning approaches that transform interfaces into dynamic, computational objects that can be optimized for users, applications, and contexts. I first discuss my contributions to UI Understanding, a class of approaches that allow machines to reliably understand the semantics (content and functionality) of UIs using the input/output modalities as humans (e.g., visual perception of rendered pixels and mouse input). I show how these capabilities enhance user interaction and unlock new possibilities for systems such as assistive technology, software testing, and UI automation. Next, I discuss my contributions to UI Generation, which enables applications to dynamically decide how to present content and functionality for new usage contexts. To this end, I present novel approaches to user modeling, UI adaptation, and code generation. Finally, I discuss future research directions and opportunities to drive advances in fields such as HCI, machine learning, and software engineering.
| 11:45am - 12:15pm: | Bring your own lunch and community socializing. |
| 12:15pm - 1:15pm: | Presentation with Q&A. Available hybrid via Zoom. |
| 1:30pm - 2:15pm: | Student meeting with speaker, held in CSE2/Gates 271. Students will walk to this from the seminar. |
Jason Wu is a Research Scientist at Apple in the Human-Centered Machine Intelligence group. Previously, he received a PhD in Human-Computer Interaction from Carnegie Mellon. In his research, Jason applies machine learning to understand, generate, and optimize user interfaces for human-computer interaction. His research has resulted in over 30 publications in top venues for human-computer interaction, user interface technology, accessibility, and machine learning, where he has received several best paper awards (CHI 2021, W4A 2021, IUI 2024) and honorable mention awards (CHI 2020, CHI 2023). His work has also been recognized outside of academic conferences by a Fast Company Innovation by Design Student Finalist Award, press coverage in major outlets such as TechCrunch and AppleInsider, and by the FCC Chair Awards for Advancements in Accessibility. Jason is a recipient of the NSF Graduate Research Fellowship and was selected as a Heidelberg Laureate Forum Young Researcher.