Multimodal Human-Computer Interaction

  • The class are compreised of mini projects that students demonstrate to one another in mini expos.
  • Selected topics on human modalities will be explored in this semsester (subject to change).
  • You will learn a lot in many in-class activies, project demonstration, and also from your classmates!
  • This class will take place at Make Reality Space, CSIE's makerspace.

Semester: 114-2 (Spring 2026)
Class number: CSIE7641 (Welcome non-CS students!)
Instructor: Shan-Yuan Teng
Time: Tuesday 789
Location: MK Innovation Hall 523 (CSIE Make Reality Space)
Language: English
Student Quota: 30
How to enroll: You must come to the first class and we will notify you by next class

Synopsis

Humans perform physical tasks with abundant multimodal information from visual, auditory, tactile, and more senses. Mainstream computers still offer very limited input and output capabilities when interacting with the users (e.g., multitouch gestures, speech input). This course provides project-based knowledge on human sensory perception & cognition and technologies that accommodate these, including topics on eye-tracking, auditory UI, robotic touch interfaces. The instructor will walkthrough user studies that deepen the understanding of modalities with hands-on programming sessions. Students will develop software-hardware applications that explore and leverage multimodal interaction and showcase them in live demonstrations. Later in the course, students will explore an open-ended research project guided by the instructor.

Prerequisites

Basic programming skills are required.

Syllabus

Week Date Content Resources (TBA)
1 2/24 Intro to class  
2 3/3 Auditory modality I  
3 3/10 Auditory modality II  
4 3/17 Mini project 1 expo (student live demos)  
5 3/24 Visual modality I  
6 3/31 Visual modality II  
7 4/7 Mini project 2 expo (student live demos)  
8 4/14 Robotic workshop  
9 4/21 Haptic modality I  
10 4/28 Haptic modality II  
11 5/5 Mini project 3 expo (student live demos)  
12 5/12 Final project: brainstorming  
13 5/19 Final project: pilot user study  
14 5/26 Final project: user study  
15 6/2 Final project: presentation (student live talk)  
16 6/9 Final project: paper submission  

Policies

  • You must come to the first class to be enrolled in this class. We will evaluate the number of students and let you know the availability before the second week.
  • Physically attending each class is important (in class user studies and demos, both graded). If you cannot attend for any reason, you should email the instructor directly and it will be handled individually.
  • No auditting. Our resources and space are limited.
  • Students should prepare their own laptops with webcam and headphones.

Questions

Email multimodal-hci@csie.ntu.edu.tw; for personal request, please email the instructor directly.

TA recruitment

If you have experiences in HCI research, user studies, psychology, neuroscience, etc, being either undergrad/grad students (CS or non-CS), email Shan-Yuan for Teaching Assistant opportunity!