多模態人機互動
Multimodal Human-Computer Interaction
開課學期 Semester： 114-2
課號 Class number：CSIE7641 資工系研究所選修（歡迎外系與大學部加選）
授課教授 Instructor： 鄧善元 Shan-Yuan Teng
上課時間 Time： 星期二 Tuesday 789
上課地點 Location：學新 523 (CSIE Make Reality Space)
上課語言 Language：English
開課人數 Student Quota：30
Synopsis
Humans perform physical tasks with abundant multimodal information from visual, auditory, tactile, and more senses. Mainstream computers still offer very limited input and output capabilities when interacting with the users (e.g., multitouch gestures, speech input). This course provides project-based knowledge on human sensory perception & cognition and technologies that accommodate these, including topics on eye-tracking, auditory UI, robotic touch interfaces. The instructor will walkthrough user studies that deepen the understanding of modalities with hands-on programming sessions. Students will develop software-hardware applications that explore and leverage multimodal interaction and showcase them in live demonstrations. Later in the course, students will explore an open-ended research project guided by the instructor.
Prerequisites
Basic programming skills are required.
Syllabus
| Week | Date | Content | Resources |
|---|---|---|---|
| 1 | 2/24 | Intro to class | Install Processing |
| 2 | 3/3 | Auditory modality I | Audio programming Audio games |
| 3 | 3/10 | Auditory modality II | |
| 4 | 3/17 | Mini project 1 expo (student live demos) | |
| 5 | 3/24 | Visual modality I | Face tracking Web hacking |
| 6 | 3/31 | Visual modality II | Eye tracking |
| 7 | 4/7 | Mini project 2 expo (student live demos) | |
| 8 | 4/14 | Robotic workshop | |
| 9 | 4/21 | Haptic modality I | Robotic arm programming |
| 10 | 4/28 | Haptic modality II | Robotic arm applications |
| 11 | 5/5 | Mini project 3 expo (student live demos) | |
| 12 | 5/12 | Final project: brainstorming | |
| 13 | 5/19 | Final project: prototyping | |
| 14 | 5/26 | Final project: user study | |
| 15 | 6/2 | Final project: presentation | |
| 16 | 6/9 | Final project: paper submission |
Policies
- You must come to the first class to be enrolled in this class. If you are not enrolled but want to take this class, come to the first class. We will evaluate the number of students and let you know the availability before the second week.
- Physically attending each class is important (in class user studies and demos, both graded). If you cannot attend for any reason, you should email the instructor directly and it will be handled individually.
- No auditting. Our resources and space are limited.
- Students should prepare their own laptops with webcam and headphones.
Questions
Email multimodal-hci@csie.ntu.edu.tw; for personal request, please email the instructor directly.
Copyright
Otherwise stated, the copyright of the materials in the class belongs to Shan-Yuan Teng.