Projects and Output

Project 1: Energy-efficient Object Segmentation for a Sustainable Future

Current forms of artificial intelligence (AI), including generative AI, demand significant computational resources, which are often neither environmentally sustainable nor cost-effective. In contrast, the human brain exhibits remarkable energy efficiency, heavily utilizing recurrence, sparse patterns of activity, and specialized neurological pathways. In this project, we will investigate brain-inspired modifications of AI architectures in terms of constrained, sparse, temporal-oriented artificial neural networks, drawing insights from how the brain processes information. We will explore these concepts in the context of visual perception, particularly focusing on the development of more efficient and generative forms of scene segmentation and object recognition.

Mentors: Alexander Ororbia, Cecilia Alm, and Reynold Bailey

Project 3: LLM-based Language Tutors for Different Languages

Large Language Models (LLMs) can generate fluent feedback, scaffold writing, and simulate conversational partners with impressive realism. However, most existing educational applications built on these models focus heavily on English, leaving open the question of how well LLMs support learners of other languages. Their ability to handle culturally appropriate expressions, non-Latin writing systems, or pragmatically sensitive contexts remain underexplored. This project invites students to design and evaluate LLM-driven applications for learners of other languages. It specifically aims to explore whether LLMs effectively support non-English language learners and, if so, how to ensure cultural appropriateness and adaptive difficulty across different linguistic contexts.

Mentors: Hakyung Sung, Cecilia Alm, and Reynold Bailey

Project 1: Sensing fatigue during video conferencing using physiological signals

Video conferencing using software tools such as Zoom has become a necessity in recent times due to the COVID-19 pandemic situation. While video conferencing is productive for many, research has shown that prolonged hours of video conferencing can lead to psychological effects such as fatigue, a.k.a. 'Zoom fatigue'. Thus, the goal of this project is to develop a Zoom fatigue sensing method that leverages physiological signals. Based on previous research, the project aims to explore eye tracking-based methods (sensing blink rates and speeds) and/or speech patterns (changes in voice energy, etc.) to detect Zoom fatigue of video conferencing participants.

Mentors: Roshan Peiris, Reynold Bailey, and Cecilia O. Alm

Project 3: Adaptive interaction for inclusive social robots

Social agents and robots are becoming more prevalent in everyday life, from Siri to autonomous robotic cleaners. However, the design of how these agents interact with humans has typically focused on the general population, and these interactions are sometimes ineffective for specific groups of users. An adaptive interaction framework may promote inclusiveness by detecting ineffective interactions, reasoning why an interaction failed, and adapting the framework’s interaction strategy appropriately. This project will rely on physiological signals and computer-vision techniques to detect when an interaction between an agent and a human is ineffective, including when this interaction is between an agent and someone who is Deaf or Hard of Hearing.

Mentors: Jamison Heard, Reynold Bailey, and Cecilia O. Alm

Project 5: Neural task assistants for resolving confusion in multimodal, complex tasks

Multimodal language modeling remains a significant challenge to this day. Addressing some of the tasks involved would benefit language-driven dialogue systems that form the backbone of various applications on mobile devices and systems in the home. In this project, students will explore statistical learning techniques for dialogue that consider human data modalities, such as facial expressions, in the implementation of more human-like agents, including those that are capable of recognizing and resolving confusion between human interlocutors. Students will also evaluate the dialogue assistants in a user study.

Mentors: Alex Ororbia, Cecilia O. Alm, and Reynold Bailey

Project 1: Measuring engagement in mixed reality applications

Mixed reality has increased in applications across education, entertainment, and manufacturing. We will study how multimodal data--skin response, pulse, facial expressions, eye gaze, speech, etc.--aids understanding of engagement levels in immersive environments with diverse users. We will seek to develop and evaluate a flexible metric of engagement for optimizing user experience in immersive applications

Mentors: Joe Geigel, Reynold Bailey, and Cecilia O. Alm

Project 3: Intelligent journalism

Automation and machine-generation of news content is changing newsrooms' practices, the news consumer's understanding of what journalism is, and how news is selected and presented. We will use multimodal behavioral data (gaze, facial expressions, biophysical signals) to study perception of machine-generated content versus stories written by journalists. Influences of credibility, objectivity, and political bias will be investigated.

Mentors: Ammina Kothari, Reynold Bailey, and Cecilia O. Alm

Project 5: Augmented transfer learning for deep amusement

Positive experiences boost human well-being and productivity. We will attempt to explore innovative methods of achieving sensing-based inference of human amusement with deep learning and modest data. Using relevant base models and recent techniques in transfer learning and data augmentation, verbal and nonverbal sensing data elicited while diverse subjects view or read amusing content will be infused into systems to infer ranges of positive expressions and intensities of amusement.

Mentors: Raymond Ptucha and Cecilia O. Alm

Project 1: Effects of immersive viewing of live performance in a virtual world

This project will build on our work in virtual theatre; which we define as shared, live performance, experienced in a virtual space, with participants contributing from different physical locales. In particular, we will explore how immersive viewing of a virtual performance (using a head mounted display such as an Oculus Rift) affects audience engagement compared viewing the same performance in a non-immersive environment (such as on a screen or monitor). Galvanic skin response, facial expression capture, and possibly eye gaze will be used to gauge audience interest and engagement. In addition, speech capture (of audience descriptions and impressions of the performance) will enable emotional prosody analysis.

Mentors: Joe Geigel, Reynold Bailey, and Cecilia O. Alm

Project 3: Multi-modal sensing and quantification of atypical attention in autism spectrum disorder

While autism spectrum disorder (ASD) is primarily diagnosed through behavioral characteristics, behavioral studies alone are not sufficient to elucidate the mechanisms underpinning ASD-related behavioral difficulties. In previous research, behavioral difficulties associated with ASD have in part been attributed to atypical attention towards stimuli. Eye-tracking has been previously used to quantify atypical visual saliency in ASD during natural scene viewing. In this project, we will explore the potential of electroencephalography (EEG) in providing insight into the neurological correlates of information processing during natural scene viewing. We will integrate EEG and eye-tracking data in machine learning models, with the goal to uncover the attentional and neurological correlates of viewing performance between typical and ASD subjects. These findings may also provide potential markers for early diagnosis of ASD. In this REU project, together with the faculty mentors, student researchers will extend the current state-of-the-art, conduct experiments to collect necessary data, and utilize machine learning and statistical methods to analyze the collected data with a goal to uncover the mechanism underlying atypical attention in ASD.

Mentors: Linwei Wang, Hadi Hosseini, and Catherine Qi Zhao

Project 5: Alignment of gaze and dialogue

Automated word alignment is a first step in traditional statistical machine translation. We have used alignment in a computational framework that establishes meaningful relationships between observers’ gaze and co-collected spoken language during descriptive image/scene inspection, for machine-annotation of image/scene regions. REU student researchers will extend the current framework to task-based dialogues with pairs of observers discussing visual content. We may also explore an alternative metric for assessing multimodal alignment quality when the alignment input consists of visual-linguistic data from more than one observer.

Mentors: Cecilia O. Alm, Preethi Vaidyanathan, Reynold Bailey, and Ernest Fokoué

Project 1: Attention and behavior of students in online vs. face-to-face learning contexts

In the United States, online learning enrollments are on the rise and competing in growth with traditional course offerings. How is this shift in STEM educational delivery impacting attention, behavior, and performance of students? This project explores how online vs. face-to-face learning contexts influence students. We will use various sensing devices that capture facial expressions, skin response, etc. We will examine what happens when students are given performance incentives, and also consider the role of content difficulty. Project outcomes will contribute to improving individualized learning experiences.

Project 3: An interactive dermatological diagnosis system

This project will investigate models and algorithms to best leverage human expertise in building machine-learning systems. Active and interactive learning techniques will be explored where the former focuses on minimizing humans’ input to the system and the latter aims to elicit and monitor the most effective user feedback to improve the learning outcome. Students will research a number of interaction strategies between human users and a learning system and help enhance existing active and interactive learning models.

Project 5: Visual-linguistic alignment with static and dynamic imagery

In statistical machine translation, bilingual word alignment is a first step in translating text from one language to another. Such algorithms can also establish meaningful relationships between observers’ gaze data and their co-collected verbal descriptions, for machine-annotation of visual content. REU student researchers will extend the current visual-linguistic alignment framework to video, consider holistic and region-based annotation for visual content with positive or negative valence, and explore a novel approach to human assessment of alignment quality within the context of different applications.

Project 1: Real-time attention and engagement monitoring

In recent work, we used a microphone to record speech and a camera to capture facial expressions of individuals engaged in computer-based tasks. The nature of the tasks were manipulated to induce moderate stress. We were later able to fuse both sources of data to reliably predict when the user was stressed. Students involved in this project will build on these ideas and develop techniques for real-time monitoring of user stress, attention, and cognitive load from multisource data towards monitoring in online learning systems.

Project 3: Multimodal interface with adaptive user feedback

We will extend techniques to systematically fuse multimodal data, representing experts’ domain knowledge, to improve image understanding. Students will enhance a prototype multimodal interactive user interface with interactive machine learning, to effectively collect and incorporate user feedback to enhance the data fusion results.

Project 5: Visual-linguistic alignment and beyond

In statistical machine translation, bitext word alignment is the first step in translating text from one language to another. Alignment algorithms can establish meaningful relationships between observers’ gaze data and their co-collected verbal image descriptions, for image annotation or classification. REU student researchers will (1) extend the bi-modality alignment framework to new modalities, (2) automate transcription with speech recognition, and (3) explore alignment quality considering factors such as concept concreteness and specificity, image domain, and observer background knowledge