Highlighted Research Projects

NRT-AI: AWARE-AI: AWAREness for Sensing Humans Responsibly with AI (AWARE-AI NRT Program)

Cecilia Alm, PI
Rennie Bailey, Co-PI
Matt Huenerfauth, Co-PI
Esa Rantanen, Co-PI
Ferat Sahin, Co-PI
S.P. Bosworth, SP
Gabe Diaz, SP
Kristen Shinohara, SP
Chris Kanan, SP

Siwei Dodge, NRT Project Coordinator

National Science Foundation

AWARE-AI's vision is to bridge critical gaps in graduate education for training a convergent and creative research workforce in innovative, responsible AI. Multiple PhD programs integrate AI technical content (algorithms) yet fail to transmit skills for sensing humans responsibly such as human data methods, human-AI teaming techniques, explainable AI, and ethical evaluation or deployment. Yet, in the nation's near future, sensing-based predictive AI will increasingly appraise, filter, generate, react to, and perform complex actions using human data from multiple sources and modalities. Responsibility and explainability are essential for human-sensing AI systems, for as they become more ubiquitous, capable, and influential than prior generation automation, their unintended consequences will be felt on a larger, societal, scale. This motivates a paradigm shift to trustworthy AI that senses humans robustly, with a transformation in the training of aware, accountable, next-generation AI researchers in NSF fields.

REU Site: Computational Sensing for Human-centered AI

Cissi Alm, PI
Rennie Bailey, Co-PI
Kristen Shinohara, Mentor
Jamison Heard, Mentor

Alexander Ororbia, Mentor
Roshan Peiris, Mentor
Zhi Zheng, Mentor
Tracy Worrell, Facilitator

Tammi Wickson, Logistics Manager

National Science Foundation

The REU Site in Computational Sensing for Human-centered Artificial Intelligence recognizes that as the boundaries between HCI and AI blur, and AI grows increasingly agile and pervasive, the next generation of computational scientists must be capable of responsibly and effectively leveraging a spectrum of sensing data from data-generating humans. With this focus, the REU Site will expand its trajectory as an attractor for highly diverse students who will gain experience with sensing hardware and software towards transformative advances in intelligent systems focused on human behaviors and cognitive processes. Enabling diverse stakeholders early in their careers to discover how to collect, fuse, make inference with, and visualize multimodal human data can transform how humans and machines engage and collaborate. The research in the REU Site will address two limitations in AI: first, that underserved populations are at risk of being marginalized with the present focus on big data AI and, second, that AI trainees often lack experience with human data collection and critical thinking about human-elicited datasets. The REU Site will stimulate novel, safe ways for systems to team up with people to address society's vexing problems while remaining fair, attuned to bias, and representative of the diverse fabric of the general population.

The National General Aviation Flight Information Database (NGAFID)

Travis Desell, PI
MITRE Corporation, Inc.
The Federal Aviation Administration (FAA)

The purpose of this work is to maintain and further develop the National General Aviation Flight Information Database (NGAFID) and its integration within the Federal Aviation Administration's Aviation Safety Information Analysis and Sharing (ASIAS) System. The NGAFID provides a free to use interface for institutions and private pilots to upload flight data, which can be analyzed to track trends in exceedences (potential flight issues), reanimate nights for educational purposes, and provide more advanced interfaces to determine flight safety at fleet level. A major focus of this work is in using AI and machine learning on geospatial and time series data to determine potential flight hazards and accident precursors to make general aviation safer.

Image of eyes (people gazing)

Improved Semantic Segmentation with Natural Gaze Dynamics

Gabriel Diaz, PI
Reynold Bailey, Co-PI
Jeff Pelz, Co-PI
Meta Reality Labs

Current methods for semantic segmentation of eye imagery into pixels that represent the skin, sclera, pupil, and iris cannot leverage information about how the eye moves across time, or gaze dynamics. This limitation is a symptom of the datasets on which models are trained, which reflect individual, temporally non-contiguous frames of the eyes looking at known locations. To address this issue, we will develop a computer graphics rendering pipeline for the generation of datasets that represent sequences of eye movements faithful to natural gaze dynamics. In addition, we will develop models that leverage these dynamics in the semantic segmentation of eye imagery on the next frame through a process of short-term prediction. Finally, these models will be used in a series of studies designed to evaluate the potential benefit of segmentation models that have been trained using synthetic gaze sequences when applied to real-world recordings of gaze sequences.

Image of Articulatory and Perceptual Constraints

Collaborative Research: Multimethod Investigation of Articulatory and Perceptual Constraints on Natural Language Evolution

Matthew Dye, PI
Andreas Savakis, Co-PI
Matt Heunerfauth, Co-PI
Corrine Occhino, Co-PI
National Science Foundation

The evolution of language is something that appears to be unique to the human species. Viewed as a cognitive tool, language is a powerful system responsible for the cultural and technological advance of human civilization. Like physical tools, such as the wheel or fire, cognitive tools have the power to shape their user. That is, languages need not only evolve by change within linguistic systems themselves, but also through changes in the organisms (humans) that use those languages. The research proposed here will focus upon one particular type of human linguistic system - signed languages - and on one aspect of human organisms - spatial visual attention. It will ask the fundamental question: to what extent have signed languages evolved over generations to conform to the human visual and articulatory systems, and to what extent is the human visual attention system shaped by the use of a signed language within the lifetime of an individual signer?

Twenty-First Century Captioning Technology, Metrics and Usability

Matt Huenerfauth, PI
DHHS ACL/Gallaudet University

Captioning plays an important role in making video and other media accessible for many people who are Deaf or Hard of Hearing. This collaborative research project with Gallaudet University will investigate the design of metrics that can predict the overall quality of text captions provided during video for people who are Deaf or Hard of Hearing (DHH). In addition, this project will study the presentation, display, and user experience of people who are DHH viewing captions. This project will include focus groups, interviews, surveys, and experimental studies with several thousand DHH participants across the U.S., with studies occurring at Gallaudet, at RIT, and at other U.S. locations, to obtain input and feedback from a diverse cross-section of the U.S. DHH population. Through this work, we will identify key requirements from stakeholders as to the quality of captions, identify factors that can be used to implement software-based automatic metrics for evaluating caption quality, and identify new methods for modifying the presentation and display of captions, to boost the overall user experience of DHH users. This project will create software tools, as well as a captioned video collection, which will serve as a critical research infrastructure for empirical research on caption quality and accessibility. Outcomes of this project include the creation of information materials and outreach to the DHH community about captioning technologies, as well as software and standards for how to best evaluate captions of videos for these users.

Image of Automatic Text Simplication and Reading Assistance

Automatic Text-Simplification and Reading-Assistance to Support Self-Directed Learning by Deaf and Hard-of-Hearing Computing Workers

Matt Huenerfauth, PI
National Science Foundation

While there is a shortage of computing and information technology professionals in the U.S., there is underrepresentation of people who are Deaf and Hard of Hearing (DHH) in such careers. Low English reading literacy among some DHH adults can be a particular barrier to computing professions, where workers must regularly "upskill" to learn about rapidly changing technologies throughout their career, often through heutagogical (self-directed) learning, outside of a formal classroom setting. There has been little prior research on self-directed learners with low literacy nor on automatic text-simplification reading-assistance systems for DHH readers, who have a unique literacy profile. Our interdisciplinary team includes researchers and educators with expertise on DHH computing education, natural language processing researchers with expertise in text simplification technologies, and accessibility researchers with expertise in conducting empirical studies with large numbers of DHH users evaluating assistive technologies. RIT is an ideal setting for this study, as it is home to the National Technical Institute for the Deaf, including DHH computing students who engage in workplace experiences as part of a senior capstone course, in which they must heutatogogically learn about new technologies for work projects. As a research vehicle for this study, we will implement a web-browser plug-in that provides automatic English text simplification (on-demand) for DHH individuals, including providing simpler synonyms or sign-language videos of complex English words or simpler English paraphrases of sentences or entire documents. By embedding this prototype for use by DHH students as they learn about new computing technologies for workplace projects, we will evaluate the efficacy of our new technologies.

Images of Automatic Speech Recognition

Critical Factors for Automatic Speech Recognition in Supporting Small Group Communication Between People who are Deaf or Hard of Hearing and Hearing Colleagues

Matt Huenerfauth, PI
National Science Foundation

To promote inclusion and success of D/HH employees in workplace communication, we investigate the use of Automatic Speech Recognition (ASR) technology for automatically providing captions for impromptu small-group interaction. We will conduct interviews with D/HH users, employers, and hearing co-workers; participatory design sessions and prototype usability testing with users; lab-based studies investigating how the presentation of ASR text output may influence the speaking behavior of hearing colleagues; experimental sessions with pairs or small groups of D/HH and hearing individuals collaborating on a problem solving-task while using a prototype ASR communication system; and observations of the use of prototype designs in real workplace settings. The project will result in human-computer interaction design and evaluation guidelines for the use of ASR in small group communication; broader impacts include societal benefits and STEM research opportunities for DHH students.


Image of Scalable Integration of Methods for Large Vocabulary Sign Recognition

Scalable Integration of Data-driven and Model-based Methods for Large Vocabulary Sign Recognition and Search

Matt Huenerfauth, PI
National Science Foundation

Sign recognition from video is still an open and difficult problem because of the nonlinearities involved in recognizing 3D structures from 2D video, as well as the complex linguistic organization of sign languages. Purely data-driven approaches are ill-suited to sign recognition given the limited quantities of available, consistently annotated data and the complexity of the linguistic structures involved, which are hard to infer. Prior research has, for this reason, generally focused on selective aspects of the problem, often restricted to limited vocabulary, resulting in methods that are not scalable. We propose to develop a new hybrid, scalable, computational framework for sign identification from a large vocabulary, which has never before been achieved. This research will strategically combine state-of-the-art computer vision, machine-learning methods, and linguistic modeling. Looking up an unfamiliar word in a dictionary is a common activity in childhood or foreign-language education, yet I there is no easy method for doing this in ASL. The above framework will enable us to develop a user-friendly, video-based sign-lookup interface, for use with online ASL video dictionaries and resources, and for facilitation of ASL annotation. This research will (1) revolutionize how deaf children, students learning ASL, or families with deaf children search ASL dictionaries; (2) accelerate research on ASL linguistics and technology, by increasing efficiency, accuracy, and consistency of annotations of ASL videos through video-based sign lookup; and (3) lay groundwork for future technologies to benefit deaf users, e.g., English-to-ASL translation, for which sign-recognition is a precursor. The new linguistically annotated video data and software tools will be shared publicly.

Images of 2 men signing

Data & AI Methods for Modeling Facial Expressions in Language with Applications to Privacy for the Deaf, ASL Education & Linguistic Research

Matt Huenerfauth, PI
National Science Foundation/Rutgers

This multi-university research project investigates robust artificial-intelligence methods for facial analytics that can be widely used across domains, applications, and language modalities (signed and spoken languages). This proposed work includes (1) extensions to ASL-based tools, AI methods, and data sharing; (2) an application to enable researchers to contribute video clips for analysis of facial expressions and head gestures; (3) an application to help ASL learners produce more accurately the types of facial expressions and head gestures that convey essential grammatical information in signed languages; and (4) a tool for real-time anonymization of ASL video communications, to preserve essential grammatical information expressed on the face/head in sign languages (SL) while de-identifying the signer in videos to be shared anonymously. Experimental user studies to assist in the in design of (3) and (4) will be conducted by Huenerfauth at RIT. Individuals who are Deaf and Hard of Hearing will be actively involved the research, including perceptual experiments to test comprehensibility and to investigate factors that influence acceptance of the ASL animations. This Phase I project lays the groundwork for a future Phase II proposal to the NSF Convergence Accelerator program.

CAREER: Brain-inspired Methods for Continual Learning of Large-scale Vision and Language Tasks

Christopher Kanan, PI
National Science Foundation

The goal of this research project is to create deep neural networks that excel in a broad set of circumstances, are capable of learning from new data over time, and are robust to dataset bias. Deep neural networks can now perform some tasks as well as humans, such as identifying faces, recognizing objects, and other perception tasks. However, existing approaches have limitations, including the inability to effectively learn over time when data is structured without forgetting past information, learning slowly by looping over data many times, and amplification of pre-existing dataset bias which results in erroneous predictions for groups with less data. To overcome these problems, this research project aims to incorporate memory consolidation processes inspired by the mammalian memory system that occur both when animals are awake and asleep. The new methods developed in this project could lead to machine learning systems that 1) are more power efficient, 2) can learn on low-powered mobile devices and robots, and 3) can overcome bias in datasets. In addition, a significant educational component involves training the next generation of scientists and engineers in deploying machine learning systems that are safe, reliable, and well tested via new courses and programs.

Image of Starcraft II

Lifelong Learning in Brain-Inspired Deep Neural Networks

Christopher Kanan, PI
Primarily supported by:
L2M – TA2 – Multi-Stage, Multi-Task Memory Transfer (M3T)
DARPA/SRI International

RI: Small: Lifelong Multimodal Concept Learning
National Science Foundation

Kanan's lab has created many new datasets and algorithms for lifelong machine learning over the past three years. In 2021, we published a review article on “replay,” a mechanism used in the brain and in deep neural networks for memory consolidation, with collaborators that juxtaposes the differences between how this happens in the brain vs. artificial neural networks today. We also pioneered continual learning for analogical reasoning by progressively training neural networks to solve Raven’s Progressive Matrices tasks (Hayes & Kanan, 2021). My lab worked with many other groups to create the Avalanche toolbox to make continual learning more reproducible and easier to conduct. My lab has also been collaborating with SRI to integrate the lifelong learning methods we developed into their AI systems for playing Starcraft 2, where we have enabled their system to rapidly learn when situations are dangerous. My lab has also been working hard to create algorithms that mitigate bias and on self-supervised learning systems for lifelong machine learning.

Image of Invasive Plants

Detecting Invasive Species via Street View Imagery

Christopher Kanan, PI
Christy Tyler, Co-PI
Primarily supported by NY DEC

As part of this project, we have compiled a large database of invasive plant species and street view images. The goal of the project is to use a deep learning system to automatically detect these species so that they can be better monitored. In 2021, we made significant progress on the algorithm and we intend to publish the results of the study in 2022 for two invasive species.

Dynamic Scene Graphs for Extracting Activity-based Intelligence image

Dynamic Scene Graphs for Extracting Activity-based Intelligence

Yu Kong, PI
Qi Yu, Co-PI

Activity-based Intelligence (ABI) is an analysis methodology, which rapidly integrates data from multiple intelligence sources around the interactions of people, events, and activities, in order to discover relevant patterns, determine and identify change, and characterize those patterns to drive collection and create decision advantage. In this project, we plan to develop Dynamic Scene Graphs over large-scale multimodal time series data for representation learning. The new representation will enable learning from complex and dynamic environment, where a variety of vision tasks can be achieved including open-set object classification and detection, event detection, question-answering, and dense captioning.

Hochgraf research group

Effective and Efficient Driving for Material Handling

Michael Kuhl, PI
Amlan Ganguly, Co-PI

Clark Hochgraf, Co-PI
Andres Kwasinski, Co-PI
National Science Foundation

In warehousing operations involving a mix of autonomous and human-operated material handling equipment and people, effective and efficient driving is critical. We propose to address a set of integrated areas of research to enable improved real-time decision making leading to improved productivity, information, and communication. These include human-robot interaction/collaboration – avoiding incidents and improving predicted actions; and robust, low latency, secure vehicle to vehicle and system communication.

Image of Developing a Hands-On Data Science Curriculum for Non-Computing Majors

Developing a Hands-on Data Science Curriculum for Non-Computing Majors

Xumin Liu, PI
Erik Golen, Co-PI
National Science Foundation

This project aims to serve the national interest by addressing the particularly high national demand for data scientists, which is estimated to grow much faster than the average for all occupations between 2018 to 2028. Moreover, recent national challenges have highlighted the nation’s ongoing need for data scientists who can rapidly create actionable results from unprecedented amounts of data being generated in various fields. Computing and mathematics departments in colleges and universities across the U.S. have increased their efforts to provide data science content for their own undergraduate majors, but typically in their senior year. This project aims to develop a data science curriculum that can attract non-computing majors. Such an option does not require a long prerequisite chain of courses in programming, data structures, and introductory databases, as well as relevant mathematics. By directly addressing this challenge of broadening the early engagement of all students in data science, this project will create a hands-on curriculum that will make learning more readily-accessible to non-computing majors.

Trilobyte image

Trilobyte – Autonomous Learning for Full-Spectrum Sims

Alexander Loui, PI
Carl Salvaggio, Co-PI
L3 Harris Technologies

Generate valuable multimodal sUAS-based image data of a single scene. Generate a corresponding DIRSIG scene and accompanying synthetic image data for training. Using deep learning techniques used in prior research to automatically build and label a synthetic scene.

Enabling Efficient 3D Perception image

Collaborative Research: SHF: Small: Enabling Efficient 3D Perception: An Architecture-Algorithm Co-Design Approach

Guoyu Lu, PI
National Science Foundation

The objective of the proposed research is to rethink the systems stack, from algorithms to hardware, for 3D perception, i.e., point cloud processing, so as to enable 3D perception as a fundamental building block in emerging domains such as autonomous driving, Augmented/Virtual Reality, and smart agriculture.


Young Investigator Program (YIP) Award: Theory and Efficient Algorithms for Dynamic and Robust L1-norm Analysis of Tensor Data

Panos Markopoulos
Air Force Office of Scientific Research

The main objective of this project is to develop theory and efficient algorithms for dynamic and robust analysis of multi-modal (tensor) data, based on LI-norm formulations. Specifically, we will first formulate Stochastic L1-norm principal-component analysis (L1-PCA) and investigate for the first time its theoretical underpinnings (graph of metric function, connection to batch formulations, connection to standard stochastic principal component analysis, etc.) Then, we will develop efficient online algorithms for the solution of this problem, based on solid stochastic approximation theory. Next, we will expand these methods for the analysis of tensor data, in the form of dynamic robust decomposition. In addition, emphasis will be placed on the development of scalable algorithms that can be used in systems with limited computational resources.

Image of Panos Markopoulos with student

Continual and Incremental Learning with Tensor-Factorized Neural Networks

Panos Markopoulos
Air Force Research Laboratory/Griffiss Institute

In this project, we will investigate the capability of tensor-based network factorization to allow for incremental and continual learning.

Image representing L1-Norm-Based Data Analysis

Collaborative Research: CDS&E: Theoretical Foundations and Algorithms for L1-Norm- Based Reliable Multi-Modal Data Analysis

Panos Markopulous, PI
Andreas Savakis, Co-PI
National Science Foundation

This project focuses on providing theoretical foundations and algorithmic solutions for reliable, L1-norm-based analysis of multi-modal data (tensors). The proposed research is organized in three main thrusts. In Thrust 1, we will focus on investigating theoretically (e.g., hardness and connections to known problems) and solving L1-norm-based Tucker and Tucker2 decompositions (L1-TUCKER and L1-TUCKER2, respectively). In Thrust 2, we will focus on developing efficient (i.e., low-cost, near-optimal) and distributed solvers for L1-TUCKER and L1-TUCKER2, appropriate for the analysis of big data and data in the cloud. In Thrust 3, we will investigate the application of the developed algorithms to tensor analysis paradigms from the fields of computer vision, deep learning, and social-network and stock-content data analytics. Overall, this project aspires to provide algorithmic solutions that will support reliable data-enabled research in a plethora of disciplines across science and engineering.

Analysis of stop signs.

Towards Adversarially Robust Neuromorphic Computing

Cory Merkel
Air Force Research Laboratory

The recent artificial intelligence (AI) boom has created a growing market for AI systems in size, weight, and power (SWaP)-constrained application spaces such as wearables, mobile phones, robots, unmanned aerial vehicles (UAVs), satellites, etc. In response, companies like IBM, Intel, and Google, as well as a number of start-ups, universities, and government agencies are developing custom AI hardware (neuromorphic computing systems) to enable AI at the edge. However, many questions related to these systems' security and safety vulnerabilities still need to be addressed. Since the early 2000's a number of studies have shown that AI and especially deep learning algorithms are susceptible to adversarial attacks, where a malicious actor is able to cause high-confidence misclassifications of data. For example, an adversary may easily be able to hide a weapon from an AI system in plain sight by painting it a particular color. A number of defense strategies are being created to improve AI algorithm robustness, but there has been very little work related to the impact of hardware-specific characteristics (especially noise in the forms of low precision, process variations, defects, etc.) on the adversarial robustness of neuromorphic computing platforms. To fill this gap, we propose the following research objectives: Objective 1. Evaluate and model the effects of noise on the susceptibility of neuromorphic systems to adversarial attacks. Objective 2. Design adversarially-robust training algorithms for neuromorphic systems. Objective 3. Develop novel adversarial attacks that leverage hardware-specific attributes of neuromorphic systems.

CAREER: A Computational Approach to the Study of Behavior and Social Interaction

Ifeoma Nwogu, PI
National Science Foundation

This work investigates the best practices for modeling emotion in speech in order to determine what speech-based emotion is most prominent during entrainment. Commonly used features for emotion analysis were compared using different models, Hidden Markov Models (HMMs), Long Short Term Memory networks (LSTMs) and 1-dimensional temporal convolutional neural networks (1D TConv). These models were trained on a benchmark dataset consisting of seven different emotions and then applied to real-life conversations to analyze the extent of entrainment. The models were evaluated on their accuracy to correctly recognize emotion, and then applied for analyzing real-life conversations. The ultimate goal is to develop technologies to support mental health disorders in the clinic. We are also investigating several other modalities in a similar manner.

Image of neural networks and fusion for learning from multiple  data modalities

Target Detection/Tracking and Activity Recognition from Multimodal Data

Eli Saber, PI
Panos Markopoulos, Co-PI
Raymond Ptucha, Co-PI

National Geospatial-Intelligence Agency

Our primary objectives in this proposal are to develop operational target detection/tracking techniques and activity recognition/understanding methods that possess the capability to leverage multimodal data in a fusion framework using deep learning algorithms and coupled tensor techniques while providing accurate and near real-time performance. The proposed methodology is divided into four major stages: (1) pre-processing, (2) co-registration and fusion, (3) target detection and tracking, and (4) activity recognition and scene understanding. The aforementioned algorithms will be benchmarked against existing state of the art techniques to highlight their advantages and distinguish their abilities. The above is intended to assist analysts to effectively and efficiently ingest large volumes of data, or perform object detection and tracking in real-time under dynamic settings.

Data-Driven Adaptive Learning for Video Analytics

Andreas Savakis, PI
Christopher Kanan, Co-PI
Panos Markopoulos, Co-PI
Air Force Office of Scientific Research

The objective of this research project is to design a Data Driven Adaptive Learning framework for visual recognition. The adaptive nature of this framework is suitable for recognition in new domains, that are different from those used for training, and in data starved environments where the number of available training samples is small. We design classification engines that learn incrementally, without full retraining, using continuous updating methods as new data become available. These adaptive learners will incorporate weakly labeled data as well as human in the loop to facilitate annotation of previously unseen data, and will adapt to new environments dynamically in cooperation with multimodal sensors and a dynamic control unit. We design, implement and test the following classifier engines as they have high potential to operate effectively within our framework: a) incremental subspace learning using robust techniques and related applications to classification and adaptation to new domains; b) adaptive deep learning and applications to recognition of people, vehicles, etc.

Image of multimodal sensng, domain adaptation and transfer learning

Multimodal Sensing, Domain Adaptation and Transfer Learning

Andreas Savakis, PI

The proposed project is to extend basic research in contemporary areas that supports the vision of artificial intelligence (AI) autonomy for real time operational support. AI for computer vision consists of Deep Learning and Machine Learning from which typical examples focus on multi-media content such as imagery; but basic research theory elements of joint multimodal DL have yet to be realized for intelligence, surveillance, and reconnaissance applications such as electro-optical (EO) and synthetic aperture radar (SAR). Additionally, there is a need to transfer from one domain (e.g., SAR) to another domain (e.g. EO) in contexts were data availability is limited.

Image of Global Surveillance Augmentation

Global Surveillance Augmentation Using Commercial Satellite Imaging Systems

Andreas Savakis, PI
USAF SBIR (with Kitware Inc.) 
NYS/UR Center for Emerging and Innovative Systems (CEIS)

The proposed project is to extend basic research in contemporary areas that supports the vision of artificial intelligence (AI) autonomy for real time operational support. AI for computer vision consists of Deep Learning and Machine Learning from which typical examples focus on multi-media content such as imagery; but basic research theory elements of joint multimodal DL have yet to be realized for intelligence, surveillance, and reconnaissance applications such as electro-optical (EO) and synthetic aperture radar (SAR). Additionally, there is a need to transfer from one domain (e.g., SAR) to another domain (e.g. EO) in contexts were data availability is limited.

Image of Multiple Attribution Network based Fake Imagery Detection

MANFID: Multiple Attribution Network based Fake Imagery Detection

Andreas Savakis, PI

We propose a novel deep learning approach for the detection of GAN generated fake images. A multiple attribution network is proposed for the detection of GAN fingerprints in images in order to determine whether an image is real or fake. The method will be trained and tested for robustness to noise, geometric transformations and compression.

Peri-procedural transmural electrophysiological imaging of scar-related ventricular tachycardia

Linwei Wang, PI
DHHS/National Institutes of Health

The majority of life-threatening ventricular arrhythmia episodes are caused by an electrical "short circuit", formed by a narrow channel of surviving tissue inside myocardial scar. An important treatment is to target and destroy these culprit slow-conducting channels. Unfortunately, with conventional catheter mapping, up to 90% of the VT circuits are too short-lived to be mapped. For the 10% “mappable” VTs, their data are available only during ablation and limited to one ventricular surface. This inadequacy of functional VT data largely limits our current ablation strategies. The goal of this proposal is to develop a novel technique to provide pre- and post-ablation functional arrhythmia data – integrated with LGE-MRI scar data in 3D – to improve ablation with pre-procedural identification of ablation targets and post-procedural mechanistic elucidation of ablation failure. Its specific objectives include: 1) To develop and validate a peri-procedural electrocardiographic imaging (ECGi) technique for mapping scar-related VT circuits. 2) To integrate LGE-MRI scar into ECGi for improved electroanatomical investigations of VT circuits. 3) To perform clinical evaluation of pre-ablation and post-ablation MRI-ECGi of scar-related VT. This project is carried out in collaboration with Siemens Healthineers, Nova Scotia Health Authority, and University of Pennsylvania.

Visualization of parts of human body and computer processing of disease processes

Utilizing Synergy between Human and Computer Information Processing for Complex Visual Information Organization and Use

Qi Yu, PI
Anne Haake, Co-PI
Rui Li, Co-PI
Pengcheng Shi, Co-PI
National Science Foundation

This project will research computational models to represent different elements in human knowledge, aiming to understand how their interaction with image data underlies human image understanding. The modeling outcomes will inform algorithmic fusion of human knowledge with image content to create novel representations of image semantics as a result of human-machine synergy.

Measurement of physiological arousal and stress in children with ASD image

Measurement and relationship of physiological arousal and stress in children with ASD and caregivers

Zhi Zheng, PI
Peter Bajorski, Co-PI
NIH/UR Center for Emerging and Innovative Systems (CEIS)

The proposed research is a cross-sectional evaluation of the bidirectional relationship between parent and child arousal in dyads that include a child with autism spectrum disorder (ASD) and their primary caregiver. We propose the use of a wearable device that captures physiological stress data in parent/child dyads. Caregivers will complete diaries to provide context to specific events during which arousal and stress are likely to be high. The focus on the multiple factors that influence caregiver health will help meet both current and future health care needs in this population, which to date are largely unexplored.  RIT researchers will be responsible for the configuration of physiological sensors, physiological data processing and analysis, as well as statistical analysis for the whole project.