Research

Dynamic Scene Graphs for Extracting Activity-based Intelligence

ARO

PI: Dr. Yu Kong, Co-PI: Dr. Qi Yu. 

May 2021 - April 2024

The primary objective of this project is to explore a dynamic learning framework that effectively extracts ABI in various highly complicated mission operations. This project plans to develop Dynamic Scene Graphs over large-scale multimodal time series data for representation learning. The new representations will enable learning from complex and dynamic environment, where a variety of vision tasks can be achieved including open-set object classification and detection, event detection, question-answering, and dense captioning. The proposed framework will achieve the following four properties: (i) Combining the strength of multimodal data for learning effective representations. (ii) Enabling open-set object classification and detection by learning the boundary of data distributions. This allows the proposed framework to understand novel objects in open-world scenarios. (iii) Effective event detection and model adaptation from limited samples. The proposed event detection model is practical as it can be efficiently adapted to novel events using few training samples. (iv) Interactive learning with human users via active question-answering and semantic description generation. This project will create a novel framework that allows users to communicate with machines and integrate human knowledge into the machines. All of these properties enable the extraction of ABI to support mission operations.

Generalized Visual Forecasting

Visual Forecasting Diagram

Generalized Visual Forecasting

Seeing into the future is one of the most appealing powers, featured in hundreds of movies and books. Researchers in computer vision has developed various methods to attempt to gain this power. Possible methods include human action prediction, intention prediction, trajectory prediction, video prediction, and so on. These prediction methods enhance technology’s capability of seeing into the future, which has broad applications in practical scenarios where high-risk events may happen in the near future, such as crime prevention, traffic accident avoidance, and health care. It is extremely necessary to predict these events before they happen, rather than performing after-the-fact analysis. In this project, we plan to focus on the visual surveillance domain, and create a group of computational algorithms for visual prediction in uncontrolled scenarios.

A Multimodal Dynamic Bayesian Learning Framework for Complex Decision-making

Data collection

A Multimodal Dynamic Bayesian Learning Framework for Complex Decision-making

PI: Dr. Qi Yu, Co-PIs: Dr. Daniel Krutz and Dr. Yu Kong.
October 2018 - September 2022

The overall goal of this project is to explore a multimodal dynamic Bayesian learning framework to facilitate complex decision-making in various highly complicated scenarios and tasks. The proposed framework aims to provide comprehensive decision support to maximize the overall effectiveness of complex decision-making. The project will focus on four main methods: (1) analyzing large-scale, heterogeneous, and dynamic data streams from multiple sources and extract high-level and meaningful features, (2) algorithmically fusing multimodal data streams and provide decision recommendations with interpretable justifications, uncertainty evaluation, and estimated costs to prioritize/coordinate multiple tasks (3) identifying sources of uncertainty and offer informative guidance for cost-effective information gathering, and (4) visualizing model outcome and allow intuitive interactions with the human for collaborative learning and continuous model improvement to achieve high-quality decisions.