Research

Utilizing Synergy between Human and Computer Information Processing

Utilizing Synergy between Human and Computer Information Processing for Complex Visual Information Organization and Use

NSF IIS Award (~$500K, July 2018- June 2023).
Co-PIs: Anne Haake, Rui Li, and Pengcheng Shi

The primary focus of this project is on complex image understanding in specialized domains. (e.g., medicine, bioinformatics, and so on) This proposed research aims to integrate human and computer capabilities to discover image semantics by (1) encoding image inspection and analysis behaviors that represent domain expertise, and  (2) algorithmically fusing human expertise with image data.

Trans-disciplinary approaches for knowledge elicitation and extraction

Trans-disciplinary approaches for knowledge elicitation and extraction

Eye movement sequence and verbal narrative

Eye movement sequence and verbal narrative

Multimodal knowledge fusion

Multimodal knowledge fusion

A Multimodal Dynamic Bayesian Learning Framework

A Multimodal Dynamic Bayesian Learning Framework for Complex Decision-making

DoD/ONR Award (~$1.6M, October 2018- September 2023)
Co-PIs: Daniel Krutz and Yu Kong

Machine Learning Data Model

Develop advanced machine learning models to support decision-making in military operations while addressing the following challenges:

  • Coordinating multiple concerns and missions (e.g., self-protection, position assets, resource allocation…)
  • Missions carry some form of risk and uncertainty
  • Data come in heterogeneous formats (sensor readings, transcribed communications, image/videos from surveillance) in real-time, and fast-changing
  • The decision model should provide interpretable recommendation and effectively incorporate human feedback