CHAI Advanced PhD Student Talk: Spandan Pyakurel, Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models
CHAI Seminar Series
Refreshments will be served.
DATE: Monday, March 31, 2025
SPEAKER: Spandan Pyakurel
Advanced Ph.D. Candidate in Computing and Information Sciences, RIT
TITLE: Be Confident in What You Know: Bayesian Parameter Efficient
Fine-Tuning of Vision Foundation Models
IN PERSON: Golisano Hall (070), Room CYB-1710/1720
ABSTRACT: Large transformer-based foundation models have been commonly used as pre-trained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning (PEFT) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through PEFT leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight Bayesian Parameter Efficient Fine-Tuning (referred to as Bayesian-PEFT) framework for large transformer-based foundation models. The framework integrates state-of-the-art PEFT techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components. Our thorough theoretical analysis justifies that the Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple PEFT techniques demonstrate the outstanding prediction and calibration performance by Bayesian-PEFT.
BIO: Spandan Pyakurel is a third-year PhD student in Computing and Information Sciences at RIT and a member of the Machine Learning and Data Intensive Computing research lab led by her advisor Dr. Qi Yu. Her research focuses on uncertainty quantification, novelty detection, and calibration. She has published papers in top machine learning conferences including ICML, and NeurIPS.
NOTE: To schedule interpreter and/or services for this event, please use https://myaccess.rit.edu.
Event Snapshot
When and Where
Who
Open to the Public
| Cost | FREE |
Interpreter Requested?
No