Publications

SWARM-FR: Benchmarking Virtual Cell Metrics

Submitted to NeurIPS, 2026

So excited to share more about my newest first-author work when it’s available! For now, here’s an abstract visualization of the work’s abstract. Is that abstract enough?

Multimodal Classification of Alzheimer’s Disease by Combining Facial and Eye-Tracking Data

Published in Proceedings of the 4th Machine Learning for Health Symposium, PMLR 259, 2025

In recent years, there has been growing interest in developing a non-invasive tool for detecting Alzheimer’s Disease (AD). Previous studies have shown that a single modality such as speech or eye-tracking (ET) data can be effective for classifying AD patients from healthy individuals. However, understanding the role of other modalities, and especially the integration of facial analysis with ET for enhancing dementia classification, remains under-explored. In this paper, we investigate whether we can leverage facial patterns in AD patients by building on EMOTION-FAN—a deep learning model initially developed for recognizing seven distinct human emotions, now fine-tuned for our facial analysis tasks. We also explore the efficacy of leveraging multimodal information by combining the results from the facial and ET data through a late fusion technique. Specifically, our approach uses a neural classifier to learn from raw ET data (VTNet) alongside the fine-tuned EMOTION-FAN model that learns from the facial data. Experimental results show that facial data gives superior results than ET data. Notably, we obtain higher scores when both modalities are combined, providing strong evidence that integrating multimodal data benefits performance on this task.

Recommended citation: Shih-Han Chou, Miini Teng, Harshinee Sriram, Chuyuan Li, Giuseppe Carenini, Cristina Conati, Thalia S. Field, Hyeju Jang, and Gabriel Murray. (2025). "Multimodal Classification of Alzheimer’s Disease by Combining Facial and Eye-Tracking Data." Proceedings of the 4th Machine Learning for Health Symposium, PMLR 259:219-232.

Personalizing Explanations of AI-Driven Hints to Users’ Characteristics: An Empirical Evaluation

Published in International Conference on Artificial Intelligence in Education (AIED 2025), 2025

The paper extends an existing Intelligent Tutoring System (ITS) that supports students’ learning via AI-driven personalized hints and can generate explanations to justify why/how the hints were generated. In this work, we investigate personalizing these hint explanations to students with low levels of two traits, Need for Cognition and Conscientiousness in order to enhance their engagement with the explanations, based on prior findings that these students generally do not ask for the explanations although they would benefit from them. We evaluate the effectiveness of the personalized hint explanations with a formal user study. Our results show that the personalization increases our target users’ interaction with the hint explanations, their understanding of the hints, and their learning. Hence, this work contributes to exiting initial evidence on the value of Personalized Explainable AI (PXAI) in education.

Recommended citation: Vedant Bahel, Harshinee Sriram, and Cristina Conati. (2025). "Personalizing Explanations of AI-Driven Hints to Users’ Characteristics: An Empirical Evaluation." Artificial Intelligence in Education (AIED 2025), Lecture Notes in Computer Science (LNAI, volume 15877):411–423.

Detecting when Users Disagree with Generated Captions

Published in ICMI '24 Companion: Companion Proceedings of the 26th International Conference on Multimodal Interaction, 2024

The pervasive integration of artificial intelligence (AI) into daily life has led to a growing interest in AI agents that can learn continuously. Interactive Machine Learning (IML) has emerged as a promising approach to meet this need, essentially involving human experts in the model training process, often through iterative user feedback. However, repeated feedback requests can lead to frustration and reduced trust in the system. Hence, there is increasing interest in refining how these systems interact with users to ensure efficiency without compromising user experience. Our research investigates the potential of eye tracking data as an implicit feedback mechanism to detect user disagreement with AI-generated captions in image captioning systems. We conducted a study with 30 participants using a simulated captioning interface and gathered their eye movement data as they assessed caption accuracy. The goal of the study was to determine whether eye tracking data can predict user agreement or disagreement effectively, thereby strengthening IML frameworks. Our findings reveal that, while eye tracking shows promise as a valuable feedback source, ensuring consistent and reliable model performance across diverse users remains a challenge.

Recommended citation: Omair Shahzad Bhatti, Harshinee Sriram, Abdulrahman Mohamed Selim, Cristina Conati, Michael Barz, and Daniel Sonntag. (2024). "Detecting when Users Disagree with Generated Captions." ICMI '24 Companion: Companion Proceedings of the 26th International Conference on Multimodal Interaction.

Initial results on personalizing explanations of AI hints in an ITS

Published in UMAP '24: Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization, 2024

Previous research on an Intelligent Tutoring System (referred to as ACSP), showed the need to personalize explanations of its AI-driven hints for users with low Need for Cognition (N4C) and low Conscientiousness (Cons.). Specifically, this work found that explanations should be provided to these users with the objective of increasing user interaction with them. In this paper, we present and evaluate design alterations to the original ACSP explanation interface aimed at achieving this objective. Our results provide initial evidence that the implemented personalization, in the form of the design alterations, had a positive impact on users with low N4C and Cons., by increasing attention to explanations and contributing to learning gains.

Recommended citation: Vedant Bahel, Harshinee Sriram, and Cristina Conati. (2024). "Initial results on personalizing explanations of AI hints in an ITS." UMAP '24: Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization.

Classification of Alzheimer’s Disease with Deep Learning on Eye-tracking Data

Published in ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction, 2023

Existing research has shown the potential of classifying Alzheimer’s Disease (AD) from eye-tracking (ET) data with classifiers that rely on task-specific engineered features. In this paper, we investigate whether we can improve on existing results by using a Deep Learning classifier trained end-to-end on raw ET data. This classifier (VTNet) uses a GRU and a CNN in parallel to leverage both visual (V) and temporal (T) representations of ET data and was previously used to detect user confusion while processing visual displays. A main challenge in applying VTNet to our target AD classification task is that the available ET data sequences are much longer than those used in the previous confusion detection task, pushing the limits of what is manageable by LSTM-based models. We discuss how we address this challenge and show that VTNet outperforms the state-of-the-art approaches in AD classification, providing encouraging evidence on the generality of this model to make predictions from ET data.

Recommended citation: Harshinee Sriram, Cristina Conati, and Thalia Field. (2023). "Classification of Alzheimer's Disease with Deep Learning on Eye-tracking Data." ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction.

Evaluating the overall sensitivity of saliency-based explanation methods

Published in IJCAI-XAI workshop, 2023

We address the need to generate faithful explanations of “black box” Deep Learning models. Several tests have been proposed to determine aspects of faithfulness of explanation methods, but they lack cross-domain applicability and a rigorous methodology. Hence, we select an existing test that is model agnostic and is well-suited for comparing one aspect of faithfulness (i.e., sensitivity) of multiple explanation methods, and extend it by specifying formal thresholds and building criteria to determine the over-all sensitivity of the explanation method. We present examples of how multiple explanation methods for Convolutional Neural Networks can be compared using this extended methodology. Finally, we discuss the relationship between sensitivity and faithfulness and consider how the test can be adapted to assess different explanation methods in other domains.

Recommended citation: Harshinee Sriram, Cristina Conati. (2023). "Evaluating the overall sensitivity of saliency-based explanation methods." IJCAI-XAI 2023 workshop.

Classification of Alzheimer’s using Deep-learning Methods on Webcam-based Gaze Data

Published in Proc. ACM Hum.-Comput. Interact, 7, ETRA, Article 157, 2023

There has been increasing interest in non-invasive predictors of Alzheimer’s disease (AD) as an initial screen for this condition. Previously, successful attempts leveraged eye-tracking and language data generated during picture narration and reading tasks. These results were obtained with high-end, expensive eye-trackers. Instead, we explore classification using eye-tracking data collected with a webcam, where our classifiers are built using a deep-learning approach. Our results show that the webcam gaze classifier is not as good as the classifier based on high-end eye-tracking data. However, the webcam-based classifier still beats the majority-class baseline classifier in terms of AU-ROC, indicating that predictive signals can be extracted from webcam gaze tracking. Hence, although our results indicate that there is still a long way to go before webcam gaze tracking can reach practical relevance, they still provide an encouraging proof of concept that this technology should be further explored as an affordable alternative to high-end eye-trackers for the detection of AD.

Recommended citation: Anuj Harisinghani, Harshinee Sriram, Cristina Conati, Giuseppe Carenini, Thalia Field, Hyeju Jang, and Gabriel Murray. (2023). "Classification of Alzheimer’s using Deep-learning Methods on Webcam-based Gaze Data." Proc. ACM Hum.-Comput. Interact, 7, ETRA, Article 157.