Event-Based Multi-Task Facial Landmark and Blink Detection

Facial landmark detection and blink detection are essential tasks in computer vision, with significant applications in behavioral analysis and human-computer interaction. Traditional frame-based cameras struggle with the high temporal resolution needed for accurate blink detection under challenging conditions. This paper presents a novel approach using event-based cameras, which offer superior temporal resolution and dynamic range. By leveraging the asynchronous nature of these cameras, we capture fine-grained motion information, enabling precise real-time detection of rapid eye blinks and facial landmarks. We propose a multi-task deep neural network that optimizes both tasks simultaneously, enhancing overall system performance through shared learning. Our method achieves state-of-the-art results on a public event dataset with a normalized mean error of 4.76% for five facial keypoints. Using a combined dataset of real and synthetic events, we scale the task from 5 facial landmarks to 98, while attaining a normalized mean error of 3.65%. This 98 landmark model simultaneously detects blinks with an F1 score of 0.905, setting a new state-of-the-art in event-based blink detection. With these findings we demonstrate the potential of event-based cameras for robust, real-time facial analysis.