Introduction
Multimodal Auth is a research and implementation project carried out by students of artificial intelligence at the Wrocław University of Science and Technology.
It deals with multi-modal authentication. We focus on the modality of the voice and the face, additionally combining modalities with plug-in fusion to perform deepfake detection.
Motivation
The motivation behind our project is the need to address the vulnerability of popular biometric systems to:
- Presentation attacks
- Sensitivity to changing environmental conditions
This challenge applies not only to large companies, but also to each and every one of us - all of us who want to use modern, secure identity verification methods.
Another challenge when integrating additional security features (e.q deepfake detection) into verification systems is:
- High computational requirements
- Long training times
Goal
Creating a biometric user authentication system using audio and video data.
Including:
- Flexible approach to variable recording quality
- Detection of DeepFake attacks
Proposed Solution
Voice Modality
In order to select the best model for the voice embedding domain, we conducted an evaluation on four-second sequences and environmental noise data. Between the models, ECAPA-TDNN, CAM++, and ReDimNet-b6, ReDimNet-b6 showed the lowest frequency of errors in both quality cases, so it was the one selected for our pipeline.
Face Modality
The pretrained SOTA model AdaFace was selected for face verification. To improve verification, several methods were tested to assess the quality of the face image in terms of their impact on metrics. CR-FIQA, SDD-FIQA, SDD-FIQA-mod (modified SDD-FIQA) and SER-FIQ methods were tested. The best results were obtained for SDD-FIQA-mod.
Pipeline
In our solution pipeline, we segment the film using a sliding window. The window size is set to 4 seconds, which is the length of the audio and video fragments used for verification. The sliding window allows us to process the entire recording, even if it is longer than 4 seconds.
We assess the audio and video quality, which increases the flexibility of the system. Depending on the selected quality threshold, we can make the system more accessible for verification under varying conditions or increase its security by rejecting low-quality recordings.
Feature extraction with domain models is performed on the fragments filtered for quality. The extracted features are passed to fusion, whose task is to determine the veracity of the recording. The verification models and the detection module return predictions on the basis of which a final rejection or acceptance decision is returned.
DeepFake Detection Module
Using late fusion in the solution has many advantages. It allows for simple personalisation and change of domain models. The saving of computational resources and the time needed for the learning process is also an important advantage. In addition, fusion ensures the scalability of the system, making it easier to expand with new modalities.
Our system performs a projection of voice and facial features into a common space, aligning the data distributions and separating true examples from manipulated ones within each modality. The fusion module then uses cross-attention to assess pairwise correspondences. The final classifier decides whether the sample is real or not.
Results
Our method achieves comparable results to competing SOTA models in the DeepFake detection task, while it is 40x smaller than SSVF and the inference time takes almost 300x less time.
Example of final voting
In our approach, we implement a hard voting mechanism to combine the predictions from multiple modalities. Each modality independently assesses the authenticity of the input, and their votes are aggregated to make a final decision.
In this particular example, audio and video modalities accepted a verification request, while the deepfake detection module rejected it. Resulting in a final decision of rejection. This illustrates how the system can effectively handle conflicting predictions from different modalities.
SynthWeave Library
Explore our open-source library on GitHub for implementation details and code.
Resources
Contacts
- inż. Izabela Majchrowska LinkedIn
- inż. Filip Strózik LinkedIn
- inż. Dawid Wolkiewicz LinkedIn
- dr inż. Piotr Syga piotr.syga@pwr.edu.pl