This paper introduces a novel approach, KFM-SLT, for continuous sign language translation that leverages continuous normalizing flows to model the probability path from sign language video features to target text representations, with keyframe conditioning to guide the flow trajectory.
Key findings
KFM-SLT captures the continuous dynamics of signing while maintaining the discrete structure of natural language.
The method includes a keyframe detector, conditional flow matcher, and text decoder for fluent target language sequence generation.
Addresses limitations in current CSLT methods, such as modeling continuous motion dynamics and generalization across signers.
Limitations & open questions
The approach's effectiveness in handling unseen sentence structures and its applicability across different sign languages remain open questions.