Video datasets suffer from huge inter-frame redundancy, which prevents deep networks from learning effectively and increases computational costs. Therefore, several methods adopt random/uniform frame sampling or key-frame selection techniques. Unfortunately, most of the learnable frame selection methods are customized for specific models and lack generality, independence, and scalability. In this paper, we propose a novel two-stage video-to-video summarization method termed FastPicker, which can efficiently select the most discriminative and representative frames for better action recognition. Independently, the discriminative frames are selected in the first stage based on the inter-frame motion computation, whereas the representative frames are selected in the second stage using a novel Transformer-based model. Learnable frame embeddings are proposed to estimate each frame contribution to the final video classification certainty. Consequently, the frames with the largest contributions are the most representative. The proposed method is carefully evaluated by summarizing several action recognition datasets and using them to train various deep models with several backbones. The experimental results demonstrate a remarkable performance boost on Kinetics400, Something-Something-v2, ActivityNet-1.3, UCF-101, and HMDB51 datasets, e.g., FastPicker downsizes Kinetics400 by 78.7% of its size while improving the human activity recognition.
|Original language||English (US)|
|Number of pages||14|
|State||Published - Jan 7 2023|
Bibliographical noteFunding Information:
We thank Ahmed Elazab, Zeyad Qasem, and Murtadha Ahmed for the fruitful discussion. This work was supported in part by the National Natural Science Foundation of China under grants U21A20455, 61972265, 11871348 and 61872429, the Natural Science Foundation of Guangdong Province of China under grant 2020B1515310008, the Educational Commission of Guangdong Province of China under grant 2019KZDZX1007, the Simons Foundation under grant 353185, and ARO under grant W911NF2110218.
© 2022 Elsevier B.V.
- Action recognition
- Deep learning
- Discriminative frame selection
- Representative frame selection
- Video-to-video summarization