HONGDA JIANG —— CFCS, Peking University
MARC CHRISTIE —— University Rennes, Inria, CNRS, IRISA
XI WANG —— University Rennes, Inria, CNRS, IRISA
LIBIN LIU —— CFCS, Peking University
BIN WANG* —— Beijing Institute for General Artificial Intelligence
BAOQUAN CHEN* —— CFCS, Peking University
(* corresponding author)
Our proposed deep-learning framework for camera keyframing offers both high-level style specification and low-level keyframe control.
In this work we present a tool that enables artists to synthesize camera motions following a learned camera behavior while enforcing user-designed keyframes as constraints along the sequence. To solve this motion in-betweening problem, we train a camera motion generator from a collection of trajectories using an additional conditioning on target keyframes. We also condition the generator with a style code automatically extracted from real film clips through the design of a gating LSTM network. This style code encodes the camera behavior defined as the correlation between the characters and camera motions. We further extend the system by incorporating a fine control of camera speed and direction via a hidden state mapping module. We then evaluate our method on two aspects: i) the capacity to synthesize camera trajectories by extracting camera behaviors from real movie film clips, and constraining them with user defined keyframes; ii) the capacity to ensure that in-between motions still comply with the reference camera behavior while satisfying the keyframe constraints. As a result, our system is the first behavior-aware keyframe in-betweening technique for camera control that balances behavior-driven automation with precise and interactive control..
ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2021)
This work was supported in part by the National Key R&D Program of China (2018YFB1403900, 2019YFF0302902). We also thank Anthony Mirabile and Yulong Zhang for the various support and helpful discussions throughout this project, as well as Yu Xiong for his help processing the MovieNet dataset.