Video frame interpolation is a process that generates new, artificial frames and inserts them between the original frames of a video. This technique can make a 30 frames-per-second (fps) video look as smooth as 60 fps, or turn standard footage into compelling slow motion. It works by analyzing the motion between two consecutive frames and synthesizing a new frame that represents a plausible moment in between.
Modern methods use artificial intelligence to achieve this, learning from vast datasets of video clips to predict motion and reconstruct images. While powerful, this process is an estimation, not a recovery of real information, and it can introduce visual errors.
Key takeaways
- Video Frame Interpolation (VFI) synthesizes new frames to increase frame rate or create slow motion; it does not recover frames that were actually captured.
- Modern AI methods work by estimating motion, warping the input frames toward an intermediate point in time, and blending them to reconstruct the final image.
- Visual errors called artifacts—such as warping, ghosting, and tearing—are common, especially with fast motion, hidden objects, and abrupt scene cuts.
- Key AI models like RIFE and FILM represent different architectural approaches. RIFE focuses on estimating intermediate motion flows for real-time performance, while FILM uses a unified network to better handle large motion (Source).
- VFI is a tool for enhancing temporal resolution (frame rate), not spatial resolution (pixel detail). It can make video smoother but will not fix blurriness and can introduce new visual flaws.
What Is Video Frame Interpolation?
Video frame interpolation synthesizes new, intermediate frames between existing ones to increase a video's frame rate. This process makes motion appear smoother or allows for the creation of slow-motion effects. It analyzes two source frames and generates a new image representing the state of the scene at a moment between them, creating more fluid temporal detail than the original footage contained.
This is different from simpler methods of changing frame rate. For example, frame duplication simply repeats an existing frame. While this can stretch a video's duration, it creates a noticeable stutter or judder because no new motion information is added.
Imagine converting a 30 fps video to 60 fps.
- Frame Duplication: [Frame A] → [Frame A again] → [Frame B]
- Frame Interpolation: [Frame A] → [Synthesized Frame A.5] → [Frame B]
The synthesized frame (A.5) shows the objects having moved halfway between their positions in Frame A and Frame B, resulting in genuinely smoother motion. The primary use cases are converting video to a higher frame rate (e.g., 24 fps to 60 fps for smoother playback on modern displays) and creating slow-motion effects without the stutter that comes from simply slowing down low-frame-rate footage.
How Does AI Video Frame Interpolation Work?
AI video frame interpolation uses a deep neural network to generate a new frame between two existing ones by estimating motion. The process involves calculating where every part of the image has moved, warping the original frames based on that motion, and blending them into a coherent new image. This allows the AI to create a plausible "in-between" moment.
Most modern VFI systems follow a general pipeline, though the specific network architectures vary. Here is a step-by-step breakdown of how a typical flow-based model works.
Input and Goal: The system takes two input frames,
Frame 0(I₀) andFrame 1(I₁), and a target timetbetween them (wheret=0isFrame 0andt=1isFrame 1). For a single interpolated frame right in the middle, the target timetis 0.5.Motion Estimation: The model first estimates the motion between
Frame 0andFrame 1. This is often done using optical flow, a technique that calculates a motion vector for every pixel, describing where it has moved from one frame to the next (Source). The result is two "flow fields," one showing forward motion (0→1) and one showing backward motion (1→0).Intermediate Flow Estimation: To create a frame at time
t, the model needs to know the motion at that specific moment. Some models, like RIFE (Real-Time Intermediate Flow Estimation), use a specialized neural network called IFNet to directly estimate the intermediate motion flows from the target timetback toFrame 0and forward toFrame 1(Source).Backward Warping: The model uses these intermediate flows to pull pixels from the source frames. For each pixel location in the new frame, it asks: "Based on the estimated motion, where in
Frame 0andFrame 1should this pixel's color come from?" This "pull" method, called backward warping, is generally more effective at avoiding holes and overlaps than "pushing" pixels forward from the source frames.Blending and Reconstruction: The two warped frames (one from
Frame 0and one fromFrame 1) are now approximations of the target frame. However, some areas may be problematic. For example, an object visible inFrame 0might be hidden (occluded) inFrame 1. The model uses a synthesis network to intelligently blend the two warped images, often using a learned "occlusion mask" to decide which warped frame is more reliable for each pixel. This network also fills in any newly revealed areas (disocclusions).
This final blending and refinement step is crucial for generating a clean output and is a key part of how AI video generation works. It's not just about moving pixels; it's about reconstructing a plausible image from imperfect motion data.
A Worked Example: From 30 fps to 60 fps
To make this process more concrete, let's walk through an illustrative example of creating one new frame to turn 30 fps video into 60 fps.
- Start: We have
Frame 0(at time t=0ms) andFrame 1(at time t=33.3ms). - Goal: Create
Frame 0.5at the halfway point, time t=16.65ms.
Let's focus on a single pixel representing a small object.
Motion Estimation: The AI model analyzes both frames and determines that a specific pixel at coordinate (100, 150) in
Frame 0has moved to coordinate (110, 154) inFrame 1. The motion vector for this pixel is therefore(+10, +4).Intermediate Flow Calculation: Our target is
t=0.5. Assuming linear motion for this simple example, the model calculates the motion from our target frame back toFrame 0and forward toFrame 1.- Motion from
Frame 0.5back toFrame 0is(-5, -2). - Motion from
Frame 0.5back toFrame 1is(+5, +2).
- Motion from
Backward Warping: The model now builds the new frame,
Frame 0.5. To determine the color of the pixel at the new, intermediate position (105, 152), it performs backward warping:- It looks at the
Frame 0pixel located at(105, 152) + (-5, -2) = (100, 150). Let's say this pixel's color is blue. - It looks at the
Frame 1pixel located at(105, 152) + (+5, +2) = (110, 154). Let's say this pixel's color is also blue.
- It looks at the
Blending and Synthesis: The model now has two candidate colors for the pixel at (105, 152) in the new frame. In this ideal case, both warped pixels point to the same object, and their colors are consistent. The synthesis network blends them (e.g., a 50/50 average) to produce the final blue pixel in
Frame 0.5. If one area was occluded, a learned mask would give 100% weight to the visible candidate and instruct the network to fill in any missing information.
This process is repeated for every pixel to construct the entire new frame.
AI Model Architectures: RIFE vs. FILM
Different AI models for video frame interpolation take different approaches to solving the core problems of motion estimation and image synthesis. While many VFI models are built around estimating motion with optical flow, other architectural families exist. Researchers have developed methods using learned convolution kernels, which combine neighboring pixels based on locally learned weights rather than an explicit motion field. Other approaches use phase information from signal processing to analyze and reconstruct motion (Source).
Below is a comparison of two prominent architectures, RIFE and FILM, which represent flow-based and unified network approaches.
| Approach | Key Idea | Strengths | Weaknesses / Trade-offs |
|---|---|---|---|
| Classical Optical Flow | Calculate an explicit motion field first, then use it to warp and blend frames. The flow and synthesis steps are often separate. | The motion representation is explicit and interpretable. Can work well for simple, small motions. | Can be slow. Errors from the flow estimation step are passed on to the synthesis step. May struggle with large motion. |
| RIFE (Intermediate Flow) | Use a single neural network to directly estimate the "intermediate" motion flows needed for warping. It avoids a separate, pre-trained flow model. (Source) | Designed for real-time applications like video processing, media players, and display devices (Source). A reported implementation can exceed 30 fps for 720p interpolation on an NVIDIA 2080 Ti (Source). An optimization called FastRIFE aims to further improve speed (Source). | May not handle extremely large or complex motion as well as models specifically designed for it. Performance depends on the IFNet architecture. |
| FILM (Unified Network) | Use a single, large network to learn motion and synthesis jointly without an explicit optical flow representation. It is trained on frame triplets. (Source) | Designed to handle large motion better than many previous methods. The unified approach can capture complex relationships between motion and appearance. | May be more computationally intensive than real-time models like RIFE. The inner workings are less interpretable than explicit flow methods. |
These different strategies highlight a key trade-off in VFI research: balancing speed, accuracy, and the ability to handle challenging scenarios like large motion.
How VFI Models Are Trained and Evaluated
VFI models are evaluated on standardized benchmarks to measure their performance. These datasets often feature different types of motion and content to test a model's robustness. For example, Vimeo90K is a widely used dataset whose training split contains 51,312 frame triplets, each at a resolution of 448 by 256 pixels (Source). To specifically test performance on challenging movement, benchmarks like SNU-FILM provide 1,240 triplets categorized into 'easy,' 'medium,' 'hard,' and 'extreme' motion subsets. A model that performs well on Vimeo90K might still struggle on SNU-FILM's 'extreme' category, highlighting why models like FILM, which are designed for large motion, are necessary (Source). Performance is not a single number but is dependent on motion type, resolution, and the specific dataset used for evaluation.
Why Do Interpolated Videos Look Warped or Ghostly?
Synthesized frames are estimates, and the AI model can make mistakes, resulting in noticeable visual errors, or "artifacts." Understanding these artifacts helps you identify the limitations of video frame interpolation and diagnose problems in your output.
- Warping and Distortion: This appears as if objects are made of liquid, bending or stretching unnaturally as they move. It is caused by incorrect motion estimation. The model misjudges the path of pixels, especially on non-rigid objects (like clothing), complex textures, or during rotational movement.
- Ghosting and Halos: A faint, transparent copy of an object may trail behind or appear around its edges. This artifact occurs when the model incorrectly blends the two warped frames. It is most common around object boundaries where the background is being covered (occlusion) or revealed (disocclusion). The model struggles to decide which frame to trust and may average them incorrectly.
- Tearing and Duplication: An object, especially a thin one like a pole or an arm, might appear to split into pieces or duplicate. This often happens with very fast motion that exceeds what the model was trained to handle. The motion estimation fails completely, and parts of the object are misplaced or copied from the wrong source frame. These errors are a failure of temporal consistency in video.
- Frozen or Stuck Areas: Sometimes, a small patch of the video appears frozen in place while the rest of the scene moves around it. This can happen when the model fails to detect subtle motion in low-texture areas, like a calm water surface or a plain wall, and assumes the region is static.
These artifacts are fundamental challenges. No current model can eliminate them entirely, especially in complex, real-world footage.
When Should You Avoid Video Frame Interpolation?
Video frame interpolation is a powerful tool, but it is not a solution for every situation. Applying it to the wrong type of footage can make the video look worse, not better. Here are some scenarios where you should avoid using VFI.
- Across Scene Cuts: Interpolating between the last frame of one scene and the first frame of another will produce a nonsensical, morphing image. There is no valid motion path between two completely different shots.
- On Footage with Fast, Unpredictable Motion: Action sequences, sports highlights, and chaotic handheld video often contain motion that is too fast, blurry, or erratic for current models to track accurately. This leads to severe warping and tearing artifacts.
- For Videos with Fine Text or User Interfaces: Small text, cursors, or thin lines in screen recordings are difficult for motion estimation algorithms to track. Interpolation often renders text unreadable and mangles UI elements.
- When Temporal Accuracy Is Critical: For scientific analysis, sports officiating, or forensic video, synthesized frames are not ground truth. They are plausible guesses made by a model and should not be treated as a factual record of what happened between the captured frames.
- On Transparent, Reflective, or Fluid Surfaces: Water, smoke, fire, rain, and reflections violate the common assumption of opaque objects with coherent motion. Models struggle to track these elements, leading to strange visual distortions.
- When You Can Record at a Higher Frame Rate: The best way to get smooth, high-quality video is to capture it at a high frame rate from the start. VFI is a post-production tool for enhancement and rescue, not a substitute for proper recording.
What to learn next
- Text to Video Prompting: A Practical Guide - Learn the fundamentals of text to video prompting. This guide provides a step-by-step process for writing effective prompts to control motio
- Temporal Consistency Video: A Guide to Why AI-Generated Videos Flicker: Learn more about the artifacts that plague AI video, including those common in frame interpolation.
- How AI Video Generation Works: A Technical Guide: Broaden your understanding of how AI creates video, from interpolation to text-to-video models.
- Image to Video AI: How Models Animate Still Images: Explore another fascinating area of video synthesis where AI creates motion from a single static picture.
Sources and further reading
- Video Frame Interpolation: A Comprehensive Survey
- Real-Time Intermediate Flow Estimation for Video Frame Interpolation
- FILM: Frame Interpolation for Large Motion
- A Unified Pyramid Recurrent Network for Video Frame Interpolation
- Real-Time Intermediate Flow Estimation for Video Frame Interpolation (ECCV Version)
- Official RIFE Implementation on GitHub
- FastRIFE: A Real-time Video Frame Interpolation Network
