Extracting video frames in unity 3d as textures

Video can be played using MovieTexture in Unity. There is no easy way of retrieving the video frame in unity though.

One of the hard way to retrieve the video frame is to render the video in to a RenderTexture and create video frame as texture by reading pixels from the render texture. Lets do it.

So let’s start and create a scene with a Camera, let’s call it MovieCamera, a cube and direction light.

Our scene will look like,

 

Let the camera be an orthographic one. We will use the cube to map the video on to it.

Playing the video

To play video you might need QuickTime Player, especially if you are not using ogg format. Keep the video in the Assets folder. Now drag and drop the video on to the cube object.

This will not start playing the video; for that you would need to write a script and call the Play method of MovieTexture.

script

Attach below given scripts to the cube game object.

using UnityEngine;
using System.Collections;

public class VideoPlayer : MonoBehaviour {

    // assign the video to this texture
    public MovieTexture movie;

    void Start () {
        GetComponent<renderer>().material.mainTexture = movie;

        // play the video
        movie.Play();
        movie.loop = true;

        // scale the cube so that it fills the screen
        float height = Camera.main.orthographicSize * 2.0f;
        float width = height * (float)Screen.width / (float)Screen.height;
        transform.localScale = new Vector3(width, height, 0.1f);
   }

   void Update () {
   }
}

The movie variable should be assigned with the video.

The inspector view of the Cube will be,

With this much of code we will be able to play the video. But our aim is to retrieve the video frames as textures.

To capture video frames we will render the video onto a RenderTexture first. For that instead of the camera rendering onto the screen we will make it render on to the RenderTexture.

 

Once video rendered on to the RenderTexture we can use ReadPixels method of Texture2D to retrieve the video frames.

Actually it is not so difficult to do, just attach the below shown script to the camera(MovieCamera) and we are done.


using UnityEngine;
using System.Collections;

public class MovieCamera : MonoBehaviour {
    private RenderTexture renderTexture = null;
    private Texture2D videoFrame;
    private Rect rect;   // Rect from where we need to read the pixels(entire  screen)
 
    void Start () {
    }

    void Update () {
    }

    void OnPreRender() {
        if (renderTexture == null) {
            Create();
        }

        // make camera’s targetTexture as current rendering target
        // and then read the pixels.
        RenderTexture orig = RenderTexture.active;
        RenderTexture.active = renderTexture;
        videoFrame.ReadPixels(rect, 0, 0, false);
        videoFrame.Apply();
        RenderTexture.active = orig;
    }

    void OnGUI () { 
        // draw the read texture on to the screen
        Graphics.Blit(videoFrame, (RenderTexture)null);
    }

    private void Create() {
        renderTexture = new RenderTexture (Screen.width, Screen.height, 8, RenderTextureFormat.ARGB32);
        videoFrame = new Texture2D(Screen.width, Screen.height, TextureFormat.ARGB32, false);
  
        // set targetTexture for the Camera
        GetComponent<camera>().targetTexture = renderTexture;
  
        // Rect to read the pixels; entire screen
        rect = new Rect(0, 0, Screen.width, Screen.height);
    }
}

Our Camera's(MovieCamera) inspector view,

In order to try this you would need a Unity pro version.

The video that I have used can be downloaded from https://peach.blender.org/download/