Motion Tracking a Smart Phone Screen with Final Cut Pro X

Creating a screen replacement effect on an iPhone using FCPX, a motion tracking plugin, and a custom marker guide.

Published Categorized as Behind The Scenes No Comments on Motion Tracking a Smart Phone Screen with Final Cut Pro X

Working on a recent spot for a client, I was required to produce a shot of a phone receiving text messages. I prefer to create effects in-camera as much as possible for the sake of realism, however, the text messages needed to match the client’s current marketing graphics (view this blog to know more about this concept in detail) and be animated, so this would have to be accomplished using visual effects.

If you don’t know, motion tracking is part of creating special visual effects. Remember that scene in Jurassic Park when the dinosaurs were running past the actors in the field?

Motion tracked tennis balls help create a digital grid on a virtual ground to aid the animation of dinosaurs. The actors’ eyelines were also tracked so the animators knew where to place the dinosaurs.
In the final shot, the motion tracking and digital dinosaurs come together and all the actors, both real and digital, appear to share the same space together.

The special effects team made a grid on the ground with tennis balls so they could recreate the field on their computers and animate the dinosaurs into the shot so it appeared they were really there. They did that by tracking the location of each ball in the field against a digital field with digital balls. Doing that insured the dinosaurs’ feet was accurately touching the “ground” and that when the camera bounced around, the dinosaurs also moved correctly as if they were sharing the same “real” space as the actors.

What I’m trying to accomplish is no where near as complicated, nor as dramatic as a field of stampeding gallimimus, but the gist of accomplishing the same motion-tracked effect is the same even 27 years later. Except faster and less expensive.

This is actually my second attempt. My first attempt was a bit frustrating but it was completed, the job was approved, and now it’s done, but I didn’t record any behind the scenes information. Because I wanted to try again, I decided to revise my methods and share the experience with you.

First, I had to create a motion tracking marker guide to display on my phone. After a bit of research I created a template in Photoshop. The centers of the markers are at precisely 50% of the area of the screen, so after you’ve done the motion track, you only have to scale up your comped media 200% to fit. I’ve made my guides available here for your HD smartphone motion tracking needs in PNG format with two backgrounds: Chroma Green or Chroma Blue.

I got the shot with my Canon EOS 6D Mark II, and some easy lighting design. The 6D’s auto focus came in very handy since both my hands were already in use.

BTS: Canon EOS 6D-II, Manfrotto head, Falcon Eyes RX-18T, Matthews grip.

After ingesting the footage into Final Cut, I used the FCPX Auto Tracker Perspective plugin from Pixel Film Studios to do a corner pin track of the phone’s screen.

After successful export of the tracking date, I then used the Draw Mask tool in Final Cut to remove the tracking markers. This became time consuming because of the length of the shot and how much movement I was performing.

After a simple chroma key using the Keyer tool in Final Cut, I was then able to replace the screen with anything I wanted. For this example I am using a photo I created for the same client back in 2018 in Austin.

With a bit of finessing to get the image to fit the screen, I then layered the image underneath the footage of the phone, and I finally got the results I wanted. The entire post-process took approximately 10 minutes this time, and about an hour on my first attempt as I figured everything out.

And there it is! I don’t like how the image seems to be “in front” of the phone. It’s Z axis, for some reason, is off, and I can’t fix it using the plugin, even though there is a control for that axis. I think it’s a bug. But, it’s not obvious for a quick shot that doesn’t move very much as is the one I originally produced for the actual spot.

In the end, the client was very pleased and quickly approved it. Here it is below, my first effort at motion tracking this way.

I hope this overview shows the potential of motion tracking in Final Cut Pro X. I also hope my 1920×1080 Motion Tracking Marker Guides above come in handy for you one day. Cheers!

Hindsight is 20/20: I could have done the animation, rendered it, shared it to my phone, and played it off my phone with my fingers synced with the animation action and accomplished the entire shot in-camera, but that’s not what happened! Instead, I wanted to try out a new motion tracking plugin for Final Cut Pro X I hadn’t put to use yet.

By Jason R. Johnston

Jason is an award-winning cinematographer, and director of commercials, branding films, native content, music videos, documentaries, and narrative films. As a full-time freelancer, he can be hired to DP or direct almost any project you have in mind. He is based in Sparta, Tennessee, and ready to travel for any gig.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.