Improving Blender Workflows with Still Image Camera Matching App

Putting objects inside images and making the whole picture look natural at the same time is a difficult task, so software developer Per Gantelius released fSpy, an open-source, cross-platform app for still image camera matching. To learn more about it, we asked him about the software and his 3D art history.

Introduction

I've always been interested in both the artistic and the technical creative process and coding is a natural way for me to combine the two. For example, I'm interested in generative visual art, audio synthesis, and parametric design. I used to work as a sound engineer in a previous life and building VST audio plugins is what got me into coding "for real". After getting a degree in engineering physics, I worked as a mobile game developer for a few years before I started working as a freelance developer, which I've been doing for the last ten years or so.

Apart from minor contributions to various projects, my open-source stuff can be found on GitHub and ranges from 3D/Blender tools to DSP, embedded systems, and electronics.

3D Art

3D art appeals to me because it's a field where being nerdy and technical can pay off artistically. I don't create 3D art on a regular basis, but sometimes I get ideas I want to explore. As someone who got into Blender around version 2.43 (I think), it's been amazing to see it grow into what it is today.

fSpy

The predecessor to fSpy was a Blender plugin called BLAM that I wrote to compute camera parameters for a photo I shot with a camera lens I had borrowed from a friend. I started playing around with modeling, which escalated into the Slussen Plan C project. BLAM's user interface was quite clunky and weird, so I decided to make a standalone app with a more polished and intuitive interface, motivated by BLAM's relative popularity and the fact that there was no such free or open-source app at the time.

Still image camera matching is the process of computing the field of view and orientation of the camera from an image given the position of vanishing points, i.e., points where parallel lines appear to meet at infinity. This is useful when doing 3D modeling based on existing images.

I'd recommend beginners to have a look at the fSpy site, where the basic concepts are explained. There are also a bunch of fSpy tutorials on YouTube.

fSpy is an Electron app where the UI is built in React and running in an embedded Chromium browser. This adds some overhead in terms of memory usage and download size but makes it possible for a single volunteer developer like myself to develop a cross-platform application in my limited spare time.

The algorithm for computing the camera parameters (focal length, etc.) is described in a paper available from the fSpy GitHub repo if you're so inclined. It is based on some not-too-complicated geometric observations related to vanishing points and the pinhole camera model.

fSpy eliminates the guesswork when trying to match a 3D camera to an existing image. Given a reasonable input image, you can set up a matching Blender camera in a matter of minutes and start modeling "in the picture".

Now, there is only one importer – for Blender, and I'm not currently planning to write any new ones for other software, but there seem to be some third-party alternatives (that I haven't tried) on GitHub. Also, I've heard of people importing to other modeling software by first importing into Blender and then exporting to a suitable format from there. Finally, fSpy supports exporting the raw camera parameters, so a sufficiently nerdy user should be able to input these into their modeling software of choice.

Conclusion

There's not much of a roadmap at the moment I'm afraid. If you're interested in very infrequent fSpy news, your best bet is to follow me on Twitter. If you feel like supporting the project, there is a donate button on the fSpy site.

Per Gantelius, fSpy Developer

Interview conducted by Theodore McKenzie