Creating a Game with Makey Makey, Unity, OpenCV and a Banana Gun

Introduction

Hello! My name is Pavel Efremov. I'm a CG artist and Art director from Moscow. A big part of my experience was working as art director at Hello Computer studio. We made a lot of fun interactive games based on the mix of physical and digital experiences, using projections, Kinect, leap motion, VR headsets, AR camera, and so on. I left studio about 5 years ago but the love for combining physical and digital in an unusual way still lives in my heart.

Idea

Some years ago, I got Makey Makey Go, which I bought but did not use a single time for a reason unknown to me. But this spring, suddenly Quarantine comes and some extra spare time with it. As I am not a developer and I've got very basic C# programming skills, I decided to make a small game with casual repetitive arcade shooter mechanics, but with fun unusual input. I am a big fan of the Rick and Morty universe, and the second half of the last season was released in March, so the lore of the game was automatically chosen. In the end, the idea was quite simple: I get the position of my pistol from AR marker looking at the laptop camera and shoot event from Makey Makey Go which simply sends “Space” button as it is connected as keyboard to Mac or PC.

Gun. Banana with Makey Makey and AR Marker

I started by making my custom banana gun controller. I used Makey Makey Go as a pistol trigger and AR marker to track the position of my banana when it was facing the web camera. To do that, I mounted a cable on banana in a place where trigger on a regular pistol usually is and made a holder for AR marker from a key ring (see the photos). I had to draw marker by hand, as I had no printer, but it works surprisingly well. And of course, you can use any object you associate a gun with, cucumber for example. So be creative!

Unity and OpenCV

I've been using the Unity engine for more than 6 years and it perfectly works not only with usual games but also with any weird projects like mine. The most popular and easy way to add augmented reality in game is OpenCV library for Unity. It also contains a lot of other cool functions like face tracking, so there is another reason to buy it and dive in if you haven’t done it before.

When adding OpenCV asset to your project you will get a lot of ready to use scenes with examples on every feature that package has. So what I did was just modify the existing AR Marker demo scene. I added an aim icon that tracks to AR object.

To make player's experience more challenging I added a few simple scripts: one to spawn enemies, and second to reduce player health as it receives damage. As mentioned, I am not a developer and didn't want to spend much time learning scripting so I asked a friend to help me with this part of code. If you lack programming skills too, you can create the same mechanics with the help of various Unity tutorials on YouTube (Breakeys is very useful) and on the free Unity Learning portal. Of course, it takes more time, but you will get useful skills out of it for sure.

Graphics

It was planned to be a small non-commercial game, so I didn’t want to spend a lot of time on development. Graphics is a very time-consuming part of any project but adding context is a need. In this case, I made assets that I could reuse several times.

For landscape, I made a simple landscape object in Cinema4D and a seamless texture that I could animate with offset in a simple way.

I made a model of a tree based on the tree from the first episode, colored it, and baked AO map on it, all in Substance Painter. I used auto UV layout in Painter and it works great for me.

While developing the game, I got an idea about who the enemies are. Rick and Morty are on a planet where their memories from all previous episodes are falling from the sky like asteroids. Every enemy has a texture of one of the screenshots from the series. So for enemies, I made custom shader in Unity shader graph. It makes a simple sphere wavy and creates a dissolve effect as it gets shot by the player.

As it is a non-commercial game, I downloaded a free Rick model (made by quentin.cdt) from Sketchfab and made auto rig and simple running animation in Mixamo. Also, I made a toon UI for the main menu, instructions, and game UI with life counter and timer.

Conclusion

As a result, I got not very many downloads and some very positive feedback from my friends. But I expected something like this.

You can also play games with less complex setup (without Makey Makey), only with an AR marker and a web camera. Just choose this option in the game menu. You can download the game from itch.io. Also if you are interested in more content about Unity, game making, computer graphics, and AR and VR projects you can subscribe to my Twitter. If you've got any questions you can contact me there or through the form on my website.

Pavel Efremov, CG Artist and Art Director