The CEO of Auki Labs and Matterless Nils Pihl have told us about the companies' recent achievements in the field of collaborative AR, discussed the challenges of the industry and limitations of the current tech, and spoke about real-time hand tracking in augmented reality.
In case you missed it
You may find these articles interesting
My name is Nils Pihl and I am the CEO at Auki Labs. I studied philosophy at the University of Lund for a year before I dropped out to join the software industry. My specialty is in behavioral engineering and memetics, and I did a lot of consulting and workshops for companies like Apple, Weibo, and the World Bank.
I've been lucky to have a chance to contribute to a lot of great projects, from an intracortical neural interface eight years before Neuralink, to real-time behavioral analytics software, to running the Data department for the world's third-largest mobile OS.
Auki Labs actually started out under a different name, Minitaur Gaming, and we were making an augmented reality companion for competitive tabletop games like Warhammer. However, we quickly learned that there were a lot of limitations to what you could accomplish in shared AR, and after banging our heads against the wall for a year and a half, lightning struck, and we invented and filed a patent for a new calibration method that made it seamless to join and calibrate into an ongoing AR session. When we had a technical demo of the invention, we quickly raised money from forward-looking investors like Outlier Ventures and Kenetic Capital that understood how disruptive shared AR will be to the current way of interacting with computers.
We also realized that the peer-to-peer and decentralized approach that we were taking to positioning provided a more hopeful alternative to the dystopian outlook for centralized visual positioning systems like those from Niantic or Google, and we found it very motivating to deploy such a solution to combat the rise of surveillance capitalism.
We have a really stellar team with a mix of industry veterans and bright young creators. Between Auki Labs and Matterless, our daughter company that creates toys and companions using our tech, we are now a team of 50 people, split roughly 50/50 between the two teams. Getting to work with wise wizards like Ted Ostrem, José Henriques, Dariusz Zolna, and Oskar Levin, just to name a few, is an incredible privilege. Since we work remotely, we've had the opportunity to hire the best people we can find in the world rather than limiting ourselves to our local market. Now we have brilliant team members everywhere, from New York to Taipei.
Collaborative Spatial Computing
We believe that what will make augmented reality feel real and get mass appeal is when it is shared, and sharing AR requires very precise collaborative positioning. While our competitors focus on helping several devices understand where they are in the world, which is a very difficult task indeed, we realized that it is often sufficient to help the devices understand where they are in relation to each other. Also, more devices have more sensory information about the world, and allowing them to collaborate can create experiences that are better than a single device could manage.
The ConjureKit SDK allows the participating devices and applications to share SLAM and semantic data in ways that improve the congruence and quality of the experience. For example, when two devices are sharing SLAM data with each other and detect that they are both looking at the same flat surface, then they can send a 70-byte message over our decentralized real-time networking service to improve their calibration relative to the flat surface.
Limitations of Current AR Tech
Environmental understanding is, of course, a very limiting factor, but we think the biggest issue for social experiences is how hard it is to get several devices to agree on their relative positioning – which is, of course, why we have made that our focus. Using competing positioning solutions also requires a lot of data, a good internet connection, and a real-time networking solution that has a really low response time.
Normally, when you play a game online, you don't get to see how other players see the world, so it is easier to mask lag – but when you are in shared AR in your own actual environment, any lag becomes a lot more palpable. That's one of the main reasons why we chose to try to decentralize Hagall, the networking service that handles collaborative spatial computing, to increase the likelihood that the experience can connect to an ultra-local mini-server.
The ConjureKit, as we call the SDK, is based on a couple of powerful modules that come together to form something greater than the sum of their parts. The Manna module handles the instant calibration, while Hagall handles real-time message passing, Vikja handles actions between participants, and Ur provides monocular 3D hand reconstruction to allow for even baseline AR devices to support touching digital objects.
Hagall is so lightweight that we've successfully run nodes on both Raspberry Pis and our CTO's router at home, which makes it easy for people to find spare compute (maybe on their work computer or their Mac Mini under the TV) and connect it to the network to help serve traffic in exchange for rewards. We are very proud of the Hagall network and excited about how the modules come together to create really awesome social AR experiences.
Real-Time Hand Tracking in AR
To create the demo attached above, we worked with Dr. Zhongpai Gao, who was our Computer Vision Lead at the time, and José Henriques to develop a monocular hand tracker based on a modified version of Mediapipe.
The challenge was getting a 3D understanding of the hand's position in world space from only the RGB feed of a single camera, but we managed to create a model that estimates the size and position of the hand in the real world. It's still not perfectly accurate, but it's a huge leap forward, especially for devices without a LIDAR, which are otherwise restricted to only detecting the hand in screen space rather than world space. I can't dive too deeply into how the magic works, partially because it's proprietary but also because I can't pretend to understand the wizardry of José and Zhongpai.
The Hagall network went into private testing back in September, and the ConjureKit went into public beta on September 30th. In the future, we plan to dive deeper into our Lighthouses framework, the scaffolding for persistent and collaborative digital reconstructions of the world. Collaborative scene reconstruction, semantic calibration, distributed light estimation to preserve battery, and UWB trilateration, allowing for privacy-preserving AR glasses that don't have a camera... we have a lot of cool things in the pipeline. If you want to know more, I highly recommend you read our whitepaper.