Ruslan Kapustin from Skywell Software overviewed the process of an AR app for Ecotoki development and talked about the problems faced during its production.
Nowadays, augmented reality development is a profitable business, but not every company is able to achieve success and recognition.
AR-based applications provide impactful visual experiences, engage customers. In addition, it helps to provide training and enhances productivity and efficiency.
This eco-friendly toy with elements of augmented reality allows the parents to captivate the child with entertaining animations that help in learning the alphabet and words. Up to date, we have created English and Russian versions of the game. The essence of the game is to use the smartphone camera on the board to activate the letter. After that, you can click on any letter via your smartphone screen to see an interactive 3D model with animation that appears in front of you depicting an animal or an object starting with this letter, as well as the word itself so that the child can quickly remember it. The game has already been bought by many parents, but in spite of the success of this AR application, its creation was quite complex and thorny, even for our team.
Stages & Problems of Ecotoki AR App Creation
The creation of every AR app begins by adding an object to the real world.
To add something to the real world, you need to “cling” it to an object in this world. Targets serve for this purpose. In other words, a target is a kind of a real object thanks to which our application can place virtual objects in the right places and appropriate proportions. In the case of Ecotoki, our target was a wooden board and we had some issues with targeting our 3D models.
For a stable search of points, we need to create a target that suits our technical specifications. When activating the smartphone’s AR camera, the components of Vuforia compare the prepared target with the visualization of the camera, for further superposition of the 3D model. From this, you can conclude that the more “points” we have on the target, the faster and more stable the imposition of virtual objects will be. Points represent a set of contrasting drops. Before defining and calculating points, the image is transferred to JPG format with RGB GrayScale. A good example of fixing points is the boundaries of discontinuous segments since there we can see a clear difference in color. If we do not have enough points, then except adding elements to the picture, we also filter the image before sending it to Vuforia for calculating points. After calculating the points, the programmer visually checks their locations to determine the quality of further tracking. The ideal option is an equable distribution of points over the target’s surface.
The application architecture is based on the MVC pattern.
MVC (Model-View-Controller) is a pattern that includes several smaller templates. When using MVC, the application’s data model, user interface, and user interaction are divided into three separate components so that modifying one of them has minimal impact on the others or does not have it at all. The main purpose of MVC is to separate data and business logic from visualization (appearance). Due to this separation, the possibility of reusing software code is increased and maintenance is simplified (changes in appearance, for example, do not affect business logic).
The basis of AR application architecture is the MVC pattern. This pattern allows the developer to quickly add new functionality to the app based on an optimized and logical distribution of tasks among the scripts.
Since the application is focused on a wide range of mobile devices, asset optimization plays an important role.
An asset is a digital object mainly consisting of data of the same type, an indivisible entity that represents a part of the content and has certain properties. From the point of view of the program model, an asset (resource) can be an object created on a certain data set and the resources are the “building blocks” of the project itself. Assets include all data used and processed by the application: geometric models, textures, individual sounds, and music tracks, dialogue texts, animation data, behavior patterns, and so on. The collection of all assets represents the content of the app.
To output a stable number of frames per second in these conditions, the scene should have a small polygon and the number of operations in the shaders should be minimal. Therefore, our AR team uses optimized shaders and only 2 texture maps – a color map and a normal map. The exception was the Ecotoki characters. For this project, we used more polygons and bigger textures sizes.
Lack of Points & Space Distortion
Since the Ecotoki board is an object with volumetric letters, we had a problem with a lack of points and space distortion.
In order to quickly determine the target and stable operation of AR, we needed to get chaotic-evenly points throughout the board area.
After fixing the board, we got dots most of which were on the borders of each letter.
The number of points was insufficient, and when the positioning of the camera changed, the marker constantly flew off. This was because the board was not flat and the letters protruded from the main plane. Since most of the points were located on the borders of the letters, when the camera view changed, they either disappeared from the visibility zone or were deformed because of what the space distortion and tracking loss occurred.
There was a need for additional pictures and patterns on the tablet to increase the number of points and subsequently, the quality of tracking. Since there were marketing restrictions on changing the overall picture, it was decided to add new patterns under the letters which allowed fixing the focus of the camera on these patterns and not on the borders of the alphabet elements.
The Problem of Similar Targets
As Ecotoki designers created two versions of the boards for each language and they were supposed to work in one application, we had some issues with the display of our models.
One important rule in the development of augmented reality is to prevent the simultaneous use of two very similar targets. The boards for one language had minor differences which caused the constant departures of the model binding from the fact that the device could not decide on which board we are directing.
To solve this problem, we proposed to create a so-called “Key”. The key is an additional removable element on the board that has enough points to identify the board.
As a result, it solved the problem of similarity of targets. Now, when you hover the device on the target, the camera spells out the tablet language and identifies the type using the key.
Although our work on the application was almost completed, we faced another problem: downloading the application to the App Store and the Google Play.
Our application did not immediately pass moderation on the App Store. Since the application could be downloaded by any user of Apple, the moderators demanded to indicate that the application only works with the Ecotoki tablet and state the official website where it can be purchased. But switching to external resources is not welcomed by App Store because any child can go through the application to the site and spend the money of the parents. The decision was the introduction of parental control: we have implemented a small task, for which the child is more likely to need some help from a parent.
With the Google Play, in addition to the issues stated above, there was also a problem with the size of the application.
Size plays an important role in AR development since the applications are designed for mobile devices that have limited memory. If your application exceeds the size of 100 MB, you will need to implement the functionality of loading the main part of the content, namely models, textures, and audio files from the server.
The maximum allowable weight of a mobile application on the Play Market is 100 MB while our application was 125 MB. Therefore, we had to minimize the application as much as possible.
In the case of the AR game Ecotoki, the Asset bundle was loaded when the application was first turned on. The app checks the data of the latest update and loads, if necessary, the missing content.
As a result, we reduced the application to 47 MB which allowed Android users to successfully use Ecotoki.
These are just some of the nuances and features that need to be considered when developing an AR application. In fact, this is a very long and laborious process, but after examining all the subtleties and pitfalls of this innovative direction, you will achieve professional success and recognition.
Ruslan Kapustin, Head of R&D in VRAR Department at Skywell Software
The fastest solution for professional video editing, audio editing, and disc authoring. Now with even more innovative creativity tools like advanced motion tracking, world-class video stabilization, and dynamic storyboarding that deliver incredible results faster than ever.