What’s the Galaxy S9’s least loved feature? That’s easy: AR Emoji, the Samsung exclusive that whips up 3D avatars. It’s a pretty transparent imitation of the iPhone X’s 3D Animoji characters — fox, monkey, swirl-capped turd — but with one major, absolutely crucial difference: You. While, yes, you can become characters like Donald Duck, you can also make Samsung’s 3D figures look like you, because they’re made from a selfie photo taken from the Galaxy S9 or larger Galaxy S9 Plus.That was the idea, anyway. In reality, AR Emoji’s creepy human avatars look very little like the real people they try to represent.
Galaxy S9’s AR emoji: How good are they?
None of the over-20 CNET staffers who made an AR Emoji recognized themselves in their avatar. Our 3D selves had plasticky faces that grimaced every time we tried to smile, developed a horrible eye twitch and lacked enough customization options to give me curly hair or clothes I’d actually wear. AR Emoji even gave a short-haired editor a man bun for literally no reason at all.
My colleague Scott Stein and former Techcrunch writer Darrel Etherington, meanwhile, would never be mistaken for twins. But their AR Emoji would have them separated at birth:In theory, AR Emoji lets you customize everything from hair to skin tone. But in reality, those shades fail to represent real life variation, and the default image often disappoints: How could Samsung, the world’s largest supplier of smartphones, have gone so wrong with AR Emoji? Turns out, Samsung didn’t make AR Emoji itself. It snapped up the license to bring these plasticky, unsettling faces to life from two-year-old startup called Loom.ai. That company’s Hollywood-heavy pedigree includes resumes that read like a who’s-who of CG animation: Lucasfilm, Dreamworks, Disney Research. And its CTO and principal engineer are each Oscar-winners who won SciTech Academy Awards for separate achievements in computer animation.So how did people who worked on the technology that controversially brought this major Star Wars: Rogue One character back to life create such a lackluster flop in AR Emoji? Because making a fast, accurate avatar from your phone’s selfie camera is really hard.AR Emoji creation is fast. Maybe too fastHollywood-level motion capture requires considerably more time and money. AR Emoji is designed to be the streamlined home version. .
Choose to setup your AR Emoji on Samsung’s Galaxy S9 (and S9 Plus), and the phone creates your 3D avatar in a matter of seconds. The front-facing camera takes a photo of your face and analyzes that photo to approximate what your features really look like.
People’s attention span is low. People lose it after 5 seconds.
Kiran Bhat, Loom.ai co-founder and CTO
“In the movies, [making a 3D avatar is] a multimillion dollar effort with lots of specialized hardware,” said Kiran Bhat, Loom.ai’s co-founder and CTO. He knows all too well the painstaking and time-consuming work that goes into Hollywood-level motion capture from his time at Lucasfilm.These avatars of Loom.ai’s founders use the same backbone as Samsung’s AR Emoji, but were rendered on a computer instead of on a phone.
Expensive cameras and tracking equipment are used to scan actors’ body movement and even specific facial expressions, which are then mapped on to computer-generated characters such as the Incredible Hulk (The Avengers), Gollum (the Lord of the Rings movies), Caesar (the recent Planet of the Apes trilogy) or Leader Snoke (The Last Jedi). (In fact, three of those four characters are brought to life by the same talented actor: Andy Serkis.)But AR Emoji was designed to be the ultimate express version of the Hollywood process. No day-long motion capture shoot, no green tracking dots on your face; just shoot a selfie instead. “We’ve simplified it into a single photograph,” said Bhat. “People’s attention span is low. People lose it after 5 seconds.”
The Galaxy S9 nails this drinking game
That’s impressive work when switching from film to phones. But if the end result gives you the heebie-jeebies, does it really matter?Samsung needs to shoulder AR Emoji blame, tooWe know that Loom.ai can make more convincing avatars when given more time. Its initial software made much more photorealistic avatars in seven minutes. Indeed, the company’s website shows what could have been: computer-rendered avatars made with the same software tools as AR Emoji take the place of employee photos. They’re still images, not animated videos, but they look good. The Galaxy S9 makes AR Emoji quickly, I’ll give it that.
But Loom.ai’s business relies on creating the framework to give its customer — in this case, Samsung — whatever it wants. Loom.ai supplied the SDK (software development kit) that drives the avatars, Bhat said, explaining that AR Emoji “uses a 2D tracker provided from Samsung on how the face moves, which is what gets fed into the SDK.” Remember, too, that unlike the iPhone X, the Galaxy S9 doesn’t have the hardware to do 3D scanning — it can only take a 2D image of your face. This photo has fewer details to grab onto, which explains why AR Emoji are rather generic and don’t track your expressions as well — the software is working with a limited topography. According to Loom.ai, Samsung picked how AR Emoji are styled and created the face-tracking software to make avatars move. Improving this tracking software would, in theory, improve the quality of the AR Emoji likeness. Samsung declined to comment on this story.A different manufacturer or app, meanwhile, could generate a better (or worse) result by plugging its own tracking algorithm into Loom.ai’s SDK. This would be instructive… if Loom.ai had any other partners ready to bring their wares to market. (Right now, Loom.ai is also working with Chinese phone and app brand Meitu, but there’s no public product yet.)How AR Emojis will get betterIt’s still early days for animated emoji on any phone, but right now Apple is far ahead of the game.AR Emojis will have to improve.
That infamous notch on the iPhone X is crammed full of key sensors not yet found on other phones, including a dot projector, flood illuminator and infrared camera. In addition to enabling FaceID, they work in concert with the selfie camera, mapping the user’s face in real-time. As a result, the facial expressions and mouth movements of Apple’s Animojis sync are much better with your movements, even if they are transferred onto a robot emoji or a cartoonish pile of poop. “The ability to give the system more data is helpful,” Bhat said. “If [a phone has] extra sensors, that’s an active area that would help the algorithm outline your face better.” He added that depth-sensing cameras run the risk of creating digital noise, however.In addition to better cameras with more sensors, Loom.ai said that tweaking computing algorithms that make your avatar face move can help AR Emoji become more lifelike, with less lag time to boot.The founders also acknowledge the need to present more varied body types, hair colors and other markers of self-identification.In the meantime, we can expect Galaxy S9 software updates to improve AR Emoji. “The exact timing is up to Samsung,” said Ramasubramanian, Loom.ai’s CEO, “But it is something you will see evolve.”The Loom.ai team will probably never win an Oscar for Galaxy S9’s AR Emoji, but a much-improved version 2.0 is exactly the sort of comeback story that Hollywood is built on.
Read next: Galaxy S9 highs and lowsRead also: Best Galaxy S9 and S9 Pro tips and tricks