These Technologies Could Power the Next ‘Pokemon Go’
By Janko Roettgers
LOS ANGELES (Variety.com) – Niantic, the company behind “Pokemon Go,” announced the acquisition of the U.K.-based computer vision and machine learning startup Matrix Mill Thursday to further build out its (AR) technology. In preparation of the announcement, Niantic invited journalists to a press event in San Francisco Wednesday, where it provided a first behind-the-scenes look at some of its work in the AR space.
At the event, Niantic CEO John Hanke announced that the company was looking to release a major update to its “Ingress’ game later this year. Niantic is also set to release a new AR game called “Harry Potter: Wizards Unite” this year , and Hanke said that the company would reveal additional games in the coming weeks. And later this year, Niantic plans to give a small number of developers direct access to the platform powering all of these games to build their own games and experiences. “We really see this as the future of entertainment,” said Hanke.
So how does this future look like? Attending journalists had a chance to try out one of the demos Niantic engineers have been developing internally, a multiplayer AR game code-named Neon. In it, users sync up their phones and then collect glowing orbs that are added the AR camera view of their phone. Once collected, these orbs can be used to battle opponents.
The game was powered by something that Niantic’s AR platform manager Diana Hu described as low-latency P2P networking, which she said enabled phones to communicate with each other with a latency of less than 10 milliseconds, compared to the typical cloud-based gameplay that would come with latencies of 100 milliseconds or more. In the demo, this was done over Wifi; Hu claimed that it could use cell towers as well, but didn’t share any further details on the company’s networking technology. (Niantic didn’t respond to follow-up questions about the subject.)
Matrix Mill chief research scientist Gabriel Brostow explained that his startup has been working on using machine learning for augmented reality. Part of this work has been around occlusion, or making AR objects blend more naturally into real-world environments. Currently, “” players can add AR Pokemons to the camera view of their phone, but the cute monsters are always placed on top of everything. Matrix Mill has been working on recognizing real-world obstacles, and placing AR objects in relationship to them. In other words, in the future, Pokemons could hide behind bushes.
Brostow explained that it was necessary to collect lots of good training data to get occlusion right. “If you have bad training data, that leads to bad machine learning,” he said. “This is basically our special sauce.”
Niantic has also been doing a lot of work on AR object recognition, according to the company’s chief scientist Hansong Zhang. Advanced AR games and apps need to understand a person’s environment, and for example know that an object is a chair in order to have a Pikachu sit in it. “Without understanding of the world, it’s like you’re blindfolded,” he said. And to truly understand the world, systems powering AR need to have their own taxonomy, or vocabulary, of common objects. “The larger the vocabulary, the richer AR will be,” Zhang said.
Finally, Niantic has been working on perfecting its own mapping data to build more powerful AR experiences. Spun out of Google in 2015, Niantic originally began its work with Google Maps data. However, the company’s AR research lead Ross Finman explained that this wasn’t enough to build AR experiences around public places that aren’t necessarily meant for commuting.
“We want to get past these road-based representations,” he said. What’s more, traditional maps are largely static, and only get updated every few months at best. AR maps on the other hand change all the time. “You need to have a highly accurate live model of the world,” he said.
That’s why Niantic is using all of its users to continuously update its own mapping data, something that will ultimately allow the company to determine locations of people “down to the centimeter,” according to Finman. He didn’t elaborate on what kind of data Niantic is exactly collecting, but claimed that Niantic’s mapping efforts could be key for AR applications far beyond gaming. “All of AR will need a fundamental layer of the world,” he said.
What does all of this mean for the future of “Pokemon Go,” or for the features the company will bring to “Harry Potter: Wizards Unite?” That’s pretty much anyone’s guess at this point; Hanke and his colleagues didn’t comment at all on the Potter game during Wednesday’s press event. The company also didn’t answer any questions from reporters, which made it challenging to evaluate how advanced Niantic’s technology truly is when compared to similar efforts from Facebook, Snap and others.
But it’s easy to imagine that Niantic could use similar social AR mechanics for some of its upcoming titles. In fact, it’s hard to look at the Neon demo and not imagine wizards casting spells against each other.
Hanke also said Wednesday that the company plans to continuously improve existing titles, so we may even see more advanced AR tech find its way to “Pokemon Go.” The ultimate goal of the company was to build and power experiences that could last for decades, he said. “In this world, the games never really have to end,” he said.
