• Follow us


How Oculus squeezed sophisticated tracking into pipsqueak hardware – TechCrunch

Making the VR experience simple and portable was the main goal of the Oculus Quest, and it definitely accomplishes that. But going from things in the room tracking your headset to your headset tracking things in the room was a complex process. I talked with Facebook CTO Mike Schroepfer (“Schrep”) about the journey from “outside-in” to “inside-out.”

When you move your head and hands around with a VR headset and controllers, some part of the system has to track exactly where those things are at all times. There are two ways this is generally attempted.

One approach is to have sensors in the room you’re in, watching the devices and their embedded LEDs closely — looking from the outside in. The other is to have the sensors on the headset itself, which watches for signals in the room — looking from the inside out.

Both have their merits, but if you want a system to be wireless, your best bet is inside-out, since you don’t have to wirelessly send signals between the headset and the computer doing the actual position tracking, which can add hated latency to the experience.

Facebook and Oculus set a goal a few years back to achieve not just inside-out tracking, but make it as good or better than the wired systems that run on high-end PCs. And it would have to run anywhere, not just in a set scene with boundaries set by beacons or something, and do so within seconds of putting it on. The result is the impressive Quest headset, which succeeded with flying colors at this task (though it’s not much of a leap in others).

What’s impressive about it isn’t just that it can track objects around it and translate that to an accurate 3D position of itself, but that it can do so in real time on a chip with a fraction of the power of an ordinary computer.

“I’m unaware of any system that’s anywhere near this level of performance,” said Schroepfer. “In the early days there were a lot of debates about whether it would even work or not.”

Our hope is that for the long run, for most consumer applications, it’s going to all be inside-out tracking. The term for what the headset does is simultaneous localization and mapping, or SLAM. It basically means building a map of your environment in 3D while also figuring out where you are in that map. Naturally robots have been doing this for some time, but they generally use specialized hardware like lidar, and have a more powerful processor at their disposal. All the new headsets would have are ordinary cameras.

“In a warehouse, I can make sure my lighting is right, I can put fiducials on the wall, which are markers that can help reset things if I get errors — that’s like a dramatic simplification of the problem, you know?,” Schroepfer pointed out. “I’m not asking you to put fiducials up on your walls. We don’t make you put QR codes or precisely positioned GPS coordinates around your house.”

quest top

“It’s never seen your living room before, and it just has to work. And in a relatively constrained computing environment — we’ve got a mobile CPU in this thing. And most of that mobile CPU is going to the content, too. The robot isn’t playing Beat Saber at the same time it’s cruising though the warehouse.”

It’s a difficult problem in multiple dimensions, then, which is why the team has been working on it for years. Ultimately, several factors came together. One was simply that mobile chips became powerful enough that something like this is even possible. But Facebook can’t really take credit for that.

More important was the ongoing work in computer vision that Facebook’s AI division has been doing under the eye of Yann LeCun and others there. Machine learning models frontload a lot of the processing necessary for computer vision problems, and the resulting inference engines are lighter weight, if not necessarily well understood. Putting efficient, edge-oriented machine learning to work inched this problem closer to having a possible solution.

Most of the labor, however, went into the complex interactions of the multiple systems that interact in real time to do the SLAM work.

“I wish I could tell you it’s just this really clever formula, but there’s lots of bits to get this to work,” Schroepfer said. “For example, you have an IMU on the system, an inertial measurement unit, and that runs at a very high frequency, maybe 1000 Hz, much higher than the rest of the system [i.e. the sensors, not the processor]. But it has a lot of error. And then we run the tracker and mapper on separate threads. And actually we multi-threaded the mapper, because it’s the most expensive part [i.e. computationally]. Multi-threaded programming is a pain to begin with, but you do it across these three, and then they share data in interesting ways to make it quick.”

Schroepfer caught himself here; “I’d have to spend like three hours to take you through all the grungy bits.”

Part of the process of creating Insight was also extensive testing, for which they used a commercial motion tracking rig as ground truth. They’d track a user playing with the headset and controllers, and using the OptiTrack setup measure the precise motions made.


Testing with the OptiTrack system

To see how the algorithms and sensing system performed, they’d basically play back the data from that session to a simulated version of it: video of what the camera saw, data from the IMU and any other relevant metrics. If the simulation was close to the ground truth they’d collected externally, good. If it wasn’t, the engineers would adjust the system’s parameters and they’d run the simulation again. Over time, the smaller, more efficient system drew closer and closer to producing the same tracking data the OptiTrack rig had recorded.

Ultimately it needed to be as good or better than the standard Rift headset. Years after the original, no one would buy a headset that was a step down in any way, no matter how much cheaper it was.

“It’s one thing to say, well my error rate compared to ground truth is whatever, but how does it actually manifest in terms of the whole experience?” said Schroepfer. “As we got towards the end of development, we actually had a couple passionate Beat Saber players on the team, and they would play on the Rift and on the Quest. And the goal was, the same person should be able to get the same high score or better. That was a good way to reset our micro-metrics and say, well this is what we actually need to achieve the end experience that people want.”

The computer vision team here, they’re pretty bullish on cameras with really powerful algorithms behind them being the solution to many problems. It doesn’t hurt that it’s cheaper, too. Lidar is expensive enough that even auto manufacturers are careful how they implement it, and time-of-flight or structured-light approaches like Kinect also bring the cost up. Yet they massively simplify the problem, being 3D sensing tools to begin with.

“What we said was, can we get just as good without that? Because it will dramatically reduce the long-term cost of this product,” he said. “When you’re talking to the computer vision team here, they’re pretty bullish on cameras with really powerful algorithms behind them being the solution to many problems. So our hope is that for the long run, for most consumer applications, it’s going to all be inside-out tracking.”

I pointed out that VR is not considered by all to be a healthy industry, and that technological solutions may not do much to solve a more multi-layered problem.

Schroepfer replied that there are basically three problems facing VR adoption: cost, friction and content. Cost is self-explanatory, but it would be wrong to say it’s gotten a lot cheaper over the years. PlayStation VR established a low-cost entry early on, but “real” VR has remained expensive. Friction is how difficult it is to get from “open the box” to “play a game,” and historically has been a sticking point for VR. Oculus Quest addresses both these issues quite well, being at $400 and, as our review noted, very easy to just pick up and use. All that computer vision work wasn’t for nothing.

Content is still thin on the ground, though. There have been some hits, like Superhot and Beat Saber, but nothing to really draw crowds to the platform (if it can be called that).

“What we’re seeing is, as we get these headsets out and in developers hands, that people come up with all sorts of creative ideas. I think we’re in the early stages — these platforms take some time to marinate,” Schroepfer admitted. “I think everyone should be patient, it’s going to take a while. But this is the way we’re approaching it, we’re just going to keep plugging away, building better content, better experiences, better headsets as fast as we can.”

Read More

Leave A Comment

More News

Engadget RSS Feed

Intrigue at the Nurburgring as Tesla unofficially beats New! 2019-09-18 05:15:00When Porsche beat the Nurburgring lap record for a four-door EV, it was sending a message to rival Tesla: "Hey, 2-second 0-60mph acceleration times a

The iPhone 11 Pro Max has 23 percent New! 2019-09-18 04:31:00When you check the iPhone 11 model's technical specs on Apple's website, you won't see specific information about its RAM and battery capacity. Tho

Google Assistant gets new voice options in nine New! 2019-09-18 04:00:00Google Assistant users outside the US haven't had much choice when it comes to voice options -- until now, as Google launches a second voice for Assi

Pinterest's latest Lens update makes it easier to New! 2019-09-18 03:02:00Online shopping can be daunting, especially if you're looking for one very specific type of item. Rather than search for "green Jaipur tapestry" or

HP's new ultrawide monitor can show two device's New! 2019-09-18 02:00:00In addition to a super light laptop, HP also unveiled an intriguing new ultrawide curved monitor today. Creatively named S430c, the 43.4-inch screen i

HP Elite Dragonfly hands-on: A really light business New! 2019-09-18 02:00:00 HP is chasing superlatives again. Last year, the company launched the Spectre Folio -- the first laptop made out of leather. It also trotted out

Chinese retailers abruptly stop selling Juul e-cigarettes New! 2019-09-18 01:09:00It appears Juul's fortunes are about to get even worse. Bloomberg reports retailers in China have pulled the company's products just one week after

Short-form streaming remake of 'The Fugitive' stars Kiefer New! 2019-09-17 23:21:00Kiefer Sutherland (24, Designated Survivor) will star in a series remake of the 1993 Harrison Ford movie, The Fugitive at Quibi. Variety reported toda

Popular podcast app Pocket Casts is now available New! 2019-09-17 22:36:00We're big fans of Pocket Casts here at Engadget. The podcast app is arguably the best option for listening to the regular episodic content, keeping t

Snapchat may add a dedicated news tab New! 2019-09-17 22:16:00Snap's efforts to bring news to Snapchat haven't gone so well. You've had to plow through entertainment and other material just to find news, and

Hulu hackathon leads to eye-tracking controls for Roku New! 2019-09-17 21:34:00Of the 40 project ideas that came out of Hulu's annual hackathon this summer, more than a quarter addressed the needs of users with disabilities. Tod

Amazon’s ‘Lord of the Rings’ series will shoot New! 2019-09-17 20:49:00Amazon Studios' Lord of the Ring series will be filmed in the lush green forests of none other than New Zealand. The streaming platform announced tod

Sciences et Avenir en temps

France: Google verse 500 millions pour arrêter des 2019-09-12 10:12:58PARIS (Reuters) - Google a accepté de verser 500 millions d'euros d'amende dans le cadre d'un accord, homologué jeudi par la justice,

Facebook ne peut aujourd'hui lancer sa monnaie Libra 2019-09-12 07:29:33PARIS (Reuters) - Facebook ne peut être autorisé à développer sa cryptomonnaie Libra en Europe, a estimé jeudi Bruno

Toyota teste une Prius alimentée par le soleil 2019-09-12 06:56:02par Kevin Buckland TOKYO (Reuters) - Toyota Motor teste un véhicule alimenté à l'énergie solaire dans le cadre d'un proj

VIDEO. Une Audi autonome éclairée par des drones 2019-09-12 04:10:54Le constructeur automobile allemand a présenté un audacieux concept car au salon automobile de Francfort. Les phares sont remplacé

Mission terminée : le robot humanoïde Fedor n'ira 2019-09-11 11:25:34La mission est terminée pour le premier robot humanoïde russe à avoir séjourné dans l'espace, Fedor, qui ne sera plu

Et des cendres du 11-septembre sortit Google News 2019-09-11 09:28:53Le jour des attentats contre le World Trade Center et le Pentagone en 2001, Google a enregistré un brusque afflux de requêtes qu&rsq

La cryptomonnaie de Facebook va solliciter une licence 2019-09-11 02:10:42ZURICH (Reuters) - Libra, le projet de cryptomonnaie de Facebook, va solliciter une autorisation en tant que système de paiement en Suisse, a-t

Apple présente l'iPhone 11 et Apple TV+, son 2019-09-10 15:12:44Apple a présenté l'iPhone 11 lors de sa conférence annuelle de présentation de ses nouveautés le 10 septembre 2019

Ford lance une offensive dans l'électrique en Europe 2019-09-10 12:54:25(Reuters) - Ford Motor a annoncé mardi le lancement de huit modèles électriques cette année en Europe, où il s'est

PSA n'exclut pas des défaillances de groupes auto 2019-09-10 05:51:17PARIS (Reuters) - PSA n'exclut pas que la décennie qui s'annonce soit marquée par des défaillances d'entreprises du secteur au

Etats-Unis: Des Etats enquêtent sur des géants de 2019-09-09 15:35:05WASHINGTON (Reuters) - Les autorités judiciaires de la 48 Etats américains, du District of Columbia et de Porto Rico ont ouvert une enqu

Nouveaux iPhone : ce que pourrait annoncer Apple 2019-09-09 12:06:43Plus de coloris, de meilleures photos ou encore une meilleure autonomie... Que nous réservent les différents iPhone présent&eacut

LEXPRESS.fr - Rss High-Tech

Sécurité des paiements en ligne : ce que 2019-09-12 08:40:50De nouvelles normes destinées à renforcer la sécurité des transactions en ligne vont entrer en vigueur samedi.

SVOD : Salto périlleux 2019-09-11 13:00:00A l'heure ou Apple déboule sur le marché de la SVOD, Salto, le "Netflix français", résistera-t-il aux obstacles sem&eac

iPhone 11, streaming, jeux vidéo... Apple présente ses 2019-09-10 14:24:00Depuis la Californie, Tim Cook a présenté ce mardi les nouveautés de la marque à la pomme pour cette rentrée 2019.

Italie : Facebook ferme les comptes de deux 2019-09-10 09:25:17Le géant américain accuse notamment ces deux organisations d'extrême droite néo-fascistes de propager la haine envers les

Wikipédia indisponible dans plusieurs pays après une attaque 2019-09-07 07:56:00Samedi matin, la Wikimedia Foundation affirmait que cette attaque était "en cours". Le site est de nouveau accessible en France depuis samedi

"Facebook Dating" : un pari ambitieux et risqué 2019-09-06 12:24:00Le réseau social a annoncé jeudi le lancement de son application de rencontres. La veille, une fuite de données de 400 millions d

Freebox Delta : Bouygues Telecom réclame 170 millions 2019-09-06 05:09:00La filiale du groupe de BTP accuse Free de pratiques commerciales trompeuses sur sa dernière box. Une contre-attaque en réponse à

Facebook : nouvelle fuite de données concernant plus 2019-09-05 06:29:21Les bases de données répertoriaient les identités des utilisateurs, les numéros de téléphone associés

Devoirs de rentrée 2019-09-04 13:00:00Les lynchages de Greta Thunberg et Eric Woerth, les dangers de Zao et FaceApp, les Hongkongais contre la censure... Ca s'est passé cet é

VIDÉO. En Chine, votre visage suffit pour payer 2019-09-04 11:42:01Des nombreuses enseignes chinoises développent un paiement par reconnaissance faciale, un système déjà omniprésent

Protection des données : 170 millions de dollars 2019-09-04 10:32:49Le géant du web est accusé d'avoir enfreint la loi en exposant les enfants à des vidéos inappropriées ou en collec

Mickey montre ses muscles face à Netflix 2019-09-04 05:55:00Disney mise sur son catalogue de super-héros et de Jedi pour tailler des croupières à Netflix. Mais, en Europe, la France et le R

Ars Technica » Gear

Preventing GPS spoofing is hard—but you can at 2019-09-12 06:45:45This GPS spoofing defense software looks promising, but it's short on detail.

AMD improves real-world boost and idle on Ryzen 2019-09-11 15:24:37The extra 25-50MHz boost speed probably means less than the lower idle, really.

iFixit tears down the newest Fairphone—how repairable is 2019-09-11 13:14:04The Fairphone 3 gets the full iFixit treatment and a 10/10 score.

Apple iPad 2019 hands-on: A 6th-generation iPad in 2019-09-10 21:23:02An older CPU meets a (sort of) newer design, but it's still a notable improvement.

Apple reveals iOS 13 and iPadOS release dates—but 2019-09-10 21:10:40watchOS and tvOS updates are also just around the corner.

Apple Watch series 5 hands-on: Software is king 2019-09-10 20:10:10New materials and sensors headline this annual update. And we have photos.

iPhone 11, iPhone 11 Pro, and iPhone 11 2019-09-10 18:46:01Apple's new iPhones bring greatly improved cameras and new colors, among other things.

Apple continues health push with three new medical 2019-09-10 16:49:16Research partners are enthusiastic, but they face hurdles with this kind of data.

Apple’s iPhone 11 Pro is official, with three 2019-09-10 14:21:24$999 5.8-inch Pro and $1,099 6.5-inch Pro Max arrive on September 20.

Apple introduces the new $699 iPhone 11—and it’s 2019-09-10 13:54:19The follow-up to the iPhone XR adds a new dual-camera system.

The $399 Apple Watch Series 5 is coming 2019-09-10 13:46:28New materials and customization options abound, plus an always-on display.

Surprise! Apple unveils its 7th-gen iPad with a 2019-09-10 13:46:19It will still cost $329 and ship on September 30.

TechCrunch » Gadgets

Nintendo shows off exercise-powered RPG for Switch, Ring 2019-09-12 13:28:43Nintendo has been at the crossroads of video games and fitness since the famous Power Pad for the NES, and the Switch is the latest to receive a game

ckbk pulls a ‘Spotify for recipes’ out of 2019-09-12 07:48:12Cooking may be under sustained attack by a wave of on-demand food delivery startups, with names that can double as gluttonous calls to action (oh hey

Why does the new iPhone 11 Pro have 2019-09-10 14:51:29On the back of the iPhone 11 Pro can be found three cameras. Why? Because the more light you collect, the better your picture can be. And we pretty mu

iOS 13 will be available on September 19 2019-09-10 14:48:12Apple announced in a press release that iOS 13 will be available on September 19. Even if you don’t plan to buy a new iPhone, you’ll be ab

Apple introduces the A13 Bionic for the iPhone 2019-09-10 14:12:13Apple is announcing new iPhone models today. The iPhone 11 uses an Apple A13 Bionic system-on-a-chip. It is a nice improvement over the A12 Bionic in

Apple introduces the Apple Watch Series 5 with 2019-09-10 13:45:55Apple today introduced some new Apple Watch models at a press conference. The Apple Watch Series 5 has an always-on display. It seems to look just lik

Watch Apple unveil the new iPhone live right 2019-09-10 13:01:57Apple is set to announce new iPhone models today. The company is holding a keynote on its campus at 10 AM PT (1 PM in New York, 6 PM in London, 7 PM i

The Polaroid Lab uses the light from your 2019-09-10 08:00:29When all of us are carrying phones that can snap a thousand photos a minute and are connected to cloud systems that can store millions, there’s

SimShine raises $8 million for home security cameras 2019-09-10 03:19:29SimShine, a computer vision startup based in Shenzhen, has raised $8 million in pre-Series A funding for SimCam, its line of home security cameras tha

FDA says Juul ‘ignored the law’ and warns 2019-09-09 17:53:38The Food and Drug Administration has put vaping giant Juul on notice with a pair of letters calling the company out for misleading statements about it

Apple could add in-screen fingerprint reader to 2020 2019-09-05 10:55:11According to a new report from Bloomberg, Apple has been working on in-screen fingerprint readers. But that feature won’t be ready for the new i

Sonos took the mic out of its smart 2019-09-05 09:00:42Sonos has a new entry-level connected speaker that will give you all of its multi-room, high-quality sound — without the onboard microphones and

Disclaimer and Notice:WorldProNews.com is not responsible of these news or any information published on this website.