My 3 weeks with the SlimeVR open source full-body tracker

SilmeVR is an IMU-only open source full body tracker (open firmware + software) for VR. For the non-initiated, normal VR only tracks your head and hands. While full body tracking in addition tracks waist, chest, ankles, and etc.. With FBT you can sit in VR and see your avatar also sitting down, with the same pose. It adds a lot to the emersion factor. And a must have if you ever want to actually lay down and sleep in VR.

I got mine second hand from my friend. Whom built an entire set from scratch, but didn't use it much. His set uses 5 top of the line BNO085 IMUs for 5 trackers. I decided to buy from him instead of enjoy DIY due to the chip shortage. Which hikes the price of the IMU to some absurd price. Obviously mine aren't the official SlimeVR set. Your milage might vary. Also I only play VRChat. Thus the following is in the context of VRChat.

tl;dr

For those are here for the easy-to-understand conclusion. SlimeVR is

• Ok accuracy
• Needs computer skills to setup
• Easy to use after initial setup
• Perfect for light use in VRChat

You should get a set if you are a tinkerer. And is too deep into VRChat to not live without FBT. But find Vive trackers too expensive.

Full body tracking and trackers

FBT has been a thing in consumer market for quite some time now. The popular ones are the Vive tracker series from HTC. These work via a combination of IMU and base stations (IR beacon). The IMU tracks rotation. And the base stations triangulate the trackers through an IR camera. Basically the same tech the movie industry uses. The benifit being both rotation and position are tracked very accurately. Downside is that 1. The base stations are not cheap. And 2. Trackers must always keep a line of sight to the base station. Otherwise it loose tracking and starts drifting significantly (wanna see your avatar yeeting to the universe?).

For me, I don't want Vive trackers. The base stations need to be connected to the PC through a long USB cable. But my PC isn't in the same room as my play space. No chance getting that working. And I don't just want full body tracking. I want to sleep in VR. Actually, I want to sleep under a blanket in VR.

SlimeVR seems to be a better solution for me. No IR tracking so no LoS issues. And open source! But it's not without its flaws. Without base stations, it has to track position by intgrating the acceleration measred by the IMU. Yet acceleration measurments are inherently noisy with a consumer grade IMU. And unlike the Vive trackers sending data over Bluetooth and USB, SlimeVR sends it's data over 2.4GHz Wifi. Which has added latency. All in all, position is not as accurate with SlimeVR.

One intresting question is how does Oculus controllers keep accurate tracking despite not have base station. Short answer is they use the headset itself as the base station. Then track the controller's position relative to the headset. I assume they did it using some conbination of IR and computer vision magic. This however, can't work for SlimeVR as trackers are likely to be hidden to the headset. (ex: on your back, on elbow, etc..)

Levles of full body tracking

Fully body tracking is not a monolithic thing. There's various degress of tracking. Called N-point tracking. The base line is 3 point tracking. This is what you get just with the VR heatset and controllers. 3 being head + the two hands. Next up is 6 point tracking. Adding waist and 2 ancles. Then 8 and 10 point tracking. Where optional knee and elbow trackers gets involved.

As of writing this post. Standard VRChat only supports non FBT (i.e. 3 point tracking) and 6 point tracking. With the IK 2.0 beta, it adds support for 8 and 10 point tracking.

My experience with SlimeVR

I'll skip the process of setting up SlimeVR. That is very well documented in their project[1] and Eirenliel's SlimeVR setup guide[2]. I'll just go over how it actually performs. But PSA first, warning: You need to at be proficient with computers to setup and use SlimeVR. Or have time to learn the more advanced parts and not be afraid to screw things up.

And I recommend this SlimeVR review to evaluate the performance and tradeoffs:

SlimeVR needs your (physical) skeleton dimensions to work correctly. Leg, torsal, etc.. length. The "auto-configuration" feature in the SlimeVR server will get you 80% there. I recomment this video[3] to get started. Afterwards just get into SteamVR to have a gaze at the tracker positions in VR space. Then adjust the skeleton manually so it matches the real world. A trial and error process

I have 5 SlimeVR trackers. So I got 8 points of tracking. To be specific, I've SlimeVR on my ankles, knees and chest. Along with the 3 poins provided by VR headset and controllers.

Right, here's a quick picture of me in VR with full body tracking. Of course, infront of a mirror.

Accuracy

Accuracy wise, SlimeVR is Ok. As expected it's not as accurate as Vive trackers. But it's enough to reflect your actual pose. One thing to note, you absolutely need to keep the tracker aligned with the your limbs. A knee tracker must be right above the knee and the IMU must orient correctly as you set in the firmware and the SlimeVR server. Any deviation will cause tracked position to be deviate from the real deal. Especially when you are making large movements. Like siting down or laying down. However this kind of drift will automatically recover when you stand up stright again.

A quick example about what I mean by "Ok accuracy" and "needs very good alignment". I didn't strap on my trackers correctly and laying down on the floor. My avatar still reflects the pose. But my legs are slightly raised and bending.

Calibration frequency

Along with it's lower accuracy. SlimeVR drifts out of the correct position more often. Forunatelly the calibration process is quite quick. Just jump into the VR virtual desktop, go to SlimeVR server and click "reset". Calabration will be done in 3 seconds. Then hop back into VRChat and click FBT calibrate. The frequency of needed calibration depends on what I'm doing. If I was always standing, walking or sitting. Then it's relatively far apart. About once per hour or less (I can't keep track of it). But if I were constantly sitting down then standing up. It's quite often. Like 10min.

Rule of thumb: If you are making more large/sudden movements, you need to calibrate more often.

Gameplay

Ah the most imporatant qustion. How it feels to play with SlimeVR FBT is up to how you play the game. I just walk around and sit down with my friends. Sometime laying down cuz I'm drunk or too tired. It works really well. Sometime I had to calibrate it. But it's not bad at all. Mine's built with a 1000mAh battery in each tracker. Which, according to SlimeVR docs, should easily last 12+ hours in active use. Pratically no worrying about battery life.

However if you are the dancers in the clubs, etc.. activity that needs high accuracy and moves your body a lot. You are much better off with a Vive tracker.

If you are going to ERP.. SlimeVR can work but may need much more frequent calibration. Don't ask me how I know that.

Some quirks

One weird thing I noticed. If the waist tracker dies mid-session. For some unknown reason. VRChat just uses the headset (HMD) position as the waist position. Which... WTF? Anyway.. if you ever see your waist position beign very off. That's likely why.

Possible improvments

I'm very intrigued by the fact SlimeVR got BFT working without base stations. It's much more convenient and allows users to do things that warn't possible. Accuracy is the Achilles' heel. So I spent a weekend on arXiv and Google scholar figuring out if there's some self-calabration algorithm, without creating LoS limitations. No, there's isn't. Both SlimeVR server and VRChat already applies Inverse Kinematics to combact inaccurate tracking. Short of someway to measure the distance between teackers, it's hard to improve beyond what IK can do.

After that I started to think more critically. The entire reason to have accurate tracking is so it reflects your actual pose. And not to calibrate the tracker oftern, pulling you away from the game. I stated thinking about ways to periodically calibrate the tracker. Some VR players are using Kinect for full body tracking. Maybe we can point a Kinect at the playspace and periodically run an additional round of IK against the SlimeVR positions - if the user is standing stright and towards the Kinect. Otherwise Kinect might bring too much noise.

Another possible way is to use IR LEDs and IR cameras. We add an additional IR LED pointing outwards from the tracker. A camera is used to triangulate the tracker. Just like the base stations. But instead of relying on that data for position. We only use it for calabration.

Maybe we can also strap multiple IMUs into the same tracker? Instead of having one for all 3 axes. We can have IMUs on different planes to each other. So we can filter out random noise and directional bias. Futhermore, maybe instead of 90 degrees. They could be at 120 degrees. So the 3 axes aren't aligned. Thus we can do some linear algebra to solve for a minimal energy solution.

Finally. The ultimate (and impratical) solution is just use a better IMU. The BNO085 is the best IMU for consumers. But there's a lot more beyond that. According to Anaylog Devices' datasheet, and my untrained eye, their lowest end navigarion IMU ADIS16475[4] is more than 10x as accurate. The only problem being the ~400USD price tag😔.

Hardware wise. I'd like to see support for the new ESP32-C6. WiFi 6 and BLE support should redule the latency or power consumption. And the newer RISC-V processor should be more power efficient than the Xtensa DSP in ESP8266/32. And more open too! The extra computing power should also enable some more advanced algorithms.

Welp, I'll keep this problem in mind. Maybe I can make a PR or even a paper one day.

Gallery

Just me showing off the trackers and FBT. I absolutely love SlimeVR! Not as accurate as I want. But good enough and very convenient.

Sitting down with a friend (also using FBT, but with Vive trackers) in VR:

Sitting down

Lower my body to play with Rusk (the name of the NPC)

Measuring my avatar's height (notice the more natural standing pose)

Enough blogging today. Gonna head back into VR.