Complete Guide: Setting Up XRoboToolkit for Robot Teleoperation with Pico 4 Ultra on WSL2

5th March 2026

A step-by-step guide to setting up XR-based robot teleoperation using the Pico 4 Ultra headset, XRoboToolkit, and MuJoCo simulation — all running on Windows WSL2.

What You’ll Build

By the end of this guide, you’ll be able to control simulated dual UR5e robot arms in real-time using your Pico 4 Ultra VR headset. Move your hands in VR, and the robot arms follow in a physics-accurate MuJoCo simulation.

Prerequisites

Windows 11 with WSL2 (Ubuntu 22.04)
Pico 4 Ultra headset
Both devices on the same WiFi network

Phase 1: Install Qt 6.6.3

The XRoboToolkit PC Service requires Qt 6.6.3. Instead of using the GUI installer (which can be tricky on WSL2), we use aqtinstall — a command-line Qt installer.

Install aqtinstall

pip install aqtinstall

Install Qt 6.6.3 with Required Components

# Install Qt 6.6.3 base with modules
aqt install-qt linux desktop 6.6.3 gcc_64 \
    -m qt3d qt5compat qtmultimedia qtquick3d \
    --outputdir ~/Qt

# Install build tools
aqt install-tool linux desktop tools_cmake --outputdir ~/Qt
aqt install-tool linux desktop tools_ninja --outputdir ~/Qt

Configure PATH

Add to your ~/.bashrc:

export QT_DIR="$HOME/Qt/6.6.3/gcc_64"
export PATH="$HOME/Qt/6.6.3/gcc_64/bin:$HOME/Qt/Tools/CMake/bin:$HOME/Qt/Tools/Ninja:$PATH"
export CMAKE_PREFIX_PATH="$HOME/Qt/6.6.3/gcc_64"

Then run source ~/.bashrc.

Verify Installation

~/Qt/6.6.3/gcc_64/bin/qmake --version
~/Qt/Tools/CMake/bin/cmake --version
~/Qt/Tools/Ninja/ninja --version

Phase 2: Build XRoboToolkit PC Service from Source

The PC Service acts as a bridge between your Pico headset and the simulation. It receives hand tracking data from the headset and forwards it to Python scripts via gRPC.

Clone the Repository

cd ~
git clone https://github.com/XR-Robotics/XRoboToolkit-PC-Service.git
cd XRoboToolkit-PC-Service

Update Qt Paths

Edit RoboticsService/qt-gcc.sh and update the Qt paths to match your installation:

# Change these lines:
QT_GCC_64=/home/<your-username>/Qt/6.6.3/gcc_64/
export QT6_TOOLS=/home/<your-username>/Qt/Tools

export PATH=/home/<your-username>/Qt/6.6.3/gcc_64/bin:$PATH
export PATH=/home/<your-username>/Qt/6.6.3/gcc_64/include:$PATH
export PATH=/home/<your-username>/Qt/Tools/CMake/bin:$PATH
export PATH=/home/<your-username>/Qt/Tools/Ninja:$PATH

Fix Build Issues (Important for aqtinstall Users)

When building with a minimal Qt install from aqtinstall, you’ll hit two build errors in RoboticsServiceProcess/CMakeLists.txt:

Issue 1: Missing libqtvirtualkeyboardplugin.so

The virtual keyboard plugin isn’t included in aqtinstall’s minimal Qt package. Find this line (around line 462):

COMMAND ${CMAKE_COMMAND} -E copy "${CMAKE_PREFIX_PATH}/plugins/platforminputcontexts/libqtvirtualkeyboardplugin.so" "${INSTALL_DIR}/plugins/platforminputcontexts"

Replace with:

COMMAND ${CMAKE_COMMAND} -E copy_directory "${CMAKE_PREFIX_PATH}/plugins/platforminputcontexts" "${INSTALL_DIR}/plugins/platforminputcontexts"

Issue 2: Missing resources directory

The resources directory doesn’t exist in aqtinstall packages. Find and remove this line:

COMMAND ${CMAKE_COMMAND} -E copy_directory "${CMAKE_PREFIX_PATH}/resources" "${INSTALL_DIR}/resources"

Keep only:

COMMAND ${CMAKE_COMMAND} -E make_directory "${INSTALL_DIR}/resources"

Build

cd ~/XRoboToolkit-PC-Service
bash RoboticsService/qt-gcc.sh

You should see [100%] Built target RoboticsServiceProcess at the end. Warnings about NULL conversion are harmless.

Run the PC Service

cd ~/XRoboToolkit-PC-Service/RoboticsService/bin
export LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$HOME/Qt/6.6.3/gcc_64/lib:$LD_LIBRARY_PATH
./RoboticsServiceProcess &

The service listens on:

Port 63901 — headset TCP connections
Port 60061 — gRPC (localhost, for Python SDK)
Port 9090 — additional service port

Phase 3: Set Up the Teleoperation Python Environment

Install Miniconda (if not already installed)

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p ~/miniconda3
rm ~/miniconda.sh
~/miniconda3/bin/conda init bash
source ~/.bashrc

If you get a Terms of Service error, accept them:

conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r

Clone and Set Up

cd ~
git clone https://github.com/XR-Robotics/XRoboToolkit-Teleop-Sample-Python.git
cd XRoboToolkit-Teleop-Sample-Python

# Create conda environment
bash setup_conda.sh --conda xrobo

# Activate and install dependencies
conda activate xrobo
bash setup_conda.sh --install

This installs MuJoCo, Placo (inverse kinematics), xrobotoolkit_sdk, PyTorch, OpenCV, and all other dependencies.

Phase 4: Set Up Pico 4 Ultra

Enable Developer Mode

On the Pico 4 Ultra:

Go to Settings > General > Developer
Enable Developer Mode and USB Debugging

Install the XRoboToolkit App

Download the pre-built APK:

curl -sL -o ~/XRoboToolkit-PICO-1.1.1.apk \
    "https://github.com/XR-Robotics/XRoboToolkit-Unity-Client/releases/download/v1.1.1/XRoboToolkit-PICO-1.1.1.apk"

Important for WSL2 users: WSL2 can’t directly access USB devices. Install the APK from Windows instead:

Install Android Platform Tools on Windows
Access the APK at \\wsl$\Ubuntu\home\<username>\XRoboToolkit-PICO-1.1.1.apk
Run from PowerShell: adb install XRoboToolkit-PICO-1.1.1.apk

Phase 5: WSL2 Network Configuration (Critical Step!)

This is the trickiest part. The Pico headset can’t directly reach WSL2’s internal IP address. You need to set up port forwarding from Windows to WSL2.

Get Your IPs

In WSL2:

hostname -I    # e.g., 172.31.200.18

In PowerShell:

ipconfig | findstr "IPv4"    # Find your WiFi IP, e.g., 192.168.5.81

Set Up Port Forwarding (PowerShell as Administrator)

# Forward PC Service ports from Windows to WSL2
netsh interface portproxy add v4tov4 listenport=63901 listenaddress=0.0.0.0 connectport=63901 connectaddress=172.31.200.18

netsh interface portproxy add v4tov4 listenport=9090 listenaddress=0.0.0.0 connectport=9090 connectaddress=172.31.200.18

# Allow through Windows Firewall
netsh advfirewall firewall add rule name="XRoboToolkit" dir=in action=allow protocol=TCP localport=63901,9090
netsh advfirewall firewall add rule name="XRoboToolkit-UDP" dir=in action=allow protocol=UDP localport=63901,9090

# Verify
netsh interface portproxy show all

On the Pico Headset

Use your Windows WiFi IP (e.g., 192.168.5.81), NOT the WSL2 IP.

Phase 6: Connect and Run

1. Start the PC Service (if not already running)

cd ~/XRoboToolkit-PC-Service/RoboticsService/bin
export LD_LIBRARY_PATH=$(pwd):$(pwd)/lib:$HOME/Qt/6.6.3/gcc_64/lib:$LD_LIBRARY_PATH
./RoboticsServiceProcess &

2. Launch the Simulation

conda activate xrobo
cd ~/XRoboToolkit-Teleop-Sample-Python
python scripts/simulation/teleop_dual_ur5e_mujoco.py

3. Connect the Pico Headset

Put on the Pico 4 Ultra
Launch the XRoboToolkit app
A connection prompt appears — enter your Windows WiFi IP
Status should show “WORKING”

4. Enable Data Streaming (Easy to Miss!)

On the XRoboToolkit app main panel:

Under Tracking, toggle “Controller” to ON
Under Data & Control, toggle “Send” to ON

This is the most commonly missed step! Without “Send” enabled, the headset connects but sends zero data.

5. Teleoperate!

Control	Action
Hold grip buttons (L/R)	Activate arm control
Squeeze triggers	Open/close grippers
Move your hands	Robot arms follow in real time

Troubleshooting

All pose values are zeros

The Pico is connected but not streaming data. Make sure “Send” is toggled ON in the XRoboToolkit app under Data & Control.

You can verify data flow with this test script:

import xrobotoolkit_sdk as xrt
import time

xrt.init()
time.sleep(2)
for i in range(5):
    lp = xrt.get_left_controller_pose()
    rp = xrt.get_right_controller_pose()
    print(f'Left: {lp[:3]}, Right: {rp[:3]}')
    time.sleep(0.5)
xrt.close()

If poses are [0.0, 0.0, 0.0], the “Send” toggle is off.

Connection error on Pico

Make sure you’re using the Windows IP (not WSL2 IP)
Verify port forwarding is set up: netsh interface portproxy show all
Check Windows Firewall isn’t blocking port 63901
Ensure PC and Pico are on the same WiFi network

Build errors with Qt

If using aqtinstall, the minimal install won’t include libqtvirtualkeyboardplugin.so or the resources directory. See the CMakeLists.txt fixes in Phase 2 above.

WSL2 IP changes after reboot

WSL2 gets a new IP on each reboot. You’ll need to update the port forwarding rules:

# Remove old rules
netsh interface portproxy delete v4tov4 listenport=63901 listenaddress=0.0.0.0
netsh interface portproxy delete v4tov4 listenport=9090 listenaddress=0.0.0.0

# Add new rules with updated WSL2 IP
netsh interface portproxy add v4tov4 listenport=63901 listenaddress=0.0.0.0 connectport=63901 connectaddress=<NEW_WSL2_IP>
netsh interface portproxy add v4tov4 listenport=9090 listenaddress=0.0.0.0 connectport=9090 connectaddress=<NEW_WSL2_IP>

MuJoCo window not appearing

WSL2 needs WSLg (built into Windows 11) for GUI apps. If you’re on Windows 10, install VcXsrv and set export DISPLAY=:0.

Available Simulation Scripts

# Dual UR5e arms
python scripts/simulation/teleop_dual_ur5e_mujoco.py

# Dual ARX A1X arms
python scripts/simulation/teleop_dual_a1x_mujoco.py

# Flexiv Rizon4s
python scripts/simulation/teleop_flexiv_rizon4s_mujoco.py
python scripts/simulation/teleop_flexiv_rizon4s_placo.py

# Shadow Hand dexterous manipulation
python scripts/simulation/teleop_shadow_hand_mujoco.py
python scripts/simulation/teleop_shadow_hand_placo.py

# Inspire Hand
python scripts/simulation/teleop_inspire_hand_placo.py

# Unitree G1 humanoid
python scripts/simulation/teleop_unitree_g1_placo.py

# ARX X7S
python scripts/simulation/teleop_x7s_placo.py

Architecture Overview

Pico 4 Ultra (XRoboToolkit App)
    |
    | WiFi (TCP port 63901)
    |
    v
Windows Port Forwarding (netsh)
    |
    v
WSL2: PC Service (RoboticsServiceProcess)
    |
    | gRPC (localhost:60061)
    |
    v
WSL2: Python Script (xrobotoolkit_sdk)
    |
    | Placo IK + MuJoCo Physics
    |
    v
MuJoCo Visualization Window

Summary

Use aqtinstall for headless Qt installation — much easier than the GUI installer on WSL2
Fix CMakeLists.txt when building from source with minimal Qt — remove references to missing plugins/directories
WSL2 networking requires port forwarding — the Pico can’t reach WSL2’s internal IP directly
The “Send” toggle is critical — connecting the Pico isn’t enough, you must enable data streaming in the app
WSL2 IP changes on reboot — you’ll need to update port forwarding rules each time

Posted 5th March 2026 at 1:22 am · Subscribe to my newsletter

Akshay Parkhi's Weblog