Remote Access streams a computer's apps and screens to a phone, a browser or another computer as live video. This post covers the performance work behind it so far: the fixes made in September, the changes from this week, and the work now in testing. It is ongoing, and every stream is now logged at both ends so each change can be measured against real use.
How the stream works
The computer captures a window or a display, encodes it as H.264, and sends it to the viewer over WebRTC, the technology browsers use for video calls. The connection runs directly between the two devices, over the local network or a private network such as Tailscale. No video passes through AI Integrator's servers.
The sending side is written in Rust on top of str0m, an open-source WebRTC library. On Windows it uses the hardware encoder in the graphics driver, and on a Mac it uses VideoToolbox.
September: a packet pacer and the Windows clock
In September, a full-screen 1440p test with constant motion reached the viewer at 2.4 frames a second while the encoder produced 51. Smaller and calmer pictures, such as an app window with text being typed into it, streamed normally. A picture like that needs only a few packets per update, while a busy full screen needs hundreds.
The cause was the interaction between the WebRTC pacer and the Windows clock. The pacer releases one packet each time it is called, then asks to be called again at once. The session loop handled that request by sleeping for zero milliseconds, and on Windows a zero-length sleep waits for the next clock tick, which is 15.6 milliseconds by default. That capped the stream at roughly 64 to 110 packets a second.
- The pacer releases one packet
- It asks to be called again now
- The loop sleeps for 0 ms
- Windows waits for the next clock tick. 15.6 ms
- About 64 packets a second
- The pacer releases one packet to It asks to be called again now
- It asks to be called again now to The loop sleeps for 0 ms
- The loop sleeps for 0 ms to Windows waits for the next clock tick
- Windows waits for the next clock tick to The pacer releases one packet: repeat
- Windows waits for the next clock tick to About 64 packets a second
The loop now sends every packet that is already due, up to 512 at a time, before it waits again.
September: bitrate, bandwidth and playback delay
Fixing the pacer exposed several more problems, each fixed in the same release cycle:
- Bitrate. NVIDIA's encoder accepted a target bitrate and then exceeded it on fast-changing pictures. A frame budget now sits in front of the encoder, sized to what the connection can carry. When the budget is repeatedly exhausted, the stream steps down to 75 percent and then 50 percent resolution, and steps back up after eight quiet seconds.
- Bandwidth estimates. The library measures available bandwidth by sending short trains of packets about a millisecond apart, which a 15.6 millisecond clock cannot do. While a stream is live, the computer now requests a one-millisecond timer from Windows, including when the app is minimized. On a loopback test, the estimate now reaches 18 to 22 megabits a second within the first second, up from a ceiling near 4.
- Keyframes. The encoder now applies its keyframe interval correctly, and repeated keyframe requests from a viewer are spaced to match how long a keyframe takes to arrive.
- Playback delay. A phone could fall 10 to 30 seconds behind the computer. Video now asks to be played as soon as it is decoded, and the computer keeps the audio clock running through silence, because the phone's player held video back to stay in step with audio that had stopped.
Stream logging
Every stream is now recorded at both ends: delay, frame rate, bitrate, and the reason a stream ended. On a computer or in a browser, Show stats in the viewer's menu shows the same numbers live. The phone's stream log is attached to bug reports, so a report about a slow stream arrives with the measurements needed to investigate it.
This week: connection path and session renewal
Two changes finished this week and are coming in the next release:
- Connection path in stats. The Connection row in Show stats used to read "Direct" for every connected stream. It now names the actual route: Local network, ZeroTier, VPN, Tailscale direct, or Tailscale relayed with the relay's region. The computer reads which addresses the video connection is using and, for Tailscale, checks the device's status in the background without delaying the stats view. The path is also written to both stream logs, which makes a relayed connection easy to spot in a slow-stream report.
- Session renewal between computers. When one computer views another, its session with that computer is now renewed in the background before it expires. Each renewal proves the device's key and account sign-in again, so it never widens what the session may do.
In testing: GPU-only encoding on Windows
On PCs with NVIDIA graphics, a new path keeps every picture on the graphics card from capture to encoding. The captured window or display is copied into a shared ring of GPU textures, scaled and converted to the encoder's colour format on the GPU, and encoded by NVENC. The current path reads each picture back to main memory, converts it on the processor, and uploads it again. Computers with AMD or Intel graphics, and Macs, keep the current path for now.
The new path also changes how the encoder behaves:
- Rate control. The encoder runs at a constant bitrate with a one-frame buffer and follows the network's bandwidth estimate directly. Under pressure it lowers picture quality instead of skipping whole frames.
- Priority while gaming. During a stream, the encoding work runs at the realtime GPU scheduling priority, so it no longer waits behind a game's frames.
- Encode on arrival. Each picture is encoded as soon as it is captured, without waiting for a fixed frame interval.
- Faster starts. A viewer who reconnects within ten minutes starts near its previous bandwidth estimate instead of ramping up again.
These measurements come from a loopback test on October 21, on a PC with an RTX 3080 running Windows 11, streaming 2560×1440 at 60 frames a second:
| Test | Current path | GPU path |
|---|---|---|
| Processor use while scrolling | 74–121% of one core | 19–33% of one core |
| Frame rate with the GPU near 100% busy | 56.7 fps | 58.4 fps |
| Added delay with the GPU busy, 95th percentile | 18 ms | 1 ms |
| Encoding time per frame with the GPU busy | 97–120 ms | 6 ms |
| Wait before each frame is encoded | 6–8 ms | 0 ms |
| Average frame size on a 4 Mbps link | 7.4 KB | 4.9 KB |
The GPU path has passed its tests and has not yet been released.
In testing: the pointer and the Mac
- Pointer drawn by the viewer. The computer's pointer is being moved out of the video. The computer sends the pointer's position and shape separately, and the viewer draws it over the picture, so moving the mouse over a still window no longer produces new video. In a test, two seconds of pointer movement over an unchanging window produced 116 encoded pictures with the pointer in the video and none with it drawn by the viewer. Older phones and computers keep the pointer in the video.
- GPU-only capture on the Mac. On a Mac, captured pictures will go straight from ScreenCaptureKit to the VideoToolbox encoder without a copy through main memory. This work is written and has not yet been run on a Mac.
Next
The changes in testing go into a combined build next, followed by streaming tests on phones and Macs over home Wi-Fi and cellular connections before release. Two limits remain. Recovering from a lost packet still requires a full keyframe, because browser video decoders depend on one. A very busy picture can also skip frames for the first second or two while the bandwidth estimate rises.
