Remote shows a window or a whole screen from a computer on a phone, as live video. It looked fine in the first demos. Then a stopwatch went on it: the encoder on the test PC was producing 51 frames a second, and the viewer was getting 2.4.
This is the story of where those frames went, and of the next three problems hiding behind that one.
How the video travels
The computer captures the screen, compresses it with the graphics card's H.264 encoder, and sends it to the viewer over WebRTC, the same technology a browser uses for video calls. The connection runs straight between the devices. There is no relay server in the middle, and none of the video passes through AI Integrator's servers.
That makes the sending side the part that has to be right. It is written in Rust on top of an open-source WebRTC library called str0m, and on Windows it uses the encoder built into the graphics driver.
The 15.6 millisecond wall
Nothing was being lost. There were no dropped packets and no full buffers. The frames were simply not leaving the computer.
The library sends video through a pacer, which releases packets at an even rate instead of in a burst. Each time it is called it releases one packet, then asks to be called again immediately. The session loop obeyed by sleeping for zero milliseconds. On Windows, a sleep of zero does not return right away. The program waits for the next tick of the system clock, and by default that tick is 15.6 milliseconds long.
- The pacer releases one packet
- It asks to be called again now
- The loop sleeps for 0 ms
- Windows waits for the next clock tick. 15.6 ms
- About 64 packets a second
- The pacer releases one packet to It asks to be called again now
- It asks to be called again now to The loop sleeps for 0 ms
- The loop sleeps for 0 ms to Windows waits for the next clock tick
- Windows waits for the next clock tick to The pacer releases one packet: repeat
- Windows waits for the next clock tick to About 64 packets a second
That is about 64 ticks a second, and one packet per tick. A video this size needs hundreds to thousands of packets every second. The measured rate was between 64 and 110. The picture was being sent, just far too slowly.
The fix was to stop sleeping when there is nothing to wait for. If the library says a packet is due now, the loop sends it now, up to 512 at a time, before it goes back to waiting.
The encoder that ignores the speed limit
With packets flowing, the next problem was the opposite one: too much video. A bitrate was set on NVIDIA's encoder. It accepted every setting tried, and then ignored them. On a 1440p picture where every pixel changed, it produced 25 to 50 megabits a second whatever the target, from 1.5 up to 30.
So the encoder is no longer trusted to keep to a budget. A small budget now sits in front of it, set to what the connection can carry, and it skips pictures when the budget runs out. If more than 30 percent of a second's pictures are skipped for two seconds in a row, the picture is made smaller, first to 75 percent and then to half, never below 640 pixels on its longest edge. After eight calm seconds it tries the larger size again.
Clocks, again
The estimate of how much the connection could carry was stuck near 4 megabits a second. To measure bandwidth, the library sends short trains of packets about a millisecond apart. On a clock that only ticks every 15.6 milliseconds, those packets bunch together, the probe fails, and the estimate stays low.
While a stream is live, the computer now asks Windows for a one-millisecond clock. It also tells Windows 11 not to ignore that request when the app is minimized, and keeps the display awake. On a loopback test the estimate now reaches 18 to 22 megabits a second in the first second. On a Mac the same job is done with the system's activity assertions.
Where it ended up
The measurements were taken on the development PC, which has an RTX 3080, with the viewer on the same machine so the network was not a factor.
Scrolling a 1440p picture at 60 frames a second arrived at 56 to 57. The worst case, a picture where every pixel changes, averaged 52 and held 59 once it settled. Added delay at the 95th percentile was 7 milliseconds for scrolling and 9 for the worst case, with a single worst frame of 41.
Then the phone fell 30 seconds behind
The same day, a bug report arrived from a phone: the picture played in order, but it drifted 10 to 30 seconds behind the computer. A slow network was rebuilt in the lab, a 4 megabit link and a relay that queued 30 seconds of video, and the computer kept up in every case. The sending side was not the cause.
That left the pieces around it, and four of them were worth fixing:
- NVIDIA's encoder sent a complete keyframe every 60 frames, because it only reads its keyframe interval when it is set before the video format.
- When a viewer asked for a keyframe, it asked again and again while the first one was still on its way. Requests are now spaced out to match how long a keyframe takes to cross the link.
- The packets now tell the phone to play frames as soon as they are decoded, instead of growing a buffer.
- When the computer is silent, it sends no audio at all, so the audio clock stopped. The phone's player holds video back to keep it in step with sound, by up to 10 seconds, and silence made it wait.
It is not yet certain that this cured that exact report, so logging was added as well. Both ends now record every stream: delay, frames, bitrate, and why a stream ended. On a computer or in a browser, open the menu and choose Show stats to see the same numbers live. If a stream lags, the report that gets sent now carries what is needed to see why.
