Multi-Codec Graphics Pipeline for RDP on Linux

Introduction

This document examines the RDP Graphics Pipeline Extension (MS-RDPEGFX) as it was designed: a multi-codec graphics system. The EGFX wire format carries per-PDU codec identifiers. The framing model supports mixed-codec frames. The capability exchange negotiates codec availability per session. Microsoft's Azure Virtual Desktop uses this multi-codec model in production, encoding text with ClearCodec, video with H.264, and photographic content with progressive refinement — all within single frame boundaries.

Most open-source RDP implementations treat EGFX as an H.264 delivery mechanism. This works, but it leaves most of the protocol's design unused. ClearCodec — the lossless text codec that the spec declares mandatory for all EGFX implementations — is not implemented in any open-source RDP server. Since approximately 80% of typical remote session content is text, this means every open-source implementation is either applying lossy video compression to its dominant content type or falling back to raw bitmap.

On Linux, the transition from X11 to Wayland creates both challenges and opportunities. The Wayland compositor's damage tracking, surface metadata, and content type signals provide information that maps directly to codec selection decisions — enabling content-aware encoding without pixel analysis.

1. The Three Eras of RDP Graphics

2. The Multi-Codec Model

EGFX supports eight codec identifiers per PDU:

Within a single frame, different tiles use different codecs. The client decodes each according to its codec ID. The server selects per-tile, per-frame.

3. ClearCodec: The Mandatory Codec Nobody Implements

The spec is unambiguous: "Implementers have to support the ClearCodec codec."

ClearCodec uses three layers: Residual (RLE for flat colors), Bands (V-Bar dictionary cache — text characters decomposed into vertical pixel columns that repeat across frames), and Subcodec (NSCodec/RLEX for complex regions). After cache warming, V-Bar hit rates exceed 80% for text content.

4. Linux-Specific Advantages

The multi-codec model requires content classification. On Windows, DWM provides this intrinsically. On Linux, the Wayland compositor provides equivalent signals without pixel analysis:

A Linux RDP server has content classification that is more reliable, lower-cost, and lower-latency than pixel analysis.

5. The Capture Pipeline: Where Timing Matters

Multi-codec encoding requires timing awareness. Empirical measurement across compositors:

Same protocol, both wlroots-based — 6× difference in frame delivery. The client cannot determine why. This gap limits adaptive encoding: without knowing capture cost, the encoder must assume worst case.

For detailed capture timing analysis, see our Capture Protocol Timing Feedback report.

6. Health Monitoring and Observability

A multi-codec pipeline produces richer telemetry. Per-codec encoder statistics enable closed-loop adaptation:

EIS stream EOF ──────────┐
Portal SessionClosed ────┤     ┌─────────────────┐
PipeWire state change ───┤────►│ Health Monitor   │──► Unified health state
Compositor restart ──────┤     └─────────────────┘
Frame ack timeout ───────┘

Session health tracking (video, input, clipboard, session lifecycle), adaptive frame rate control (5–60 FPS based on activity), latency governance (interactive/balanced/quality modes), and priority-based event multiplexing ensure that graphics congestion never affects input responsiveness.

7. Current State of Open-Source RDP

8. Summary

The EGFX protocol defines a multi-codec graphics architecture that Microsoft uses in production. The combination of EGFX's per-PDU codec selection, Wayland's content metadata, health monitoring, and capture pipeline timing creates a natural pipeline for adaptive remote desktop on Linux that is architecturally superior to what a monolithic approach can achieve.

ClearCodec is ~2,500–3,000 lines. Progressive is ~3,000–3,500 lines. These are well-specified protocol mechanisms, not open research problems. The scope is tractable and the benefit is immediate.