Media over QUIC (MoQ) Implementation: Technical Achievement & Browser Reality

📺 Live Demo

URL: https://moq.wink.co/moq-player.html

Requirements: Chrome or Edge with WebTransport enabled

What You'll See: Real-time video with 200-300ms latency from camera to screen

💻 Source Code

GitHub: https://github.com/winkmichael/mediamtx-moq

Language: Go (backend) + JavaScript (player)

License: Open Source

Integration: Full MediaMTX compatibility

The H.264 Puzzle

MediaMTX provides Access Units containing multiple NAL units. WebCodecs expects complete frames in Annex B format. Here's what we learned:

func processH264AccessUnit(au *unit.H264, h264Format *format.H264) []byte {
    var frameData []byte
    hasIDR := false
    startCode := []byte{0x00, 0x00, 0x00, 0x01}
    
    // Check if this AU contains an IDR frame
    for _, nalu := range au.AU {
        nalType := nalu[0] & 0x1F
        if nalType == 5 { // IDR frame
            hasIDR = true
            break
        }
    }
    
    // Critical: Add SPS/PPS before every IDR frame
    if hasIDR && h264Format.SPS != nil && h264Format.PPS != nil {
        frameData = append(frameData, startCode...)
        frameData = append(frameData, h264Format.SPS...)
        frameData = append(frameData, startCode...)
        frameData = append(frameData, h264Format.PPS...)
    }
    
    // Process NAL units - only include VCL NALUs (types 1-5)
    for _, nalu := range au.AU {
        nalType := nalu[0] & 0x1F
        
        // Skip non-VCL NAL units (SEI, AUD, etc.)
        if nalType < 1 || nalType > 5 {
            continue
        }
        
        // Add VCL NAL unit with start code
        frameData = append(frameData, startCode...)
        frameData = append(frameData, nalu...)
    }
    
    return frameData
}
                

Key discoveries:

• SEI NAL units (type 6) crash WebCodecs decoder
• SPS/PPS must be included with every keyframe
• Start codes are mandatory (no length prefixing)
• Access Unit ≠ Frame (can contain multiple slices)

The Audio Challenge

Audio would play for 1 second then stop. The problem: Web Audio API scheduling conflicts when buffering too many chunks.

// WRONG - This kills audio after ~1 second
audioChunks.forEach(chunk => {
    playAudioChunk(chunk, audioContext.currentTime + offset);
    offset += duration;
});

// CORRECT - Sequential scheduling
class AudioPlayer {
    constructor() {
        this.audioContext = new AudioContext();
        this.audioWorklet = null;
        this.nextPlayTime = 0;
        this.audioQueue = [];
        this.setupWorklet();
    }
    
    async setupWorklet() {
        // Use Web Worker for audio decoding to prevent main thread blocking
        await this.audioContext.audioWorklet.addModule('audio-processor.js');
        this.audioWorklet = new AudioWorkletNode(this.audioContext, 'aac-processor');
        this.audioWorklet.connect(this.audioContext.destination);
    }
    
    scheduleAudio(audioData) {
        // Decode AAC to PCM
        const pcmData = this.decodeAAC(audioData);
        
        // Create buffer
        const audioBuffer = this.audioContext.createBuffer(
            2, // stereo
            pcmData.length / 2,
            44100 // sample rate
        );
        
        // Copy data to buffer
        audioBuffer.copyToChannel(pcmData.left, 0);
        audioBuffer.copyToChannel(pcmData.right, 1);
        
        // Schedule playback
        const source = this.audioContext.createBufferSource();
        source.buffer = audioBuffer;
        source.connect(this.audioContext.destination);
        
        // Play at the right time
        if (this.nextPlayTime < this.audioContext.currentTime) {
            this.nextPlayTime = this.audioContext.currentTime;
        }
        source.start(this.nextPlayTime);
        this.nextPlayTime += audioBuffer.duration;
    }
}
                

Key insight: Don't schedule everything at once. Process audio sequentially to maintain sync.

💡 Recommendation

For B2B/Enterprise: Use MoQ if you can control browser choice

For B2C/Consumer: Stick with WebRTC or LL-HLS until Safari supports WebTransport

For Future-Proofing: Build MoQ support now, but keep fallbacks

CPU Usage

MediaMTX Process:
├── RTMP Ingestion: ~5%
├── H.264 Processing: ~8%
├── MoQ Encoding: ~2%
└── Network I/O: ~3%
Total: ~18%

Browser:
├── WebTransport: ~10%
├── Video Decode: ~15%
├── Audio Decode: ~5%
├── Rendering: ~10%
└── JavaScript: ~5%
Total: ~45%
                

Resource Requirements

• Server Memory: ~70MB for MediaMTX + MoQ
• Browser Memory: ~150MB (includes decoded frame buffer)
• Network Bandwidth: ~630kbps per viewer (500kbps video + 96kbps audio + overhead)

Enable Debug Logging

# mediamtx.yml
logLevel: debug
logDestinations: [stdout]

# Enable MoQ specific logs
servers:
  moq:
    enable: yes
    debug: true
                

Chrome WebTransport Setup

# Enable experimental features
chrome://flags/#enable-experimental-web-platform-features

# Check WebTransport status
chrome://webrtc-internals/
                

Network Monitoring

# Monitor QUIC traffic
sudo tcpdump -i any -nn port 4443 or port 4444

# Test with moq-rs tools
moq-relay --listen 0.0.0.0:4444
moq-pub --url https://localhost:4444/test video.mp4