🚀 Complete MoQ Implementation Guide: Every Line of Code, Every Lesson Learned

🏆 Achievement Summary: What We Built

Latency Comparison: The Numbers Don't Lie

Key Innovation: Dual Transport Architecture

We discovered (through frustration) that browsers can't use native QUIC. Instead of giving up, we implemented both:

• WebTransport (Port 4443): For browser playback using HTTP/3 upgrade
• Native QUIC (Port 4444): For server-to-server relay

This "mistake" became our biggest feature - maximum compatibility for different use cases.

🎬 Live Demo & Resources

🏗️ System Architecture

Data Flow Through the System

1. Input Stream: MediaMTX receives RTMP/RTSP stream
2. Stream Processing: MediaMTX decodes into Access Units
3. MoQ Adaptation: Convert Access Units to MoQ Objects
4. Transport: Send via WebTransport or QUIC
5. Browser Decode: WebCodecs API processes frames
6. Playback: Canvas rendering + Web Audio output

🔥 Implementation Challenges: The Hard Parts

// This crashes - stream.Desc is nil!
stream.AddReader(reader, nil, nil, callback)

// Wait for stream.Desc to be populated
for attempts := 0; attempts < 50; attempts++ {
    if stream.Desc != nil {
        break
    }
    time.Sleep(100 * time.Millisecond)
}

if stream.Desc == nil {
    return fmt.Errorf("stream not ready after 5 seconds")
}

// Now find the H.264 format
var videoFormatH264 *format.H264
videoMedia := stream.Desc.FindFormat(&videoFormatH264)

// Register with proper format
stream.AddReader(reader, videoMedia, videoFormatH264, callback)

// This will NEVER work in browsers - security policy blocks it
const transport = new QuicTransport('quic://server:4444');

// WebTransport for browsers (HTTP/3 upgrade required)
wtServer := webtransport.Server{
    H3: http3.Server{
        Addr:      ":4443",
        TLSConfig: tlsConfig,
    },
}

// Native QUIC for server-to-server
quicListener, _ := quic.ListenAddr(":4444", tlsConfig, &quic.Config{
    MaxIncomingStreams:    256,
    MaxIncomingUniStreams: 256,
    EnableDatagrams:       true,
})

func (s *MoQSession) buildAnnexBFrame(au *formatprocessor.H264, isKeyframe bool) []byte {
    var frame []byte
    startCode := []byte{0x00, 0x00, 0x00, 0x01}
    
    // CRITICAL: WebCodecs needs SPS/PPS with every keyframe for Annex B format
    if isKeyframe && s.h264Format.SPS != nil {
        // Add SPS (Sequence Parameter Set)
        frame = append(frame, startCode...)
        frame = append(frame, s.h264Format.SPS...)
        
        // Add PPS (Picture Parameter Set)
        frame = append(frame, startCode...)
        frame = append(frame, s.h264Format.PPS...)
    }
    
    // Add only VCL NAL units - SEI and AUD break WebCodecs!
    for _, nalu := range au.Units {
        nalType := nalu[0] & 0x1F
        
        // Types 1-5 are VCL (Video Coding Layer) units
        // Type 1: Non-IDR slice
        // Type 5: IDR slice (keyframe)
        if nalType >= 1 && nalType <= 5 {
            frame = append(frame, startCode...)
            frame = append(frame, nalu...)
        }
        // Skip:
        // Type 6: SEI (Supplemental Enhancement Information)
        // Type 7: SPS (we add it separately)
        // Type 8: PPS (we add it separately)
        // Type 9: AUD (Access Unit Delimiter)
    }
    
    return frame
}

// This causes audio to stop after ~1 second
audioChunks.forEach(chunk => {
    playAudioChunk(chunk, audioContext.currentTime + offset);
    offset += duration;
});

// Use Web Worker to prevent main thread blocking
class AudioProcessor extends AudioWorkletProcessor {
    constructor() {
        super();
        this.port.onmessage = (e) => {
            // Decode audio in worker thread
            const audioData = this.decodeAAC(e.data);
            this.port.postMessage(audioData);
        };
    }
    
    process(inputs, outputs, parameters) {
        // Process audio without blocking main thread
        const output = outputs[0];
        // ... audio processing ...
        return true;
    }
}

// Main thread: Sequential scheduling
let audioPlaybackTime = audioContext.currentTime;

function scheduleNextAudio() {
    if (audioPending.length > 0) {
        const audioData = audioPending.shift();
        
        // Create buffer
        const audioBuffer = audioContext.createBuffer(
            2, // stereo
            audioData.samples.length / 2,
            audioData.sampleRate
        );
        
        // Fill buffer
        audioBuffer.getChannelData(0).set(audioData.samples.left);
        audioBuffer.getChannelData(1).set(audioData.samples.right);
        
        // Schedule playback
        const source = audioContext.createBufferSource();
        source.buffer = audioBuffer;
        source.connect(audioContext.destination);
        source.start(audioPlaybackTime);
        
        // Update time for next chunk
        audioPlaybackTime += audioBuffer.duration;
    }
}

📡 Protocol Implementation Details

MoQ Message Flow

Varint Encoding Implementation

MoQ uses variable-length integers for all numeric values to save bandwidth:

func encodeVarInt(value uint64) []byte {
    // 1 byte for values < 64
    if value < 0x40 {
        return []byte{byte(value)}
    }
    
    // 2 bytes for values < 16384
    if value < 0x4000 {
        return []byte{
            byte(0x40 | (value >> 8)),
            byte(value),
        }
    }
    
    // 4 bytes for values < 1073741824
    if value < 0x40000000 {
        return []byte{
            byte(0x80 | (value >> 24)),
            byte(value >> 16),
            byte(value >> 8),
            byte(value),
        }
    }
    
    // 8 bytes for large values
    return []byte{
        byte(0xC0 | (value >> 56)),
        byte(value >> 48),
        byte(value >> 40),
        byte(value >> 32),
        byte(value >> 24),
        byte(value >> 16),
        byte(value >> 8),
        byte(value),
    }
}

func decodeVarInt(reader io.Reader) (uint64, error) {
    firstByte := make([]byte, 1)
    if _, err := reader.Read(firstByte); err != nil {
        return 0, err
    }
    
    // Check the two high bits to determine length
    prefix := firstByte[0] >> 6
    
    switch prefix {
    case 0: // 1 byte
        return uint64(firstByte[0]), nil
        
    case 1: // 2 bytes
        secondByte := make([]byte, 1)
        reader.Read(secondByte)
        return uint64(firstByte[0]&0x3F)<<8 | uint64(secondByte[0]), nil
        
    case 2: // 4 bytes
        rest := make([]byte, 3)
        reader.Read(rest)
        return uint64(firstByte[0]&0x3F)<<24 | uint64(rest[0])<<16 | 
               uint64(rest[1])<<8 | uint64(rest[2]), nil
        
    case 3: // 8 bytes
        rest := make([]byte, 7)
        reader.Read(rest)
        // ... combine all 8 bytes ...
    }
}

MoQ Object Structure

Each MoQ object is self-describing with metadata:

type ObjectHeader struct {
    TrackAlias uint64  // Which track (video=1, audio=2)
    GroupID    uint64  // Increments with each keyframe
    ObjectID   uint64  // Frame number within group
    Priority   uint8   // Delivery priority (0=highest)
}

func (s *MoQSession) sendMoQObject(frameData []byte, isKeyframe bool) {
    // Update group on keyframe
    if isKeyframe {
        s.currentGroup++
        s.currentObject = 0
    }
    
    // Prepare object header
    header := ObjectHeader{
        TrackAlias: 1, // video track
        GroupID:    s.currentGroup,
        ObjectID:   s.currentObject,
        Priority:   0, // high priority for video
    }
    
    // Add timestamp to payload (moq-rs expects this)
    timestamp := uint64(time.Now().UnixMicro())
    payload := append(encodeVarInt(timestamp), frameData...)
    
    // Open new unidirectional stream for this object
    stream, _ := s.connection.OpenUniStream()
    
    // Write header fields
    stream.Write(encodeVarInt(header.TrackAlias))
    stream.Write(encodeVarInt(header.GroupID))
    stream.Write(encodeVarInt(header.ObjectID))
    stream.Write([]byte{header.Priority})
    
    // Write payload length and data
    stream.Write(encodeVarInt(uint64(len(payload))))
    stream.Write(payload)
    
    stream.Close()
    s.currentObject++
}

Subscribe Message Implementation

type Subscribe struct {
    SubscribeID   uint64
    TrackAlias    uint64
    Namespace     string
    TrackName     string
    StartGroup    *uint64 // nil = latest
    StartObject   *uint64
    EndGroup      *uint64 // nil = infinite
    EndObject     *uint64
}

func (m *Subscribe) Encode() []byte {
    var buf []byte
    
    // Message type
    buf = append(buf, encodeVarInt(MessageTypeSubscribe)...)
    
    // Subscribe ID for tracking
    buf = append(buf, encodeVarInt(m.SubscribeID)...)
    
    // Track alias for fast lookups
    buf = append(buf, encodeVarInt(m.TrackAlias)...)
    
    // Namespace and track name as length-prefixed strings
    buf = append(buf, encodeVarInt(uint64(len(m.Namespace)))...)
    buf = append(buf, []byte(m.Namespace)...)
    
    buf = append(buf, encodeVarInt(uint64(len(m.TrackName)))...)
    buf = append(buf, []byte(m.TrackName)...)
    
    // Optional start/end positions
    if m.StartGroup != nil {
        buf = append(buf, 0x01) // has start
        buf = append(buf, encodeVarInt(*m.StartGroup)...)
        buf = append(buf, encodeVarInt(*m.StartObject)...)
    } else {
        buf = append(buf, 0x00) // no start (latest)
    }
    
    return buf
}

🔧 MediaMTX Integration: The Secret Sauce

MediaMTX's brilliant design made our integration surprisingly clean. The path manager abstraction meant we could tap into any stream source without modifying existing code.

How MediaMTX Path System Works

// MediaMTX uses a Path Manager that routes streams between producers and consumers
// This is genius - any input (RTMP/RTSP/HLS) becomes available to any output

type MoQServer struct {
    pathManager *core.PathManager  // MediaMTX's central routing system
    parent      *core.Core         // Access to MediaMTX internals
    address     string             // WebTransport address
    addressQuic string             // Native QUIC address
    tlsConfig   *tls.Config
    sessions    map[string]*MoQSession
    mu          sync.RWMutex
}

func (s *MoQServer) Initialize() error {
    // Register with MediaMTX core
    s.parent.AddServer(s)
    
    // Start listening for connections
    go s.runWebTransport()
    go s.runNativeQUIC()
    
    return nil
}

The Critical Path Manager Integration

func (s *MoQSession) setupStream(pathName string) error {
    // Request stream from PathManager
    res := s.pathManager.AddReader(defs.PathAddReaderReq{
        Author: s,
        AccessRequest: defs.PathAccessRequest{
            PathName: pathName,
            IP:       s.remoteAddr,
        },
    })
    
    if res.Error != nil {
        return res.Error
    }
    
    // THIS IS THE CRITICAL BUG WE FOUND!
    // MediaMTX returns a stream immediately, but stream.Desc is nil
    // until the publisher sends actual data. This crashed everything.
    
    // Our hacky but working solution:
    for attempts := 0; attempts < 50; attempts++ {
        if res.Stream.Desc != nil {
            break
        }
        time.Sleep(100 * time.Millisecond)
        s.log.Debug("Waiting for stream description... attempt %d", attempts)
    }
    
    if res.Stream.Desc == nil {
        return fmt.Errorf("stream not ready after 5 seconds")
    }
    
    // Now we can safely access the format
    var h264Format *format.H264
    videoMedia := res.Stream.Desc.FindFormat(&h264Format)
    if videoMedia == nil {
        return fmt.Errorf("no H.264 video found in stream")
    }
    
    // Find AAC audio
    var aacFormat *format.AAC
    audioMedia := res.Stream.Desc.FindFormat(&aacFormat)
    
    // Store everything
    s.stream = res.Stream
    s.h264Format = h264Format
    s.aacFormat = aacFormat
    
    // Register callbacks for data
    res.Stream.AddReader(
        s,
        videoMedia,
        h264Format,
        s.onVideoData,
    )
    
    if audioMedia != nil {
        res.Stream.AddReader(
            s,
            audioMedia,
            aacFormat,
            s.onAudioData,
        )
    }
    
    return nil
}

Stream Reader Callback - Where the Magic Happens

func (s *MoQSession) onVideoData(unit formatprocessor.Unit) {
    // MediaMTX gives us an "Access Unit" - but what is it really?
    au := unit.(*formatprocessor.H264)
    
    // An Access Unit contains multiple NAL units for a single timestamp
    // This confused us for days - it's NOT a single frame!
    
    // We need to:
    // 1. Find if this has a keyframe (IDR NAL)
    // 2. Combine NAL units into a frame
    // 3. Add SPS/PPS for keyframes
    // 4. Send as MoQ object
    
    hasIDR := false
    for _, nalu := range au.Units {
        nalType := nalu[0] & 0x1F
        if nalType == 5 { // IDR slice (keyframe)
            hasIDR = true
            break
        }
    }
    
    // Build the frame in Annex B format
    frameData := s.buildAnnexBFrame(au, hasIDR)
    
    // Calculate timestamp (MediaMTX gives us PTS/DTS)
    timestamp := au.PTS
    if timestamp == 0 {
        timestamp = time.Duration(time.Now().UnixNano())
    }
    
    // Send as MoQ object
    s.sendVideoObject(frameData, hasIDR, timestamp)
}

func (s *MoQSession) onAudioData(unit formatprocessor.Unit) {
    au := unit.(*formatprocessor.AAC)
    
    // AAC in ADTS format for browser compatibility
    adtsFrame := s.buildADTSFrame(au.Units[0], s.aacFormat)
    
    // Send as MoQ audio object
    s.sendAudioObject(adtsFrame, au.PTS)
}

func (s *MoQSession) buildADTSFrame(aacData []byte, format *format.AAC) []byte {
    // ADTS header (7 bytes) for AAC
    frameLength := len(aacData) + 7
    
    header := []byte{
        0xFF, 0xF1, // Sync word + MPEG-4, no CRC
        0x50,       // AAC LC, 44.1kHz
        0x80 | (byte(frameLength>>11) & 0x03), // Channel config + frame length high
        byte(frameLength >> 3),                 // Frame length middle
        byte((frameLength&0x07)<<5) | 0x1F,    // Frame length low + buffer fullness
        0xFC,                                    // Buffer fullness low
    }
    
    return append(header, aacData...)
}

🚀 Dual Transport Implementation

Our dual-transport approach came from frustration but works brilliantly. When we couldn't get browsers to connect to native QUIC, we implemented both protocols.

WebTransport Server (For Browsers)

func (s *MoQServer) runWebTransport() error {
    // WebTransport requires HTTP/3 server with special upgrade handling
    
    mux := http.NewServeMux()
    
    // The magic endpoint that browsers connect to
    mux.HandleFunc("/moq", func(w http.ResponseWriter, r *http.Request) {
        // Check for WebTransport upgrade
        if r.Header.Get("Upgrade") != "webtransport" {
            http.Error(w, "not a webtransport request", http.StatusBadRequest)
            return
        }
        
        // Perform the upgrade dance
        w.Header().Set("Sec-Webtransport-Http3-Draft", "draft02")
        w.WriteHeader(http.StatusOK)
        w.(http.Flusher).Flush()
        
        // Get the underlying connection
        // This is where it gets tricky - we need quic-go's special sauce
        conn := r.Body.(*webtransport.Conn)
        
        // Create session wrapper
        session := &WebTransportSession{
            conn:      conn,
            streams:   make(map[uint64]*WebTransportStream),
            incoming:  make(chan *WebTransportStream, 100),
        }
        
        // Handle as MoQ connection
        s.handleConnection(session)
    })
    
    // Start HTTP/3 server with WebTransport support
    server := &http3.Server{
        Addr:      s.address,
        TLSConfig: s.tlsConfig,
        Handler:   mux,
        QuicConfig: &quic.Config{
            MaxIncomingStreams:    256,
            MaxIncomingUniStreams: 256,
            EnableDatagrams:       true,
            // IMPORTANT: These timeouts matter for browsers!
            MaxIdleTimeout:        30 * time.Second,
            KeepAlivePeriod:      10 * time.Second,
        },
    }
    
    s.log.Info("WebTransport server listening on %s", s.address)
    return server.ListenAndServe()
}

Native QUIC Server (For Server-to-Server)

func (s *MoQServer) runNativeQUIC() error {
    // Native QUIC is much simpler - no HTTP/3 overhead
    
    listener, err := quic.ListenAddr(s.addressQuic, s.tlsConfig, &quic.Config{
        MaxIncomingStreams:    256,
        MaxIncomingUniStreams: 256,
        EnableDatagrams:       true,
    })
    if err != nil {
        return err
    }
    
    s.log.Info("Native QUIC server listening on %s", s.addressQuic)
    
    for {
        conn, err := listener.Accept(context.Background())
        if err != nil {
            s.log.Error("Failed to accept QUIC connection: %v", err)
            continue
        }
        
        // Wrap in our connection interface
        session := &QUICSession{
            conn:     conn,
            streams:  make(map[uint64]quic.Stream),
            incoming: make(chan quic.Stream, 100),
        }
        
        go s.handleConnection(session)
    }
}

Connection Abstraction Layer

This interface saved us - same code handles both transports!

// Common interface for both WebTransport and native QUIC
type Connection interface {
    OpenUniStream() (SendStream, error)
    AcceptUniStream(context.Context) (ReceiveStream, error)
    OpenBidiStream() (BidiStream, error)
    AcceptBidiStream(context.Context) (BidiStream, error)
    CloseWithError(uint64, string) error
    RemoteAddr() net.Addr
}

// WebTransport implementation
type WebTransportConnection struct {
    session *webtransport.Session
}

func (c *WebTransportConnection) OpenUniStream() (SendStream, error) {
    stream, err := c.session.OpenUniStreamSync(context.Background())
    if err != nil {
        return nil, err
    }
    return &wtSendStream{stream}, nil
}

func (c *WebTransportConnection) AcceptUniStream(ctx context.Context) (ReceiveStream, error) {
    stream, err := c.session.AcceptUniStream(ctx)
    if err != nil {
        return nil, err
    }
    return &wtReceiveStream{stream}, nil
}

// Native QUIC implementation
type QuicConnection struct {
    conn quic.Connection
}

func (c *QuicConnection) OpenUniStream() (SendStream, error) {
    stream, err := c.conn.OpenUniStreamSync(context.Background())
    if err != nil {
        return nil, err
    }
    return &quicSendStream{stream}, nil
}

func (c *QuicConnection) AcceptUniStream(ctx context.Context) (ReceiveStream, error) {
    stream, err := c.conn.AcceptUniStream(ctx)
    if err != nil {
        return nil, err
    }
    return &quicReceiveStream{stream}, nil
}

🌐 Browser Implementation: JavaScript Video Hell

WebTransport Connection

class MoQPlayer {
    constructor(videoElement, statusElement) {
        this.canvas = videoElement;
        this.ctx = this.canvas.getContext('2d');
        this.statusElement = statusElement;
        
        // WebTransport connection
        this.transport = null;
        this.controlStream = null;
        
        // Video decoder using WebCodecs
        this.videoDecoder = null;
        this.frameBuffer = [];
        this.maxBufferSize = 30; // 1 second at 30fps
        
        // Audio context for playback
        this.audioContext = null;
        this.audioWorklet = null;
        this.audioStartTime = 0;
        
        // Statistics
        this.stats = {
            framesDecoded: 0,
            framesDropped: 0,
            bytesReceived: 0,
            latency: 0
        };
    }
    
    async connect(url) {
        try {
            // Create WebTransport connection
            this.transport = new WebTransport(url);
            
            // Wait for connection to be established
            await this.transport.ready;
            console.log('WebTransport connected');
            
            // Setup control stream for MoQ protocol
            await this.setupControlStream();
            
            // Start receiving data streams
            this.receiveStreams();
            
            // Setup video decoder
            await this.setupVideoDecoder();
            
            // Setup audio
            await this.setupAudio();
            
            // Start render loop
            this.startRenderLoop();
            
        } catch (error) {
            console.error('Connection failed:', error);
            throw error;
        }
    }
    
    async setupControlStream() {
        // Open bidirectional stream for control messages
        this.controlStream = await this.transport.createBidirectionalStream();
        const writer = this.controlStream.writable.getWriter();
        const reader = this.controlStream.readable.getReader();
        
        // Send SETUP message
        const setup = this.encodeSetup();
        await writer.write(setup);
        
        // Read SETUP_OK response
        const { value } = await reader.read();
        const response = this.decodeMessage(value);
        
        if (response.type !== 'SETUP_OK') {
            throw new Error('Setup failed');
        }
        
        // Subscribe to video track
        const subscribeVideo = this.encodeSubscribe('video', 1);
        await writer.write(subscribeVideo);
        
        // Subscribe to audio track
        const subscribeAudio = this.encodeSubscribe('audio', 2);
        await writer.write(subscribeAudio);
        
        writer.releaseLock();
        reader.releaseLock();
    }
}

Video Decoding with WebCodecs

async setupVideoDecoder() {
    // Check if WebCodecs is available
    if (!('VideoDecoder' in window)) {
        throw new Error('WebCodecs API not supported');
    }
    
    this.videoDecoder = new VideoDecoder({
        output: (frame) => {
            // Don't render immediately - buffer it
            if (this.frameBuffer.length >= this.maxBufferSize) {
                // Drop oldest frame if buffer is full
                const droppedFrame = this.frameBuffer.shift();
                droppedFrame.close();
                this.stats.framesDropped++;
            }
            
            this.frameBuffer.push(frame);
            this.stats.framesDecoded++;
        },
        error: (e) => {
            console.error('Decoder error:', e);
            // Try to recover
            this.resetDecoder();
        }
    });
    
    // Configure for H.264 Annex B format
    // This took forever to get right!
    this.videoDecoder.configure({
        codec: 'avc1.42001e', // H.264 Baseline Profile Level 3.0
        codedWidth: 640,
        codedHeight: 360,
        hardwareAcceleration: 'prefer-hardware',
        optimizeForLatency: true
    });
}

async receiveStreams() {
    // Accept incoming unidirectional streams
    const reader = this.transport.incomingUnidirectionalStreams.getReader();
    
    while (true) {
        const { value: stream, done } = await reader.read();
        if (done) break;
        
        // Process each stream (contains one MoQ object)
        this.processStream(stream);
    }
}

async processStream(stream) {
    const reader = stream.getReader();
    let buffer = new Uint8Array(0);
    
    try {
        while (true) {
            const { value, done } = await reader.read();
            if (done) break;
            
            // Concatenate chunks
            const newBuffer = new Uint8Array(buffer.length + value.length);
            newBuffer.set(buffer);
            newBuffer.set(value, buffer.length);
            buffer = newBuffer;
        }
        
        // Parse MoQ object
        const object = this.parseMoQObject(buffer);
        
        if (object.trackAlias === 1) {
            // Video track
            await this.processVideoObject(object);
        } else if (object.trackAlias === 2) {
            // Audio track
            await this.processAudioObject(object);
        }
        
    } catch (error) {
        console.error('Stream processing error:', error);
    }
}

async processVideoObject(object) {
    // Extract timestamp and frame data
    const view = new DataView(object.payload.buffer);
    const timestamp = this.decodeVarInt(view, 0);
    const frameData = object.payload.slice(8); // Skip timestamp
    
    // Create EncodedVideoChunk for WebCodecs
    const chunk = new EncodedVideoChunk({
        type: object.groupID > this.lastGroupID ? 'key' : 'delta',
        timestamp: timestamp,
        data: frameData
    });
    
    // Decode the chunk
    this.videoDecoder.decode(chunk);
    
    this.lastGroupID = object.groupID;
    this.stats.bytesReceived += frameData.length;
}

Audio Playback with Web Audio API

async setupAudio() {
    // Create audio context
    this.audioContext = new (window.AudioContext || window.webkitAudioContext)({
        latencyHint: 'interactive',
        sampleRate: 48000
    });
    
    // Load and register audio worklet for processing
    await this.audioContext.audioWorklet.addModule('audio-processor.js');
    
    this.audioWorklet = new AudioWorkletNode(
        this.audioContext,
        'aac-decoder',
        {
            numberOfInputs: 0,
            numberOfOutputs: 1,
            outputChannelCount: [2] // Stereo
        }
    );
    
    // Connect to speakers
    this.audioWorklet.connect(this.audioContext.destination);
    
    // Handle decoded audio from worklet
    this.audioWorklet.port.onmessage = (event) => {
        const audioData = event.data;
        this.scheduleAudioPlayback(audioData);
    };
}

scheduleAudioPlayback(audioData) {
    // Create audio buffer
    const audioBuffer = this.audioContext.createBuffer(
        2, // channels
        audioData.samples.length / 2,
        audioData.sampleRate
    );
    
    // Copy samples to buffer
    const leftChannel = audioBuffer.getChannelData(0);
    const rightChannel = audioBuffer.getChannelData(1);
    
    for (let i = 0; i < audioData.samples.length / 2; i++) {
        leftChannel[i] = audioData.samples[i * 2];
        rightChannel[i] = audioData.samples[i * 2 + 1];
    }
    
    // Create and schedule buffer source
    const source = this.audioContext.createBufferSource();
    source.buffer = audioBuffer;
    source.connect(this.audioContext.destination);
    
    // Calculate when to play this chunk
    if (this.audioStartTime === 0) {
        this.audioStartTime = this.audioContext.currentTime + 0.1; // Small buffer
    }
    
    source.start(this.audioStartTime);
    this.audioStartTime += audioBuffer.duration;
}

Frame Timing and Rendering

startRenderLoop() {
    let lastFrameTime = performance.now();
    const targetFrameInterval = 1000 / 30; // 30 FPS
    
    const render = () => {
        const now = performance.now();
        const elapsed = now - lastFrameTime;
        
        // Only render if enough time has passed
        if (elapsed >= targetFrameInterval) {
            if (this.frameBuffer.length > 0) {
                // Get next frame
                const frame = this.frameBuffer.shift();
                
                // Draw to canvas
                this.ctx.drawImage(frame, 0, 0, this.canvas.width, this.canvas.height);
                
                // CRITICAL: Close frame to free memory
                frame.close();
                
                lastFrameTime = now;
                
                // Update stats
                this.updateStats();
            } else {
                // No frames available - we're behind!
                console.warn('Frame buffer empty - stuttering likely');
            }
        }
        
        requestAnimationFrame(render);
    };
    
    requestAnimationFrame(render);
}

updateStats() {
    const latency = this.frameBuffer.length * (1000 / 30); // Estimate based on buffer
    
    this.statusElement.textContent = `
        Frames: ${this.stats.framesDecoded} | 
        Dropped: ${this.stats.framesDropped} | 
        Buffer: ${this.frameBuffer.length} | 
        Latency: ~${latency.toFixed(0)}ms
    `;
}

// This runs in a separate thread to prevent blocking
class AACDecoder extends AudioWorkletProcessor {
    constructor() {
        super();
        
        this.decoder = new AACDecoderWASM(); // WebAssembly AAC decoder
        this.sampleRate = 48000;
        
        this.port.onmessage = (event) => {
            // Receive AAC data from main thread
            const aacData = event.data;
            
            // Decode AAC to PCM
            const pcmData = this.decoder.decode(aacData);
            
            // Send back to main thread for scheduling
            this.port.postMessage({
                samples: pcmData,
                sampleRate: this.sampleRate,
                timestamp: event.timestamp
            });
        };
    }
    
    process(inputs, outputs, parameters) {
        // This is called 128 samples at a time
        // We don't use it for decoding, just for keeping worklet alive
        return true;
    }
}

registerProcessor('aac-decoder', AACDecoder);

📊 Performance Analysis

CPU Usage Breakdown

Memory Usage

• Server: ~70MB for MediaMTX + MoQ server
• Browser: ~150MB (includes decoded frame buffer)
• Frame Buffer: ~30MB (30 frames @ 640x360)

Network Bandwidth

Latency Sources (200-300ms Total)

1. Encoding (50-100ms): FFmpeg x264 encoding
2. Network (20-50ms): QUIC transport
3. Buffering (50-100ms): Frame buffer in browser
4. Decoding (30-50ms): WebCodecs processing
5. Rendering (33ms): One frame at 30fps

⚠️ Known Limitations

Not Production Ready For:

• High packet loss networks (>5%)
• Mobile devices (performance issues)
• Large scale broadcasting (no CDN integration)
• DRM protected content
• Live captions/subtitles
• Consumer applications (no Safari/iOS)

🌐 Browser Support: The Harsh Reality

🔮 What Needs to Happen for MoQ to Succeed

📁 Code Structure

🐛 Debugging Tips

Enable Debug Logging

logLevel: debug
logDestinations: [stdout]

# MoQ specific debugging
moq:
  logLevel: debug
  dumpPackets: true  # Dumps all MoQ messages

Chrome WebTransport Debugging

# Enable in Chrome
chrome://flags/#enable-experimental-web-platform-features

# View WebTransport logs
chrome://webrtc-internals/  # Yes, it works for WebTransport too!

Monitor Network Traffic

# Watch WebTransport traffic
sudo tcpdump -i any -nn port 4443

# Watch native QUIC
sudo tcpdump -i any -nn port 4444

# Decode QUIC packets (requires Wireshark)
sudo tcpdump -i any -w moq.pcap port 4443
wireshark moq.pcap  # Set decode as QUIC

Test with moq-rs Tools

# Install moq-rs
cargo install moq-relay moq-pub moq-sub

# Test relay
moq-relay --listen 0.0.0.0:4444

# Publish test video
moq-pub --url https://localhost:4444/test video.mp4

# Subscribe to stream
moq-sub --url https://localhost:4444/test

Common Issues

🙏 Credits & Acknowledgments