A fixed camera bolted to a pole over a road produces 622 Mbit/s of raw video and sends it to you at 600 kbit/s. That is roughly a thousand to one, and it is not a compliment to the encoder. It is a statement about the scene. This is what the encoder is actually doing to get there, trick by trick, and the specific thing in the field that defeats each one.
We have written twice about which codec to pick and how to tune the one you have. Neither piece explains what the thing is doing. This one does, mechanism by mechanism, with the part we care about most: each trick has a specific way of failing in the field, and once you know which trick a scene is defeating you know what to go and fix.
The arithmetic nobody runs
Take an ordinary 1080p25 camera. The sensor hands the encoder 1920 by 1080 luma samples, and colour at half resolution in each direction, which is the 4:2:0 arrangement almost every camera produces. That works out to 1.5 bytes per pixel, so 3,110,400 bytes per frame, 25 times a second.
A thousand to one sounds impossible until you look at what is actually in the pictures. Twenty five times a second the camera sends the same tarmac, the same kerb, the same sign, the same building behind it. The sky changes slowly. The lane markings never change at all. Somewhere in the middle of all that, a few cars move a predictable distance in a predictable direction.
H.264 has four ways of exploiting that, and they stack.
One: most frames are not pictures
The encoder cuts each frame into macroblocks of 16 by 16 pixels. At 1080p that is 120 across and 68 down, since the 1080 is padded up to 1088 to divide evenly, so 8,160 blocks per frame. For every one of them it asks a single question: can I describe this with a pointer to something I have already sent?
There are three answers. It can say nothing at all, which means this block is identical to the one behind it and the decoder should copy it, the cheapest outcome available. It can send a motion vector, which says this block is the same as a patch of the previous frame but shifted by this much, and here is a small correction. Or it can give up on prediction and code the actual detail, which is the expensive answer.
A full picture, where every block takes the expensive answer, is an I-frame, and it is the only kind of frame a decoder can start from. The frames that lean on what came before are P-frames, and those that lean on frames both before and after are B-frames. On a roadway camera the great majority of blocks in the great majority of frames take one of the two cheap answers.
What breaks it
Everything that makes a block fail to match its predecessor. Three of these account for most of the trouble we see.
Auto-exposure hunting. A cloud crosses the sun, the camera re-evaluates exposure, and every pixel in the frame shifts in brightness by a little. Nothing is identical to anything any more, so the free answer disappears across the whole picture at once. This is the single most common cause of a bitrate spike on a camera that is otherwise well behaved, and it is nastier on a cellular or radio path than a gradual rise would be, because it arrives as a step. If the camera lets you lock or narrow exposure, that setting is a bandwidth setting.
Foliage, rain and snow. Leaves in wind and falling precipitation move randomly and at a fine grain. Motion estimation looks for a matching patch, finds nothing that matches well enough, and falls back to coding new detail. A tree line filling a third of the frame can cost more than the traffic you installed the camera to watch.
The camera moving. Mount flex in wind defeats prediction across the entire frame at once, for the same reason exposure hunting does. So does a PTZ mid-move, which is why our tuning numbers hold up far less well during a pan than on a fixed shot. When we measured how much a live encoder can still save, a rigidly mounted camera over a quiet scene was the best case and a handheld reference was the floor, at 21%. The spread between those two is almost entirely this mechanism.
The useful way to read a bitrate graph
A camera whose bitrate is flat and low has a scene that holds still. A camera that sits at its cap all day is sending new detail constantly, and the question is which of the three causes above is doing it. That is a diagnosis you can make from a graph before you ever look at the picture, and the fix is nearly always at the camera rather than in the encoder.
Two: colour is cheap because the eye is bad at it
Human vision resolves brightness detail far better than colour detail. Colour television exploited this in the 1950s so that a colour signal would fit in the channel allocated to a monochrome one, and every codec since has kept the trick. The picture is separated into one brightness channel and two colour channels, and the colour channels are stored at half resolution horizontally and vertically.
The result is the 1.5 bytes per pixel from the arithmetic above. Store colour at full resolution and it would be two thirds of your data. At quarter resolution it is one third. That is a 50% saving taken before any of the clever parts of the codec have run, and almost nobody notices it is happening.
Two consequences matter in our world. The first is that detail survives unevenly, and it survives in the channel that carries brightness. A licence plate is high contrast and essentially a brightness pattern, so it holds up at bitrates where other things have fallen apart. The colour of a vehicle, the state of a signal head, the wording on a coloured sign: those live in the channels you threw three quarters of away. Worth knowing before you promise anyone reliable vehicle colour off a 600 kbit/s stream.
The second is a windfall. At night, with infrared illuminators on, the camera gives you a monochrome scene. The colour channels go flat, and flat channels cost almost nothing to code. Some of the bitrate headroom you appear to gain after dark is this, and it is real. It is also the only good news in this article about night, because of what comes next.
Three: throwing away the detail nobody looks at
When a block has to be coded properly, the encoder does not send pixels. It converts the block into frequency coefficients using a small integer transform, 4 by 4 or 8 by 8 depending on profile, which rearranges the information so that the broad shape of the block sits in a few coefficients and the fine detail sits in many. Then it divides those coefficients by a quantiser and rounds. Rounding is where the data actually goes away, and because the fine detail lives in the small coefficients, fine detail is what rounds to zero first.
That quantiser is the single most important number in the encoder. Raise it and you get a smaller stream and a blurrier, blockier picture. Rate control, in the end, is a machine for deciding what the quantiser should be from one block to the next.
Sensor noise is the most expensive thing a camera can hand you
Noise is fine grained, random, and different in every frame. That makes it the worst possible content for this codec on two counts at once: prediction cannot match it, so blocks fall through to being coded properly, and once coded it lands squarely in the high frequency coefficients that eat bits. A noisy night feed can cost several times what the same camera costs at noon while carrying less usable information. Noise is not detail. Denoise it before the encoder sees it, not after.
We have measured this from the other direction. When we backported AV1's CDEF filter into our pipeline expecting it to help, a plain denoiser did better, which told us the problem was never the sophistication of the filter. One caution on reading any of those results, ours included: quality metrics score the encode against the feed the camera delivered, not against the scene in front of it. A denoiser is therefore penalised for removing noise that was faithfully recorded. Whether an operator prefers the denoised night picture is a question the numbers cannot settle, and we have been careful to say so.
Four: the lossless squeeze at the end
After all the discarding is done, what remains is a stream of symbols with a lot of statistical structure left in it. Small coefficients are far more common than large ones, certain vectors are far more common than others. The final stage codes those symbols by probability, spending fewer bits on the likely ones, and it adapts its model as it goes. Nothing is lost here. It is typically worth somewhere between 5 and 15% over the simpler alternative, for free.
The adaptivity has a cost that matters more to us than the saving does. Because the decoder is maintaining a model that depends on everything it has already read, and because P-frames are defined relative to earlier frames, a lost packet does not damage one block. It damages a region, and the damage persists in every frame that predicts from it, until the decoder reaches a frame it can start from cleanly. That is the real reason keyframe interval is a transport decision on a lossy path rather than a convenience for scrubbing, and it is a large part of why we keep arguing for H.264 on cellular links in the first place.
What a real encoder does with these
The four mechanisms are what the standard allows. What an encoder actually decides, frame by frame, is a separate question, and the clearest way to see it is libx264, the reference open source H.264 encoder. It is what our measurements were run with, it is what anyone can install, and its settings map almost one to one onto the four tricks above.
Where to put the I-frames. You can set a fixed interval, but the encoder also watches for scene cuts and will insert one when the picture changes enough that predicting from the old frame is worse than starting fresh. On a fixed camera a genuine scene cut is rare, which is why the interval you configure is usually the interval you get. On a PTZ it is not, and the encoder will spend I-frames during a move whether you asked for them or not.
Looking ahead before deciding quality. This is the setting that produced the largest result we have measured, and the mechanism is worth understanding because it explains why a fixed camera benefits so much more than a moving one. With lookahead enabled the encoder buffers a short window of future frames and works out which blocks are going to be referenced repeatedly by the frames that follow. Blocks that many later frames will predict from get more bits, once. Everything that predicts from them is then cheap and accurate.
On a roadway camera the background is referenced by every single subsequent frame until the next I-frame. Spending extra bits on the tarmac once, and then having several hundred frames copy it cleanly, is an enormous win. On handheld footage nothing is referenced for long, so there is nothing to invest in. That is the whole of the gap between our 43 to 45% best case and our 21% floor, and it is mechanism one being exploited deliberately rather than incidentally.
It also explains the cost. Looking ahead means holding frames before encoding them, which is latency you cannot get back. Turning lookahead off is exactly what the low latency tuning does, and it is why that tuning costs bitrate rather than saving it.
Where to spend the quantiser. Adaptive quantisation moves bits toward flat areas and away from busy ones, which sounds backwards until you remember what quantisation does. Fine detail hides quantisation error. A smooth surface does not, so an under-coded flat area shows visible banding and blocking while an under-coded textured area just looks slightly softer. A large expanse of road surface or sky is exactly the case this protects.
Recovering without a full I-frame. There is an alternative to the periodic IDR worth knowing about on lossy paths. Rather than sending one expensive full picture every few seconds, the encoder can cycle a band of fully coded blocks across the frame continuously, so that over the course of a cycle every part of the picture has been refreshed. Recovery is spread out instead of arriving in one spike, which suits a constrained link, and it removes the bitrate sawtooth that a keyframe cadence produces. The trade is that there is no single frame a decoder can start from instantly, so joining takes up to a full cycle.
What this is and is not
libx264 is the reference implementation and our measurement harness. It is what makes this checkable: anyone can install it and reproduce the behaviour described here against their own cameras. WINK Forge is our own software, and nothing in this section describes our implementation.
The camera is already doing some of this
If you run Axis, Hikvision or Dahua cameras you have almost certainly seen a setting called Zipstream, or H.264+, or simply smart codec. These are not new codecs and they are not proprietary formats. The output is ordinary H.264 that any decoder can read. What they are is the camera applying the same four mechanisms more aggressively than a general purpose encoder would dare, using an advantage no transcoder downstream has: the camera can see the scene, and it knows which parts of it are interesting.
Axis describes Zipstream as three algorithms running together. Dynamic ROI finds areas of interest, such as people and moving objects, and compresses everything else more heavily, with the region expanding, shrinking, splitting and merging as content changes. Dynamic GOP sends I-frames less often when the scene is still. Dynamic FPS captures at full frame rate but declines to encode frames it judges unnecessary. Axis puts the individual contributions at roughly 10 to 50% for ROI and 0 to 50% each for the other two, all heavily scene dependent. Cameras ship with the strength at a conservative default and dynamic GOP and FPS switched off, which Axis presents as the setting compatible with everything.
Hikvision's H.264+ and H.265+ take a different route to the same place. The headline technique is to build a model of the static background and use it as a long term reference, so a frame predicts from a clean background picture rather than only from the frame immediately before it. Alongside that sit background noise suppression, which is the denoising argument from earlier in this article applied at the sensor end where it belongs, and a bitrate control that averages its target over a long window, typically around 24 hours.
Every one of those is a mechanism from this article, pushed into the camera. Dynamic GOP is mechanism one. Dynamic ROI and noise suppression are mechanism three. Background modelling as a long term reference is mechanism one again, done better than a general encoder can manage because the camera knows the background does not walk away.
| Camera feature | What it actually does | Mechanism | What it costs you downstream |
|---|---|---|---|
| Dynamic ROI | Quantises harder outside areas judged interesting | Transform and quantisation | Its idea of interesting comes from analytics. Check what it decides at night |
| Dynamic GOP | Sends I-frames less often while the scene is still | Prediction | Lengthens recovery from packet loss precisely when the scene is quiet |
| Dynamic FPS | Captures full rate, encodes fewer frames | Prediction | Anything downstream that assumes a constant frame rate |
| Background as long term reference | Static areas predict from a background model | Prediction | Recovery and late joining are less predictable than a plain IDR cadence |
| Long window bitrate control | Averages the target over roughly 24 hours | Rate control | Instantaneous rate can sit far above the nominal figure |
These features are tuned for the recorder, not for your link
Every vendor saving figure for smart codec is a storage figure, measured over a long recording of a scene the vendor chose. That is a fair measurement of the thing they are selling. It is not the number that matters if you are carrying the stream somewhere. Two settings deserve attention before you let a camera decide for itself. Cap the maximum GOP length rather than leaving it unbounded, because on a cellular or radio path an error persists until the next I-frame and dynamic GOP removes I-frames exactly when the scene is quiet enough that you had stopped worrying. And size the link on observed peak rather than the long term average, because a controller averaging over 24 hours is explicitly permitted to spend well above nominal when something finally happens, which tends to be the moment the footage matters.
None of which is an argument against turning them on. For recording they are close to free, they solve a real problem, and the industry adopted them for good reason. The point is only that the camera is optimising for the disk, and if the stream is also going somewhere, the constraints are different and nobody has told the camera about them. Where we bridge a site to a central viewer we would rather carry the camera's output untouched than re-encode it, because the camera spent real effort deciding where its bits should go and a transcode throws that work away.
Where this works, and where it does not
Putting the four mechanisms together, the compressibility of a camera feed is mostly a property of the scene and the mount, not of the encoder.
| Scene | What holds still | Which mechanism is defeated |
|---|---|---|
| Fixed camera, quiet road, daylight | Nearly all of the frame | None. This is the best case |
| Fixed camera, heavy traffic | Background, sky, markings | Prediction, locally and predictably |
| Fixed camera, tree line in wind | Everything except the trees | Prediction, in a fine grained patch |
| Fixed camera, night, IR on | Geometry holds, colour is free | Quantisation, via sensor noise |
| Fixed camera, sun going in and out | Geometry only | Prediction, across the whole frame |
| Camera on a flexing pole | Nothing reliably | Prediction, across the whole frame |
| PTZ mid-move | Nothing | Prediction, plus motion blur |
The third column is the point. These are not seven different bandwidth problems, they are three mechanisms failing in seven situations, and the remedy follows from which one it is. Prediction failing across the whole frame is a mount or an exposure problem. Prediction failing in a patch is a framing problem, and sometimes the answer really is to point the camera slightly away from the tree. Quantisation drowning in noise is a denoising problem. None of those are solved by changing codec.
Why this explains the codec argument
It also explains something we have published numbers on and been asked about repeatedly. At the bitrates security and roadway cameras actually run, typically 400 to 800 kbit/s, we measure newer codecs delivering around 2 to 3% over H.264. People find that implausibly small, because the same codecs show far larger gains in published comparisons.
Both things are true. The newer codecs win with larger and more flexible block partitioning, better intra prediction and more sophisticated in-loop filtering, and those tools pay off on complex, detailed, high bitrate content where there is a lot of hard-to-predict information to handle well. On a static scene at 600 kbit/s, almost all of the available redundancy has already been taken by the four mechanisms above, every one of which H.264 already has. There is very little left for a cleverer codec to find, and what it charges for the attempt, in encode latency and CPU, is the part that lands on us rather than on a benchmark.
Which is why, when we went looking for savings, we found them inside H.264 rather than outside it. Half a second of encoder lookahead cut bitrate 43 to 45% on roadway cameras at 1.1 times the CPU, across 1,586 encodes. That is an order of magnitude more than switching codec would have bought, from the codec we were already running.
The cheapest bitrate saving is not a codec
In rough order of what it costs you to do and what it returns: mount the camera so it does not move, narrow or lock exposure if the camera allows it, denoise before the encoder at night, frame out the foliage you do not need, set keyframe interval for the path rather than by habit, then tune the encoder. Changing codec comes after all of that, and on these scenes it is the smallest item on the list.