A　pruning　algorithm　is　proposed　to　reduce　the　encoding　bits　for　the　map　information　of　residual　coefficients　in　H.264.　Video　sequences　are　predicted,　transformed,　quantized,　and　scanned.　Then　residual　coefficients　obtained　from　scanning　are　divided　into　linear　groups,　and　then　the　groups　are　pruned　out　using　the　rate-distortion　optimization　(RDO)　sense.　Furthermore,　the　pruning　results　of　each　macroblock　with　different　quantization　steplengths　are　calculated　in　a　given　range.　Finally　the　remaining　coefficients　with　the　minimal　rate-distortion　cost　are　encoded　and　transformed.　Experimental　results　show　that　the　proposed　algorithm　has　better　RD　performance　than　the　residual　coding　method　in　H.264　at　the　relatively　high　bitrates.　The　proposed　algorithm　gains　an　0.9　dB　increase　on　the　peak　signal　to　noise　ratio(PSNR)　at　the　same　bitrate,　or　equivalently,　a　17%　rate　saving　at　the　same　PSNR.　Moreover,　the　proposed　algorithm　does　not　change　the　bit-stream　structure　of　H.264,　and　is　compatible　with　the　H.264　standard.