There　is　an　inevitable　requirement　of　transcoding　from　MPEG-2　to　H.264/AVC　in　many　video　applications.　Due　to　the　use　of　variable　block　sizes　and　rate-distortion　optimization　techniques　in　H.264,　there　is　extremely　high　computational　complexity　in　the　reference　cascaded　transcoder.　To　reduce　the　complexity,　this　paper　presents　a　fast　inter　mode　decision　algorithm　for　P　frames　in　MPEG-2　to　H.264/AVC　transcoding.　Firstly,　with　the　statistical　analysis　of　the　correlation　between　the　MB＇s　coding　mode　in　MPEG-2　and　the　corresponding　mode　in　H.264/AVC,　the　proposed　algorithm　directly　determines　the　candidate　modes　for　some　special　MBs　based　on　the　mode　mapping　rules.　Then　for　other　MBs,　machine　learning　tools　are　used　to　exploit　the　correlation　between　the　residual　MB　information　in　MPEG-2　and　the　coding　mode　in　H.264/AVC,　and　the　decision　tree　is　derived　to　determine　the　candidate　modes.　The　proposed　algorithm　efficiently　reduces　the　number　of　candidate　modes　in　H.264/AVC　encoder.　Experimental　results　show　that　the　proposed　algorithm　can　achieve　60　%　of　transcoding　time　saving　on　average　with　the　less　PSNR　degradation　and　bitrate　increase,　comparing　with　the　reference　cascaded　transcoder.