Video　object　segmentation　is　important　for　object　tracking,　video　surveillance　and　semantic　classification.　In　order　to　overcome　the　limitation　of　existing　video　object　segmentation　methods　under　dynamic　background,　a　video　segmentation　algorithm　based　on　motion　cue　and　color　information　is　proposed　in　this　paper.　At　first,　a　new　motion　trajectory　classification　method　is　designed.　The　proposed　method　can　accurately　divide　the　motion　trajectory　set　into　background　and　moving　object　ones　by　combining　the　low　rank　constraint　and　cumulative　acknowledgment　strategy.　Then,　the　superpixels　are　acquired　by　over-segmenting　method.　And　the　color　similarity　of　adjacent　superpixels　is　computed　according　to　their　common　boundary.　At　last,　taking　the　superpixels　as　node,　an　energy　function　of　Markov　random　field　model　is　designed,　which　has　combined　motion　trajectory　classification　information　and　color　similarity　of　superpixels.　The　classification　of　each　superpixel　can　be　obtained　by　finding　the　minimum　of　the　energy　function.　The　proposed　algorithm　is　tested　on　several　publicly　available　videos.　Experimental　results　demonstrate　that　the　proposed　method　can　accurately　segment　the　moving　objects　from　the　dynamic　background,　and　it　has　better　segmentation　accuracy　compared　with　classical　methods.