Our　previous　study　has　shown　that　image　distortions　cause　saliency　distraction,　and　that　visual　saliency　of　a　distorted　image　differs　from　that　of　its　distortion-free　reference.　Being　able　to　measure　such　distortion-induced　saliency　variation　(DSV)　significantly　benefits　algorithms　for　automated　image　quality　assessment.　Methods　of　quantifying　DSV,　however,　remain　unexplored　due　to　the　lack　of　a　benchmark.　In　this　paper,　we　build　a　benchmark　for　the　measurement　of　DSV　through　a　subjective　study.　Sixteen　experts　in　computer　vision　were　asked　to　compare　saliency　maps　of　distorted　images　to　the　corresponding　saliency　maps　of　the　original　images.　All　saliency　maps　were　rendered　from　ground　truth　human　fixations.　A　statistical　analysis　is　performed　to　reveal　the　behaviours　and　properties　of　human　assessment　of　the　saliency　variation.　The　benchmark　is　made　publicly　available　to　the　research　community.　©　2019　IEEE.