Object　detection　is　a　fundamental　and　important　task　in　the　analysis　of　remote　sensing　images　(RSIs),　and　existing　deep　learning-based　object　detection　models　in　this　literature　strongly　rely　on　predefined　anchor　boxes　and　encounter　redesigned　difficulties　related　to　anchors.　In　addition,　they　often　ignore　the　scene-contextual　information　that　objects　are　usually　closely　related　to　their　surrounding　scene.　To　deal　with　these　problems,　we　propose　an　anchor-free　network,　referred　to　as　scene-relevant　anchor-free　network　(SRAF-Net),　for　object　detection　in　RSIs.　The　SRAF-Net　first　captures　the　scene-contextual　features　of　objects　by　using　a　designed　scene-enhanced　feature　pyramid　network　(SE-FPN)　and　then　performs　more　accurate　detection　by　implementing　a　scene　auxiliary　detection　head　(SADH),　which　can　predict　the　existence　of　the　objects　with　the　help　of　the　scene-contextual　features　extracted　from　the　SE-FPN.　To　deal　with　insufficient　scene　diversity　in　the　training　stage,　a　simple　yet　effective　data　augmentation　module,　termed　balanced　mixup　data　augment　(BMDA),　is　introduced　by　linearly　expanding　the　training　dataset　to　improve　the　generalization　of　SRAF-Net.　Comprehensive　experiments　on　three　publicly　available　challenging　remote　sensing　datasets　demonstrate　the　effectiveness　of　the　proposed　method.　The　codes　will　be　made　publicly　available　at　https://github.com/Complicateddd/SRAF-Net.