Challenge Overview
Problem Statement | |||||||||||||
Prize Distribution
Prize USD
1st $6,000
2nd $3,000
3rd $1,500
Best F-score, Las Vegas $1,000
Best F-score, Paris* $1,000
Best F-score, Shanghai $1,000
Best F-score, Khartoum $1,000
Early Incentive** $1,000
Total Prizes $15,500
* The winning entry on the Paris data set must score higher than 400,000, which is above a score derived solely from OpenStreetMaps (OSM). ** To win the Early Incentive your solution must be the first to reach a threshold of 400,000 for the average F-score of all the cities. Overview of changes from round 1This contest is the second edition of the SpaceNet building footprint detection challenge. Most of the problem statement is identical to the specification used in the first challenge. The most important differences are the following:
There are other, less significant differences, so if you participated in the first contest please read this document carefully to make sure you are aware of changes in the details. BackgroundThe commercialization of the geospatial industry has led to an explosive amount of data being collected to characterize our changing planet. One area for innovation is the application of computer vision and deep learning to extract information from satellite imagery at scale. DigitalGlobe, CosmiQ Works, and NVIDIA have partnered to release the SpaceNet data set to the public to enable developers and data scientists. Today, map features such as roads, building footprints, and points of interest are primarily created through manual techniques. We believe that advancing automated feature extraction techniques will serve important downstream uses of map data. For example, this type of map data is needed for planning large-scale international events, as well as humanitarian and disaster response as recently observed in Haiti to aid the response to Hurricane Matthew. Furthermore, we think that solving this challenge is an important stepping stone to unleashing the power of advanced computer vision algorithms applied to a variety of remote sensing data applications in both the public and private sector. ObjectiveCan you help us automate mapping? In this challenge, competitors are tasked with finding automated methods for extracting map-ready building footprints from high-resolution satellite imagery. Moving towards more accurate fully automated extraction of buildings will help bring innovation to computer vision methodologies applied to high-resolution satellite imagery, and ultimately help create better maps where they are needed most. Your task will be to extract polygonal areas that represent buildings from satellite images. The polygons your algorithm returns will be compared to ground truth data, the quality of your solution will be judged by the combination of precision and recall. See Scoring for details. Input FilesSatellite images Four types of images are available for the target areas:
The images were collected by the DigitalGlobe Worldview-3 satellite over Las Vegas, Paris, Shanghai and Khartoum. The training data contains more than 10,000 images, each image covers 200m x 200m on the ground. Worldview-3 is sensitive to light in a wide range of wavelengths, see the specification of the frequency bands here. This extended range of spectral bands allows Worldview-3 imagery to be used to classify the material that is being imaged. Images are provided in GeoTiff format. Since most image viewer applications will not be able to display 16-bit images and 8-band images, we provide a visualizer tool that you can use to look at the data. See the Visualizer description for details. Building footprints The location and shape of known buildings are referred to as ���ground truth��� in this document. These data are described in CSV files using the following format: ImageId,BuildingId,PolygonWKT_Pix,PolygonWKT_Geo img1,1,"POLYGON ((103.7 205.4 0,107.8 201.9 0, 100.5 203.2 0,94.4 208.0 0,93.0 215.7 0,92.1 226.1 0,89.6 228.8 0, 95.0 233.9 0,92.4 236.6 0,95.4 239.8 0,116.7 221.0 0,116.7 216.1 0, 103.7 205.4 0))","POLYGON ((-43.681699199999969 -22.981289 0, ... [truncated for brevity]...))" img2,-1,POLYGON EMPTY, POLYGON EMPTY img339,1,"POLYGON ((213 269 0,184 221 0, 130 249 0,154 293 0,162 288 0,165 292 0,169 290 0,180 285 0, 213 269 0),(151 253 0,177 242 0,189 263 0,164 275 0,151 253 0))" ,"POLYGON ((...[truncated for brevity]...))" (The sample above contains 4 lines of text, extra line breaks are added only for readability.)
Usually a building is defined by a single polygon that represents the exterior boundary edge of its shape. This is sometimes not enough, see building #6 on image AOI_5_Khartoum_img1308 for an example shape that contains a hole. In such cases two (or more) polygons are needed, the first one always defines the exterior edge, the second (third, fourth, etc) defines the interior rings (holes). (See line #4 above for an example of the required syntax in this case.) The PolygonWKT_Pix coordinates refer to the PAN images. Note that because of the different resolutions, the coordinates of the same object on the MUL images are different. Notes
Downloads Input files are available for download from the spacenet-dataset AWS bucket. A separate guide is available that details the process of obtaining the data. See also this page for description of the Spacenet AWS data and download instructions. Note that the bucket holds much more data than what is required for this challenge. The minimal list of data files that you should download is the following:
Where
A sample submission file that scores non-0 on the spacenet_sample dataset is available here. A note on image ids and image names The format of an imageId is AOI_<n>_<city>_img<i> where the <n>_<city> part can take one of the following values: 2_Vegas, 3_Paris, 4_Shanghai, 5_Khartoum. <i> is a 1-based integer that is unique for an image within a set that belongs to a city. Images within the PAN, MUL, etc folders have names like PAN_<imageId>.tif, MUL_<imageId>.tif, etc. Output FileYour output must be a CSV file with almost identical format to the building footprint definition files. ImageId,BuildingId,PolygonWKT_Pix,Confidence Your output file may or may not include the above header line. The rest of the lines should specify the buildings your algorithm extracted, one per line. The required fields are:
Your output must be a single file with .csv extension. Optionally the file may be zipped, in which case it must have .zip extension. The file must not be larger than 150MB and must not contain more than 2 million lines. FunctionsThis match uses the result submission style, i.e. you will run your solution locally using the provided files as input, and produce a CSV or ZIP file that contains your answer. In order for your solution to be evaluated by Topcoder���s marathon system, you must implement a class named BuildingDetector2, which implements a single function: getAnswerURL(). Your function will return a String corresponding to the URL of your submission files. You may upload your files to a cloud hosting service such as Dropbox or Google Drive, which can provide a direct link to the file. To create a direct sharing link in Dropbox, right click on the uploaded file and select share. You should be able to copy a link to this specific file which ends with the tag "?dl=0". This URL will point directly to your file if you change this tag to "?dl=1". You can then use this link in your getAnswerURL() function. If you use Google Drive to share the link, then please use the following format: "https://drive.google.com/uc?export=download&id=" + id Note that Google has a file size limit of 25MB and can���t provide direct links to files larger than this. (For larger files the link opens a warning message saying that automatic virus checking of the file is not done.) You can use any other way to share your result file, but make sure the link you provide opens the filestream directly, and is available for anyone with the link (not only the file owner), to allow the automated tester to download and evaluate it. An example of the code you have to submit, using Java:
public class BuildingDetector2 {
public String getAnswerURL() {
//Replace the returned String with your submission file���s URL
return "https://drive.google.com/uc?export=download&id=XYZ1";
}
}
Keep in mind that your complete code that generates these results will be verified at the end of the contest if you achieve a score in the top six of the average F-score for all cities or in the top two of the highest F-score per city, as described later in the "Requirements to Win a Prize" section, i.e. participants will be required to provide fully automated executable software to allow for independent verification of software performance and the metric quality of the output data. ScoringA full submission will be processed by the Topcoder Marathon test system, which will download, validate and evaluate your submission file. Any malformed or inaccessible file, or one that exceeds the maximum file size (150 MB) or the maximum number of lines (2 million) will receive a zero score. If your submission is valid, your solution will be scored using the following algorithm.
The precision and recall of your algorithm are defined as Precision = TP / (TP + FP) Recall = TP / (TP + FN) The F-score of your algorithm is defined as 0 if either Precision or Recall is 0. Otherwise: F_score = 2 * Precision * Recall / (Precision + Recall) Your overall score is calculated as 1000000 * F_score. Note that your returned confidence values are used only to sort the building footprints you extract so that buildings with higher confidence values are tried to be matched first. The exact values don���t matter, they only establish a ranking among the buildings. Note also that due to clipping building footprints at image chip boundaries very small buildings may appear. As it is not realistic to expect that your algorithm can find such buildings, all buildings with area less than 20 (measured in pixels on the PAN imagery) will be ignored both in the ground truth and in your solution. For the exact algorithm of the registration and scoring see the visualizer source code. Example submissions can be used to verify that your chosen approach to upload submissions works. The tester will verify that the returned String contains a valid URL, its content is accessible, i.e. the tester is able to download the file from the returned URL. If your file is valid, it will be evaluated, and precision, recall and F-score values will be available in the test results. The example evaluation is based on a small subset of the training data, the contents of the spacenet_sample.tar.gz file is used for this purpose. Full submissions must contain in a single file all the extracted building polygons that your algorithm found in all images of the AOI_<n>_<city>_Test_public folders, i.e. all images of all 4 cities should be processed into a single submission file. Final ScoringThe top six competitors for average F-score across all cities and the top two for the highest F-score per city, according the provisional scores, will be invited to the final testing round. In the past such final testing of 'submit data' type contests was done on AWS VM instances where contestants deployed an executable version of their systems which was given new test data to determine the final rankings. This time the process will be different. If your score qualifies you as a top 6 average F-score per city or top 2 F-score per city competitor, you will have to submit a Docker container that the organizers can use to score your solution. The exact details of this process will be announced not later than 2 weeks before the end of the provisional testing phase. The most important requirements for your final submission will be the following:
Your solution will be subjected to three tests: First, your solution will be validated, i.e. we will check if it produces the same output file as your last submission, using the same input files used in this contest. Note that this means that your solution must not be improved further after the provisional submission phase ends. (We are aware that it is not always possible to reproduce the exact same results. E.g. if you do online training then the difference in the training environments may result in different number of iterations, meaning different models. Also you may have no control over random number generation in certain 3rd party libraries. In any case, the results must be statistically similar, and in case of differences you must have a convincing explanation why the same result can not be reproduced.) Second, your solution will be tested against a set of new image files. The number and size of these new set of images will be similar to the one you downloaded as testing data. Also the scene content will be similar. Third, the resulting output from the steps above will be validated and scored. The final rankings will be based on this score alone. Competitors who fail to provide their solution as expected will receive a zero score in this final scoring phase, and will not be eligible to win prizes. Additional Resources
General Notes
Requirements to Win a PrizeEarly incentive To encourage early participation a bonus prize will be awarded to the first participant who reaches a threshold of 400,000 on the average F-score over all 4 cities. In order to receive this bonus prize you must package the source codes you used to generate the submission that passes this threshold, and within two days you must send it to walrus71@copilots.topcoder.com and tim@copilots.topcoder.com so that we can verify that your submission was generated algorithmically. We won't try to run your code at this point, so you don't have to package data files, model files or external libraries, this is just a superficial check to see whether your system looks convincingly automatized. Final prizes In order to receive a final prize, you must do all the following: Achieve a score in the top three of the average F-score of all cities, or the highest F-score per city, according to system test results. See the "Final scoring" section above. Once the final scores are posted and winners are announced, the prize winner candidates (on overall F-score and on individual city F-score) have 7 days to submit a report outlining their final algorithm explaining the logic behind and steps to its approach. You will receive a template that helps creating your final report. If you place in a prize winning rank but fail to do any of the above, then you will not receive a prize, and it will be awarded to the contestant with the next best performance who did all of the above. | |||||||||||||
Definition | |||||||||||||
| |||||||||||||
Examples | |||||||||||||
| 0) | |||||||||||||
| |||||||||||||
This problem statement is the exclusive and proprietary property of TopCoder, Inc. Any unauthorized use or reproduction of this information without the prior written consent of TopCoder, Inc. is strictly prohibited. (c)2020, TopCoder, Inc. All rights reserved.