Spacenet - 6: Multi-Sensor All Weather Mapping

Prize Distribution

Prize                    USD

1st                  $20,000

2nd                  $10,000

3rd                   $7,500

4th                   $5,000

5th                   $2,500

Top Graduate          $2,500

Top Undergraduate     $2,500

Total Prizes         $50,000

Multi-Sensor All Weather Mapping is the first SpaceNet challenge to feature two modalities of overhead data: Synthetic Aperture Radar (SAR) and traditional electro-optical (EO) imagery.  SAR sensors are unique as they can penetrate clouds and collect during all weather conditions. Furthermore, radar satellites do not require illumination and can capture data during both the day and the night. Consequently, overhead collects from SAR satellites could be particularly valuable in the quest to aid disaster response, when weather and cloud cover can obstruct traditional electro-optical sensors. 

In this challenge participants will be asked to extract building footprints using a combination of these two data-sources over the port of Rotterdam.  For training data, participants are allowed to leverage both the electro-optical and SAR datasets. However, for scoring performance, only the SAR data is available to participants to map buildings. The stakeholders of this challenge hope that such a structure will incentivize new data fusion methods and other approaches such as domain adaptation.

CosmiQ Works’ blog, The DownLinQ, provides additional background information on this challenge, an introduction to Synthetic Aperture Radar imagery (101) , a more detailed look at SAR (201), and an overview of the dataset used in the challenge.
   

Input files

Satellite images

Five types of images are available for the target area, but note that only the SAR image type is available for provisional and final scoring, the four EO types are provided only for the training set. For all data types areas containing no valid data are set to 0.

 

1. SAR-Intensity: 1 channel per radar polarization type for a total of 4 bands (4*Float32, ~50cm resolution.) Each channel displays the intensity of backscatter (the amount of transmitted radar signal that the imaging surface redirects back to the sensor) for each polarization. The units of each image are calculated by setting any raw intensity value <0 to 0. The logarithm of raw intensity is then calculated and this value is multiplied by 10. Any value that again falls below 0 is set to 10^-5. This stretches data to a range between 10^-5 and 92.88.

2. PAN: EO panchromatic (single channel, 16-bit grayscale, ~50 cm resolution)

3. RGBNIR: EO 4-band multi-channel (4*16-bit, ~1m resolution).

4. PS-RGB: EO pan-sharpened version of Red-Green-Blue bands from the multispectral product (3 channels, 3*8-bit, ~50 cm resolution). This is formed by using the PAN image to interpolate 3 bands of the RGBNIR dataset to increase the resolution of the Red, Green and Blue bands.

5. PS-RGBNIR: EO pan-sharpened version of RGBNIR (4 channels, 8*16 bit, ~50 cm resolution)

 

The two imagery datasets (SAR and EO) are collected within days of one another and are centered over the Rotterdam port. Imagery is provided in ~400x400 meter chips. RGBNIR chips are 450 pixels wide and tall, all other image types are 900x900 pixels.

Images are provided in GeoTiff format. Since most image viewer applications are not able to display 16-bit images and 4-band images, we provide a visualizer tool that you can use to look at the data. See the Visualizer description for details.

Building footprints

The location and shape of known buildings are referred to as ‘ground truth’ in this document. These data are described in CSV files using the following format:

ImageId,TileBuildingId,PolygonWKT_Pix,Mean_Building_Height,Median_Building_Height,StdDev_Building_Height

 

img1,1,"POLYGON ((103.7 205.4,107.8 201.9,

100.5 203.2,94.4 208.0,93.0 215.7,92.1 226.1,89.6 228.8,

95.0 233.9,92.4 236.6,95.4 239.8,116.7 221.0,116.7 216.1,

103.7 205.4))",1.75,1.75,0

 

img2,0,POLYGON EMPTY,0,0,0

 

img339,1,"POLYGON ((213 269,184 221,

130 249,154 293,162 288,165 292,169 290,180 285,

213 269),(151 253,177 242,189 263,164 275,151 253))"

,1.75,1.75,0

 

(The sample above contains 4 lines of text, extra line breaks are added only for readability.)

• ImageId is a string that uniquely identifies the image.

• TileBuildingId is an integer that identifies a building in the image, it is unique only within an image.

• PolygonWKT_Pix specifies the points of the shape that represents the building in Well Known Text format. Only the POLYGON object type of the WKT standard is supported. The special POLYGON EMPTY construct is used for cases when there are no buildings present in the image. TileBuildingId is ignored in this case. (See line #3 for an example above.) The coordinate values represent pixels, the origin of the coordinate system is at the top left corner of the image, the first coordinate is the x value (positive is right), the second coordinate is the y value (positive is down). Note that the polygons must be closed: the first and last entries in their point list must be the same.

• Mean_Building_Height, Median_Building_Height, StdDev_Building_Height are variables describing the estimated height of the building in meters. These values are provided for information only and have no relevance in scoring.

Usually a building is defined by a single polygon that represents the exterior boundary edge of its shape. This is sometimes not enough, see tile_5554 in the training set for an example shape that contains a hole. In such cases two (or more) polygons are needed, the first one always defines the exterior edge, the second (third, fourth, etc.) defines the interior rings (holes). (See line #4 above for an example of the required syntax in this case.)

The PolygonWKT_Pix coordinates refer to the SAR-Intensity images. Note that because of the different resolutions, the coordinates of the same object on the RGBNIR images are different.

Additional metadata

SAR_orientations.txt describes whether a given SAR image strip was captured from a north facing or south facing view. Each line of this file contains two timestamps and an orientation tag. The timestamps correspond to those present in imageId, see later. The orientation tag is 0 for north, 1 for south. The same SAR_orientations.txt file is valid for all three test sets (training, provisional testing, final testing).

Notes on data 

• Many tiles contain regions from where no image and no building footprint data is available. Such regions are shown in black in the visualizer tool. The SAR images contain a value of 0 for no data. Your algorithm should handle such cases of missing data.

• The way of representing holes is different from how the WKT standard specifies them: the standard mandates that the points of a polygon must be enumerated in an anti-clockwise order if the polygon represents a shape with positive area (exterior rings), and in a clockwise order if the polygon represents a shape with negative area (interior rings). However, what appears clockwise in the latitude/longitude based geographic coordinate system appears anti-clockwise in the image space where the y axis points downwards, so in the hope of causing less confusion we chose a simpler approach: the first polygon is always positive, the rest are always negative.

• For ground truth data we use existing labels provided by the BAG3D Dataset, with building footprints initially curated by the Netherlands land registry and national mapping agency. CosmiQ subsequently performed quality control on these labels, removing incorrect labels and adding in missed buildings.  Labels are then simplified to contiguous structures, as buildings like apartments and townhouses are difficult to map from space. Nevertheless, as in all real-life problems, the data set may contain errors.

• The task of finding building footprints is different from finding the outline of the building’s shape on an image. In most cases the difference is not significant, but you should be aware of this difference, especially when a high building is viewed at an angle.

 

Downloads

Input files are available for download from the spacenet-dataset AWS bucket. A separate guide is available that details the process of obtaining the data. See also this page for description of the SpaceNet AWS data and download instructions. 

Note that the same bucket holds data for the previous SpaceNet challenges as well, you need only a subset of the bucket content for this challenge: download only the files in the spacenet/SN6_buildings/tarballs directory (~56GB compressed). 

Here

• SN6_buildings_AOI_11_Rotterdam_train_sample.tar.gz contains a few sample images and labels from the training set. Use this if you want to get familiar with the data without having to download any of the large files.

• train.tar.gz is the training set. It contains imagery in the 5 formats described above and also contains building footprints in a CSV file.

• test_public.tar.gz is the testing set that you should use to submit results during the provisional testing phase of the challenge. It contains SAR imagery but does not contain building footprints.

Note that individual files or smaller subsets are available from the spacenet/SN6_buildings/train/ or spacenet/SN6_buildings/test_public/ folders. Use these if you want to get familiar with the data without having to download the full sets.

Notes on image ids and image names.

• The format of an imageId is
  <ts1>_<ts2>_tile_<i>

• <ts1> and <ts2> are two timestamps that represent when image collection started and stopped for each SAR image strip.

• All imageIds are case sensitive.

• The format of an image name is
  SN6_<prefix>_AOI_11_Rotterdam_<type>_<imageId>.tif

• <type> is one of: SAR-Intensity, RGBNIR, PAN, PS-RGBNIR, PS-RGB. Images of different type are organized into folders having the same name as the 5 values listed here.

• <prefix> is a string that is the same for images belonging to the same test set, an example is Test_Public. 

 

Output file

Your output must be a CSV file with similar format to the building footprint definition files.

• ImageId,PolygonWKT_Pix,Confidence

Your output file may or may not include the above header line. The rest of the lines should specify the buildings your algorithm extracted, one per line.

The required fields are:

• ImageId is a string that uniquely identifies the image within a test set.

• PolygonWKT_Pix specifies the points of the shape that represents the building you found. The format is exactly the same as given above in the Input files section.

• Confidence is a positive real number, higher numbers mean you are more confident that this building is indeed present. See the details of scoring for how this value is used.

Your output must be a single file with .csv extension. The file must not be larger than 150MB and must not contain more than 2 million lines.

Constraints

• A single file must contain building footprints for ALL images in the test set.

• The file may (and typically does) contain multiple lines for the same ImageId.

• If you found no buildings on an image then you must use the LINESTRING EMPTY construct. In this case your file must not contain other lines for the same ImageId. The value for confidence is ignored in this case.

 

Submission format and code requirements

This match uses a combination of the "submit data" and "submit code" submission styles. The required format of the submission package is specified in a submission template document. This current document gives only requirements that are either additional or override the requirements listed in the template.

• Your algorithm must process the image tiles of the test set one by one, that is, when you are predicting building footprints you must not use information from other tiles of the test set.

• You must not submit more often than 3 times a day. The submission platform does not enforce this limitation, it is your responsibility to be compliant to this limitation. Not observing this rule may lead to disqualification.

• An exception from the above rule: if your submission scores 0, then you may make a new submission after a delay of 1 hour. 

• The /solution folder of the submission package must contain the solution.csv file, which should be formatted as specified above in the Output file section and must list all extracted buildings from all images in the test set. 

Scoring

During scoring your solution.csv file (as contained in your submission file during provisional testing, or generated by your docker container during final testing) will be matched against  expected ground truth data using the following algorithm (closely related to Algorithm 2 of ImageNet). 

If your solution is invalid (e.g. if the tester tool can't successfully parse its content), you will receive a score of 0.

If your submission is valid, your solution will be scored as follows.

1. Sort the building polygons you returned in decreasing order of confidence. (Equal values will be sorted in the order they appear in your output file.)

2. For each polygon find the best matching one from the set of the ground truth polygons. Loop over the truth polygons, and:

   1. Skip this truth polygon if it was already matched with another solution polygon.

   2. Skip this truth polygon if it belongs to a different ImageId than the solution polygon.

   3. Otherwise calculate the IOU (Intersection over Union, Jaccard index) of the two polygons.

   4. Note the truth polygon which has the highest IOU score if this score is higher than or equal to 0.5. Call this the ‘matching’ polygon.

3. If there is a matching polygon found above, increase the count of true positives by one (TP).

4. If there is no matching polygon found, increase the count of false positives by one (FP).

5. When all solution polygons are processed then for each truth polygon that is left unmatched, increase the count of false negatives by one (FN).

The precision and recall of your algorithm are defined as:

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

The F1-score of your algorithm is defined as 0 if either Precision or Recall is 0. Otherwise:

F1_score = 2 * Precision * Recall / (Precision + Recall)

Your overall score is calculated as 100 * F1_score.

Note that your returned confidence values are used only to sort the building footprints you extract so that buildings with higher confidence values are tried to be matched first. The exact values don’t matter, they only establish a ranking among the buildings.

Note also that due to clipping building footprints at image chip boundaries very small buildings may appear. As it is not realistic to expect that your algorithm can find such buildings, all buildings with area less than 80 pixels (measured in pixels on the SAR imagery) will be ignored both in the ground truth and in your solution.

Note that geographic metadata will be stripped from the private test set.

For the exact algorithm of scoring see the visualizer source code.

Final testing

This section details the final testing workflow and the requirements against the /code folder of your submission are also specified in the submission template document. This current document gives only requirements or pieces of information that are either additional or override those given in the template. You may ignore this section until you decide you start to prepare your system for final testing. 

• The signature of the train script is as given in the template:
  train.sh <data_folder>
  The supplied <data_folder> parameter points to a folder having raw SpaceNet satellite data in the same structure as is available for you during the coding phase, tar.gz files already extracted. The supplied <data_folder> will be the parent folder of the SAR-Intensity, PAN, SummaryData, etc. folders.

• The allowed time limit for the train.sh script is 8 GPU-days (2 days on a p3.8xlarge). Scripts exceeding this time limit will be truncated.

• A sample call to your training script (single line) follows. Note that folder names are for example only, you should not assume that the exact same folders will be used in testing.
  ./train.sh /data/SN6_buildings/train/AOI_11_Rotterdam/
  In this sample case the training data looks like this:
    data/

    SN6_buildings/
      train/

        AOI_11_Rotterdam/

          PAN/

          RGBNIR/
          SAR-Intensity/

          geojson_buildings/
          ...
          SummaryData/
            SAR_orientations.txt
            SN6_Train_AOI_11_Rotterdam_Buildings.csv
     
 

• The signature of the test script:
  test.sh <data_folder> <output_file>
  The testing data folder contains similar SAR imagery as is available for you during the coding phase. The supplied <data_folder> will be the parent folder of the SAR-Intensity folder.

• The allowed time limit for the test.sh script is 12 GPU-hours (3 hours on a p3.8xlarge) when executed on the full provisional test set (the same one you used for submissions during the contest). Scripts exceeding this time limit will be truncated.

• A sample call to your training script (single line) follows. Again, folder and file names are for example only, you should not assume that the exact same names will be used in testing.
  ./test.sh /data/SN6_buildings/test_public/AOI_11_Rotterdam/ /wdata/my_output.csv
  In this sample case the testing data looks like this:
    data/

    SN6_buildings/
      test_public/
        AOI_11_Rotterdam/

          SAR-Intensity/
            ... // SAR .tif image tiles
      

• The verification workflow will be different from what is described in the template. All submissions will be retrained first, and final scores will be established by using the retrained models on the final test set.

• To speed up the final testing process the contest admins may decide not to build and run the dockerized version of each contestant's submission. It is guaranteed however that at least the top 10 ranked submissions (based on the provisional leader board at the end of the submission phase) will be final tested.

• Hardware specification. Your docker image will be built, test.sh and train.sh scripts will be run on a p3.8xlarge Linux AWS instance. Please see here for the details of this instance type.

 

Additional Resources

 

• A visualizer is available that you can use to test your solution locally. It displays satellite images, your extracted building footprints and the expected ground truth. It also calculates detailed scores so it serves as an offline tester.

• Further details on the scoring metric and also a Python implementation can be found here (link to be shared later).

• See the algorithm description and source code of winning entries in previous SpaceNet contests here.

 

General Notes

 

• This match is NOT rated.

• Teaming is allowed. Topcoder members are permitted to form teams for this competition. After forming a team, Topcoder members of the same team are permitted to collaborate with other members of their team. To form a team, a Topcoder member may recruit other Topcoder members, and register the team by completing this Topcoder Teaming Form. Each team must declare a Captain. All participants in a team must be registered Topcoder members in good standing. All participants in a team must individually register for this Competition and accept its Terms and Conditions prior to joining the team. Team Captains must apportion prize distribution percentages for each teammate on the Teaming Form. The sum of all prize portions must equal 100%. The minimum permitted size of a team is 1 member, the maximum permitted team size is 5 members. Only team Captains may submit a solution to the Competition. Topcoder members participating in a team will not receive a rating for this Competition. Notwithstanding Topcoder rules and conditions to the contrary, solutions submitted by any Topcoder member who is a member of a team on this challenge but is not the Captain of the team are not permitted, are ineligible for award, may be deleted, and may be grounds for dismissal of the entire team from the challenge. The deadline for forming teams is 11:59pm ET on the 21th day following the date that Registration & Submission opens as shown on the Challenge Details page. Topcoder will prepare a Teaming Agreement for each team that has completed the Topcoder Teaming Form, and distribute it to each member of the team. Teaming Agreements must be electronically signed by each team member to be considered valid. All Teaming Agreements are void, unless electronically signed by all team members by 11:59pm ET of the 28th day following the date that Registration & Submission opens as shown on the Challenge Details page. Any Teaming Agreement received after this period is void. Teaming Agreements may not be changed in any way after signature.
  The registered teams will be listed in the contest forum thread titled “Registered Teams”.

• Organizations such as companies may compete as one competitor if they are registered as a team and follow all Topcoder rules.

• Relinquish - Topcoder is allowing registered competitors or teams to "relinquish". Relinquishing means the member will compete, and we will score their solution, but they will not be eligible for a prize. Once a person or team relinquishes, we post their name to a forum thread labeled "Relinquished Competitors". Relinquishers must submit their implementation code and methods to maintain leaderboard status.

• In this match you may use any programming language and libraries, including commercial solutions, provided Topcoder is able to run it free of any charge. You may also use open source languages and libraries, with the restrictions listed in the next section below. If your solution requires licenses, you must have these licenses and be able to legally install them in a testing VM (see “Requirements to Win a Prize” section). Submissions will be deleted/destroyed after they are confirmed. Topcoder will not purchase licenses to run your code. Prior to submission, please make absolutely sure your submission can be run by Topcoder free of cost, and with all necessary licenses pre-installed in your solution. Topcoder is not required to contact submitters for additional instructions if the code does not run. If we are unable to run your solution due to license problems, including any requirement to download a license, your submission might be rejected. Be sure to contact us right away if you have concerns about this requirement.

• You may use open source languages and libraries provided they are equally free for your use, use by another competitor, or use by the client.

• If your solution includes licensed software (e.g. commercial software, open source software, etc.), you must include the full license agreements with your submission. Include your licenses in a folder labeled “Licenses”. Within the same folder, include a text file labeled “README” that explains the purpose of each licensed software package as it is used in your solution.

• External data sets and pre-trained networks are allowed for use in the competition provided the following are satisfied:

  • The external data and pre-trained network dataset are unencumbered with legal restrictions that conflict with its use in the competition.

  • The data source or data used to train the pre-trained network is defined in the submission description.

  • The external data source must be declared in the competition forum in the first 45 days of the competition to be eligible in a final solution. References and instructions on how to obtain are valid declarations (for instance in the case of license restrictions). If you want to use a certain external data source, post a question in the forum thread titled “Requested Data Sources”. Contest stakeholders will verify the request and if the use of the data source is approved then it will be listed in the forum thread titled “Approved Data Sources”.

  • Using OpenStreetMap or similar, street level data sources that explicitly describe the targeted test city is not allowed.

  • Using georeferencing of the data (utilizing the geographic coordinates) is not allowed in the testing phase.

  • Using adjacent tiles to improve prediction scores is not allowed.

• Use the match forum to ask general questions or report problems, but please do not post comments and questions that reveal information about possible solution techniques.

 

Award details and requirements to Win a Prize 

 

Final prizes

In order to receive a final prize, you must do all the following:

Achieve a score in the top five according to final system test results. See the "Final scoring" section above.

Comply with all applicable Topcoder terms and conditions.

Once the final scores are posted and winners are announced, the prize winner candidates  have 7 days to submit a report outlining their final algorithm explaining the logic behind and steps to its approach. You will receive a template that helps when creating your final report.

If you place in a prize winning rank but fail to do any of the above, then you will not receive a prize, and it will be awarded to the contestant with the next best performance who did all of the above.

Top undergraduate / top graduate award

The highest scoring undergraduate university student (someone actively pursuing a Bachelor’s degree), or team of such students is awarded $2,500. The same applies to the highest scoring graduate student (someone pursuing a master’s or doctoral degree) or team of such students.

Both prizes are based on the rankings after the final tests. The top undergraduate / graduate prize and the final prizes are not exclusive, the same contestant / team may win a final prize and also one of these two awards. For teams to be eligible for one of these awards all team members must be eligible for the same award. 
 

Eligibility

To be eligible to win the Graduate and Undergraduate prizes, individuals must provide proof of enrollment in an accredited degree program prior to the end of the challenge.

Employees of (or contractors for) In-Q-Tel, Maxar Technologies (DigitalGlobe, Radiant Solutions, SSL, and MDA), Amazon Web Services (AWS), Intel, TopCoder, Capella Space, the Geoscience and Remote Sensing Society (GRSS) of IEEE are not allowed to participate in the contest.

  The challenge opens for submissions on March 16th.