Finders

One satellite image needed per AIKP

Every AIKP takes some input data, either rasters, DEMs, point clouds, or satellite EO data. In the latter case, OCLI comes with Finders, Python components that allow for searching, filtering, and retrieving of satellite EO data directly from OCLI, with no need to go off platform. This is the case for Sentinel-1, Sentinel-2, and Landsat-8 data, which all have integrated Finders.

If data is manually stored in OCLI storage folder via rsync command (same as ROIs), for satellite imagery that does not have an internal Finder but has Processor for Stack creation (like ASTER or WordView-3 images), than the commands for linking stored data to the task are:

task set master_path=PATH_TO_IMAGE
task set master=PRODUCT_ID --force

Now back to the beginning, on how to select needed satellite images with Finders directly from OCLI. Keep in mind that the activation of the Finder component is automatic based on the AIKP used, with no additional action needed from the user. Meaning, if Composite single AIKP for Sentinel-2 is selected for the creation of a task, Sentinel-2 Finder is automatically initiated as a part of that task.

There are several filters that can be set for satellite image search, all with the task set key=value command, while keys can be found with a task show command:

task set finder.startDate=YYYY-MM-DD
task set finder.completionDate=YYYY-MM-DD

The default date settings are based on the start of the current year. It is possible to change the start and end dates by setting them to the desired timeframe (important when you need the data from a specific time range). Do not set these too long though, because the finder commands might take too long. The optimal setting should not extend 1 year but could be shorter.

In addition to time-based filtering, for optical imagery, finder.cloudCover parameter is available for restricting the results based on the percentage of cloud cover in the images. This parameter works with both Sentinel-2 and Landsat-8 data. The cloud cover filter can be set as:

task set finder.cloudCover=10

or as a range:

task set finder.cloudCover=[MIN_VALUE,MAX_VALUE]

Pay attention to no space between values. OCLI will take space as a separator between arguments: task set finder.cloudCover=[0,  10] is incorrect syntax. Quotes of the range value can solve it:
task set finder.cloudCover=”[0,  10]” is correct syntax.

Remember that the loaded satellite data are only the ones that correspond with the previously chosen AOI or ROI, and in the time range specified by start and end date in task parameters. The basic command to load and see the list of products available are straightforward:

product load
product list

By default, the product list command will list all the products loaded to the database. Depending on the number of loaded products, it may or may not be easier to use additional filters (--where, --column, --limit, --fit, or --aggregate options) and sort (--sort option) commands to limit the number of products displayed. The full list of available columns can be viewed with the product list --columns command.

Each product can be inspected for more information:

product show -i PRODUCT NAME or PRODUCT ID

If AIKP only uses one satellite image, and you do not need to ensure two satellite acquisitions with a complete overlap (as in clustering or composite per AIKP), then the satellite acquisition with a satisfying timestamp and fit can be chosen directly from the list with:

task set master=PRODUCT_ID

For optical imagery (Sentinel-2 and Landsat-8), next command is simply

task get -m --data

and with that, the selected image is ready for the next step, for example creation of a Stack using appropriate Processor.

For SAR imagery (Sentinel-1), some additional commands are available. Detailed command flow can be found in the Annex of the Manual.

task get -m

fetches only metadata of the selected master image. Sentinel-1 images are recorded in 3 swaths and 9 bursts per swath. To reduce the space requirements needed for the analytics and simplify the flow, the user can check and set the swaths and bursts based on the coverage of AOI or ROI:

task show --swath

that gives swath and burst coverage over ROI. In this case, the value of ‘1’ indicates a 100% ROI coverage by just one swath. Values lower than 1 indicate only partial coverage of the ROI by a certain swath. The same logic applies to burst. By default, all swaths and bursts are included in further analytics; this step only helps with reduction of needed space requirements, as additional swaths and bursts with 0 coverage do not add any precision to the final result. To reduce, or set, the swath and range of burst to be used, the command for the example above is:

task set swath=IWx firstBurstIndex=xx lastBurstIndex=xx

The command to download full data for the selected swath and burst of the selected master to OCLI storage is:

task get --master --data

Two satellite image needed per AIKP

If the AIKP calls for two overlapping acquisitions, it is wise to use buckets. For example, for SAR data processing, the algorithm uses an interferometric pair consisting of master and slave images (two acquisitions; before and during an event of interest). The same approach applies to the time series processing, where a single master is compared to a series of slave images. At the minimum, one 1 master (newer) and 1 slave (older) image are needed for comparisons.

When choosing the acquisitions to be used, best practice is to search for the image pairs in the data buckets. Think of buckets as groups of raw images considering the same relative orbit number and cycle direction. This ensures the best overlap between the two acquisitions (newer/master and older/slave) used in further processing. Buckets are powerful sets of time series, which could be used for coherence estimation or surface displacement calculations. Additionally, buckets are not linked to a particular task. The same bucket could be used by many tasks and even re-used in future analysis. The commands to list the available buckets, after product load command, is:

bucket list

To ensure the best results, it is advisable to choose the bucket:

  • with the time range (from - to columns) for your time frame of interest,

  • with at least 2 (preferably more) acquisitions (Cnt column),

  • with the fit over the ROI/AOI as close to 1 as possible (fit column)

After the bucket selection, the command to show products inside the bucket of users choice is:

bucket show BUCKET NAME or BUCKET ID

Please note that not all of the images are actually downloaded (especially on the remote machines). To check if the files are downloaded or present add the --check option to the bucket show command:

  • + next to the product name will verify that the data exists

  • ~ symbol means that only metadata are loaded

Buckets can also be exported in json, txt, csv or SAFE format files (with the path, file name and appropriate file extension).

bucket show # --column title,relativeOrbitNumber --export "~/OCLI/demo.json"
bucket show # --column title,relativeOrbitNumber --export "~/OCLI/demo.txt"
bucket show # --column title,relativeOrbitNumber --export "~/OCLI/demo.csv"
bucket show # --export_linkdir “~/OCLI/demo”

Once the choice is made for master (newer) and slave (older) image acquisition has been made, they can easily be set with:

task set master=PRODUCT_ID
task set slave=PRODUCT_ID

At this point you can continue the same flow as for one satellite image

task get --master --slave  #additional swath and burst setting described above for Sentinel-1 
task get --master --slave --data  #for Sentinel-2