Upload Strategy

The UploadStrategy class manages the process of uploading output files and related data to cloud storage (e.g., S3). It supports both default upload strategies and custom implementations if specified by the AIKP template.

See also: GeoJSON generation and document schema in GeoJSON Strategy.

Workflow

  1. Initialization During initialization, UploadStrategy attempts to check the current execution context and retrieve upload configurations such as cos_key and dry_run parameters from the context metadata. If a custom upload logic is defined in the AIKP template, it is loaded and stored in the _self._upload_ property.

  2. Default file processing In the absence of custom upload logic, the UploadStrategy falls back to its default behavior to process the following files:

    • COG (Cloud Optimized GeoTIFF) files and associated outputs.

    • DEM and normalized DEM files.

    • Metadata documents in .geojson format.

    • Files from out_additional_dir, which expands the flexibility of the process by including any additional files present in this directory.

The directory out_additional_dir, if present, is scanned for files. Any files found are uploaded to the target location in the cloud using the following file structure:

   S3: <cos_key>/<file_name>

  1. Custom upload handling If the AIKP template contains a RECIPE_DEFAULTS dictionary with a custom upload module specified, the logic in the module is used by invoking its execute() method.

  2. Dry-run simulation The UploadStrategy supports a dry-run mode that simulates file uploads without performing any actual operations. This mode provides comments on the files and destinations to be processed.

Key Methods

  • _upload_common() Handles the default upload process by gathering file mappings and invoking upload logic.

  • _default_files_processing() Processes the default file set, including additional files from out_additional_dir if available.

  • upload() Entry point for the upload process. Decides whether to execute custom or default uploading behavior.

Considerations

  1. Custom upload logic

    1. Option 1

    Similar to other strategies, the UploadStrategy allows specifying a custom upload module in the RECIPE_DEFAULTS dictionary of the AIKP template. The custom module should provide an execute() method with a cos_key and other parameters as needed. 2. Option 2

    Implement upload(cls, task: Task, cos_key: str | None, dry_run: bool) function inside your TaskTemplate implementation

  2. Cloud storage requirements The default implementation requires valid COS credentials to interact with the target bucket. If these are not provided in the recipe, the process will throw an exception.

Usage Example

To execute the upload strategy, instantiate the class and invoke the upload() method as follows:

strategy = UploadStrategy()
strategy.upload()

UI document types and default uploads

This strategy cooperates with GeoJsonStrategy outputs and type-specific documents.

  • Raster-based via GeoJsonStrategy

    • Types: Image, DEM, and raster-style Scalar

    • Default uploads:

      • .tiff

      • .tiff.geojson (produced by ai makegeojson)

      • .dem.tiff (optional)

      • .dem_normals.tiff (optional)

    • The backing COG and its GeoJSON use the same ResultKey stem.

  • Static/document types

    • Types: Vector, Shape, Cluster, Media, Mesh, PointCloud, Report, Actor

    • These are typically external/static payloads referenced by properties.source.url (prefer /s3/$FILE_PATH).

    • Place files under out_additional_dir to have them uploaded to <cos_key>/<file_name>.

  • Collection/document listing

    • If <ResultKey>.document.geojson exists (collection listing for the UI), it will be included in the upload set when discovered.

cos_key resolution and overrides

  • The effective upload root (cos_key) is taken from the current execution context or the recipe defaults.

  • When invoked via CLI with --cos-key, a leading + appends to the existing key; otherwise it replaces it. See GeoJSON Strategy for CLI tips.

Dry run

  • Use dry-run mode to see the final mapping of local files → <cos_key>/... without actually uploading.