Searching VEDA STAC API and inspecting catalogs with intake

Demonstrates using intake-stac to inspect and load data from VEDA STAC API.
Author

Greg Corradini, Julia Signell

Published

February 2, 2023

Run this notebook

You can launch this notebook in VEDA JupyterHub by clicking the link below.

Launch in VEDA JupyterHub (requires access)

Learn more

Inside the Hub

This notebook was written on the VEDA JupyterHub and as such is designed to be run on a jupyterhub which is associated with an AWS IAM role which has been granted permissions to the VEDA data store via its bucket policy. The instance used provided 16GB of RAM.

See (VEDA Analytics JupyterHub Access)[https://nasa-impact.github.io/veda-docs/veda-jh-access.html] for information about how to gain access.

Outside the Hub

The data is in a protected bucket. Please request access by emailng aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). The team will help you configure the cognito client.

You should then run:

%run -i 'cognito_login.py'

Approach

  1. Use pystac_client to open the STAC catalog and retrieve items
  2. Inspect available items using intake
  3. Read collection using intake
import intake
from pystac_client import Client

import rasterio as rio
from rasterio.plot import show
STAC_API_URL = "https://staging-stac.delta-backend.com/"

Discover items for time of interest

Use pystac_client to search the STAC for a particular time of interest.

catalog = Client.open(STAC_API_URL)
search = catalog.search(
    max_items=100,  # this is max number of records that will be returned
    limit=25,  # this is the page size of results per page (so based on `max_items` above we will have four pages of results)
    datetime="2001-01-01/2003-01-01",
    sortby="datetime",
)
items = list(search.items())
len(items)
100

intake requires data be casted to pystac.ItemCollection(s) before we can inspect the catalog:

collection = intake.open_stac_item_collection(search.item_collection())

Basic Inspection of Catalog/Collection(s) with instake-stac

# list the `features` of a STAC `FeatureCollection`
list(collection)[:25]
['SSH_2017_minus_1992.cog',
 'NDVI_2000_2009_cog',
 'LST_Night_1km_2000_2009_cog',
 'LST_Day_1km_2000_2009_cog',
 'houston-lst-diff_2000_2019',
 'HOUS_AOD_difference_2000_2019',
 'HOUS_AOD_2000_2009_cog',
 'TREND_MEAN_TWS.cog',
 'MODIS_LC_2001_BD_v2.cog',
 'landcover_Z_f_2001_cog',
 'HOUS_DFAL_urbanization_2001_2019',
 'FLDAS_NOAH01_SoilMoi00_10cm_tavg_C_GL_MA_ANOM200101_20010101',
 'LE07_L2SR_001113_20010104_20200917_02_T2_SR',
 'LT05_L2SP_160029_20010106_02_T1',
 'LT05_L2SP_161029_20010113_02_T2',
 'LE07_L2SR_005113_20010116_20200917_02_T2_SR',
 'LT05_L2SP_126051_20010124_02_T1',
 'LT05_L2SP_161028_20010129_02_T1',
 'LT05_L2SP_161029_20010129_02_T1',
 'LT05_L2SP_127051_20010131_02_T1',
 'FLDAS_NOAH01_SoilMoi00_10cm_tavg_C_GL_MA_ANOM200102_20010201',
 'LT05_L2SP_161028_20010214_02_T1',
 'LT05_L2SP_161029_20010214_02_T1',
 'LT05_L2SP_127051_20010216_02_T1',
 'FLDAS_NOAH01_SoilMoi00_10cm_tavg_C_GL_MA_ANOM200103_20010301']
# list the `assets` of a particular `FeatureCollection.<feature_id>`
list(collection["TREND_MEAN_TWS.cog"])
['cog_default']
# if the entries count is too high to `list` all of them in a notebook session you can "search" children of the `FeatureCollection`
for id, entry in collection.search("TREND_MEAN_TWS").items():
    print(id)
TREND_MEAN_TWS.cog.cog_default
TREND_MEAN_TWS.cog
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=https://landsatlook.usgs.gov/stac-browser/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_SR_B4.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_SR_B5.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_SR_B7.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_ANG.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_MTL.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_MTL.xml>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_MTL.json>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_QA_PIXEL.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_QA_RADSAT.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/001/113/LE07_L2SR_001113_20010104_20200917_02_T2/LE07_L2SR_001113_20010104_20200917_02_T2_thumb_large.jpeg>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=https://landsatlook.usgs.gov/stac-browser/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_SR_B4.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_SR_B5.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_SR_B7.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_ANG.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_MTL.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_MTL.xml>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_MTL.json>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_QA_PIXEL.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_QA_RADSAT.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20010116_20200917_02_T2/LE07_L2SR_005113_20010116_20200917_02_T2_thumb_large.jpeg>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=https://landsatlook.usgs.gov/stac-browser/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_SR_B4.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_SR_B5.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_SR_B7.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_ANG.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_MTL.txt>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_MTL.xml>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_MTL.json>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_QA_PIXEL.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_QA_RADSAT.TIF>
  warnings.warn(
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_stac/catalog.py:512: UserWarning: STAC Asset "type" missing, assuming default type=application/rasterio:
<Asset href=s3://usgs-landsat/collection02/level-2/standard/etm/2001/005/113/LE07_L2SR_005113_20011202_20200917_02_T2/LE07_L2SR_005113_20011202_20200917_02_T2_thumb_large.jpeg>
  warnings.warn(
# or inspect the metadata for a `Feature` outright as `yaml`
collection["TREND_MEAN_TWS.cog"]
cog_default:
  args:
    chunks: {}
    urlpath: s3://veda-data-store-staging/EIS/COG/LIS_TWS_TREND/TREND_MEAN_TWS.cog.tif
  description: Default COG Layer
  direct_access: allow
  driver: intake_xarray.raster.RasterIOSource
  metadata:
    description: Cloud optimized default layer to display on map
    href: s3://veda-data-store-staging/EIS/COG/LIS_TWS_TREND/TREND_MEAN_TWS.cog.tif
    plots:
      geotiff:
        cmap: viridis
        data_aspect: 1
        dynamic: true
        frame_width: 500
        kind: image
        rasterize: true
        x: x
        y: y
    raster:bands:
    - data_type: float64
      histogram:
        buckets:
        - 343348.0
        - 0.0
        - 0.0
        - 0.0
        - 0.0
        - 0.0
        - 0.0
        - 0.0
        - 0.0
        - 93900.0
        count: 11.0
        max: 8.917984008789062
        min: -9999.0
      nodata: 0.0
      offset: 0.0
      sampling: area
      scale: 1.0
      statistics:
        maximum: 8.917984008789062
        mean: -7851.75869965132
        minimum: -9999.0
        stddev: 4105.965757426157
        valid_percent: 100.0
    roles:
    - data
    - layer
    title: Default COG Layer
    type: image/tiff; application=geotiff; profile=cloud-optimized
# now walk from the `FeatureCollection` to a specific asset URL
collection["TREND_MEAN_TWS.cog"]["cog_default"].metadata["href"]
's3://veda-data-store-staging/EIS/COG/LIS_TWS_TREND/TREND_MEAN_TWS.cog.tif'
# view this single cloud optimized geotiff source
cog = rio.open(collection["TREND_MEAN_TWS.cog"]["cog_default"].metadata["href"])
show(cog)

Loading data items with intake-stac

After identifying an asset you want you can then load it into an xarray.DataArray using the .to_dask() function

import hvplot.xarray

da = collection["TREND_MEAN_TWS.cog"]["cog_default"].to_dask().squeeze()
da = da.where(da != -9999)
da.hvplot.image(x="x", y="y")
/srv/conda/envs/notebook/lib/python3.11/site-packages/intake_xarray/raster.py:107: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(ds2.dims),