Local index¶
CatalogIndex is a SQLite index over the catalog for near-instant, offline
repeat searches. Build it yourself (umbra index build), refresh it
incrementally (umbra index update), or fetch the prebuilt weekly snapshot
(umbra index fetch).
A full catalog scan belongs on that snapshot, not on a paged hosted
/search. umbra index fetch downloads it. The flags are --db, --url,
--insecure-url, --if-changed, and --force. --db defaults to
$UMBRA_INDEX_DB or ~/.cache/umbra-py/catalog.db and overwrites an existing
file. --url overrides the release asset URL and must be https unless
--insecure-url (or UMBRA_INSECURE_INDEX_URL=1). --if-changed skips the
body when the published ETag or Last-Modified still matches, then validates
the file and swaps it in atomically; run it while no server holds the index
open. --force applies only with --if-changed: download even when
unchanged, and accept a snapshot much smaller than the current index. The
file is
catalog.db.
catalog.sha256
is the GNU sha256sum manifest beside that file. umbra index fetch checks
catalog.db against it when the manifest is present, and warns and continues
when the manifest is missing (HTTP 403, 404, or 410). A manifest that does
not match is refused. After the fetch, umbra search --local reads the file.
The baked SAR previews travel separately, as an opt-in catalog.thumbs.db
sidecar (umbra index fetch-thumbnails), so the metadata download stays small.
The weekly release stores that file as catalog.thumbs.manifest.json and
hash-named parts under GitHub's 2 GiB asset limit; fetch reassembles them.
A release that still has a single catalog.thumbs.db and no manifest is
downloaded as that file. umbra index fetch-thumbnails --if-changed will
not replace a local sidecar that already has more previews. The weekly job
caches its bake between runs and will not publish fewer matching previews
than the released sidecar, /healthz, or
.github/thumbnail-preview-floor.json, except to seed a release that has
no sidecar yet.
umbra index build quarantines source items that cannot be published: a
datetime before 2018-01-01 or more than a day ahead of the build, a missing
or invalid footprint, no public HTTPS asset href, or any href into the
private prod-prod-processed bucket. Duplicate ids are collapsed, keeping
the latest updated (or datetime), then the record with the most assets.
The dropped rows are listed in umbra index coverage and in
index-validation.json. They are omitted from the coverage percentage and
from the missing list; a sidecar that was never indexed still counts as
missing. --validation points the report at a saved index-validation.json. The weekly
coverage.md
and
coverage.json
on the catalog-index release are that report: indexed vs bucket acquisitions
by prefix (sar-data/tasks/, sar-data/task-data/, open-data/) and by
product, whether pct_indexable meets the 99% publication bar, each
quarantined row with its reason, and the duplicate count.
index-validation.json
is the same list from the build. umbra index validate re-checks the index and,
with --parquet, the geoparquet export, and exits non-zero if any violation
remains. Duplicate ids are counted in SQL and from the parquet id column;
rows are checked a batch at a time. A snapshot exported before asset schema
v2 has one assets field per filename (tens of thousands); v2 uses a fixed
set of product keys, and validation still batches so a v1 file is not loaded
all at once.
Asset schema v2 keys each product by a stable name instead of its filename.
The filename is title (when the asset had none) and umbra-py:filename.
A second file of the same product is gec_2, gec_3, … in filename order;
the canonical *_GEC.tif / *_CPHD.cphd / … name keeps the unsuffixed key.
umbra-py:products stays the uppercase type list (GEC, SICD, …), one
entry per type, lined up with those keys. The collection document and the
parquet file metadata (stac-geoparquet → collections → umbra-open-data)
both carry umbra-py:asset_schema: 2, and so does each item.
| Key | File |
|---|---|
gec |
*_GEC.tif, or a legacy bare <stem>.tif |
csi |
*_CSI.tif (color sub-aperture GeoTIFF) |
sidd |
*_SIDD.nitf |
csi_sidd |
*_CSI-SIDD.nitf (color sub-aperture SIDD) |
sicd |
*_SICD.nitf |
cphd |
*_CPHD.cphd, or a legacy bare <stem>.cphd |
metadata |
*_METADATA.json |
stac_metadata |
*_STAC-METADATA.json |
di_gif |
*_DI-GIF.gif detected-image preview |
di_gif is the only thumbnail-like product file in the open archive. It is
not the baked quicklook PNG (catalog.thumbs.db). A file that matches none
of these is file or file_<ext>.
asset_href, download_asset, and umbra download --asset accept a product
type in any case (GEC), a v2 key including a suffix (gec, gec_2,
csi_sidd, file_tif), or the original filename (umbra-py:filename).
gec is the primary file; gec_2 is the next file of that product. A miss
lists the v2 keys on the item. That lookup works on an index row that still
stores filename or uppercase keys; the API rewrites keys on the way out, and
selection uses the same key space.
The snapshot is what Umbra has published, rebuilt weekly. umbra index info
reports built_at and end (the newest plausible acquisition). As of
2026-10-14 that was a build on 2026-10-08 whose newest plausible acquisition
is 2026-09-13. Gaps of weeks are Umbra's publish cadence.
A datetime after properties.created — or after built_at when created
is missing — that is still inside the quarantine window is kept and flagged
datetime_suspect. It is omitted from end, from rank_by=recency, and
from other latest-pass selections. datetime_suspect_count is how many
rows are flagged. quarantined_future_count is how many source rows the
build removed for a datetime more than a day ahead (the upstream 2035–2036
items); those are not in the index, and the count is what keeps them visible.
created and published are nullable columns (schema version 6; version 5
is datetime_suspect). Opening a version-5 index adds the two columns and
keeps an existing flag. Schema version 7 adds items.canonical (1 for the
copy to keep). Opening a version-6 index adds that column with default 1,
so every existing row stays canonical until the next build links copies.
created is STAC properties.created when the sidecar
has one; legacy
_METADATA.json collects leave it empty. published is the sidecar's S3
LastModified, read from the ListObjectsV2 page the crawl already
fetches. created minus datetime is the lag until Umbra's STAC
generation, not delivery latency, and published is when the object
appeared in the public bucket. Open data is a curated batch-published
subset — see Limitations.
Republished captures¶
Umbra publishes the same collect under more than one prefix, usually
sar-data/task-data/<task_id>/... and
sar-data/tasks/<campaign>/<task_id>/.... Those sidecars are different
STAC items (different ids) that share umbra:task_id and
umbra:collect_id. The index keeps every copy. umbra index build groups
the ones whose datetime values fall within 60 seconds of the earliest in
the cluster and marks one canonical. A wider gap is a different imaging
window and is split. Items with no datetime are not grouped.
The canonical copy is the first of these that differs:
- Newest
properties.created. A missingcreatedloses to any present one. - Higher
processing:version(dotted integers, so5.2.10beats5.2.8), or the version insideprocessing:softwarewhen the field is absent. - Newest acquisition
datetime. - Most product assets present (
GEC,CSI,SIDD,SICD,CPHD). - An href under
sar-data/task-data/. - Lowest STAC id.
duplicate_confidence is high only for a pair that is one task-data
href and one tasks/<campaign> href. Every other linked group is low.
Every item gets umbra-py:canonical (true when it is the copy to keep,
including items that are not in a group). A grouped item also gets
umbra-py:canonical_id, umbra-py:duplicate_ids, and
umbra-py:duplicate_confidence. A non-canonical item has a STAC link
rel: canonical to the preferred sidecar. Every member links to each
sibling with rel: related. The index and the parquet file keep those
sidecar URLs. API responses rewrite both rels to
/collections/umbra-open-data/items/{id}. IANA duplicate means a byte-for-byte copy;
these items differ in id, processor version, and often in which products
they carry, so related is the relation that does not claim the bytes are
interchangeable.
items on /healthz, umbra index info, and the coverage report stays
the row count. unique_captures is how many of those rows are canonical.
index-validation.json records republished_captures
(groups, extra_items, unique_captures, low_confidence_groups).
The coverage gate still counts rows, so linking copies does not change it.
Filter to the canonical copy once this build is what the API and the
parquet file are serving. GET /search?canonical=true and POST /search
with {"umbra-py:canonical": {"eq": true}} (or a top-level
"canonical": true, which wins over the query object) are the same filter.
search_catalog(canonical=True) is the MCP form. Omitting the flag returns
every item. In DuckDB the column name contains a colon, so it is quoted as
"umbra-py:canonical". HTTP, pystac-client, and DuckDB programs are in
Replace your crawler.
A local SQLite index of Umbra acquisitions.
Open (creating the database and schema if needed) with a path, or no path
to use :func:default_index_path. Usable as a context manager, which
commits and closes on exit::
with CatalogIndex() as index:
index.build(area="centerfield") # walk S3 once, persist
for item in index.search(area="centerfield"): # local, instant
print(item.summary())
readonly=True opens an existing file with mode=ro and does not
migrate, relocate previews, or create the file. That is the public host's
serving connection: a write through it fails instead of changing the
shared snapshot. umbra index fetch and other maintenance opens stay
writable (the default).
Copy inline items.thumbnail blobs into item_thumbnails.
Idempotent and safe to run on a schema-7 database that was not rebuilt: each batch commits and checkpoints so the freed overflow pages are reused instead of growing the file by the preview payload. Returns how many rows moved. A no-op when the move has already finished.
classmethod
¶
Download the published prebuilt index and open it.
Umbra has no STAC API, so a fresh install would otherwise crawl the
whole S3 bucket (minutes) before search returns anything. This
fetches the weekly-rebuilt catalog.db snapshot from the project's
rolling catalog-index GitHub release straight to path (default:
:func:default_index_path) and returns an open index over it, so
whole-catalog local search works out of the box -- no crawl. Re-run any
time to refresh; the download is resume-safe and always overwrites the
existing file. url overrides the release asset location (e.g. to
pull from a fork or a mirror).
Abort the running statement once cancelled is set.
interrupt is forgotten when no statement is running, so a deadline
that fires before the query starts would not stick. The progress
handler re-checks the flag. The server arms this on the worker that
owns the connection and calls :meth:interrupt_query from the event
loop when the request deadline passes.
Remove the progress handler so commit is not aborted with the query.
Abort the statement on this connection. Safe to call from another thread.
How many rows are the canonical copy of their capture.
A republished collect counts once. Items not in a duplicate group are
canonical, and so is every row of an index that has not been linked
yet (the column defaults to 1), so this equals :meth:__len__ until
:func:umbra_py.validate.apply_duplicate_links runs.
While preview blobs are still inline, canonical sits after the
blob and this returns the row count instead of scanning overflow.
Upsert one item (does not commit). Returns False (and skips) an
item with no sidecar href, since the href is the row's identity.
On a re-index (same href) every STAC-derived column is refreshed, but the
baked place label is deliberately left untouched -- it is a derived
denormalization keyed on the footprint, not on the STAC document, so an
umbra index update that re-reads the sidecar must not clear a label an
umbra index bake already computed. created and published are
refreshed with the other STAC-derived columns: they come from the crawl,
not from a bake.
Walk the live catalog and persist every matching item.
Accepts the same keyword filters as :meth:UmbraCatalog.search
(bbox, start, end, area, product_types, limit,
max_per_task) to scope the build. Pass no filters to index the
whole bucket -- the one-time crawl that makes every later
search(local) instant. Idempotent: re-running refreshes existing
rows and adds new ones, so an index can be grown incrementally.
progress, if given, is called with the running count of items
written -- a full-bucket build lists every task and takes a while, so
the CLI uses this to show a live tally. Returns how many rows the walk
upserted, which can be more than :meth:__len__ afterwards: known-bad
source items are quarantined and duplicate ids collapsed (see
:func:umbra_py.validate.sanitize_index). A violation that survives
that pass raises :class:~umbra_py.exceptions.IndexValidationError.
Cheaply refresh the index by re-walking only recent acquisitions.
A full :meth:build fetches a sidecar for every acquisition in
scope; on an index only days old, almost all of that work re-reads
unchanged data. update instead derives an acquisition-date lower
bound from what the index already holds -- the maximum indexed
acq_date minus overlap_days -- and passes it as start to the
live walk. The walk prunes older acquisitions' sidecar fetches (see
:meth:UmbraCatalog.search), so a weekly refresh reads only the new
passes rather than the whole catalog, and every returned row is upserted
exactly as :meth:build does. It is the incremental companion to
:meth:from_release: fetch the weekly snapshot once, then update to
catch acquisitions published since.
The bound is on acquisition date, not publish date, so a scene
acquired before the bound but published after the last build is not
picked up. overlap_days (default 1) re-scans a little past the newest
indexed date to catch the common near-real-time lag; widen it (or run a
full :meth:build) when completeness over back-dated late arrivals
matters. An empty index has no bound to derive, so update falls back
to a full build (start=None). Pass since to force a specific
lower bound instead of deriving one.
Extra keyword filters (bbox, area, product_types, limit,
max_per_task) scope the walk exactly as :meth:build does -- pass
the same scope the index was built with. start may not be passed
(the bound is what update computes); use since to override it.
Returns an :class:UpdateResult tallying new vs. refreshed rows.
Reverse-geocode each item's footprint once and cache the place label.
Reverse geocoding is rate-limited (OpenStreetMap's Nominatim allows one
request per second) and, until now, ran at render time -- so labelling
a whole catalog in a map or the umbra demo explorer was impractical.
This bakes the label in ahead of time: for every indexed acquisition that
has a footprint but no label yet, it resolves the footprint centroid to a
human place name (e.g. "Reykjavík, Iceland") and stores it in the
place column, so every later search/get yields it on
:attr:UmbraItem.place for free -- turning the shared index into a
labelled demo backend.
It is idempotent: only items whose place is still NULL are
geocoded, so a re-run labels just what was added since (and an item whose
geocode returns nothing is retried on the next run rather than marked).
limit caps how many geocode calls this run makes (to bake a large
catalog in bounded batches); zoom is the Nominatim address
granularity (3 = country ... 10 = city ... 18 = building). progress,
if given, is called with the running count of calls made.
With by_site the bake geocodes once per site rather than once per
acquisition: Umbra files every pass over a site under one task, so the
passes sharing a task and a :data:_SITE_CELL_DEGREES cell are resolved
together from their mean centroid and all take that one label (see
:func:_site_groups). A repeat-imaged catalog is mostly repeat passes,
so this collapses the throttled ~1 req/s call count by roughly the
average passes-per-site -- which is what makes labelling a whole
catalog (and shipping it pre-labelled in the published snapshot) practical
rather than an overnight job. The label is a coarse place name for a
footprint a few km across, so one per site is the same answer per-item
geocoding would converge on; the default stays per-item.
geocoder is an injectable (lat, lon) -> label | None callable; the
default wraps :func:umbra_py.viz._reverse_geocode, which self-throttles
to Nominatim's policy and caches in-process. Passing a stand-in keeps the
whole path offline-testable. Returns the number of items newly labelled.
bake_thumbnails(renderer=None, *, asset='GEC', max_size=PUBLISHED_THUMBNAIL_SIZE, limit=None, newest_first=False, progress=None)
Render a small SAR quicklook per acquisition once and cache it.
Every gallery, umbra demo preview and umbra serve quicklook
otherwise re-streams a scene's cloud-optimized GeoTIFF overview from S3
at render time, so the first view of a whole catalog is network-bound
and slow. This bakes the preview ahead of time: for every indexed
acquisition that carries the asset (default GEC) but no thumbnail
yet, it renders a max_size-pixel PNG and stores the bytes in the
additive thumbnail column, so a later
:meth:get_thumbnail -- and the GET /artifacts/thumbnail/{id}.png
server endpoint that wraps it -- is an instant, offline file read.
It is the render-side sibling of :meth:bake_places, and shares its
discipline: idempotent (items whose thumbnail is still NULL,
or whose recorded thumbnail_size is smaller than this call's
max_size, are rendered -- so a weekly bump from 128 px to 512 px
actually upgrades the sidecar instead of leaving the old bake in
place), with limit capping how many are rendered this call (to bake
a large catalog in bounded batches). An item whose asset can't be
rendered (no asset, a decode error, a network blip) is skipped --
its thumbnail stays as it was so a later run retries it -- and one bad
scene never aborts the batch, mirroring the gallery contact sheet.
newest_first chooses which acquisitions a capped run spends its
budget on: by default the batch is taken in href order, which is
arbitrary with respect to time, so a bounded bake over a whole catalog
leaves the freshest scenes -- the ones a demo or a monitoring view opens
on -- unbaked the longest. With newest_first=True the most recently
acquired are rendered first, which is what makes a per-run cap a
priority rather than a lottery. Items with no acquisition date sort
last, as they carry no claim to being recent.
asset and max_size are recorded beside the bytes
(thumbnail_asset / thumbnail_size, read back by
:meth:get_preview), because a preview is only interchangeable with the
picture a caller asked for when the two are of the same product: without
the record every consumer had to assume the default bake. They describe
what this call was asked to render, so an injected renderer that
ignores them records a claim it did not honour.
renderer is an injectable (UmbraItem) -> bytes | None callable
returning PNG bytes (or None to skip); the default wraps
:func:umbra_py.viz._thumbnail_png, which streams only the overview for
max_size and needs the viz extra. Passing a stand-in keeps the
whole path offline-testable. progress, if given, is called with the
running count of items processed. Returns the number newly thumbnailed.
Return the baked quicklook PNG for this STAC id, or None.
The retrieval complement to :meth:bake_thumbnails: an
idx_items_id-backed point lookup for the cached preview bytes (never
loaded by :meth:search/:meth:get, which would bloat every
:class:~umbra_py.models.UmbraItem with a PNG). None means the id is
absent or its thumbnail has not been baked -- both mean "render it
instead". If two sidecars share an id, the first by href order wins,
as in :meth:get.
Which of ids have a baked PNG, without reading the bytes.
One indexed lookup against item_thumbnails for the page, then a
second only for ids that are still missing, against the legacy inline
items.thumbnail column (length(thumbnail) IS NOT NULL, the
same test as :meth:_thumbnail_row). Neither statement reads the
PNG. An empty input does not query. Duplicate ids are looked up once.
The caller is a public search page, at most 500 ids, under SQLite's older 999-variable cap, so this is one statement rather than a loop.
Return the baked quicklook and what it is a picture of, or None.
The provenance-carrying form of :meth:get_thumbnail, for the one
consumer that cannot treat a preview as interchangeable pixels:
umbra describe --preview hands the picture to a vision model, so a
reading of a CSI bake is not a reading of the GEC one that was
asked for. It reads the same row, so the extra provenance costs nothing;
a preview baked before the index recorded it reports None for both
fields (see :class:BakedPreview) rather than claiming the default.
Write every baked thumbnail to a transportable sidecar database.
Baking a quicklook costs a cloud-optimized GeoTIFF overview streamed
from S3 per acquisition, so it is the one derived artifact nobody should
recompute: this writes the bytes already baked into dest
(:data:_THUMBS_SCHEMA -- href, id, png) so they can be
published beside catalog.db and merged into any other index with
:meth:import_thumbnails. That is what makes the weekly publish
incremental: each run re-imports the previous sidecar first and then
bakes only the acquisitions added since, instead of re-streaming the
whole archive every Monday.
The sidecar is deliberately separate from the index (rather than a
published catalog.db carrying its thumbnail column) because the
pixels are far larger than the metadata and not every caller wants them.
Writing is an upsert into an existing file, so exporting twice is safe.
Each row carries the bake's asset and size beside the bytes, so
the receiving index knows what it merged rather than assuming it (an
older sidecar is widened in place before writing). Returns the number of
thumbnails written.
Merge a thumbnail sidecar (:meth:export_thumbnails) into this index.
The consume side of the published catalog.thumbs.db: it fills the
thumbnail column for the acquisitions the sidecar covers, so
umbra serve's GET /artifacts/thumbnail/{id}.png, the umbra
demo preview and a --local gallery all read local bytes without a
single COG range read. Rows the index does not hold are ignored (a
sidecar built from a newer crawl is not an error).
A local bake is kept rather than clobbered -- with one exception the
sidecar's own record makes safe: when both sides say what they are and
the incoming preview is a larger bake of the same product, it wins.
That is the case the published sidecar creates, since it is baked at
:data:PUBLISHED_THUMBNAIL_SIZE (512 px): a merge
used to keep whichever arrived first, which made the resolution of a
preview a fact about the order two commands were run in. Where either
side is unrecorded the two are not comparable and the local bake stays.
overwrite=True replaces unconditionally, as before. Returns the
number of thumbnails applied.
The merge is one transaction: a failure part-way rolls back every row
it applied, not only the ones SQLite itself rolls back on database or
disk is full. It runs in place,
so it needs about twice the sidecar's PNG payload free under WAL; the
boot refresh uses :func:merge_thumbnails_atomically instead.
Whether area would select at least one indexed task.
The same filter as :meth:search (substring, or the fuzzy token
match), stopped at the first row. Closing the generator finishes
the SQL cursor before the caller's own search runs.
search(*, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, canonical=None, limit=None, max_per_task=None)
Yield indexed items matching the filters.
Same semantics as :meth:UmbraCatalog.search, answered from local SQL.
Only returns acquisitions already present in the index; build or refresh
it with :meth:build first. fuzzy=True widens area to the same
deterministic token-wise match the live path uses
(:func:umbra_py.fuzzy.matching_tasks): the distinct task names are
read from the index and matched in Python, so both backends agree.
intersects (the exterior-ring form from
:func:umbra_py._geometry.parse_geometry) keeps only items whose
footprint intersects the polygon. Its bounding box is pushed into SQL as
a cheap prefilter and the exact polygon test then runs in Python on each
candidate, so the result matches :meth:UmbraCatalog.search exactly.
The acquisition-property filters (polarizations, min_incidence /
max_incidence, max_resolution) mean exactly what they do on
:meth:UmbraCatalog.search. They are read from each item's stored STAC
document (already reconstructed here) and applied in Python via
:meth:UmbraItem.matches_filters, the same way the polygon test runs, so
both backends agree without a schema change.
canonical keeps the preferred copy of a republished collect
(True) or only the other copies (False). None, the
default, returns every row. The flag is the canonical column
written when the index is linked; an unlinked row defaults to
canonical. See :mod:umbra_py.duplicates.
A bbox (or the bbox of intersects) on a read-only open can be
narrowed by the attached catalog.rtree.db sidecar when that file
matches this catalog and the box is selective
(:func:_rtree_selective). CROSS JOIN is not reordered, so the
R-tree stays the outer loop even when a product filter matches almost
every row. The R-tree stores coordinates as float32 rounded outward,
so every candidate still has to pass the same column overlap the scan
uses. A wide first page, and every writable open, uses that column
predicate directly. The ordered index can then stop after limit.
A deep page, or a limit the box cannot fill, uses the tree so that
walk does not run to the last late-sorting hit.
search_live(catalog=None, *, overlap_days=1, refresh=True, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, canonical=None, limit=None, max_per_task=None)
Read-through search: the index for the bulk, a live delta for what's new.
:meth:search is instant but only returns what the index already holds;
:meth:UmbraCatalog.search is always current but re-walks the whole
bucket every call. This is the transparent middle the codebase analysis
named as "make the index the default path": the index answers the whole
query from local SQL, and a bounded
live walk covers only acquisitions at or after the index's freshness
horizon -- its maximum indexed acq_date minus overlap_days -- so
the walk fetches sidecars only for recent passes rather than the whole
catalog (the same pruning :meth:update relies on). The two streams are
merged in (task, acq_date, datetime, id, href) order and de-duplicated by
sidecar href, so an acquisition the index already knows is never yielded
twice; the result is what a single fresh search would return, without
paying for a full crawl.
With refresh=True (the default) each genuinely new acquisition the
live delta discovers is upserted into the index as it is yielded -- the
"read-through cache warms" behavior -- so the next call needs an even
smaller (often empty) walk. Set refresh=False to leave the index
untouched (e.g. when it is a shared read-only snapshot); a read-only
database also disables warming automatically rather than failing the
search. The write-back is committed only when at least one new row was
added, and built_at is re-stamped then, exactly as :meth:update.
The keyword filters (bbox, start, end, product_types,
area, fuzzy, the acquisition-property filters polarizations /
min_incidence / max_incidence / max_resolution, limit,
max_per_task) mean exactly what they
do on :meth:search / :meth:UmbraCatalog.search; start bounds both
streams (the live delta never walks older than the caller asked for, even
when the freshness horizon is older). overlap_days (default 1)
re-scans a little past the newest indexed date to catch near-real-time
publish lag; the bound is on acquisition date, so a back-dated late
arrival still wants a widened overlap or a full :meth:build. An empty
index has no horizon, so the live walk covers the caller's full window
(and, with refresh, this doubles as a first :meth:build).
rank_sites(*, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, top=20, min_passes=2, rank_by='passes', active_since=None, active_before=None, first_since=None, first_before=None, max_revisit_days=None, median_revisit_days=None, min_span_days=None, max_span_days=None)
Rank the most repeat-imaged sites across the whole index.
The index-native form of :func:umbra_py.coverage.rank_site_coverage,
and the reason it exists: that function ranks whatever pool it is handed,
so umbra sites --local capped the pool at --limit acquisitions and
a site with many passes just outside the first --limit rows read as
shallower than it is. Umbra files every pass of a site under one task, so a
site's depth is a GROUP BY task the index can answer over its entire
contents -- no pool cap, so a deeply-imaged site is ranked by all its
passes rather than by the arbitrary window a limit happened to admit.
The ranking is :func:umbra_py.showcase.select_featured_sites' exactly --
dated passes per task, most first, task name breaking ties, keeping those
with at least min_passes -- so this, umbra sites and the featured
gallery cannot disagree about what "most repeat-imaged" means. Each site
is summarised by :func:umbra_py.coverage.site_coverage; the result is
the top best, best-first.
The filters mean exactly what they do on :meth:search. The
SQL-expressible ones (bbox, start / end, area / fuzzy,
product_types) are counted directly in a GROUP BY, so only the top
tasks' documents are then read to summarise -- cheap even whole-archive.
The polygon (intersects) and acquisition-property (polarizations,
min_incidence / max_incidence / max_resolution) filters run per
item in Python, so when any is set this ranks the full uncapped matching
stream instead: still whole-archive (no limit), identical to the pool
path, just without the cap this method exists to remove.
rank_by is one of :data:umbra_py.coverage.SITE_RANKINGS. "passes"
(the default) is answerable in SQL -- a COUNT(*) per task orders the
candidates, so only the top top tasks' documents are read.
"comparable" ranks by analysable depth (comparable_passes), which
depends on each pass's polarization inside the document JSON and so is not a
COUNT: every qualifying task's documents are read and summarised, then
re-ranked by the analysable subset and truncated to top. The temporal
rankings "recency" (newest dated pass first), "span" (longest
observation baseline first) and "cadence" (tightest typical revisit gap
first) are likewise not the SQL COUNT order -- the top-by-count tasks are
not the top-by-recency, -span or -cadence -- so they take the same
read-every-task-then-re-rank path (the raw-count LIMIT is dropped when the
ranking is not "passes", so a recently-active, long-baseline or
tightly-revisited site outside the raw top-top is not truncated before it
can be promoted). That is heavier than the raw path (it reads every
repeat-imaged task rather than the top top), but still whole-archive and
correct. All five rankings share :func:umbra_py.coverage._rank_sort_key, and
the temporal trio reads the last / span_days / median_revisit_days
off the same summarised :class:SiteCoverage the pool path reduces from the
passes, so this and the pool path order every ranking identically.
min_passes gates on the same depth rank_by ranks by
(:func:umbra_py.coverage._min_passes_depth): under "comparable" the
HAVING COUNT(*) >= min_passes clause is a superset pre-filter (comparable
depth is never above the raw count) and the true floor on comparable_passes
is applied in Python before the re-rank, so --rank-by comparable
--min-passes N returns only sites whose differenceable series is at least
N passes deep. Under "passes" the SQL floor is exact and this and the
pool path qualify a site identically.
active_since keeps only sites still imaged on or after that date -- a
recency filter on each site's newest dated pass. It is answered in the
same HAVING clause the pass-count floor is (MAX(acq_date) >= ?), so
it costs nothing beyond the group already computed and is exact under either
ranking (a site's latest pass is independent of the polarization grouping
"comparable" re-ranks by). Whole-archive like the rest of this method,
and byte-identical to the pool path's :func:umbra_py.coverage.rank_site_coverage
recency gate. It is orthogonal to start / end (those bound which
rows the GROUP BY counts; this selects whole sites by their latest and
keeps every counted pass in the summary). None applies no recency filter.
active_before is the complement -- keep only sites whose newest dated
pass is on or before that date (a dormant series), answered by the twin
MAX(acq_date) <= ? clause in the same HAVING, so with active_since
the two bound the site's latest pass to a window. A span expression snaps to
its last day (symmetric with end). Byte-identical to the pool path's
upper-recency gate. None applies no upper bound.
first_since / first_before are the onset (first-seen) twins of the
active_* pair -- they gate each site's earliest dated pass rather than
its newest, selecting newly-appeared series (first_since, first pass on
or after the date) and long-established ones (first_before, first pass
on or before it); set together they bound the onset to a window. Like the
recency pair they are pure SQL aggregates, answered by MIN(acq_date) >= ? /
MIN(acq_date) <= ? clauses in the same HAVING -- costing nothing beyond
the group already computed and exact under either ranking (a site's earliest
pass, like its latest, does not depend on the polarization grouping
"comparable" re-ranks by), so unlike the cadence and span filters they do
not force the raw-count LIMIT to be dropped. MIN skips NULL
acq_date, so a group with no dated pass yields NULL and is dropped
(NULL >= ? / NULL <= ? is never true), matching select_featured_sites
dropping a site with no datable pass. first_before snaps a span expression
to its last day (symmetric with active_before / end). Byte-identical to
the pool path's onset gate. None applies no onset filter.
max_revisit_days keeps only sites revisited at least this often -- a
cadence filter on each site's worst-case revisit gap. Unlike the recency
and depth filters it is not a SQL aggregate: the worst gap is between
consecutive passes (and, under "comparable", over the largest
single-polarization subset the document JSON defines), which no HAVING
clause on a column expresses. It is applied in Python on the same per-task
items this method already reads to summarise, using the same
:func:umbra_py.coverage._passes_cadence the pool path uses -- so the two are
byte-identical -- and when it is set the raw-count SQL LIMIT is dropped
(as it is for the comparable ranking), because a site that passes the cadence
filter but sits just outside the raw top-top must not be truncated before
the filter runs. Gated on the same depth rank_by measures (the analysable
series' cadence under "comparable"), orthogonal to the recency filters,
and dropping a site with fewer than two passes in the gated series. None
applies no cadence filter; a non-positive value is a ValueError.
median_revisit_days is the typical-cadence twin of max_revisit_days --
keep only sites whose median revisit gap is at most this many days (a site
usually imaged often, tolerating the odd long outage the worst-case bound
rejects). Like the worst-case filter it is not a SQL aggregate -- a median of
consecutive gaps is no more a HAVING clause than a max of them -- so it is
applied in Python on the same per-task items this method already reads, using the
same :func:umbra_py.coverage._passes_median_revisit the pool path uses (so the
two are byte-identical), gated on the analysable subset under "comparable",
and it drops the raw-count SQL LIMIT when set (as the cadence and span
filters do) so a usually-tight site outside the raw top-top is promoted rather
than truncated before the filter runs. A site with fewer than two passes in the
gated series has no measurable cadence and is dropped. None applies no
typical-cadence filter; a non-positive value is a ValueError.
min_span_days keeps only sites imaged over at least this long -- a
baseline filter on each site's observation span (whole days from first
dated pass to last). Like the cadence filter it is applied in Python rather
than SQL: under "comparable" the span is over the largest
single-polarization subset the document JSON defines (not a column
expression), and keeping the two rankings byte-identical is worth more than a
SQL HAVING on the raw case alone, so it uses the same
:func:umbra_py.coverage._passes_span the pool path does on the same per-task
items this method already reads, and drops the raw-count SQL LIMIT when
set (as the comparable ranking and the cadence filter do) so a long-baseline
site outside the raw top-top is promoted rather than truncated before the
filter runs. Gated on the same depth rank_by measures (the analysable
series' span under "comparable"), orthogonal to the recency and cadence
filters, and dropping a site with fewer than two passes in the gated series.
None applies no span filter; a non-positive value is a ValueError.
max_span_days is the upper twin of min_span_days -- keep only sites
imaged over at most this long (a short-lived series), the complement of the
floor, and set with it a window bounding each site's baseline
(min_span_days <= span <= max_span_days), as active_since /
active_before bound the newest pass. Like the floor it is applied in Python
with the same :func:umbra_py.coverage._passes_max_span the pool path uses (so
the two paths stay byte-identical), gated on the analysable subset under
"comparable", drops the raw-count SQL LIMIT when set, and drops a site
with no measurable span so the window admits only a confirmed baseline.
None applies no span ceiling; a non-positive value is a ValueError.
Return the indexed item with this STAC id, or None if absent.
The keyed point-lookup complement to :meth:search's listing: where
filtering a full search by id would scan the ordered result set,
this is an idx_items_id-backed lookup, so it stays fast as the
published catalog.db snapshot grows. STAC ids are unique per
acquisition in Umbra's catalog; in the unlikely event two sidecars
share an id, the first by href order is returned deterministically.
Return the indexed item with this STAC sidecar URL, or None.
The href-keyed complement to :meth:get. MCP tools that take a
stac_href from search_catalog look the snapshot up here so
they do not re-fetch a live sidecar that may have grown extra
assets or different product filenames than the index the search
already answered from.
Return items whose asset hrefs contain this filename or stem.
name is a product filename (<stem>_GEC.tif, <stem>_SICD.nitf)
or the bare acquisition stem YYYY-MM-DD-HH-MM-SS_UMBRA-NN. A
filename matches an asset href or key with that basename. A stem
matches when it is a path segment of an asset href (or of the sidecar
href) or the start of a product filename. Ordered by sidecar href.
An unrecognised name returns an empty list.
The published index stores the public href, and that href is the only
place the layout (sar-data/tasks/, sar-data/task-data/, or
open-data/) is recorded — the filename does not carry it. _ in
the stem is escaped so it is not a LIKE wildcard.
Number of distinct STAC item ids (rows can repeat an id when one
collect is published under both tasks/ and task-data/).
Summary counts for umbra index info: item count, acquisition-date
span, number of distinct tasks, how many items carry a baked place label
(labeled; see :meth:bake_places), how many carry a baked quicklook
thumbnail (thumbnailed; see :meth:bake_thumbnails), and the date the
index was last built (built_at, None for an index written before
build stamping).
end is the newest acquisition date that is not
datetime_suspect (a datetime after properties.created, or after
built_at when created is missing). Those rows stay in items.
datetime_suspect_count is how many are flagged.
quarantined_future_count is how many source rows the last build
dropped for a datetime more than a day ahead (see
:func:umbra_py.validate.sanitize_index); they are not in the index.
Acquisition-date span (start, end), the same dates as :meth:stats.
end excludes datetime_suspect rows once preview blobs have left
the row. While they are still inline, datetime_suspect sits after
the blob, so this uses MAX(acq_date) and does not walk overflow.
dataclass
¶
Outcome of an incremental :meth:CatalogIndex.update.
added counts acquisitions whose href was not already in the index;
refreshed counts those whose existing row was replaced; scanned is
their sum (every item the scoped walk yielded). start is the
acquisition-date lower bound the walk used -- None when the index was
empty and update fell back to a full build.
Where the index lives by default.
$UMBRA_INDEX_DB overrides everything; otherwise it sits under the XDG
cache dir ($XDG_CACHE_HOME or ~/.cache) at
umbra-py/catalog.db.
Where the baked-thumbnail sidecar lives by default.
It sits beside the index (catalog.db -> catalog.thumbs.db) so the
two travel together while staying separate files: the pixels are opt-in and
an order of magnitude larger than the metadata, so keeping them out of
catalog.db is what lets the published index stay small (the same split
:func:umbra_py.embed.default_scene_embed_path makes for vectors). Pass
index_path to derive the sibling name from a non-default index location.
Download the published baked-thumbnail sidecar.
The weekly index workflow bakes a quicklook per acquisition and publishes it
on the rolling catalog-index release beside catalog.db /
catalog.pmtiles, so a fresh install gets scene previews without
streaming a cloud-optimized GeoTIFF overview per acquisition -- the
thumbnail sibling of :meth:CatalogIndex.from_release and
:func:umbra_py.pmtiles.fetch_prebuilt_pmtiles. A current release ships
catalog.thumbs.manifest.json plus hash-named parts (GitHub rejects one
asset of 2 GiB or more); those parts are concatenated back to
catalog.thumbs.db. A release with no manifest is the legacy single
catalog.thumbs.db asset. This fetches the sidecar to dest (default:
:func:default_thumbs_path) and returns its path;
:meth:CatalogIndex.import_thumbnails merges it into a local index. Re-run
any time to refresh; the download is resume-safe and always overwrites the
existing file. url overrides the release asset location (a sidecar, or
a catalog.thumbs.manifest.json).