Local index¶
CatalogIndex is a SQLite index over the catalog for near-instant, offline
repeat searches. Build it yourself (umbra index build), refresh it
incrementally (umbra index update), or fetch the prebuilt weekly snapshot
(umbra index fetch).
The baked SAR previews travel separately, as an opt-in catalog.thumbs.db
sidecar (umbra index fetch-thumbnails), so the metadata download stays small.
A local SQLite index of Umbra acquisitions.
Open (creating the database and schema if needed) with a path, or no path
to use :func:default_index_path. Usable as a context manager, which
commits and closes on exit::
with CatalogIndex() as index:
index.build(area="centerfield") # walk S3 once, persist
for item in index.search(area="centerfield"): # local, instant
print(item.summary())
classmethod
¶
Download the published prebuilt index and open it.
Umbra has no STAC API, so a fresh install would otherwise crawl the
whole S3 bucket (minutes) before search returns anything. This
fetches the weekly-rebuilt catalog.db snapshot from the project's
rolling catalog-index GitHub release straight to path (default:
:func:default_index_path) and returns an open index over it, so
whole-catalog local search works out of the box -- no crawl. Re-run any
time to refresh; the download is resume-safe and always overwrites the
existing file. url overrides the release asset location (e.g. to
pull from a fork or a mirror).
Upsert one item (does not commit). Returns False (and skips) an
item with no sidecar href, since the href is the row's identity.
On a re-index (same href) every STAC-derived column is refreshed, but the
baked place label is deliberately left untouched -- it is a derived
denormalization keyed on the footprint, not on the STAC document, so an
umbra index update that re-reads the sidecar must not clear a label an
umbra index bake already computed.
Walk the live catalog and persist every matching item.
Accepts the same keyword filters as :meth:UmbraCatalog.search
(bbox, start, end, area, product_types, limit,
max_per_task) to scope the build. Pass no filters to index the
whole bucket -- the one-time crawl that makes every later
search(local) instant. Idempotent: re-running refreshes existing
rows and adds new ones, so an index can be grown incrementally.
progress, if given, is called with the running count of items
written -- a full-bucket build lists every task and takes a while, so
the CLI uses this to show a live tally. Returns the total written.
Cheaply refresh the index by re-walking only recent acquisitions.
A full :meth:build fetches a sidecar for every acquisition in
scope; on an index only days old, almost all of that work re-reads
unchanged data. update instead derives an acquisition-date lower
bound from what the index already holds -- the maximum indexed
acq_date minus overlap_days -- and passes it as start to the
live walk. The walk prunes older acquisitions' sidecar fetches (see
:meth:UmbraCatalog.search), so a weekly refresh reads only the new
passes rather than the whole catalog, and every returned row is upserted
exactly as :meth:build does. It is the incremental companion to
:meth:from_release: fetch the weekly snapshot once, then update to
catch acquisitions published since.
The bound is on acquisition date, not publish date, so a scene
acquired before the bound but published after the last build is not
picked up. overlap_days (default 1) re-scans a little past the newest
indexed date to catch the common near-real-time lag; widen it (or run a
full :meth:build) when completeness over back-dated late arrivals
matters. An empty index has no bound to derive, so update falls back
to a full build (start=None). Pass since to force a specific
lower bound instead of deriving one.
Extra keyword filters (bbox, area, product_types, limit,
max_per_task) scope the walk exactly as :meth:build does -- pass
the same scope the index was built with. start may not be passed
(the bound is what update computes); use since to override it.
Returns an :class:UpdateResult tallying new vs. refreshed rows.
Reverse-geocode each item's footprint once and cache the place label.
Reverse geocoding is rate-limited (OpenStreetMap's Nominatim allows one
request per second) and, until now, ran at render time -- so labelling
a whole catalog in a map or the umbra demo explorer was impractical.
This bakes the label in ahead of time: for every indexed acquisition that
has a footprint but no label yet, it resolves the footprint centroid to a
human place name (e.g. "Reykjavík, Iceland") and stores it in the
place column, so every later search/get yields it on
:attr:UmbraItem.place for free -- turning the shared index into a
labelled demo backend.
It is idempotent: only items whose place is still NULL are
geocoded, so a re-run labels just what was added since (and an item whose
geocode returns nothing is retried on the next run rather than marked).
limit caps how many geocode calls this run makes (to bake a large
catalog in bounded batches); zoom is the Nominatim address
granularity (3 = country ... 10 = city ... 18 = building). progress,
if given, is called with the running count of calls made.
With by_site the bake geocodes once per site rather than once per
acquisition: Umbra files every pass over a site under one task, so the
passes sharing a task and a :data:_SITE_CELL_DEGREES cell are resolved
together from their mean centroid and all take that one label (see
:func:_site_groups). A repeat-imaged catalog is mostly repeat passes,
so this collapses the throttled ~1 req/s call count by roughly the
average passes-per-site -- which is what makes labelling a whole
catalog (and shipping it pre-labelled in the published snapshot) practical
rather than an overnight job. The label is a coarse place name for a
footprint a few km across, so one per site is the same answer per-item
geocoding would converge on; the default stays per-item.
geocoder is an injectable (lat, lon) -> label | None callable; the
default wraps :func:umbra_py.viz._reverse_geocode, which self-throttles
to Nominatim's policy and caches in-process. Passing a stand-in keeps the
whole path offline-testable. Returns the number of items newly labelled.
bake_thumbnails(renderer=None, *, asset='GEC', max_size=256, limit=None, newest_first=False, progress=None)
Render a small SAR quicklook per acquisition once and cache it.
Every gallery, umbra demo preview and umbra serve quicklook
otherwise re-streams a scene's cloud-optimized GeoTIFF overview from S3
at render time, so the first view of a whole catalog is network-bound
and slow. This bakes the preview ahead of time: for every indexed
acquisition that carries the asset (default GEC) but no thumbnail
yet, it renders a max_size-pixel PNG and stores the bytes in the
additive thumbnail column, so a later
:meth:get_thumbnail -- and the GET /artifacts/thumbnail/{id}.png
server endpoint that wraps it -- is an instant, offline file read.
It is the render-side sibling of :meth:bake_places, and shares its
discipline: idempotent (only items whose thumbnail is still
NULL are rendered, so a re-run bakes just what was added since), with
limit capping how many are rendered this call (to bake a large
catalog in bounded batches). An item whose asset can't be rendered (no
asset, a decode error, a network blip) is skipped -- its thumbnail
stays NULL so a later run retries it -- and one bad scene never
aborts the batch, mirroring the gallery contact sheet.
newest_first chooses which acquisitions a capped run spends its
budget on: by default the batch is taken in href order, which is
arbitrary with respect to time, so a bounded bake over a whole catalog
leaves the freshest scenes -- the ones a demo or a monitoring view opens
on -- unbaked the longest. With newest_first=True the most recently
acquired are rendered first, which is what makes a per-run cap a
priority rather than a lottery. Items with no acquisition date sort
last, as they carry no claim to being recent.
asset and max_size are recorded beside the bytes
(thumbnail_asset / thumbnail_size, read back by
:meth:get_preview), because a preview is only interchangeable with the
picture a caller asked for when the two are of the same product: without
the record every consumer had to assume the default bake. They describe
what this call was asked to render, so an injected renderer that
ignores them records a claim it did not honour.
renderer is an injectable (UmbraItem) -> bytes | None callable
returning PNG bytes (or None to skip); the default wraps
:func:umbra_py.viz._thumbnail_png, which streams only the overview for
max_size and needs the viz extra. Passing a stand-in keeps the
whole path offline-testable. progress, if given, is called with the
running count of items processed. Returns the number newly thumbnailed.
Return the baked quicklook PNG for this STAC id, or None.
The retrieval complement to :meth:bake_thumbnails: an
idx_items_id-backed point lookup for the cached preview bytes (never
loaded by :meth:search/:meth:get, which would bloat every
:class:~umbra_py.models.UmbraItem with a PNG). None means the id is
absent or its thumbnail has not been baked -- both mean "render it
instead". If two sidecars share an id, the first by href order wins,
as in :meth:get.
Return the baked quicklook and what it is a picture of, or None.
The provenance-carrying form of :meth:get_thumbnail, for the one
consumer that cannot treat a preview as interchangeable pixels:
umbra describe --preview hands the picture to a vision model, so a
reading of a CSI bake is not a reading of the GEC one that was
asked for. It reads the same row, so the extra provenance costs nothing;
a preview baked before the index recorded it reports None for both
fields (see :class:BakedPreview) rather than claiming the default.
Write every baked thumbnail to a transportable sidecar database.
Baking a quicklook costs a cloud-optimized GeoTIFF overview streamed
from S3 per acquisition, so it is the one derived artifact nobody should
recompute: this writes the bytes already baked into dest
(:data:_THUMBS_SCHEMA -- href, id, png) so they can be
published beside catalog.db and merged into any other index with
:meth:import_thumbnails. That is what makes the weekly publish
incremental: each run re-imports the previous sidecar first and then
bakes only the acquisitions added since, instead of re-streaming the
whole archive every Monday.
The sidecar is deliberately separate from the index (rather than a
published catalog.db carrying its thumbnail column) because the
pixels are far larger than the metadata and not every caller wants them.
Writing is an upsert into an existing file, so exporting twice is safe.
Each row carries the bake's asset and size beside the bytes, so
the receiving index knows what it merged rather than assuming it (an
older sidecar is widened in place before writing). Returns the number of
thumbnails written.
Merge a thumbnail sidecar (:meth:export_thumbnails) into this index.
The consume side of the published catalog.thumbs.db: it fills the
thumbnail column for the acquisitions the sidecar covers, so
umbra serve's GET /artifacts/thumbnail/{id}.png, the umbra
demo preview and a --local gallery all read local bytes without a
single COG range read. Rows the index does not hold are ignored (a
sidecar built from a newer crawl is not an error).
A local bake is kept rather than clobbered -- with one exception the
sidecar's own record makes safe: when both sides say what they are and
the incoming preview is a larger bake of the same product, it wins.
That is the case the published sidecar creates, since it is baked at
128 px where umbra index bake-thumbnails defaults to 256: a merge
used to keep whichever arrived first, which made the resolution of a
preview a fact about the order two commands were run in. Where either
side is unrecorded the two are not comparable and the local bake stays.
overwrite=True replaces unconditionally, as before. Returns the
number of thumbnails applied.
search(*, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, limit=None, max_per_task=None)
Yield indexed items matching the filters.
Same semantics as :meth:UmbraCatalog.search, answered from local SQL.
Only returns acquisitions already present in the index; build or refresh
it with :meth:build first. fuzzy=True widens area to the same
deterministic token-wise match the live path uses
(:func:umbra_py.fuzzy.matching_tasks): the distinct task names are
read from the index and matched in Python, so both backends agree.
intersects (the exterior-ring form from
:func:umbra_py._geometry.parse_geometry) keeps only items whose
footprint intersects the polygon. Its bounding box is pushed into SQL as
a cheap prefilter and the exact polygon test then runs in Python on each
candidate, so the result matches :meth:UmbraCatalog.search exactly.
The acquisition-property filters (polarizations, min_incidence /
max_incidence, max_resolution) mean exactly what they do on
:meth:UmbraCatalog.search. They are read from each item's stored STAC
document (already reconstructed here) and applied in Python via
:meth:UmbraItem.matches_filters, the same way the polygon test runs, so
both backends agree without a schema change.
search_live(catalog=None, *, overlap_days=1, refresh=True, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, limit=None, max_per_task=None)
Read-through search: the index for the bulk, a live delta for what's new.
:meth:search is instant but only returns what the index already holds;
:meth:UmbraCatalog.search is always current but re-walks the whole
bucket every call. This is the transparent middle the codebase analysis
named as "make the index the default path": the index answers the whole
query from local SQL, and a bounded
live walk covers only acquisitions at or after the index's freshness
horizon -- its maximum indexed acq_date minus overlap_days -- so
the walk fetches sidecars only for recent passes rather than the whole
catalog (the same pruning :meth:update relies on). The two streams are
merged in the usual (task, acq_date) order and de-duplicated by
sidecar href, so an acquisition the index already knows is never yielded
twice; the result is what a single fresh search would return, without
paying for a full crawl.
With refresh=True (the default) each genuinely new acquisition the
live delta discovers is upserted into the index as it is yielded -- the
"read-through cache warms" behavior -- so the next call needs an even
smaller (often empty) walk. Set refresh=False to leave the index
untouched (e.g. when it is a shared read-only snapshot); a read-only
database also disables warming automatically rather than failing the
search. The write-back is committed only when at least one new row was
added, and built_at is re-stamped then, exactly as :meth:update.
The keyword filters (bbox, start, end, product_types,
area, fuzzy, the acquisition-property filters polarizations /
min_incidence / max_incidence / max_resolution, limit,
max_per_task) mean exactly what they
do on :meth:search / :meth:UmbraCatalog.search; start bounds both
streams (the live delta never walks older than the caller asked for, even
when the freshness horizon is older). overlap_days (default 1)
re-scans a little past the newest indexed date to catch near-real-time
publish lag; the bound is on acquisition date, so a back-dated late
arrival still wants a widened overlap or a full :meth:build. An empty
index has no horizon, so the live walk covers the caller's full window
(and, with refresh, this doubles as a first :meth:build).
rank_sites(*, bbox=None, intersects=None, start=None, end=None, product_types=None, area=None, fuzzy=False, polarizations=None, min_incidence=None, max_incidence=None, max_resolution=None, top=20, min_passes=2, rank_by='passes', active_since=None, active_before=None, first_since=None, first_before=None, max_revisit_days=None, median_revisit_days=None, min_span_days=None, max_span_days=None)
Rank the most repeat-imaged sites across the whole index.
The index-native form of :func:umbra_py.coverage.rank_site_coverage,
and the reason it exists: that function ranks whatever pool it is handed,
so umbra sites --local capped the pool at --limit acquisitions and
a site with many passes just outside the first --limit rows read as
shallower than it is. Umbra files every pass of a site under one task, so a
site's depth is a GROUP BY task the index can answer over its entire
contents -- no pool cap, so a deeply-imaged site is ranked by all its
passes rather than by the arbitrary window a limit happened to admit.
The ranking is :func:umbra_py.showcase.select_featured_sites' exactly --
dated passes per task, most first, task name breaking ties, keeping those
with at least min_passes -- so this, umbra sites and the featured
gallery cannot disagree about what "most repeat-imaged" means. Each site
is summarised by :func:umbra_py.coverage.site_coverage; the result is
the top best, best-first.
The filters mean exactly what they do on :meth:search. The
SQL-expressible ones (bbox, start / end, area / fuzzy,
product_types) are counted directly in a GROUP BY, so only the top
tasks' documents are then read to summarise -- cheap even whole-archive.
The polygon (intersects) and acquisition-property (polarizations,
min_incidence / max_incidence / max_resolution) filters run per
item in Python, so when any is set this ranks the full uncapped matching
stream instead: still whole-archive (no limit), identical to the pool
path, just without the cap this method exists to remove.
rank_by is one of :data:umbra_py.coverage.SITE_RANKINGS. "passes"
(the default) is answerable in SQL -- a COUNT(*) per task orders the
candidates, so only the top top tasks' documents are read.
"comparable" ranks by analysable depth (comparable_passes), which
depends on each pass's polarization inside the document JSON and so is not a
COUNT: every qualifying task's documents are read and summarised, then
re-ranked by the analysable subset and truncated to top. The temporal
rankings "recency" (newest dated pass first), "span" (longest
observation baseline first) and "cadence" (tightest typical revisit gap
first) are likewise not the SQL COUNT order -- the top-by-count tasks are
not the top-by-recency, -span or -cadence -- so they take the same
read-every-task-then-re-rank path (the raw-count LIMIT is dropped when the
ranking is not "passes", so a recently-active, long-baseline or
tightly-revisited site outside the raw top-top is not truncated before it
can be promoted). That is heavier than the raw path (it reads every
repeat-imaged task rather than the top top), but still whole-archive and
correct. All five rankings share :func:umbra_py.coverage._rank_sort_key, and
the temporal trio reads the last / span_days / median_revisit_days
off the same summarised :class:SiteCoverage the pool path reduces from the
passes, so this and the pool path order every ranking identically.
min_passes gates on the same depth rank_by ranks by
(:func:umbra_py.coverage._min_passes_depth): under "comparable" the
HAVING COUNT(*) >= min_passes clause is a superset pre-filter (comparable
depth is never above the raw count) and the true floor on comparable_passes
is applied in Python before the re-rank, so --rank-by comparable
--min-passes N returns only sites whose differenceable series is at least
N passes deep. Under "passes" the SQL floor is exact and this and the
pool path qualify a site identically.
active_since keeps only sites still imaged on or after that date -- a
recency filter on each site's newest dated pass. It is answered in the
same HAVING clause the pass-count floor is (MAX(acq_date) >= ?), so
it costs nothing beyond the group already computed and is exact under either
ranking (a site's latest pass is independent of the polarization grouping
"comparable" re-ranks by). Whole-archive like the rest of this method,
and byte-identical to the pool path's :func:umbra_py.coverage.rank_site_coverage
recency gate. It is orthogonal to start / end (those bound which
rows the GROUP BY counts; this selects whole sites by their latest and
keeps every counted pass in the summary). None applies no recency filter.
active_before is the complement -- keep only sites whose newest dated
pass is on or before that date (a dormant series), answered by the twin
MAX(acq_date) <= ? clause in the same HAVING, so with active_since
the two bound the site's latest pass to a window. A span expression snaps to
its last day (symmetric with end). Byte-identical to the pool path's
upper-recency gate. None applies no upper bound.
first_since / first_before are the onset (first-seen) twins of the
active_* pair -- they gate each site's earliest dated pass rather than
its newest, selecting newly-appeared series (first_since, first pass on
or after the date) and long-established ones (first_before, first pass
on or before it); set together they bound the onset to a window. Like the
recency pair they are pure SQL aggregates, answered by MIN(acq_date) >= ? /
MIN(acq_date) <= ? clauses in the same HAVING -- costing nothing beyond
the group already computed and exact under either ranking (a site's earliest
pass, like its latest, does not depend on the polarization grouping
"comparable" re-ranks by), so unlike the cadence and span filters they do
not force the raw-count LIMIT to be dropped. MIN skips NULL
acq_date, so a group with no dated pass yields NULL and is dropped
(NULL >= ? / NULL <= ? is never true), matching select_featured_sites
dropping a site with no datable pass. first_before snaps a span expression
to its last day (symmetric with active_before / end). Byte-identical to
the pool path's onset gate. None applies no onset filter.
max_revisit_days keeps only sites revisited at least this often -- a
cadence filter on each site's worst-case revisit gap. Unlike the recency
and depth filters it is not a SQL aggregate: the worst gap is between
consecutive passes (and, under "comparable", over the largest
single-polarization subset the document JSON defines), which no HAVING
clause on a column expresses. It is applied in Python on the same per-task
items this method already reads to summarise, using the same
:func:umbra_py.coverage._passes_cadence the pool path uses -- so the two are
byte-identical -- and when it is set the raw-count SQL LIMIT is dropped
(as it is for the comparable ranking), because a site that passes the cadence
filter but sits just outside the raw top-top must not be truncated before
the filter runs. Gated on the same depth rank_by measures (the analysable
series' cadence under "comparable"), orthogonal to the recency filters,
and dropping a site with fewer than two passes in the gated series. None
applies no cadence filter; a non-positive value is a ValueError.
median_revisit_days is the typical-cadence twin of max_revisit_days --
keep only sites whose median revisit gap is at most this many days (a site
usually imaged often, tolerating the odd long outage the worst-case bound
rejects). Like the worst-case filter it is not a SQL aggregate -- a median of
consecutive gaps is no more a HAVING clause than a max of them -- so it is
applied in Python on the same per-task items this method already reads, using the
same :func:umbra_py.coverage._passes_median_revisit the pool path uses (so the
two are byte-identical), gated on the analysable subset under "comparable",
and it drops the raw-count SQL LIMIT when set (as the cadence and span
filters do) so a usually-tight site outside the raw top-top is promoted rather
than truncated before the filter runs. A site with fewer than two passes in the
gated series has no measurable cadence and is dropped. None applies no
typical-cadence filter; a non-positive value is a ValueError.
min_span_days keeps only sites imaged over at least this long -- a
baseline filter on each site's observation span (whole days from first
dated pass to last). Like the cadence filter it is applied in Python rather
than SQL: under "comparable" the span is over the largest
single-polarization subset the document JSON defines (not a column
expression), and keeping the two rankings byte-identical is worth more than a
SQL HAVING on the raw case alone, so it uses the same
:func:umbra_py.coverage._passes_span the pool path does on the same per-task
items this method already reads, and drops the raw-count SQL LIMIT when
set (as the comparable ranking and the cadence filter do) so a long-baseline
site outside the raw top-top is promoted rather than truncated before the
filter runs. Gated on the same depth rank_by measures (the analysable
series' span under "comparable"), orthogonal to the recency and cadence
filters, and dropping a site with fewer than two passes in the gated series.
None applies no span filter; a non-positive value is a ValueError.
max_span_days is the upper twin of min_span_days -- keep only sites
imaged over at most this long (a short-lived series), the complement of the
floor, and set with it a window bounding each site's baseline
(min_span_days <= span <= max_span_days), as active_since /
active_before bound the newest pass. Like the floor it is applied in Python
with the same :func:umbra_py.coverage._passes_max_span the pool path uses (so
the two paths stay byte-identical), gated on the analysable subset under
"comparable", drops the raw-count SQL LIMIT when set, and drops a site
with no measurable span so the window admits only a confirmed baseline.
None applies no span ceiling; a non-positive value is a ValueError.
Return the indexed item with this STAC id, or None if absent.
The keyed point-lookup complement to :meth:search's listing: where
filtering a full search by id would scan the ordered result set,
this is an idx_items_id-backed lookup, so it stays fast as the
published catalog.db snapshot grows. STAC ids are unique per
acquisition in Umbra's catalog; in the unlikely event two sidecars
share an id, the first by href order is returned deterministically.
Summary counts for umbra index info: item count, acquisition-date
span, number of distinct tasks, how many items carry a baked place label
(labeled; see :meth:bake_places), how many carry a baked quicklook
thumbnail (thumbnailed; see :meth:bake_thumbnails), and the date the
index was last built (built_at, None for an index written before
build stamping).
dataclass
¶
Outcome of an incremental :meth:CatalogIndex.update.
added counts acquisitions whose href was not already in the index;
refreshed counts those whose existing row was replaced; scanned is
their sum (every item the scoped walk yielded). start is the
acquisition-date lower bound the walk used -- None when the index was
empty and update fell back to a full build.
Where the index lives by default.
$UMBRA_INDEX_DB overrides everything; otherwise it sits under the XDG
cache dir ($XDG_CACHE_HOME or ~/.cache) at
umbra-py/catalog.db.
Where the baked-thumbnail sidecar lives by default.
It sits beside the index (catalog.db -> catalog.thumbs.db) so the
two travel together while staying separate files: the pixels are opt-in and
an order of magnitude larger than the metadata, so keeping them out of
catalog.db is what lets the published index stay small (the same split
:func:umbra_py.embed.default_scene_embed_path makes for vectors). Pass
index_path to derive the sibling name from a non-default index location.
Download the published baked-thumbnail sidecar.
The weekly index workflow bakes a quicklook per acquisition and ships it as
catalog.thumbs.db on the rolling catalog-index release beside
catalog.db / catalog.pmtiles, so a fresh install gets scene previews
without streaming a cloud-optimized GeoTIFF overview per acquisition -- the
thumbnail sibling of :meth:CatalogIndex.from_release and
:func:umbra_py.pmtiles.fetch_prebuilt_pmtiles. This fetches the sidecar to
dest (default: :func:default_thumbs_path) and returns its path;
:meth:CatalogIndex.import_thumbnails merges it into a local index. Re-run
any time to refresh; the download is resume-safe and always overwrites the
existing file. url overrides the release asset location (e.g. a fork).