ML chips & export¶
Cut scenes into fixed-size, georeferenced training tiles with per-chip metadata
(look angle, resolution, polarization, license) in a .jsonl, .geojson, or
stac-geoparquet manifest. Chipping needs the [load] extra; the GeoParquet
manifest and catalog export need [export].
The complex products are chippable too: asset="SICD" geocodes each acquisition
through the conversion pipeline — optionally terrain-orthorectified,
terrain-flattened and radiometrically calibrated (see SicdConversion) — and cuts
the identical tiles from the result, so a training set can carry a physical
backscatter coefficient from the full-resolution archive. That path needs the
[convert] extra alongside [load].
bbox= (umbra chips --clip-bbox) chips one area of interest out of each
acquisition rather than the whole raster, numbering each chip's row/col from
that window's corner. On the complex path it is also the
conversion's own clip, so each scene is geocoded over the site
rather than whole — which is where the cost of chipping the complex archive
actually lives. It is lon/lat whatever the raster's CRS is, matching
to_stack(bbox=…).
--speckle-filter / speckle_filter= on SicdConversion carries the
conversion's speckle averaging to a training set, which is a
different trade there than it is for one scene: a single-look chip teaches a model
the interference pattern as much as the surface, and a filtered chip teaches it a
surface at coarser resolution. Because that decides what a model can learn to
see, every ChipRecord carries speckle_filter and speckle_window — read back
from the geocoded raster's own tags, so the manifest reports the processing rather
than the request.
Where the conversion inferred the noise floor (--noise-model estimated /
estimated-range), the run also says which of its scenes that estimate should
not be trusted on. Each ChipRecord carries the scene's own
noise_floor_margin_db and noise_floored_fraction, so a training loader can
filter the manifest instead of opening rasters, and ChipDataset.noise
(NoiseSummary) counts the scenes that had too little dark ground to read. It is
an advisory, never a refusal — a uniformly bright scene is legitimate imagery, and
the honest fix where the margin matters is --noise-model measured.
Some acquisitions cannot support the measurement at all. Radiometric calibration
needs the SICD's Radiometric scale factors and --noise-model measured needs
its stated noise floor, and Umbra's open products generally carry neither — so a
batch over a mixed archive used to end on the first product that came up short,
losing every scene already chipped. The refusal itself is right (an invented
scale factor is indistinguishable in the output from a measured one), so what it
gained is a type: UnsupportedMeasurementError, the family of refusals that
are facts about a product rather than about the request.
write_chips(skip_unsupported=True) (umbra chips --skip-unsupported) catches
exactly that type, records it on ChipDataset.skipped as a
SkippedAcquisition — which pass, and in the product's own words why — and
moves to the next acquisition, so the dataset states its hole instead of
having one. Nothing else is caught: a download failure or a corrupt product
still ends the run, because a batch that swallows unknown errors is a batch
whose output nobody can trust. The check now also runs off the product's
metadata before its pixels are read, so a scene that cannot answer costs its
header rather than a full complex read.
Chips¶
chip_item(item, out_dir, *, asset='GEC', chip_size=512, stride=None, db=False, fmt='geotiff', min_valid=0.0, prefix=None, bbox=None, speckle_filter=None, speckle_window=SPECKLE_WINDOW_DEFAULT, conversion=None, work_dir=None, preparer=None, clip_report=None)
Cut one acquisition into fixed-size, georeferenced training tiles.
Reads band 1 of the item's geocoded GeoTIFF (the GEC cloud-optimized
GeoTIFF by default) one window at a time via HTTP range requests, and writes
each full chip_size x chip_size tile to out_dir as a GeoTIFF (or
a NumPy .npy array). Returns a :class:ChipRecord per written chip.
With asset="SICD" the complex product is downloaded and geocoded first
(see :class:SicdConversion) and the resulting COG is chipped by this same
loop, so a training set can be cut from the full-resolution complex archive
-- optionally terrain-orthorectified, terrain-flattened and radiometrically
calibrated -- rather than only from the derived amplitude products.
Parameters¶
item:
The acquisition to chip.
out_dir:
Directory to write chips into (created if needed).
asset:
Which product to read. "GEC" (the default) and "CSI" are
amplitude rasters, streamed tile by tile. "SICD" is the complex
slant-plane product: it has no map grid, so it is fetched whole and
geocoded before chipping (the [convert] extra). CPHD is phase
history rather than a focused image and is not chippable.
chip_size:
Tile edge in pixels. Only full tiles are emitted; a partial strip along
the right/bottom edge is dropped, so every chip has this exact shape.
stride:
Step between tile origins in pixels. Defaults to chip_size
(non-overlapping). A smaller stride overlaps tiles (dense inference /
augmentation); it must be positive.
db:
Write the decibel (20*log10(amplitude)) scale instead of linear
amplitude. Non-positive / nodata pixels become NaN either way.
fmt:
"geotiff" (georeferenced, the default) or "npy" (a bare
float32 array; the geo metadata lives in the manifest record).
min_valid:
Drop a tile whose fraction of valid (finite, positive) pixels is below
this. 0.0 keeps every full tile; e.g. 0.5 drops mostly-nodata
corners of a rotated footprint.
prefix:
Filename stem for this item's chips (defaults to a slug of item.id).
Chips are named <prefix>_r<row>_c<col>.<ext>.
bbox:
Optional area of interest (min_lon, min_lat, max_lon, max_lat) in
WGS-84 degrees (whatever the raster's own CRS is, as in
:func:umbra_py.to_stack). Only tiles inside that window are cut, and
row / col are numbered from its corner. For a complex asset
it also becomes the conversion's own clip, so the geocoding step is
sized to the area of interest rather than to the scene — which is where
the cost of chipping the complex archive actually lives. None chips
the whole raster.
speckle_filter:
Optionally average speckle down, one of
:data:umbra_py.convert.SPECKLE_FILTERS -- so a tile teaches a model the
surface rather than the interference pattern coherent illumination made
on it, whose standard deviation equals its mean on a single look. It runs
wherever it is most correct for the asset: on a published amplitude
raster the tiles themselves are averaged, which is the first (and only)
point at which those pixels exist in this library at all; on a
SICD it is routed into the conversion
(:attr:SicdConversion.speckle_filter) and runs in the radar's own image
space before geocoding, where speckle is one independent sample per pixel.
None (the default) filters nothing: what a window spends is
resolution, so it is a request rather than a default.
On the raster path each tile is read with a ``speckle_window // 2`` halo
and cropped back after filtering, so every chip pixel averages the
neighbours it would have had in a whole-scene filter -- which is what
makes two overlapping tiles agree about the ground they share -- and
``"lee"``'s speckle parameter is read once for the acquisition
(:func:`_scene_speckle`) rather than per tile. Both numbers land in every
:class:`ChipRecord`.
speckle_window:
Edge of the odd, centred window speckle_filter averages over, in
pixels (:data:umbra_py.convert.SPECKLE_WINDOW_DEFAULT). Wider removes
more speckle and more detail; it costs no more to compute.
conversion:
How to geocode a complex asset before chipping it. Ignored for the
amplitude rasters; defaults to :class:SicdConversion's flat-earth
geocoding.
work_dir:
Where the downloaded product and the geocoded COG are kept when chipping
a complex asset. None uses a temporary directory removed
afterwards, so disk stays bounded to one scene; naming a directory keeps
both files, which makes the expensive step resumable -- a re-run reuses a
COG already geocoded with the same settings instead of rebuilding it.
preparer:
Override for the download-and-geocode step (the test seam; defaults to
:func:_prepare_sicd).
clip_report:
Optional callback, invoked once with a
:class:umbra_py.convert.ClipSavings when bbox clipped the read --
pricing the pixels this acquisition read against the pixels its whole
product holds, the same figure umbra convert --clip-bbox prints. On
an amplitude asset it comes from the tile window
(:func:_clip_pixel_window) against the source raster's own size; on a
SICD it is the conversion's own clip report, so it describes the
scene rather than the already-clipped COG that reaches the tile loop.
Not called when bbox is None (nothing was clipped), nor on a
SICD prepared by a custom preparer or served from a
work_dir cache (no conversion ran to price). A caller that does not
pass it is unchanged.
Returns¶
list[ChipRecord] One record per written chip, in row-major order.
write_chips(items, out_dir, *, asset='GEC', chip_size=512, stride=None, db=False, fmt='geotiff', min_valid=0.0, bbox=None, speckle_filter=None, speckle_window=SPECKLE_WINDOW_DEFAULT, manifest='manifest.jsonl', skipped_manifest='skipped.jsonl', progress=None, conversion=None, work_dir=None, preparer=None, skip_unsupported=False, preflight=False, preflight_progress=None, preflight_workers=None)
Chip a whole search result into a training dataset with a manifest.
Iterates items, calls :func:chip_item on each, and writes a combined
manifest (out_dir/manifest) describing every chip. Returns a
:class:ChipDataset summarising the run.
manifest is the manifest filename inside out_dir (.jsonl,
.geojson, or .parquet -- the last needs the [export] extra);
pass None to skip writing it and just collect the records. progress
is called (index, total, item, chips_written) after each item, for a CLI
progress line.
skipped_manifest is the filename of the sidecar that states what the run
could not include (see :func:write_skipped_manifest), written beside the
manifest and only when there is something to record -- so a dataset with
no hole in it is exactly the set of files it was before, and the file's
presence is itself the statement that there is a hole. It follows
manifest: manifest=None means "collect the records, write nothing",
and that stays true. Pass None to suppress the sidecar on its own.
Writing it at all is the difference between a run that knows what it left
out and a dataset that does. :attr:ChipDataset.skipped and the --json
payload describe the hole to whoever watched the run; a training loader
reading out_dir months later sees only the files, and without the sidecar
a dataset that dropped half its passes is indistinguishable from one that was
only ever offered half.
bbox restricts every acquisition to one area of interest (see
:func:chip_item) -- the usual shape of a dataset build, where the site is
the subject and the scenes are just the passes over it. What the clip read
instead of the whole scene is rolled up across the run onto
:attr:ChipDataset.clip (a :class:ClipSummary), the batch form of the
clipped line umbra convert --clip-bbox prints.
speckle_filter / speckle_window average speckle down in every scene
(see :func:chip_item), which for a complex asset means routing the
request into conversion -- so the settings the summary reports are the
ones that ran, whichever path they took.
conversion / work_dir / preparer apply when asset is a
complex product (see :func:chip_item). Each acquisition is prepared and
chipped in turn, so a run over many SICDs holds one scene on disk at a time
unless work_dir is set to keep them.
skip_unsupported decides what a batch does when one acquisition's own
metadata cannot support the measurement asked of it -- a product with no
Radiometric block under calibration=, no stated noise floor under
noise_model="measured", or no stated collection geometry under rtc=.
The default raises, which is right for a run
over one product's worth of scenes: if the archive cannot answer, the answer
is not a smaller dataset. Over a mixed archive it is the wrong default,
because the twenty scenes already chipped are lost to the twenty-first, so
True records the refusal on :attr:ChipDataset.skipped and moves to the
next acquisition. The dataset then states its hole rather than having one:
which passes are missing, and in each product's own words why.
Only :class:~umbra_py.exceptions.UnsupportedMeasurementError is skipped.
Everything else -- a download failure, a missing asset, a corrupt product --
still ends the run, because a batch that swallows unknown errors is a batch
whose output nobody can trust.
preflight asks the same question before any product is downloaded.
skip_unsupported makes a refusal survivable, but it is still discovered
by attempting the conversion, and for a complex asset that means the whole
multi-gigabyte NITF is fetched to learn that its metadata cannot answer --
twenty times over a site's twenty passes. With preflight=True each
acquisition's SICD XML is read over the wire instead (see
:mod:umbra_py.preflight: a NITF states its own layout, so two range
requests and a few tens of kilobytes locate and fetch it), the conversion's
own support check is run against it, and the passes that positively cannot
answer never reach the download at all. They are recorded on
:attr:ChipDataset.skipped exactly as a survived refusal is -- because a
dataset with a hole in it has to say so however cheaply the hole was found --
with stage="preflight" and a :class:PreflightSummary roll-up saying
what the check cost and what it saved.
An acquisition whose metadata read fails on the wire is kept: a
transport failure is not a product declaring anything, so the run proceeds to
find out the expensive way rather than dropping a scene over a blip. One
whose read fails on the product -- the item lists no such asset, nothing is
at the href, what is there is not a NITF or carries no SICD XML -- is dropped
like a refusal, because that is what it is. Keeping those was never the
cautious half of the choice: such a pass fails inside :func:chip_item as a
plain read error, which skip_unsupported deliberately does not catch, so
the run ends on an acquisition its own preflight had already ruled out. The
two are counted apart on :class:PreflightSummary (missing against
unreadable) so the summary says which kind of hole a dataset has.
preflight is only meaningful for a complex asset -- a GEC or CSI
carries no SICD metadata to ask -- and asking for it on a raster asset is
refused rather than quietly ignored. It composes with skip_unsupported,
which stays worth passing: the preflight only asks the two questions the
metadata answers (calibration and a measured noise floor), so a refusal from
anywhere else still arrives at conversion time.
preflight_workers is how many of those metadata reads run at once
(None takes :data:umbra_py.preflight.DEFAULT_PREFLIGHT_WORKERS). The
check costs round trips rather than bytes, so a serial one puts a stall in
front of the batch that grows with the number of passes -- which is the one
cost the preflight would otherwise have added to a run over a large site.
Write chip records to a manifest, format chosen by path's extension.
.jsonl (default) writes one JSON record per line -- the standard ML
manifest format, streamable and append-friendly. .geojson writes a
FeatureCollection of chip footprint polygons (each carrying the full
record as properties) for QGIS / geopandas; both are stdlib-only.
.parquet writes a stac-geoparquet table (one column-oriented file DuckDB /
geopandas can query without loading every line) and needs the [export]
extra.
Write chip records as a stac-geoparquet manifest (needs the [export] extra).
Each chip becomes one STAC Item row (footprint geometry + record properties),
so a large chip set is queryable by DuckDB / geopandas / pyarrow without
reading every line -- what the .jsonl / .geojson manifests can't offer
at scale. Reuses the same stac_geoparquet.arrow writer as
:func:umbra_py.export.export_geoparquet.
dataclass
¶
ChipRecord(path, item_id, asset, row, col, window, crs, transform, bbox, units, valid_fraction, datetime=None, place=None, platform=None, product_type=None, polarizations=list(), incidence_angle_deg=None, resolution_range_m=None, resolution_azimuth_m=None, calibration=None, noise_subtraction=None, noise_floored_fraction=None, noise_floor_margin_db=None, speckle_filter=None, speckle_window=None, speckle_enl_before=None, speckle_enl_after=None, speckle_looks=None, rtc_model=None, license=DATA_LICENSE, attribution=ATTRIBUTION)
One training tile's manifest entry.
Carries where the chip is (path, geographic bbox, crs,
transform, grid row / col, source pixel window), what the
acquisition is (item_id, datetime, place, platform,
product_type, polarizations, incidence_angle_deg, the
resolution_* pair), and how usable it is (valid_fraction -- the
fraction of finite, positive pixels). license / attribution travel
with every record.
A chip cut from a complex product also carries what the conversion did to
its pixels -- calibration, noise_subtraction, speckle_filter /
speckle_window and rtc_model, read back from the geocoded raster's own
provenance tags rather than from the request, so the record reports the
processing that actually ran. calibration, noise_subtraction and
rtc_model are None for a chip read straight from an amplitude raster
-- those steps need a complex product -- and the full tag set travels in the
chip GeoTIFF itself.
The shape is public API and published as
docs/schemas/chip-record.schema.json: it is what a training loader parses
without ever having printed it, in whichever of the three manifest formats it
reads (one .jsonl line, one .geojson feature's properties, one
.parquet row -- the same record).
The speckle pair is the one that says what a chip's resolution is as
opposed to its pixel size: a 5x5-filtered chip resolves ground five pixels
across, which is what a model trained on it can learn to see. It is filled in
on either path, because an amplitude raster can be filtered too -- on the
published GEC the tiles themselves are averaged (see :func:chip_item's
speckle_filter), on a SICD the scene is, in the radar's own image space
before it is geocoded.
speckle_enl_before / speckle_enl_after / speckle_looks are that
filter's diagnostics (see :class:umbra_py.convert.SpeckleFiltering): the
scene's equivalent number of looks either side of the window, and the looks
"lee" assumed for the speckle it was separating from structure. Like the
noise diagnostics they describe the acquisition a chip was cut from rather
than the chip, so every chip of one scene carries the same three; the ratio
of the pair is what says whether the resolution the window spent bought
anything, and either level on its own reads low on a textured scene.
noise_floored_fraction and noise_floor_margin_db come from the same
tags and are the noise subtraction's two diagnostics (see
:class:umbra_py.convert.NoiseSubtraction): how much of the scene the floor
drove to the sensor's sensitivity limit, and -- for an inferred floor -- how
far the scene's own median power sat above it. They describe the scene the
chip was cut from rather than the chip, so every chip of one acquisition
carries the same pair; a training loader that wants to drop the scenes whose
dark tail was ground rather than receiver can filter the manifest on the
second without opening a raster.
dataclass
¶
ChipDataset(out_dir, manifest_path, records, chip_size, stride, asset, units, fmt, conversion=None, skipped=(), preflight=None, skipped_path=None, clip=None)
The result of a chipping run: the written chips plus their manifest.
records are the :class:ChipRecord entries (also written to
manifest_path); the summary fields describe the run for a --json
caller or an agent deciding what to train on.
skipped is what the run could not include: the acquisitions whose own
metadata could not support the measurement that was asked for, present when
write_chips(skip_unsupported=True) let the run carry on past them or when
write_chips(preflight=True) dropped them before downloading them. An
empty tuple is the default and means what it says -- every acquisition
offered was chipped.
preflight is the roll-up of that pre-download check when one ran: what
reading the archive's headers cost, and the download it removed.
clip is the roll-up of what a --clip-bbox run read instead of the
whole scene, present only when the run was clipped to an area of interest.
Unlike :attr:noise and :attr:speckle it is accumulated during the run
rather than derived from records: the clip saving is deliberately not a
:class:ChipRecord field, so there is nothing in the manifest to derive it
from (see :class:ClipSummary).
skipped_path is where that hole was written, when there was one --
the sidecar beside the manifest (see :func:write_skipped_manifest), so a
loader reading the directory rather than the run can see it too. None
when nothing was skipped, which is the same thing the empty skipped
tuple says.
:meth:to_dict is what umbra chips --json prints, and its shape is
published as docs/schemas/chip-dataset.schema.json. Its conditional keys
are part of that contract: conversion, noise, speckle, clip,
skipped and preflight appear only when the run had something to say
with them, so an ordinary raster run's payload is unchanged by any of those
features existing.
property
¶
The run's noise-subtraction roll-up, or None when none ran.
Derived from records rather than accumulated during the run, so the
summary can never disagree with the manifest it sits beside.
property
¶
The run's speckle-filtering roll-up, or None when none ran.
Derived from records like :attr:noise, so it cannot disagree with
the manifest beside it.
dataclass
¶
NoiseSummary(scenes, models, margin_scenes, low_margin_scenes, margin_warn_db, min_margin_db=None, max_floored_fraction=None)
What the noise subtraction did across a whole chipping run.
:class:umbra_py.convert.NoiseSubtraction's diagnostics are per scene, and
umbra convert prints them for the one raster it wrote. A chip run
converts many scenes, so the equivalent there is not a line each -- it is
the question a dataset builder actually has: were any of these scenes ones
the estimator should not have been used on? The inferred floors work
because a SAR scene's dark surfaces are a different population from its
backscatter; where they aren't, the subtraction takes real backscatter off,
and the tell is a narrow margin between the floor and the scene's median
power. That is a property of some scenes in a batch and not others, which is
exactly what a roll-up is for.
Counted per acquisition rather than per chip: the numbers describe the
scene each chip was cut from, so counting chips would weight a wide scene
more heavily than a narrow one for no reason. Scenes that produced no chips
(everything dropped by min_valid) are not in the batch this describes.
Attributes¶
scenes:
Acquisitions in this run whose chips carry a noise subtraction.
models:
The distinct UMBRA_NOISE_SUBTRACTION values across them, sorted --
normally one, since a run converts every scene the same way.
margin_scenes:
How many of scenes reported a margin at all. Only the inferred
floors do: a measured floor is the product's own metadata and assumes
nothing about the scene, so it has nothing to report.
low_margin_scenes:
How many of margin_scenes sat below margin_warn_db. This is the
number the roll-up exists for.
margin_warn_db:
The advisory threshold applied (convert.NOISE_MARGIN_WARN_DB).
min_margin_db:
The narrowest margin in the batch -- the worst scene, None when no
scene reported one.
max_floored_fraction:
The largest fraction of a scene the floor drove to the sensor's
sensitivity limit. Reported by both models.
dataclass
¶
One acquisition left out of a run because it could not support the request.
A dataset built with skip_unsupported=True is a dataset with a hole in
it, so the hole is part of the result rather than a line on a console
somebody may not have been watching: item_id and datetime say which
pass is missing, reason is the refusal's own words (the product's
metadata, not a paraphrase), and hint carries the recovery step where
the refusal named one.
What produces one is a fact about a product, and only that: a
:class:~umbra_py.exceptions.UnsupportedMeasurementError (the metadata was
read, and it cannot support the request) or, from a preflight, an
:class:~umbra_py.exceptions.UnreadableProductError (there is no readable
product at the acquisition's href to ask). Both are final, which is what
makes carrying on to the next scene a defensible response, where carrying on
past an unknown error -- a download failure, a corrupt file, a transport
hiccup -- would be a way of hiding one.
stage says when it was discovered, because that is the one thing the
routes to it do not share. "conversion" means the product was downloaded
and then refused (skip_unsupported=True); "preflight" means its
metadata was read over the wire and it was never downloaded at all
(preflight=True). The reason is the product's own words either way -- the
conversion's own check, or the reader's -- so the dataset's hole is described
the same and only its cost differs.
bbox is where the missing pass was -- the acquisition's own footprint
(UmbraItem.bbox, EPSG:4326 [min_lon, min_lat, max_lon, max_lat]), so
the sidecar locates the hole in space as datetime does in time. A loader
reconstituting a time series over an area of interest can then tell a hole
that falls over the site it cares about from one that never overlapped it,
without re-running the search that produced the selection. Null when the
source item stated no footprint.
Published as docs/schemas/chip-skipped.schema.json -- one line of the
skipped.jsonl sidecar and one entry of the dataset summary's skipped
array, which is one contract because it is one record.
dataclass
¶
SicdConversion(dem=None, geoid=None, rtc=False, rtc_model='cosine', rtc_reference_deg=None, calibration=None, noise_subtract=False, noise_model='measured', speckle_filter=None, speckle_window=SPECKLE_WINDOW_DEFAULT, resolution=None, resampling='bilinear', gcp_grid=15, projection_type='HAE', bbox=None)
How a complex SICD is geocoded before it is chipped.
Every field is passed straight to :func:umbra_py.convert.sicd_to_geocoded_cog
and means exactly what it means there -- this is the chipper's handle on that
pipeline, not a second implementation of it. The defaults are the flat-earth
geocoding, which is what a training set wants unless the site has relief.
The one option deliberately not exposed is decibels: the chipper's own
db flag already chooses the scale, so the conversion always writes linear
amplitude and the chip loop takes the logarithm. That keeps one code path for
both asset kinds, and keeps a calibrated chip's decibels the decibels of the
calibrated quantity.
bbox is set from :func:chip_item's own bbox rather than passed
separately: chipping an area of interest out of a complex product means
geocoding only that area, so the two are one decision. It is part of
:meth:cache_key like every other field, so a clipped conversion never
stands in for a whole-scene one in work_dir.
A short stable digest of these settings.
Two conversions of one acquisition differ only by these values, so the
digest is what makes a cached geocoded COG in work_dir safe to reuse:
change a setting and the name changes with it, rather than silently
chipping the previous product.
Catalog export¶
Write items to a stac-geoparquet file; return how many were written.
Items without a footprint geometry are skipped — the geometry column is
the point of geoparquet, and the writer requires one — so the return
value can be less than the number of items passed in. Raises
:class:~umbra_py.UmbraError when nothing is exportable, rather than
writing an empty (and schema-less) file.