topo-detect

topo-detect scans a single polygon layer for gap/overlap coverage defects and reports them, without fixing anything. Extracted out of topo-clean's own issue-detection stage (core/topo_clean/_02_issues.py, moved verbatim to core/topo_detect/_02_issues.py) so it can be run standalone, the same way assign/edge-clip/edge-stitch were extracted as reusable primitives out of edge-match. topo-clean still detects defects as its first real step, it just calls core.topo_detect's stage function directly instead of owning the logic itself, the same pattern edge-match/edge-mosaic use for core.assign/ core.edge_clip/core.edge_stitch, and edge-match/change use for core.edge_extend's. See docs/adr/0028.

Usage¶

topo-tools topo-detect example.geojson

from topo_tools.api.topo_detect import detect

detect("example.parquet", "example_issues.parquet")

OUTPUT_FILE (positional, optional) defaults to INPUT_FILE with an _issues suffix.

Run topo-tools topo-detect --help for the full, always-current option list.

Pipeline¶

1. _01_inputs: reads and reprojects to EPSG:4326 via the shared core.io.read_and_reproject() helper, without an auto-clean pre-check. This is deliberate: topo-detect's whole purpose is to report defects in the raw input, so the detection stage needs to see them, not a table ST_CoverageClean has already silently rewritten.
2. _02_issues: detects gap/overlap regions, writing one issues table ({name}_02). Gap detection always runs; overlap detection is skipped (written empty directly) whenever has_invalid_edges() is already False, see "Skipping overlap detection when the coverage is already valid" below.
3. _03_outputs: exports {name}_02 to the issues file. No hard gate: unlike every other tool, topo-detect never modifies geometry, so there is nothing to validate against, the same "no topology hard-gate" precedent change already sets for its own read-only comparison.

Skipping overlap detection when the coverage is already valid¶

main() checks has_invalid_edges() (ST_CoverageInvalidEdges_Agg, the shared core/coverage.py) before running _build_overlaps (_02_issues.py's bbox-prefiltered O(n^2) self-join), and writes an empty overlaps table directly when it's already False: a coverage with no invalid edges cannot contain an overlapping or nested pair either. Gap detection (_build_gaps) always runs regardless: unlike overlaps, no cheaper GEOS primitive answers "are there any gaps" without doing the same whole-table union _build_gaps itself needs to extract them. See docs/adr/0007-skip-overlap-detection-when-valid.md.

_build_overlaps's bbox columns are precomputed once on {table}_narrow, not called inline per pairwise comparison in the join's ON clause: the latter makes DuckDB recompute the envelope per comparison instead of once per row, which hangs indefinitely on a table with even a few very-high-vertex-count polygons. See docs/adr/0014-bbox-inline-recompute-in-join.md.

has_invalid_edges() alone cannot stand in for gap detection: it only detects overlaps/mismatched edges, never gaps (see docs/reference/shared.md).

Sliver detection was removed¶

Earlier versions also detected (but topo-clean never auto-fixed) slivers, near-miss boundary mismatches. Dropped entirely: never fixable without re-noding the whole coverage (an unacceptable side effect for unattended batch use), and detection itself reproducibly OOM'd on real data even at small scale. Any near-miss boundary mismatch is now an upstream data-quality issue outside this tool's scope; fixing it remains a human decision (re-digitizing, manual QGIS/ArcGIS editing), just without an automated detector flagging candidates. See docs/adr/0006-sliver-detection-removed.md.

Issues table schema¶

key VARCHAR, kind VARCHAR, area_m2 DOUBLE, max_width_m DOUBLE, thinness_ratio DOUBLE, unit_a BIGINT, unit_b BIGINT, geom GEOMETRY. kind is 'gap' or 'overlap'. thinness_ratio is populated only for gap rows; unit_a/unit_b (the two fids involved) only for overlap rows. Geometry is always Polygon, so any of edge-extend's four export formats (including Shapefile) can hold the issues file. topo-clean's own issues output extends this schema further with measured fix outcomes (fixed, filled_area_m2, unit_a_area_change_m2, unit_b_area_change_m2), see docs/explanation/topo_clean.md; topo-detect's own output never has those columns, since nothing was fixed.

Units: decimal degrees in, meters out¶

area_m2/max_width_m are computed from raw degree-space geometry via a centroid-latitude cos_lat_factor (core/units.py, one degree of longitude shrinks by cos(latitude), scaled by the dataset's centroid latitude), approximate over very large north-south extents, but adequate for a reporting column. topo-detect reports every detected gap and overlap regardless of size; no floating-point noise floor is applied, after testing found no native GEOS jitter to guard against. See docs/adr/0010-noise-floor-removed-no-jitter-found.md.

Resilience¶

Each of the two detection queries (gap/overlap) falls back to an empty result for that one kind (logged) on a GEOS failure, rather than raising, consistent with edge-match's "failed group is logged and dropped, not fatal" precedent, applied per-detection-kind here instead of per-group.

Portolan-scale profiling¶

Real admin-boundary layers, --debug, Apple Silicon/10 logical cores (run before sliver detection/reporting was removed from what was then topo-clean's own issues stage; the wall time/RSS figures below no longer include a sliver-detection pass, which was the most expensive part at scale; expect faster/lighter runs now):

This run surfaced and fixed two scale bugs in _02_issues.py's _build_overlaps, both only visible past a few thousand fids: the overlap join predicate was too permissive, and the self-join ran against a wide table instead of a narrow projection. See docs/adr/0011-overlap-detection-scale-bugs.md.

A later run against Colombia admin3 surfaced a third scale bug in the same function: inline ST_XMin/ST_XMax/ST_YMin/ST_YMax calls in the overlap self-join's ON clause, which never finished (57+ min, killed) on a table with a few 20k-54k-vertex polygons, fixed by precomputing bbox columns instead of calling the envelope functions inline. See docs/adr/0014-bbox-inline-recompute-in-join.md.

Philippines admin3 exceeded the 4 GB container target (5.15 GB peak) on this run, driven mostly by the (since-removed) sliver-detection pass; gap/overlap detection alone is expected to be substantially lighter. topo-detect has no resampling knob to fall back on (nor does edge-extend anymore, see docs/explanation/voronoi-memory.md), so per this repo's "document, don't gate" policy this is noted here rather than runtime-checked.

A more recent real-world run, detecting issues on a 56,550-row/111-country combined layer (portolan adm0 subset), measured _02_issues's gap-union + overlap self-join at ~17 minutes, independent of and before any fix stage. That standalone cost, and the value of inspecting a layer's defects without committing to fixing them, is what motivated pulling detection out of topo-clean into its own tool. See docs/adr/0028.