AeroBench

AeroBench is an independent, sliced stress-test for drone-vision detectors — counter-UAS detection today, GNSS-denied visual localisation next.

Get a free evaluation Submit your model

What you get

A per-slice “where it breaks” report — where your model holds, where it breaks, with example frames — in 5 working days, free. Delivered as report.json (machine-readable) and report.html (tables by slice).

Per-slice mAP

mAP@0.5 and mAP@0.5:0.95 overall and sliced by target size (tiny/small/medium/large, by long-side pixels), time of day (day/dusk/night), weather (clear/haze/rain/fog) and backlighting (yes/no).

False positives on negatives

Your false-positive rate against the 500 unlabelled negative frames — empty backgrounds, not distractor objects. What your detector fires at when there is nothing there.

Example failure frames

Example frames for your worst-performing slice, so the number comes with the pictures behind it rather than a score on its own.

Read your report the way we read our own: a structured, honest view of where a model’s behaviour shifts across conditions on our synthetic slices — not a calibrated forecast of how your model will perform on some other real-world test set we haven’t run.

How it works

  1. You send predictions, or a container. COCO results format (per-detection image_id, category_id, bbox, score) for the DUT Anti-UAV test split — the simplest path, no infrastructure needed on your side — or a Docker image exposing an inference endpoint we can call. The format is written out in SUBMIT.md.
  2. We run aerobench eval --track detect. The same command that produces every number on the leaderboard below, including our own baselines, against the same ground truth. We do not re-run your model; we re-run the evaluator on your predictions — no black-box scoring.
  3. You get the per-slice report in 5 working days. Free, no strings. Your results stay private unless you opt into a public leaderboard entry, in writing and separately.

Optional, after the report

A hard-case pack + re-evaluation (€2,000–5,000, 2 weeks): synthetic frames targeted at the specific gaps your report shows, then a second report to see if the gap moved. We’re honest that this isn’t guaranteed to work — our own v2 attempt at exactly this made things worse; see the evidence below.

Evidence

We have measured how a model trained on AeroBench transfers to real footage, rather than asserted that it does. One seed, DUT Anti-UAV real test set, 2,200 images.

ExpTraining dataEval set mAP50mAP50-95Reading
E1 Synthetic only (AeroBench Detect v1) Real (zero-shot) 0.375 0.195 Fails a 0.5 usability bar. Synthetic-only does not replace real labels.
E2 Synthetic pre-train + 10% real Real 0.761 0.468 Matches E3 on mAP50 (noise); +0.034 mAP50-95 — suggestive box-quality gain, one seed, not established.
E3 COCO pre-train + 10% real (no synthetic) Real 0.759 0.434 The fair baseline for “is synthetic pre-training worth anything.”
E4 100% real (5,200 images) Real 0.912 0.580 The ceiling. Real labels dominate everything above.
v2 (E1v2) Synthetic v2 (distractors, near-field, capture realism) Real (zero-shot) 0.159 0.063 Worse than v1. Our own attempt to close the gap regressed it. Published, not hidden.

What this does and doesn’t support: synthetic data is not a substitute for real labels on this evidence. Synthetic pre-training may help label efficiency at the margin — one seed, no error bar, call it a direction, not a claim. What AeroBench sells is the measurement itself: a reproducible, per-slice stress-test, applied first to our own model and published honestly (including the v2 regression), then applied to yours.

Full methodology, per-slice breakdowns, and failure examples: synthetic→real transfer on DUT Anti-UAV (E0–E4) and the Detect v2 re-run. Both write-ups are reproduced in full further down this page.

Nav (GNSS-denied localisation) — coming. A baseline exists: nadir tracking from a ground-truth-seeded start reaches p50 4.7 m, 87% of frames within 10 m. Cold-start localisation is unsolved, and this has not yet been validated against real imagery. We are not offering a Nav evaluation yet; we’ll open one once real-imagery validation exists.

Leaderboard

AeroBench Detect, one row per evaluated model. Every number is produced by the same command — aerobench eval --track detect — over the submitted predictions; per-bucket columns are mAP50 by long side in pixels (tiny < 16, small 16–32, medium 32–96, large ≥ 96). Read the “real frames” column before the score column: a model trained on thousands of real labelled frames and one trained on none are not competing at the same thing. Rows are sorted by mAP50 across the whole table, but a score is only comparable within one eval set — check the eval-set column before reading two rows against each other. Rows marked internal baseline are our own runs, not submissions; rows whose eval set is — did not declare what they were scored on. See SUBMIT.md for how to submit a model.

modeleval settraining datareal frames mAP50mAP50-95 tinysmallmediumlarge datereport
COCO → full real DUT train internal baseline AeroSynth · yolov8n · e4_real_full DUT Anti-UAV test (2 200 images, 2 245 boxes) COCO weights then full DUT train, 5 200 real images real 5200 0.912 0.580 0.535 0.816 0.913 0.968 2026-09-04 report
AeroSynth Detect v1 — sanity check on synthetic val internal baseline AeroSynth · yolov8n · e0_sanity AeroBench Detect v1 synthetic val (500 images) — NOT DUT AeroBench Detect v1 - 4 500 synthetic frames, no real data synthetic 0 0.905 0.697 0.706 0.929 0.996 0.999 2026-09-04 report
AeroSynth Detect v2 — sanity check on synthetic val internal baseline AeroSynth · yolov8n · e0v2_sanity AeroBench Detect v2 synthetic val (500 images) — NOT DUT AeroBench Detect v2 - 4 500 synthetic frames, no real data synthetic 0 0.826 0.645 0.466 0.820 0.931 0.980 2026-09-05 report
AeroSynth Detect v1 → 10 % real internal baseline AeroSynth · yolov8n · e2_syn_then_real10 DUT Anti-UAV test (2 200 images, 2 245 boxes) Detect v1 (4 500 synthetic) then 520 real DUT train images mixed 520 0.761 0.468 0.242 0.744 0.717 0.848 2026-09-04 report
COCO → 10 % real (no synthetic) internal baseline AeroSynth · yolov8n · e3_real10_only DUT Anti-UAV test (2 200 images, 2 245 boxes) COCO weights then 520 real DUT train images, no synthetic real 520 0.759 0.434 0.156 0.713 0.743 0.824 2026-09-04 report
AeroSynth Detect v2 → 10 % real internal baseline AeroSynth · yolov8n · e2v2_syn_then_real10 DUT Anti-UAV test (2 200 images, 2 245 boxes) Detect v2 (4 500 synthetic) then 520 real DUT train images mixed 520 0.750 0.448 0.242 0.703 0.733 0.837 2026-09-05 report
AeroSynth Detect v1 — synthetic only internal baseline AeroSynth · yolov8n · e1_syn_only DUT Anti-UAV test (2 200 images, 2 245 boxes) AeroBench Detect v1 - 4 500 synthetic frames, no real data synthetic 0 0.375 0.195 0.038 0.201 0.381 0.458 2026-09-04 report
AeroSynth Detect v2 — synthetic only internal baseline AeroSynth · yolov8n · e1v2_syn_only DUT Anti-UAV test (2 200 images, 2 245 boxes) AeroBench Detect v2 - 4 500 synthetic frames, no real data synthetic 0 0.159 0.063 0.071 0.203 0.262 0.118 2026-09-05 report

Everything below is the benchmark’s own working record, built by scripts/build_site.py from the report files and write-ups in the repository. Every number is measured, none is typed in by hand. Open a section to read it.

Datasets

Dataset cards (4)

AeroBench Detect v0 (composite fast path)

docs/datasets/detect_v0.md

AeroBench Detect v1 (exposure-matched sprites)

docs/datasets/detect_v1.md

What changed vs v0

v0's sprites (assets/sprites) were rendered at a fixed Blender world Strength of 1.0, independent of each HDRI's own exposure. The background panorama, however, is tonemapped by load_pano_ldr to a fixed target of 0.18 mean luminance (hdri.pano_exposure, commit 676c6ee) — for these 40 HDRIs that means scaling the raw HDR by 0.19x-2.72x (median 0.34x) before compositing. v0 lit its drones at the raw, un-scaled strength, so most were rendered several times brighter than the exposure their background was tonemapped to. render-sprites now passes --strength <pano_exposure(hdr)> per HDRI, so sprite lighting matches the background's tonemap exactly. assets/sprites_v1 is a full re-render with this fix: same seed (1), same 40 HDRIs, same per-HDRI count (30); the drone silhouettes (alpha, hence bounding boxes) are unaffected, only RGB brightness changes.

Exposure and mean opaque-pixel brightness (grayscale, alpha>0 pixels only, averaged over each HDRI's 30 sprites), v0 vs v1, all 40 HDRIs:

HDRI exposure v0 opaque-mean v1 opaque-mean v1/v0 sprites.json sha256 (v1, 12c)
aarfontein_dirt_road 0.255 112.3 60.0 0.53 583130adbeb3
aarfontein_dusk 0.272 115.8 61.8 0.53 f77f07b0fa3b
abandoned_church 0.416 88.1 57.0 0.65 0ee8cdd8d0df
abandoned_hopper_terminal_01 0.338 112.7 66.9 0.59 fd37e4e87bb8
abandoned_hopper_terminal_02 0.397 101.2 64.7 0.64 2d792230cfc4
abandoned_hopper_terminal_03 0.274 118.9 64.0 0.54 0106229612c9
abandoned_hopper_terminal_04 0.284 121.9 67.6 0.55 d01d9f852fc3
abandoned_parking 0.296 113.8 63.2 0.56 0deb72c780f0
abandoned_pathway 0.478 84.4 59.0 0.70 4f1965764a09
abandoned_slipway 0.350 112.2 67.6 0.60 4d28a53ec78a
abandoned_tank_farm_01 0.420 91.1 59.4 0.65 22e39f674ca1
abandoned_tank_farm_02 0.437 90.0 59.9 0.67 c0569dec6e2f
abandoned_tank_farm_03 0.339 113.8 68.2 0.60 a2851b3e88f9
abandoned_tank_farm_04 0.392 93.5 59.3 0.63 9f9ddfa9f35d
abandoned_tank_farm_05 0.302 124.6 70.5 0.57 ee9dca80dcd6
altanka 0.263 128.0 68.9 0.54 2634f94a808f
arboretum 0.335 115.2 68.2 0.59 f3775e4e9b88
autumn_forest_04 0.262 114.8 61.8 0.54 ee9f0dab10ca
aviation_museum 0.249 130.7 68.3 0.52 9e19caab7324
bambanani_sunset 0.275 111.6 59.8 0.54 e7fa0d57dd48
beach_parking 0.500 86.9 61.7 0.71 ac004c4056a8
belfast_sunset 0.277 106.7 57.1 0.54 3baac07acc34
belfast_sunset_puresky 0.192 136.4 62.8 0.46 d4bfddbbe134
bell_park_dawn 0.477 98.6 68.9 0.70 cb76cd6b2535
bell_park_pier 0.435 97.4 65.0 0.67 b5ee0375ea54
between_bridges 0.209 145.1 73.9 0.51 a0ecb1a6542c
blaubeuren_church_square 0.574 93.2 71.2 0.76 77ac68eb864e
blaubeuren_night 0.464 101.3 70.5 0.70 25799956eed7
blue_lagoon_night 0.286 128.2 71.2 0.56 3bc8503d7417
clarens_night_01 0.285 116.6 65.0 0.56 dcb667ffe565
clarens_night_02 0.288 100.5 54.7 0.54 68dfbb511e74
cobblestone_street_night 0.318 112.1 66.3 0.59 dba6cee8fbd2
courtyard_night 0.905 67.8 64.5 0.95 3517e8304e49
dikhololo_night 2.722 44.6 74.7 1.67 1c2b388c0458
donkin_reserve 0.246 129.1 67.0 0.52 97fdefa65e77
golden_bay 0.348 111.3 67.5 0.61 717a3e58fe1b
hansaplatz 0.279 120.4 66.3 0.55 b734086d41cc
kloppenheim_02 0.775 76.5 67.5 0.88 1311e14ab2ad
kloppenheim_02_puresky 0.788 79.3 70.7 0.89 aac319021b1d
rooftop_night 0.401 97.6 62.6 0.64 2699a3e7cc00

40/40 HDRIs, mean opaque-pixel brightness (0-255) averaged across all 30 sprites/HDRI: v0 106.1 -> v1 65.1 overall (per-HDRI v1/v0 ratio mean 0.64, range 0.46-1.67). The three HDRIs spot-checked at render time (rooftop_night, abandoned_church, golden_bay) land at 88-111 (v0) -> 57-68 (v1), matching exposures 0.35-0.42 as expected. Not every HDRI gets darker: courtyard_night (exposure 0.91) and dikhololo_night (exposure 2.72, the one HDRI in this set whose raw panorama is under-exposed) end up roughly the same or brighter in v1 — the fix follows each panorama's own exposure rather than uniformly dimming every sprite.

Brightness-consistency check (scripts/brightness_check.py)

For the first 1000 positive images (by image id), for every box: mean grayscale brightness inside the box vs. an 8px ring immediately outside it (clipped to the frame edges).

$ python scripts/brightness_check.py out/detect_v0
images: 1000
boxes: 1508
median_ratio: 0.984
share_ratio_gt_2: 0.036

$ python scripts/brightness_check.py out/detect_v1
images: 1000
boxes: 1508
median_ratio: 0.907
share_ratio_gt_2: 0.019

Same 1000 images and the same 1508 boxes at the same pixel coordinates (see the self-check bullet above), so this is a paired before/after comparison, not two independent samples. The share of egregiously-overbright boxes (inside/ring > 2) roughly halves, 3.6% -> 1.9% — that is the tail the root-cause commit (676c6ee) describes, and it moves the direction the fix predicts.

The median moving from 0.984 to 0.907 is not closer to 1, and is worth calling out rather than glossing over: a box's rectangle is mostly transparent alpha for a multirotor's spread-out arms/props, so a whole-box mean is diluted by however much background shows through inside that rectangle, in both v0 and v1 — it mixes drone-pixel and background-pixel brightness rather than isolating the drone. That dilutes the metric everywhere, so the aggregate median is a blunt read on a fix that mainly removes an overbright peak (see the per-box example below), not a uniform brightness shift. scripts/brightness_check.py is left as specified (whole box vs. ring, no alpha) because the composited JPEGs it reads don't carry an alpha channel to isolate opaque pixels with — the raw-sprite table above is the cleaner signal for the underlying fix; this check is the closest thing derivable from the shipped dataset files alone, and it does move in the right direction on the metric that most directly targets "egregiously bright."

Per-box detail for the two examples used in the visual check below (both [x,y,w,h] in pixels, ratio = inside/ring, max = brightest pixel in the box):

image box v0 inside/ring/ratio/max v1 inside/ring/ratio/max
000006.jpg (quadcopter, clarens_night_02) [74,66,86,41] 107.1 / 99.3 / 1.08 / 241 88.6 / 99.3 / 0.89 / 146
000016.jpg (fixed-wing, golden_bay) [181,285,68,62] 86.4 / 74.1 / 1.17 / 191 71.2 / 74.1 / 0.96 / 132

Both boxes cross from ratio > 1 (v0, drone brighter than its ring) to ratio < 1 (v1); the brightest pixel in each box (the part most likely to look like an overbright artifact) drops by 95 and 59 (out of 255) respectively.

Visual check

Read on out/detect_v0/images/000006.jpg vs out/detect_v1/images/000006.jpg (same box, a quadcopter over a rain-streaked dusk scene, clarens_night_02): in v0 the hub is a distinct, near-white glowing patch that reads as pasted onto the muted, rain-hazed backdrop; in v1 the same hub is a soft pale blue-grey that sits inside the scene's tonal range — still the lightest part of the drone, but no longer an obvious sprite seam. 000016.jpg (golden_bay, dusk, fixed-wing): v0's fuselage is a saturated, almost glowing orange that pops out of the dim building silhouette behind it; v1's is a darker, more brownish orange that recedes into the same silhouette. 000000.jpg (daytime, two boxes, both small/thin drones): visually near-identical between v0 and v1 — daytime HDRIs in this catalog sit closer to pano_exposure ~1.0 (less scaling needed), so the fix has little to correct there, consistent with the inside-mean deltas being only 3-6 (daytime) vs 15-19 (the two night/dusk examples above).

Remaining limitations

Procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video). The whole-box brightness-consistency check above is a blunt instrument for sparse-silhouette boxes (transparent gaps inside the rectangle dilute the mean toward background) — it understates the fix's effect on the aggregate median while still correctly showing the overbright tail shrink; the raw-sprite opaque-pixel table is the more direct evidence for the underlying exposure fix.

AeroBench Detect v2 (distractors, near field, capture realism)

docs/datasets/detect_v2.md

For reference, v1 on the same 40 panoramas: 5000 frames, 500 negatives, 6760 boxes, zero distractors, no near field, wall time 7m47s.

History: this is the second v2 build

The first v2 build (2026-09-04, commit 09d522d) shipped with two defects, both found by review after it had already been trained on:

That build's transfer numbers (E1v2 mAP50 0.154 and the ablation arms) are recorded in docs/experiments/2026-09-detect-v2.md as the v2-rc result and are kept there unchanged: they are what that data did, and §5 of that write-up is what identified C2. This card describes the rebuild at 6cf56d0, which fixes both. The rebuilt set has since been trained and evaluated, and it did not help: E1v2 mAP50 0.159 against v2-rc's 0.154 and v1's 0.375. The card's claim is therefore only that two measured data defects are gone — not that the data is better. v1 remains the recommended set; see that write-up's §3 and §5.

What changed vs v2-rc
v2-rc (09d522d) v2 (6cf56d0)
box vs pixels 636/5972 at IoU < 0.9, 51 < 0.5 5893/5972 at IoU >= 0.9, 0 < 0.5 (below)
distractor_px [12, 260], median 136 [10, 96], median 53, max 95
clutter instances >= 96 px 3824 0
distractor kinds 6 5 (helicopter dropped)
distractor instances 5787 5787 (unchanged count)
boxes / frames / negatives 5972 / 5000 / 1000 5972 / 5000 / 1000

The job stream is otherwise the same: the two builds plan the same 5972 targets in the same frames, so the difference is exactly "the clutter got smaller, the helicopters left, and the boxes now follow the skew".

Why the clutter is capped at 96 px. annotate.SIZE_BUCKETS["large"] starts at 96 px, and distractor_px: [10, 96) is now strictly below it, so no unlabelled object lands in the same size bucket as a labelled large drone (0 instances vs 1319 labelled large boxes, against 3824 vs 1318 in v2-rc). This is v3 hypothesis 1 from the experiment write-up made real — and E1 has now falsified it as the main cause: with zero size-matched clutter, large mAP50 recovered only 0.064 → 0.118 against v1's 0.458. The clutter cap is still the right thing to do; it is not what was breaking the large bucket.

Why helicopter was dropped. Its main-rotor disc is rendered with the same translucent-disc trick as the drone propellers, so under --merge-classes it is an unlabelled object built out of the labelled class's most distinctive feature. Its sprites are still on disk (160 of the 960 distractor renders); they are filtered out by sprites.load_extra_index, which skips any distractor record whose kind is no longer in scenario.DISTRACTOR_KINDS.

What changed vs v1

v2 turns three of the five failure hypotheses from docs/experiments/2026-09-transfer-dut.md §4 into data. Hypotheses 3 (real CAD airframes) and 4 (real photographs as backgrounds) are not addressed — see the last section. annotate.SIZE_BUCKETS was deliberately left untouched, so the v1 and v2 size slices stay comparable.

1. Unlabelled distractors (hypothesis 1 — the big one)

A Blender sprite family rendered by render_extras.py --kind distractors with the same procedural style, the same per-HDRI seed and the same per-HDRI world strength (hdri.pano_exposure) as the drones. Five kinds, cycled so each is guaranteed to appear:

kind what it is why it is here
bird_soaring elongated dark body, near-horizontal wings the classic sky false positive
bird_flapping same body, wings raised 20-55 degrees a second bird silhouette, not one pose
airliner_far fuselage + swept wing + fin + two nacelles, long lens at 8-20x distance "aircraft far away", named in §4
mast_with_cables vertical pole, 2-4 cross arms, 2-5 thin guy cables, camera below the horizon thin man-made structure
rooftop_hardware 2-4 AC boxes, 1-2 antennas, a parapet slab exactly what E1 fired on at conf 0.25

They are composited before the drones and are never annotated, so a drone is never occluded by an unlabelled object and its alpha-derived box stays exact. Config knobs: distractor_prob: 0.50 (half of the positive frames), negative_distractor_prob: 0.85 (almost every negative), distractors_per_frame: [1, 3], distractor_px: [10, 96]. Expected share of frames with clutter: 0.20*0.85 + 0.80*0.50 = 0.57; measured 0.575 (2876/5000). Negatives are up from 10 % in v1 to 20 % (negative_fraction: 0.20), and 841 of the 1000 carry clutter (target 0.85).

5787 distractor instances were composited in total: airliner_far 1179, bird_flapping 1174, bird_soaring 1119, mast_with_cables 1136, rooftop_hardware 1179 (counted from the planned job list; the per-frame tags["distractors"] in annotations.json sum to the same 5787, and distractors_missing is 0, so every planned instance found a sprite). Their planned long side is min 10 / median 53 / max 95 px.

2. Near field (part of hypothesis 2)

scenario.SIZE_RANGES_PX gained a fifth bucket near: (200, 1100) — from "fills a third of the frame" up to "wider than the 640 px frame", so the sprite is genuinely clipped by the frame border. size_mix gives it 0.10; 621 of the 5972 targets (0.104) were planned as near.

These are not upscaled v1 sprites. render_extras.py --kind nearfield re-renders the same procedural drones (build_multirotor / build_fixed_wing, class cycled CLASSES[k % 3]) at 768 px with a close camera (dist_mult=(1.05, 1.5)) and a wide lens (24-35 mm), i.e. real near-field perspective. render_frame prefers this pool whenever size_bucket == "near".

Rendered result: 615 boxes with a long side >= 200 px (near_field_boxes), long side min 200 / median 492 / max 640 px. 8 planned near targets came out under 200 px, cut down by the frame border, so 613 of the 615 come from the near bucket; the other two are large targets planned at exactly 199 px whose box crossed 200 px when the rolling shutter sheared it (1.2 px of skew, one pixel of width). 653 boxes touch a frame edge, and they come from where you would expect: 512 near, 111 large, 30 medium. Note that annotate.SIZE_BUCKETS still ends at large: (96, inf), so per_size.large (1319) contains both the v1-style large drones and the near-field ones; near_field_boxes is the separate key.

In v1 this whole slice was empty, and §4 of the plan-3 report showed a real DUT near-field drone (GT 865 x 305 px) breaking into pieces (best box IoU 0.109).

3. Capture realism (hypothesis 5)

Applied per frame, after compositing, in this order: atmosphere -> rolling shutter -> sensor noise -> shot noise -> JPEG requantisation. The rolling shutter also shears the boxes (camera. rolling_shutter_bbox), which is the C1 fix.

effect config measured on this build
motion blur along the drone's own velocity vector motion_blur_velocity: true, blur_prob: 0.40 all 5972 targets carry a vel_deg (v1 drew a fresh random angle inside the compositor instead); blur is actually applied on the 1985 frames (0.397) where blur_px > 0, min 1.00 / median 2.47 / max 4.00 px
Gaussian sensor noise, sigma drawn per frame noise_sigma_range: [2.0, 9.0] all 5000 frames, sigma min 2.00 / median 5.42 / max 9.00 (v1 used a fixed 3.0 day / 6.0 night)
signal-dependent shot noise shot_noise_k: 6.0 all 5000 frames
rolling-shutter skew (pixels and boxes) rolling_shutter_prob: 0.25, rolling_shutter_px: [1.0, 6.0] 1258 frames (0.252), shift min 1.00 / median 3.50 / max 6.00 px, carrying 1496 boxes
in-pipeline JPEG requantisation jpeg_quality_range: [55, 95] all 5000 frames: 50-59 598, 60-69 1237, 70-79 1245, 80-89 1197, 90-99 723

The in-pipeline requantisation is deliberately a second compression: the file is still written at quality 92 by build_detect, so every frame goes through two JPEG passes, like a real frame recompressed on export.

Sprites

family per HDRI total command render wall time
drones (v1, reused) 30 1200 aerosynth render-sprites assets/hdris assets/sprites_v1 --per-hdri 30 --seed 1 --size 256 --samples 32 (v1; copied, not re-run) — (copy only)
distractors 24 960 (800 used) aerosynth render-extras assets/hdris assets/sprites_v2 --kind distractors --per-hdri 24 --seed 5 --size 256 --samples 32 19m18s (1158 s)
near field 9 360 aerosynth render-extras assets/hdris assets/sprites_v2 --kind nearfield --per-hdri 9 --seed 6 --size 768 --samples 32 15m46s (946 s)

The copy was cp -r assets/sprites_v1 assets/sprites_v2 (1200 PNGs + 40 sprites.json), then the two render-extras runs wrote the new families beside the drone sprites. 40/40 HDRIs ok, zero FAIL, on both families; total extra-sprite render time 35m04s (2104 s) on the founder's RTX 3070 Ti. The sprites were not re-rendered for this build — the helicopter change is a filter at index time, not a re-render, so the two builds composite bit-identical sprite pixels for every kind they share.

Indexed for this build: 800 distractor sprites (160 per kind x 5) and 360 near-field sprites. The 160 helicopter renders remain on disk and in distractors.json (which is what the manifest's count: 24 and its sha256 cover), and are skipped when the index is built.

One byte-level change to the PNGs. Blender 4.2 writes two identical eXIf chunks into every PNG it renders, so cv2.imread printed libpng warning: eXIf: duplicate on each of the ~12000 sprite reads a build does. sprites.strip_duplicate_exif was run once over all 2520 sprite PNGs (and both render-* commands now do it automatically for what they produce): the repeated chunk is dropped, the decoded pixels are unchanged (verified on a random sample of 12), and the sprites.json sha256 in the v1 manifest still matches 40/40, because the manifest hashes the JSON, not the PNG bytes. The build log for this dataset contains zero libpng warnings.

Build and self-checks

$ aerosynth build-detect configs/detect_v2.yaml out/detect_v2 --sprites assets/sprites_v2
frames: 5000
negatives: 1000
annotations: 5972
per_size: {'tiny': 1345, 'small': 1742, 'medium': 1566, 'large': 1319}
per_time_of_day: {'day': 1629, 'night': 1717, 'dusk': 1654}  per_weather: {'clear': 2863, 'rain': 460, 'haze': 1180, 'fog': 497}
frames_with_distractors: 2876
negatives_with_distractors: 841
near_field_boxes: 615
boxes_touching_border: 653
per_jpeg_quality: {'50-59': 598, '60-69': 1237, '70-79': 1245, '80-89': 1197, '90-99': 723}
# 559 s wall, manifest failed_frames: 0, no WARN line

Boxes vs pixels (the C1 check). For every one of the 5972 boxes, the target's sprite alpha was placed on an empty frame with placement arithmetic written independently of composite.py, sheared by that frame's own rolling shutter, thresholded at the same alpha > 8, and its tight box compared with the shipped annotation:

boxes checked: 5972  (frames whose box count did not replay: 0)
min IoU: 0.7778   mean IoU: 0.9928
boxes with IoU < 0.9: 79   < 0.5: 0
low-IoU boxes by size bucket: {'tiny': 79}
self-check, unskewed frames: 4476/4476 boxes identical to the pixels
counterfactual on the 1496 boxes of skewed frames: the pre-fix (unsheared) box
  would score IoU < 0.9 on 625 of them

Read that in order. The self-check line is what makes the rest trustworthy: on the 4476 boxes of frames with no skew, the independent re-derivation lands on exactly the same box, to the pixel, so it is measuring the compositor and not itself. 5008 of the 5972 boxes are pixel-identical, 5893 (98.7 %) are at IoU >= 0.9, and none is below 0.5 (v2-rc: 636 below 0.9, 51 below 0.5). The 79 exceptions are all 6-8 px boxes (worst 0.778): the shear is a closed form on integer pixel coordinates, so on a box a handful of pixels wide a sub-pixel displacement rounds to a one-pixel difference against the interpolated pixels, and one pixel on a 7 px box is 0.12 of IoU. That residual is a property of representing an interpolated shear with an integer box, not a displacement — the v2-rc defect was the box sitting up to 6 px away from its drone.

Dataset QA (hard checks).

$ python scripts/detect_qa.py out/detect_v2 --sample 12 --img-out docs/datasets/img
image_size: (640, 640)
...
negative_share: 0.200
frames_with_distractors: 2876 (0.575 of frames)
negatives_with_distractors: 841 (0.841 of negatives)
near_field_boxes: 615 (>= 200 px on the long side)
boxes_touching_border: 653
zero_area_boxes: 0
hard checks: 0 problem(s)
OK

Exit code 0, and no check_coverage warning at all. The hard checks that would have returned a non-zero exit code: no box with w<=0 or h<=0, no box outside the frame, COCO and YOLO labels agree row by row in both directions, every file_name exists, every annotation points at an existing image. The coverage warnings are now measured against this build's own config, read from manifest.json (negative fraction, whether the near bucket was planned), so silence here means "it delivered what it asked for", not "the check was skipped".

Perfect predictions. annotations.json fed back in as COCO results with score 1.0 ([{"image_id", "category_id", "bbox", "score": 1.0}, ...], one record per annotation):

$ aerobench eval --track detect --gt out/detect_v2/annotations.json \
    --pred out/detect_v2/perfect_preds.json --out out/detect_v2/report_perfect
mAP50=1.000 mAP50-95=1.000

out/detect_v2/report_perfect/report.json: overall mAP50 1.000 / mAP50-95 1.000; every size slice (tiny/small/medium/large) 1.000; every time-of-day, weather and backlit slice 1.000; negatives {"images": 1000, "false_positives": 0, "fp_per_image": 0.0}; counts images 5000, annotations 5972, predictions 5972.

Determinism.

$ aerosynth build-detect configs/detect_v2.yaml out/detect_v2_rep --sprites assets/sprites_v2 --limit 300

Comparing the two annotations.json on the 300 shared image ids (images + annotations + categories, json.dumps(..., sort_keys=True)): same annotations: True, sha256 315f0e54062a8e24... on both sides. This covers the new JPEG requantisation, both noise terms and the box shear, since all of them derive from the frame's seeded RNG.

Unlabelledness (the central invariant). Distractors must never produce a box. Replaying the job list (make_jobs on the same config and the same HDRI order build_detect uses) and comparing against the shipped annotations:

planned jobs: 5000, images: 5000, annotations: 5972
frames with boxes > planned drones (a distractor would have been labelled): 0
negative frames (0 planned drones) carrying a box: 0
frames with distractors: 2876
  boxes in those frames: 3023; planned drones in those frames: 3023
frames tagged distractors_missing: 0
total planned drones: 5972; total annotations: 5972; dropped: 0

So in the 2876 frames that carry clutter, the box count equals the drone count exactly, and no negative frame got a box. distractors_missing: 0 also means every planned distractor found a sprite pool — in particular, dropping helicopter from the kind list did not silently leave planned clutter undrawn.

Brightness consistency (scripts/brightness_check.py, first 1000 positive images, box mean vs an 8 px ring outside it). Exposure handling did not change between v1 and v2, so this is a "did anything drift" check, not a fix:

$ python scripts/brightness_check.py out/detect_v2
images: 1000
boxes: 1494
median_ratio: 0.913
share_ratio_gt_2: 0.017

v1 on the same script: 1508 boxes, median_ratio 0.907, share_ratio_gt_2 0.019 (v0: 0.984 / 0.036). v2 sits right next to v1 on both numbers, which is what "same sprites, same exposure" predicts. This is not the paired comparison the v1 card ran against v0: v2's first 1000 positive images are a different set of frames with different boxes (1494 vs 1508), because the job stream changed (more negatives, a fifth size bucket), so treat it as two samples from the same generator, not a before/after on identical boxes.

Visual check

The QA sampler above wrote 12 annotated frames; five are kept in docs/datasets/img/ (boxes drawn from annotations.json; the caption strip is the frame's own tags).

image what to look at
negative with distractors qa_negative_000926.jpg Negative frame, 2 distractors, 0 boxes. An airliner_far (68 px, lower left) and a bird_soaring (55 px, right) against dusk cloud. Nothing is boxed, and this frame counts towards the false-positive metric. jpeg=75, sigma 7.86.
clutter the size of the drones qa_random_002919.jpg The C2 fix, in one frame. Two multirotor_medium drones (75 and 81 px) are boxed; a bird_soaring (45 px), an airliner_far (79 px, right) and a mast_with_cables (46 px, left of centre) are not. All three unlabelled objects are now the same scale as the labelled ones — in v2-rc that airliner could have been 260 px wide next to a 12 px drone. boxes=2 distractors=3.
near field clipped by the frame, skewed frame qa_near_000291.jpg Near field on a rolling-shutter frame — the C1 fix. A near-field fixed_wing (planned 491 px) with rolling_shutter_px=2.88: the box [71, 163, 492, 273] sits on the airframe rather than 2-3 px to its left, which is what v2-rc would have shipped. A 9 px tiny multirotor is boxed at the right edge; the frame's two distractors (airliner_far 90 px, bird_soaring 36 px) are not.
frame-filling near-field drone qa_near_001759.jpg Frame-filling near field, in rain, backlit, skewed 5.17 px. A near-field multirotor planned at 959 px: the box [3, 315, 637, 325] is clipped by the frame at the right and the bottom, starts 3 px from the left edge, and its top edge lands on the rotor discs. This is the perspective the near-field sprite family exists for — not an upscaled 256 px sprite. jpeg=55.
tiny drones among unlabelled birds qa_tiny_001001.jpg Tiny drones next to bigger clutter. A 28 px multirotor_small and a 9 px multirotor_medium are boxed; three unlabelled birds (94, 83, 67 px) share the sky. The birds are still larger than the drones — the cap is at 96 px, not at "smaller than every target" — but none of them reaches the large bucket any more. boxes=2 distractors=3.

What is still wrong

AeroBench Nav v0 (Teruel)

docs/datasets/nav_v0.md

Rebuild note (2026-09-03): this is a full rebuild of v0 from the same seed/config after the final-review fix wave — the SUN lamp's azimuth now matches the Nishita sky's visible sun (previously every frame's shadows fell on the opposite side of the sun from where they should), texture_year is pinned to 2024 (previously an assumed-current mosaic), exposure moved to per-sequence tags, and manifest["conventions"] documents the pose/frame convention (all described above and below). Poses, tags, and per-sequence sun/haze/exposure values are byte-identical to the pre-rebuild build (seeded generation does not depend on rendering — verified directly, see Determinism below); only the rendered pixels and the manifest/config metadata changed. The previous build is kept at out/nav_v0_old for comparison.

Dataset layout: what's publishable

aerosynth build-nav writes only dataset files under OUT — as of the F2 fix, the Blender render inputs (heights.npy, texture.png, world.json: the exact orthophoto pixels and terrain each frame was rendered from, i.e. the localization answer key) are exported under cache_dir/_world_<cfg.name>/ instead, and never appear under OUT at all (build_nav used to write them to OUT/_world/).

The publishable file set — everything a build-nav run leaves under OUT — is:

Anything else found under a working out/nav_v0 directory in this repo's history (_sanity_*.png, *_preds.csv, report_*/) is not written by build-nav — those are ad hoc self-check/debug artifacts from validating this card (baseline predictions, eval reports, visual spot-checks) and should be stripped before publishing the dataset; they were never part of the tracked file set and carry no answer-key risk of their own (predictions/reports derived from the public data, not inputs used to render it), but they are also not part of what a consumer of the dataset needs. Confirmed directly on the 2026-09-03 rebuild: out/nav_v0/ has no _world/ (verified with ls); the pre-rebuild _world/, _sanity_*.png, and *_preds.csv are preserved for comparison under out/nav_v0_old/, which predates the F2 fix.

Conventions

Also written verbatim as manifest["conventions"] (kept in sync with navexport.CONVENTIONS) so a consumer of poses.csv/manifest.json can interpret x/y/yaw/pitch/roll without reading the render code:

field meaning
origin SW corner of patch in EPSG:25830 (origin_e, origin_n)
axes x east, y north, z up (metres above z0)
yaw_deg compass heading, clockwise from north; image top points along the heading
pitch_deg -90 nadir, -60 oblique
roll_deg right-wing-down positive
blender_euler (90+pitch, roll, -yaw) XYZ degrees — the Euler angles render_nav.py::camera_rotation feeds to the Blender camera
poses_csv_columns frame,t_s,x,y,z,yaw_deg,pitch_deg,roll_deg,agl_m

Tag distribution (frame counts; each sequence contributes all 80 of its frames to one bucket per tag)

altitude (AGL at t=0) sequences frames
low (< 120 m) 2 160
mid (120-200 m) 7 560
high (> 200 m) 3 240
view (pitch) sequences frames
nadir (pitch <= -80°) 4 320
oblique 8 640
time_of_day sequences frames
day (sun elevation > 15°) 12 960
map_gap_years sequences frames
1-3 12 960

pitch_choices: [-90, -90, -60] in configs/nav_v0.yaml samples nadir with probability 2/3 per sequence on average (rng.integers(0, len(pitch_choices)), two of three indices give -90°) — for 12 sequences the expected split is ~8 nadir/4 oblique, but seed 20260904's 12 draws landed on 4 nadir / 8 oblique (checked directly in src/aerosynth/navscenario.py:96; this is sampling variance for n=12, not a bug). This matters for reading the baseline table below: its coverage figures should be compared against the actual nadir share (1/3), not the config's long-run average (~2/3).

Sun elevation across the 12 sequences ranges ~22-59° (config range [10, 65]); haze fired on 5/12 sequences (haze_prob: 0.3 config, magnitudes 0.13-0.32 where present) — again plausible variance for n=12 Bernoulli(0.3) draws (expected ~3.6).

Self-check 1: perfect predictions

Ground-truth poses.csv fed straight back in as predictions (sequence,frame,x,y), evaluated with aerobench eval --track nav. perfect_preds.csv isn't written by any CLI command (it's the self-check's own input, not a dataset file — see "what's publishable" above) — generate it by concatenating every sequence's poses.csv into sequence,frame,x,y rows, e.g.:

$ python -c "
import csv, glob, os
rows = ['sequence,frame,x,y']
for d in sorted(glob.glob('out/nav_v0/seq_*')):
    with open(os.path.join(d, 'poses.csv'), newline='') as f:
        for r in csv.DictReader(f):
            rows.append(f\"{os.path.basename(d)},{r['frame']},{r['x']},{r['y']}\")
open('out/nav_v0/perfect_preds.csv', 'w', newline='').write('\n'.join(rows) + '\n')
"

$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/perfect_preds.csv --out out/nav_v0/report_perfect
p50=0.00 m  success<50m=1.000  coverage=1.000
p50 p90 p99 success<10m success<50m coverage frames
overall 0.0 m 0.0 m 0.0 m 1.000 1.000 1.000 960

Exactly the expected result (identical for every tag slice too) — confirms the eval pipeline itself introduces zero error when predictions equal ground truth.

Self-check 2: aided NCC baseline

aerobench baseline nav is a nadir-only aided template matcher: it knows AGL and yaw (compass heading) for each frame, rescales the nadir frame to the reference map's 1 m GSD (footprint_gsd = agl_m / fx_px), derotates it to north-up using the known yaw, and NCC-matches it against a search window of the reference map. Oblique frames have no well-defined north-up rectangular footprint, so the baseline does not attempt them at all — it predicted exactly 320/960 frames, i.e. all and only the nadir frames, under every variant below.

Correction on the search window: by default (--init-from-gt, the CLI's default) the very first frame of each sequence is seeded at the true ground-truth position ± a 150 m search window, not matched against the whole map — only later frames search around the previous fix. This means init_success and the nadir p50/p90/p99 below are not a measurement of the baseline finding its position from nothing; they measure how well it tracks once handed a correct starting fix. --no-init-from-gt runs the honest cold-start version, where frame 0 of every sequence is matched against the entire reference map with no positional hint at all — that is the fair number for "no external aiding of position". Both are reported below.

$ aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_preds.csv
baseline nav: 320 predictions -> out\nav_v0\baseline_preds.csv

$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline
p50=inf m  success<50m=0.333  coverage=0.333
p50 p90 p99 success<10m success<50m coverage frames predicted
overall inf inf inf 0.291 0.333 0.333 960 320
nadir 4.7 m 10.8 m 18.0 m 0.872 1.000 1.000 320 320
oblique inf inf inf 0.000 0.000 0.000 640 0

init_success: 0.333 (4/12 sequences — exactly the 4 nadir sequences: only they get a first-frame lock), mean_drift_pct: 1.79%, unmatched_predictions: 0 (every prediction row matched a GT sequence,frame — the F6 sanity check has nothing to flag here).

Per nadir sequence (the 8 oblique sequences all show p50: null, init_ok: false):

sequence p50 (m) final error (m) path length (m) drift %
seq_0003 3.8 2.4 433 0.56
seq_0004 4.5 2.5 622 0.41
seq_0007 4.1 6.6 601 1.10
seq_0009 9.4 21.5 423 5.08
--align-first variant (still GT-seeded start)

Aligning on the first predicted frame before computing errors (removes any constant first-fix offset, so what's left is drift rather than absolute bias):

$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline_align --align-first
p50=inf m  success<50m=0.333  coverage=0.333
p50 p90 p99 success<10m success<50m coverage
overall inf inf inf 0.256 0.333 0.333
nadir 5.1 m 16.1 m 31.6 m 0.769 1.000 1.000

init_success: 0.333, mean_drift_pct: 1.17% (lower than the un-aligned 1.79% — removing the fixed first-frame offset reduces path-relative drift even though the raw p50/p90/p99 land slightly higher than without alignment, since the aligning transform itself carries some of the first fix's own error into every later frame). init_success here is computed from the un-aligned first-frame error (the F4 fix — see below), so it correctly still reads 0.333 rather than being inflated by the alignment transform's own trivially-zero first-frame residual.

Cold start (--no-init-from-gt) — the fair "no positional aiding" number

Same aided NCC matcher (still given AGL and yaw), but frame 0 of every sequence is now matched against the entire reference map with no ground-truth position hint — this is what "the baseline localizes itself" actually means, as opposed to the GT-seeded numbers above which measure tracking-after-a-correct-fix.

$ aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_cold_preds.csv --no-init-from-gt
baseline nav: 320 predictions -> out\nav_v0\baseline_cold_preds.csv

$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold
p50=inf m  success<50m=0.250  coverage=0.333

$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold_align --align-first
p50=inf m  success<50m=0.256  coverage=0.333
p50 p90 p99 success<10m success<50m coverage
nadir, cold start 6.8 m 844.1 m 1367.9 m 0.628 0.750 1.000
nadir, cold start, align-first 8.8 m 824.5 m 1232.9 m 0.538 0.769 1.000

init_success drops from 0.333 (GT-seeded) to 0.250 (cold, no --align-first) — one of the four nadir sequences, seq_0007, fails to lock on at all without the GT seed: p50=756.3 m, final_error=564.1 m, drift_pct=93.9%, i.e. the matcher locks onto the wrong part of the map on frame 0 and never recovers. Under --align-first this sequence's aligned errors change (p50=757.7 m, final_error=1122.9 m, drift_pct=187.0%) but init_ok is still correctly False — this is the F4 fix: init_ok is now judged on the un-aligned first-frame error (which is what actually matters — "did the baseline find itself" — not on the post-alignment residual, which is trivially ~0 for whichever frame the alignment is anchored to regardless of how wrong that anchor was). Pre-rebuild, the same align-first case incorrectly showed init_ok: True for seq_0007 (the bug this card is now rebuilt without) and init_success: 0.333, one sequence too high; the correct value in both cold-start variants is 0.250 (3 of 4 nadir sequences acquire correctly; seq_0007 fails in both). The other three nadir sequences (seq_0003, seq_0004, seq_0009) localize correctly from a cold start with numbers close to the GT-seeded run (e.g. seq_0003: p50=3.8 m cold vs. 3.8 m GT-seeded). mean_drift_pct jumps to 24.99% (cold) / 47.76% (cold, aligned) — both dominated by seq_0007's divergence; excluding it, the other three sequences drift at roughly the same ~0.3-2% seen in the GT-seeded run.

Comparison across all four variants (nadir frames only — oblique is always coverage: 0 regardless of start mode)
variant p50 p90 p99 success<10m success<50m init_success mean drift %
aided, GT-seeded start (CLI default) 4.7 m 10.8 m 18.0 m 0.872 1.000 0.333 1.79
aided, GT-seeded start, align-first 5.1 m 16.1 m 31.6 m 0.769 1.000 0.333 1.17
aided, cold start (--no-init-from-gt) 6.8 m 844.1 m 1367.9 m 0.628 0.750 0.250 24.99
aided, cold start, align-first 8.8 m 824.5 m 1232.9 m 0.538 0.769 0.250 47.76

(init_success for cold-start align-first reads 0.250 here, not the pre-rebuild card's 0.333 — see the F4 note above; that column changed because a metrics bug was fixed, not because the baseline's actual behaviour changed.)

Is this a working pipeline or a broken one?

Nadir p50 under every variant (4.7-8.8 m) is far below the "investigate before writing the card" threshold (300 m) the task brief flagged, so no template-scale/yaw/exposure debugging was needed. But p50 alone is a misleading headline once the GT-seeded start is understood for what it is: the GT-seeded numbers describe tracking quality given a correct starting fix (a plausible real scenario — e.g. a last-known GPS position before entering a GNSS-denied zone), while the cold-start numbers describe genuine from-scratch localization, and there the tail is dramatically worse (p90/p99 in the hundreds to over a thousand metres, driven by seq_0007 failing to acquire at all). Read together, this is a real, if imperfect, aided tracker (87% of nadir frames within 10 m when correctly seeded) whose initial acquisition is unreliable roughly 1 in 4 times among nadir sequences in this build — both facts are true and neither should be reported without the other. Error otherwise grows roughly with path length/altitude rather than being uniformly random. seq_0009, the sequence with the largest GT-seeded drift, is altitude: high (agl ≈ 262 m) with roll_deg up to ~5.7° (mean ~1.7° across its 80 frames) — the dominant cause is roll, not the haze: 0.266 also present on this sequence: the baseline corrects for yaw but treats every nadir frame as perfectly nadir, so a rolled frame's true ground footprint centre is offset from the camera's (x, y) by roughly agl·tan|roll| (a purely geometric parallax effect, independent of image content). That formula gives ~19.5 m at the last frame's roll (4.25°) and a ~7.8 m mean across the sequence — closely tracking the observed final_error=21.5 m and p50=9.4 m respectively (checked directly against out/nav_v0/seq_0009/poses.csv). Haze reduces template contrast/matching confidence but has no comparable direct positional-bias mechanism, so it is at most a secondary contributor here.

Before/after the rebuild: the GT-seeded aided-baseline nadir p50 is 4.7 m both before and after this rebuild (10.78 m vs. the pre-rebuild card's 10.8 m on p90, 18.03 m vs. 18.0 m on p99 — the same to the precision reported). Two independent things changed the render between builds — the SUN lamp azimuth fix (F1) and pinning texture_year from an assumed "current mosaic" to the explicitly-probed 2024 flight (F3) — and either could plausibly have shifted NCC-vs-2021-map matching quality, yet the aided baseline's numbers are essentially unchanged, sequence-by-sequence (see the per-nadir-sequence table above, unchanged to 1 decimal place for all four sequences). The most likely explanation: PNOA's PNOA_CURRENT WMS layer (what the pre-rebuild config's texture_year: null fetched) already reflected 2024-era imagery at this bbox — i.e. pinning texture_year: 2024 mostly corrected the dataset's metadata (map_gap_years bucket, the manifest's stated year) rather than changing which pixels got rendered. The larger movements are in the cold-start tail (p50 9.3→8.8 m, p90/p99 shift by single-digit percentages) and are consistent with ordinary Cycles sampling noise on a chaotic, already-diverging case (seq_0007) rather than a systematic effect of either fix — see the frame-lighting comparison below for the actual, isolated visual effect of the F1 fix on this build.

Visual sanity check (2 nadir frames from seq_0003, cross-referenced against reference_map.png cropped at the frame's ground-truth (x, y) — row = 2000 - y, col = x): - Frame 0 (x=960.5, y=1544.7): the frame shows a curving road/railway above a badlands scrub patch, with a cluster of terracotta-roofed houses below-left. The matching 400x400 crop of the reference map at that location shows the same curving road, the same scrub patch, and the same house cluster in the same relative arrangement (rotated to account for the frame's yaw, ~83-89°). - Frame 40 (x=1179.1, y=1562.5): the frame shows a straight road/bridge crossing diagonally through reddish eroded terrain with scattered trees. The corresponding reference-map crop shows the same diagonal road/bridge crossing the same reddish terrain, with a building complex and a second road visible at the same relative offset.

Both crops are recognizably the same ground truth location as their frames — the NCC localizer is matching real content, not degenerating to a random/constant guess.

Lighting: out/nav_v0 (rebuilt, F1-fixed sun lamp) vs. out/nav_v0_old (pre-fix). Compared seq_0003 frames 000000.jpg and 000040.jpg directly (Read tool) between the two build directories. seq_0003's sun (sun_elevation_deg: 48.51, sun_azimuth_deg: 211.71) is identical in both builds — poses, tags, and sun/haze parameters are seeded and don't depend on rendering (confirmed byte-identical poses.csv, see Determinism below) — so any visual difference is purely the F1 lamp fix, not a different sun setting. Framing/content is pixel-for-pixel the same scene in both (same camera pose over the same baked PNOA texture): a curving road/railway, a scrub hillside, and a cluster of terracotta-roofed houses, immediately recognizable as the same location in both builds.

The lighting difference itself is real but subtle at this location, not a dramatic shadow flip: a Gaussian-smoothed (σ=9 px) per-pixel diff (new minus old, greyscale) over the full 640x480 frame has range roughly -3 to +4 (out of 0-255) for both frames 0 and 40, with a small positive mean (+0.45, i.e. the corrected render is very slightly brighter here on average) — a few-percent effect, not a reversal. The diff pattern is mottled and follows scene content (tree-crown and roof edges) rather than forming one clean directional gradient across the frame. This is expected given how render_nav.py::build_terrain works: building and tree shadows are baked into the flat orthophoto texture itself and are identical in both builds (the mesh has no separate building/tree geometry to shadow); only shading from the DEM's terrain relief (a 256x256 mesh over the 2000 m patch, ~7.8 m/vertex) responds to the corrected sun direction, and the relief within this particular nadir frame's ~300-450 m footprint is gentle — Teruel's terrain here doesn't provide the kind of tall, isolated obstruction that would cast an unambiguous, dramatic cast shadow within one frame. The F1 smoke test (tests/test_render_nav_smoke.py::test_shadow_falls_away_from_sun, a synthetic 60 m bump on flat ground) is the clean, isolated demonstration of the actual fixed mechanism (128-point west/east contrast) — real orthophoto terrain at this site simply doesn't hand the fix as dramatic a natural showcase within a single frame.

Determinism

$ aerosynth build-nav configs/nav_v0.yaml out/nav_v0_repeat --cache assets/geo --samples 16 --limit-sequences 1
sequences: 1  frames: 80  failed: 0  device: GPU:OPTIX

$ python -c "import pathlib; a=pathlib.Path('out/nav_v0/seq_0000/poses.csv').read_bytes(); b=pathlib.Path('out/nav_v0_repeat/seq_0000/poses.csv').read_bytes(); print('same poses:', a==b)"
same poses: True

seq_0000/poses.csv is byte-identical between the full 12-sequence build and an independent single-sequence rebuild from the same seed and config — the seeded pose generation is deterministic and does not depend on how many other sequences are built alongside it. Re-verified on the 2026-09-03 rebuild (same command/output as above). Also checked directly across builds: out/nav_v0/seq_0009/poses.csv is byte-identical to out/nav_v0_old/seq_0009/poses.csv — poses, tags, and sun/haze/exposure parameters are entirely unaffected by the render-side fixes (F1 sun lamp, F3 texture year), exactly as expected since pose/scenario generation never touches Blender.

Limitations

Reproduce

export AEROSYNTH_BLENDER=/path/to/blender.exe   # or put blender on PATH
source .venv/Scripts/activate

aerosynth fetch-geo configs/nav_v0.yaml --cache assets/geo
aerosynth build-nav configs/nav_v0.yaml out/nav_v0 --cache assets/geo --samples 16

aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_preds.csv          # GT-seeded start (default)
aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline

aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_cold_preds.csv --no-init-from-gt   # cold start, no positional aiding
aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold

Results

Report files (28)

out/detect_v0/report_perfect/report.jsondetect

metricvalue
mAP501.000
mAP50-951.000
images5000
annotations6760
predictions6760
fp/image0.000

out/detect_v1/report_perfect/report.jsondetect

metricvalue
mAP501.000
mAP50-951.000
images5000
annotations6760
predictions6760
fp/image0.000

out/detect_v2/report_perfect/report.jsondetect

metricvalue
mAP501.000
mAP50-951.000
images5000
annotations5972
predictions5972
fp/image0.000

out/detect_v2_rc/report_perfect/report.jsondetect

metricvalue
mAP501.000
mAP50-951.000
images5000
annotations5972
predictions5972
fp/image0.000

out/eval/e0_sanity/report.jsondetect

metricvalue
mAP500.905
mAP50-950.697
images500
annotations654
predictions4227
fp/image0.000

out/eval/e0v2_sanity/report.jsondetect

metricvalue
mAP500.826
mAP50-950.645
images500
annotations590
predictions7498
fp/image0.140

out/eval/e0v2_sanity_rc/report.jsondetect

metricvalue
mAP500.825
mAP50-950.609
images500
annotations590
predictions9083
fp/image0.150

out/eval/e1_syn_only/report.jsondetect

metricvalue
mAP500.375
mAP50-950.195
images2200
annotations2245
predictions235584

out/eval/e1v1n_syn_only/report.jsondetect

metricvalue
mAP500.380
mAP50-950.194
images2200
annotations2245
predictions189794

out/eval/e1v2_syn_only/report.jsondetect

metricvalue
mAP500.159
mAP50-950.063
images2200
annotations2245
predictions174433

out/eval/e1v2_syn_only_rc/report.jsondetect

metricvalue
mAP500.154
mAP50-950.059
images2200
annotations2245
predictions290192

out/eval/e1v2a_syn_only/report.jsondetect

metricvalue
mAP500.334
mAP50-950.164
images2200
annotations2245
predictions225556

out/eval/e1v2b_syn_only/report.jsondetect

metricvalue
mAP500.231
mAP50-950.093
images2200
annotations2245
predictions256154

out/eval/e2_syn_then_real10/report.jsondetect

metricvalue
mAP500.761
mAP50-950.468
images2200
annotations2245
predictions69961

out/eval/e2v2_syn_then_real10/report.jsondetect

metricvalue
mAP500.750
mAP50-950.448
images2200
annotations2245
predictions59196

out/eval/e2v2_syn_then_real10_rc/report.jsondetect

metricvalue
mAP500.768
mAP50-950.459
images2200
annotations2245
predictions62457

out/eval/e3_real10_only/report.jsondetect

metricvalue
mAP500.759
mAP50-950.434
images2200
annotations2245
predictions89529

out/eval/e4_real_full/report.jsondetect

metricvalue
mAP500.912
mAP50-950.580
images2200
annotations2245
predictions25670

out/nav_v0/report_baseline/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.291
success<50m0.333
coverage0.333
frames960
init success0.333
mean drift %1.785

out/nav_v0/report_baseline_align/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.256
success<50m0.333
coverage0.333
frames960
init success0.333
mean drift %1.171

out/nav_v0/report_baseline_cold/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.209
success<50m0.250
coverage0.333
frames960
init success0.250
mean drift %24.989

out/nav_v0/report_baseline_cold_align/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.179
success<50m0.256
coverage0.333
frames960
init success0.250
mean drift %47.761

out/nav_v0/report_perfect/report.jsonnav

metricvalue
p50 m0.000
p90 m0.000
p99 m0.000
success<10m1.000
success<50m1.000
coverage1.000
frames960
init success1.000
mean drift %0.000

out/nav_v0_old/report_baseline/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.291
success<50m0.333
coverage0.333
frames960
init success0.333
mean drift %1.785

out/nav_v0_old/report_baseline_align/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.256
success<50m0.333
coverage0.333
frames960
init success0.333
mean drift %1.171

out/nav_v0_old/report_baseline_cold/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.209
success<50m0.250
coverage0.333
frames960
init success0.250
mean drift %24.973

out/nav_v0_old/report_baseline_cold_align/report.jsonnav

metricvalue
p50 m—
p90 m—
p99 m—
success<10m0.176
success<50m0.256
coverage0.333
frames960
init success0.333
mean drift %48.070

out/nav_v0_old/report_perfect/report.jsonnav

metricvalue
p50 m0.000
p90 m0.000
p99 m0.000
success<10m1.000
success<50m1.000
coverage1.000
frames960
init success1.000
mean drift %0.000

Experiments

Experiment summary (8)
experimenttrain dataweights initepochs train time (s)mAP50mAP50-95fp/imagepredicted boxes
E0 out/splits/syn_v1/data.yaml yolov8n.pt 30 928.100 0.905 0.697 0.000 4227
E1 — D:\gamesss\Projects2\aerosynth\out\models\syn_v1\weights\best.pt 0 0.000 0.375 0.195 0.000 235584
E2 out/splits/dut10/data.yaml D:\gamesss\Projects2\aerosynth\out\models\syn_v1\weights\best.pt 20 1550.100 0.761 0.468 0.000 69961
E3 out/splits/dut10/data.yaml yolov8n.pt 20 1525.800 0.759 0.434 0.000 89529
E4 out/splits/dut_full/data.yaml yolov8n.pt 20 5286.200 0.912 0.580 0.000 25670
E0v2 out/splits/syn_v2/data.yaml yolov8n.pt 30 2511.400 time covers epochs 15–30 only (run resumed from a checkpoint) 0.826 0.645 0.140 7498
E1v2 — D:\gamesss\Projects2\aerosynth\out\models\syn_v2\weights\best.pt 0 0.000 0.159 0.063 n/a 174433
E2v2 out/splits/dut10/data.yaml D:\gamesss\Projects2\aerosynth\out\models\syn_v2\weights\best.pt 20 1245.100 0.750 0.448 n/a 59196
Experiment write-ups (3)

Detect v2 vs v1: synthetic→real transfer on DUT Anti-UAV (E0v2, E1v2, E2v2)

docs/experiments/2026-09-detect-v2.md

Run 2026-09-05 on branch plan5-detect-v2 at 8e4a015, on the rebuilt Detect v2 (dataset built at 6cf56d0), same predict/eval code as every earlier arm. Every number below is read from out/eval/summary.json, the per-experiment out/eval/*/report.json, the out/models/*/results.csv epoch timers and the training logs under out/eval/logs/ — all git-ignored — or computed by the counting script described in §7.4. Nothing is typed from memory and nothing is rounded up.

Headline: fixing the data defects did not fix the transfer, and v2 is still worse than v1. The synthetic-only model trained on the rebuilt AeroBench Detect v2 scores mAP50 0.159 on the DUT Anti-UAV test set against the spec §8 threshold of 0.5. The first v2 build — which carried a 10.7 % label-displacement defect and clutter up to 260 px — scored 0.154. v1 scores 0.375. Repairing both defects moved the headline number by +0.005, inside noise. This is a negative result about the v2 design, not about the v2 build quality: unlabelled clutter plus near field plus capture realism transfers worse than v1 on every size bucket except tiny, and the large-object collapse (v1 0.458 → v2 0.118) survives capping every unlabelled object below 96 px. v1 remains the recommended synthetic set.

1. Setup

Everything about the experiment is unchanged from 2026-09-transfer-dut.md. What changed is the synthetic training set.
Detector yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
Training imgsz 640, batch 32, seed 0, --workers 0, deterministic=True, default augmentation
Epochs 30 (E0v2, synthetic) / 20 (E2v2, real)
Inference scripts/predict_coco.py, conf 0.001, imgsz 640, Ultralytics' default max_det 300 (COCOeval still scores only the top 100 per image — the same caveat as in the plan-3 write-up)
Metrics aerobench eval --merge-classes
Driver scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2
Real test set DUT Anti-UAV test, 2 200 images, 2 245 boxes — the identical file used in plan 3

What is the same as v1, deliberately: the same 40 Poly Haven panoramas in the same order, the same drone sprites (assets/sprites_v2 is a byte copy of assets/sprites_v1, 40/40 sprites.json sha256 match — see the card), the same detector, the same hyperparameters, the same splits recipe (4 500 / 500), the same DUT test set and the same evaluator. So a v1↔v2 difference is a difference in the generator, not in the experiment.

What v2 changed — full details and the build's self-checks in the dataset card docs/datasets/detect_v2.md:

  1. Unlabelled distractors (hypothesis 1 of the plan-3 §4 list, the one we rated largest): five procedural kinds — bird_soaring, bird_flapping, airliner_far, mast_with_cables, rooftop_hardware — composited before the drones and never annotated. 5 787 instances across 2 876 frames (0.575 of the set); negatives doubled from 10 % to 20 %, and 841 of the 1 000 negatives carry clutter. Their long side is drawn from distractor_px: [10, 96] — min 10 / median 53 / max 95 px — so zero unlabelled objects land in the large bucket against 1 319 labelled large boxes.
  2. Near field (part of hypothesis 2): a fifth size bucket near: (200, 1100) at 0.10 of the mix, fed by a separate 768 px sprite family rendered with a close camera (1.05–1.5× object distance, 24–35 mm lens) — real perspective, not an upscaled 256 px sprite. 615 boxes with a long side ≥ 200 px; 653 boxes touch a frame edge.
  3. Capture realism (hypothesis 5): motion blur along each drone's own velocity vector, per-frame Gaussian sensor noise (σ 2–9), signal-dependent shot noise, rolling-shutter skew on 25 % of frames — shearing the boxes with the pixels — and in-pipeline JPEG requantisation at quality 55–95 on every frame.

What v2 did NOT change, and the report must not pretend otherwise: hypothesis 3 (real CAD airframes — the drones are still procedural primitives) and hypothesis 4 (real photographs as backgrounds — still HDRI panoramas). Those were a controller decision, on licence and effort grounds, before any of this ran.

One structural difference beyond the three: v2 has 20 % negative frames where v1 had 10 %. That change rides along with the distractors in every v2 arm here and in §5, and is not isolated by any of them.

This is the second v2 build. The first (09d522d, trained 2026-09-04) shipped with two defects — displaced boxes on skewed frames and clutter up to 260 px — and its runs are kept in §8 as v2-rc, because they are what that data did and because §5's ablation was measured on it. Everything in §1–§4 and §6–§7 is the rebuilt set.

Size buckets and what they are worth

Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS, untouched by plan 5 exactly so these slices stay comparable): tiny < 16, small 16–32, medium 32–96, large ≥ 96.

bucket tiny small medium large total
GT boxes in DUT test 15 385 1 087 758 2 245

The tiny column is 15 boxes. It is a direction, never a measurement. The large column is 758 boxes and is the one that carries this write-up's finding.

2. Results

Exp Train data Init weights Eval set mAP50 mAP50-95 tiny small medium large train time
E0 syn_v1 (4 500) yolov8n.pt synthetic v1 val (500) 0.905 0.697 0.706 0.929 0.996 0.999 ≈ 4 408 s ‡
E0v2 syn_v2 (4 500) yolov8n.pt synthetic v2 val (500) 0.826 0.645 0.466 0.820 0.931 0.980 ≈ 4 608 s ‡
E1 — (no training) E0 best.pt DUT test (2 200) 0.375 0.195 0.038 0.201 0.381 0.458 n/a
E1v2 — (no training) E0v2 best.pt DUT test (2 200) 0.159 0.063 0.071 0.203 0.262 0.118 n/a
E2 dut10 (520 real) E0 best.pt DUT test (2 200) 0.761 0.468 0.242 0.744 0.717 0.848 1 550.1 s
E2v2 dut10 (520 real) E0v2 best.pt DUT test (2 200) 0.750 0.448 0.242 0.703 0.733 0.837 1 245.1 s
E3 † dut10 (520 real) yolov8n.pt DUT test (2 200) 0.759 0.434 0.156 0.713 0.743 0.824 1 525.8 s
E4 † dut_full (5 200 real) yolov8n.pt DUT test (2 200) 0.912 0.580 0.535 0.816 0.913 0.968 5 286.2 s

Per-bucket columns are mAP50. The same buckets at mAP50-95:

Exp tiny small medium large
E0v2 0.242 0.563 0.787 0.889
E1 0.007 0.043 0.174 0.314
E1v2 0.023 0.042 0.097 0.072
E2 0.024 0.355 0.430 0.591
E2v2 0.024 0.301 0.426 0.584
E3 † 0.025 0.301 0.405 0.549
E4 † 0.134 0.373 0.554 0.723

† E3 and E4 were not retrained for this report. Their rows are reused verbatim from the plan-3 run (out/eval/summary.json, rows E3/E4, produced at e05a131). They are control arms that contain no synthetic data at all, so nothing in plan 5 could change them; they were computed with the same predict_coco.py / aerobench eval code path that E0v2–E2v2 use here.

‡ Neither E0 nor E0v2 ran straight through, and neither driver figure is the training cost. E0's 928.1 s in summary.json covers only a resumed segment; the plan-3 write-up reconstructs ≈ 4 408 s from results.csv. E0v2 was resumed too: summary.json records resumed_from_epoch: 14, out/eval/logs/E0v2.log line 2408 carries a ===== RESUME … from epoch 14 ===== marker followed by Resuming training … from epoch 15 to 30 total epochs, and out/models/syn_v2/results.csv shows the epoch timer resetting between epoch 14 (2 134.66 s) and epoch 15 (167.80 s). The driver's 2 511.4 s therefore covers epochs 15–30 only (16 epochs completed in 0.687 hours = 2 473.2 s in the log); adding the first segment's 2 134.7 s gives ≈ 4 608 s of training time, in line with v1. Whatever wall-clock elapsed during the interruption itself is in neither figure. E2v2 ran straight through (20 epochs completed in 0.332 hours, resumed_from_epoch: null), so its 1 245.1 s is the whole run; the gap to E2's 1 550.1 s for the same 20 epochs on the same 520 images is machine load, not data.

E0v2 is on its own evaluation set (held-out v2 synthetic val), so it is not comparable to E0 as a score — both are pipeline sanity checks. The gate the plan set was E0v2 mAP50 ≥ 0.8 or stop and debug the training: 0.826, passed. Ultralytics' own best-epoch val number in the log agrees (MAP50_BEST 0.82758). Training is not broken; the data is what changed. That 0.826 is still a validation number picked on the same 500 images Ultralytics selected best.pt with — mildly optimistic by construction, exactly as in v1, and irrelevant to every DUT number here.

One metric that is meaningful — and two that are not

fp_per_image is measured on drone-free frames. DUT test has none (negatives.images: 0 in every real report), so fp_per_image: 0.000 for E1v2 and E2v2 is an artifact and must be ignored, as in plan 3.

For E0v2 it is real: the v2 synthetic val split carries 100 negative frames (v2 is 20 % negatives), and out/eval/e0v2_sanity/report.json records 14 false positives over 100 negatives = 0.14 per image. v1's E0 saw 54 negatives and 0 false positives. On its own domain, the v2-trained model still fires on empty frames more than the v1-trained one did.

3. Verdict against spec §8

Gate 1 — E1v2 ≥ 0.5 mAP50 on real data with no real training images: FAILED. 0.159 against 0.5. It is not a near miss and not progress towards the threshold; it is worse than v1's 0.375 by 0.216, far outside the ~0.02 noise band this test set was shown to have in plan 3. Two failures in a row, and the second is a regression that survived a full dataset rebuild. (The Drone-vs-Bird half of the spec's real-data gate remains untested and still needs the founder's signature — see §6.)

At mAP50-95 the picture is the same, 0.063 against v1's 0.195. Per bucket, v2 is behind v1 everywhere except tiny — and tiny is 15 boxes:

bucket (mAP50-95) E1 (v1) E1v2 (v2) Δ
tiny (15 boxes) 0.007 0.023 +0.016
small (385) 0.043 0.042 −0.001
medium (1 087) 0.174 0.097 −0.077
large (758) 0.314 0.072 −0.242

Gate 2 — E2v2 > E3 (is it worth pre-training on this synthetic set at all): no signal in either direction.

E2v2 (syn_v2 → 10 % real) E2 (syn_v1 → 10 % real) E3 (COCO → 10 % real)
mAP50 0.7498 0.7605 0.7590
mAP50-95 0.4480 0.4682 0.4340
Δ mAP50 vs E3 −0.0092 +0.0015 —
Δ mAP50-95 vs E3 +0.0140 +0.0342 —

Read honestly:

Release decision: v1 stays the recommended synthetic set. v2 is kept as a documented negative result together with its defect history; nothing in it beats v1 zero-shot, and it does not beat v1 as pre-training either.

4. What actually changed in the errors

All counts below are our own greedy highest-score-first matching at IoU 0.5 over the prediction JSONs (§7.4), never the evaluator.

The box flood got better; the detections did not
arm boxes at conf 0.001 per image images with > 100 boxes boxes at conf 0.25 TP FP FN precision recall
E1 (v1) 235 584 107.1 963 6 297 1 187 5 110 1 058 0.189 0.529
E1v2 (v2) 174 433 79.3 592 2 500 629 1 871 1 616 0.252 0.280
E1v2-rc (§8) 290 192 131.9 1 333 4 585 879 3 706 1 366 0.192 0.392
E2v2 59 196 26.9 63 1 949 1 585 364 660 0.813 0.706
E4 25 670 11.7 9 2 231 1 999 232 246 0.896 0.890

This is the one place the rebuilt clutter did what hypothesis 1 promised. Precision at the working threshold rose 0.189 → 0.252 (v2-rc, with its oversized clutter, had managed 0.192), the diffuse box count fell by a quarter against v1 and by 40 % against v2-rc, and the number of images drowning in low-confidence boxes fell from 963 to 592. Recall fell from 0.529 to 0.280 to pay for it. The model did not learn "a railing is not a drone"; it learned "fire less", and it fires less at drones too.

The model still localizes, and still refuses to commit

For every GT box: is there any prediction over it at IoU ≥ 0.5, at any score; if so, does the highest-scoring such prediction clear conf 0.25; and what is the median of those scores.

arm bucket localized of those, clear 0.25 median score
E1 (v1) small 260/385 = 0.675 0.642 0.479
medium 808/1 087 = 0.743 0.681 0.703
large 594/758 = 0.784 0.785 0.772
E1v2 (v2) small 270/385 = 0.701 0.559 0.308
medium 768/1 087 = 0.707 0.397 0.135
large 543/758 = 0.716 0.311 0.086

Localization is essentially intact — v2 puts a well-fitted box on 72 % of the large real drones, slightly fewer than v1's 78 % but nowhere near a collapse — and then scores it at a median of 0.086 against a 0.25 threshold. The failure is confidence, not sight, exactly as in v2-rc. The difference from v2-rc is where it now falls: v2-rc's suppression was concentrated at large (its small/medium medians were 0.451 / 0.360, close to v1's 0.479 / 0.703); the rebuilt v2 has dragged every bucket down (medium 0.703 → 0.135). Capping the clutter below 96 px did not lift the large bucket back to v1; it lowered the middle to match.

Recall at conf 0.25 by bucket, same matching:

arm tiny small medium large
E1 (v1) 4/15 = 0.267 167/385 = 0.434 550/1 087 = 0.506 466/758 = 0.615
E1v2 (v2) 4/15 = 0.267 151/385 = 0.392 305/1 087 = 0.281 169/758 = 0.223
The two frames from the v2-rc report, re-measured

The two figures below were drawn from the v2-rc prediction files and are kept as they were; the rebuilt model's numbers on the same two frames are given in the text. (Left panel = E1 on v1, right = E1v2 on v2-rc; green = ground truth, red = prediction at conf ≥ 0.25, yellow = the model's best-IoU box when it falls below 0.25.)

a) The large-object collapse — img/00103.jpg, GT 185 × 102 px

v1 detects a large drone confidently, v2-rc localizes it and scores it 0.004

v1: one box, IoU 0.979, score 0.904. v2-rc: zero boxes clear 0.25; best-IoU box 0.991 at score 0.0038. The rebuilt v2 is no better and slightly worse: still zero boxes over 0.25 (its highest-scoring prediction anywhere in the frame is 0.010), best-IoU box 0.694 at score 0.0012. n = 1, but it is the §3 table in one image.

b) The near-field case from plan 3 — img/01124.jpg, GT 865 × 305 px

near-field drone: v1 fragments it, v2-rc gets a 0.88-IoU box at score 0.062

This is the one frame where the rebuilt v2 wins outright. v1 fragments the aircraft (best-IoU box anywhere 0.445, at score 0.153 — below threshold; its best scoring box, 0.672, covers a 432 px slice). v2-rc gets the extent right — IoU 0.877 — but scores it 0.062, below threshold. The rebuilt v2 gets IoU 0.877 at score 0.488, i.e. a confident, correct detection of a frame-filling drone, which neither of the other two models produced. The near-field sprites work; the suppression that hid this in v2-rc is gone here, and not in (a).

5. The causal reading, revised

The v2-rc write-up named a cause: unlabelled clutter, and specifically clutter in the same size bucket as the targets. The rebuild was that hypothesis made real — distractor_px: [10, 96], 0 unlabelled instances ≥ 96 px against 1 319 labelled large boxes, and the two large aerial kinds (helicopter, plus the already-narrow airliner_far shrunk with the rest) no longer able to appear at drone scale. It was a direct test with a stated falsifier: large mAP50 returns towards v1's 0.458 while precision at conf 0.25 keeps some of v2b's 0.259.

Half of that came true and the half that mattered did not. Precision went to 0.252. large mAP50 went 0.064 → 0.118, against v1's 0.458: about 14 % of the gap, on one seed. So:

The "large unlabelled clutter outnumbers large labelled drones" explanation is at most partial. It cannot be the main cause of the large-object collapse, because removing it entirely left 86 % of the collapse in place. The v2-rc report should be read as having over-committed to it.

The label defect is refuted as a driver too: C1 displaced 10.7 % of v2-rc's boxes, the rebuild has none below IoU 0.5 and 98.7 % above 0.9, and the headline moved 0.154 → 0.159.

What the evidence still supports, all of it one seed per arm:

6. Honesty rules and what is next

Rules applied throughout, same as the plan-3 write-up:

v3 hypotheses — none of these has been tested

v2 changed four things at once (clutter, near field, capture realism, negative fraction) and the rebuild changed a fifth. That is why we cannot say which one costs the 0.216. The next round should start from v1 and add exactly one thing per arm, each measured by the same E1 — train on synthetic only, evaluate on DUT test, report per-bucket — with nothing else touched. Each is a guess consistent with §4 and §5, and each is one 30-epoch run plus one inference pass (≈ 1.5 h on an RTX 3070 Ti):

  1. v1 + 20 % negatives only. No clutter, no near field, no capture realism — only negative_fraction: 0.20. This removes the confound §5 names last and is the cheapest of the three. If E1 drops materially below 0.375, the reticence is about empty frames and none of the rest of v2 needs explaining.
  2. v1 + capture realism only. Motion blur, sensor/shot noise, rolling shutter (boxes sheared), JPEG requantisation, at the v1 size mix and 10 % negatives. Isolates hypothesis 5 for the first time.
  3. v1 + small ground clutter only. bird_soaring, bird_flapping, mast_with_cables, rooftop_hardware at distractor_px: [10, 96], no airliner_far, v1 negatives, v1 size mix. Isolates the thing §5 still suspects most, without the negative-fraction change riding along.

The near field is deliberately not on that list: §4b says it works, and it is the only change with a positive result to its name. Add it back after one of 1–3 explains the loss.

Then the two hypotheses v2 never touched, in expected-value order: hypothesis 3 (real CAD / photogrammetric airframes instead of procedural primitives) and hypothesis 4 (real photographs as backgrounds alongside HDRIs). Both were deferred on licence and effort grounds and both remain completely open. Given that four data changes in a row have not moved E1 towards 0.5, they may matter more than anything in this plan did.

Founder action still required — unchanged from plan 3 §5. The WOSDETC Drone-vs-Bird dataset — the one real benchmark that actually contains birds, and therefore the honest test of whether bird distractors help — cannot be downloaded. Access needs an email to wosdetc@googlegroups.com and a signed data usage agreement from a named person (https://github.com/wosdetc/challenge). Nothing in that can be automated. Until it exists, "our synthetic birds teach the model to reject real birds" is untestable, and every clutter conclusion here is measured only against DUT, which contains no birds at all.

7. Reproducing this

Repo root, project venv with the training extra (uv pip install -e ".[dev,train]"). assets/real/dut/** is fetched exactly as in 2026-09-transfer-dut.md §6.1 — that step is unchanged and not repeated here.

7.1 The dataset
aerosynth render-extras assets/hdris assets/sprites_v2 --kind distractors --per-hdri 24 --seed 5 --size 256 --samples 32
aerosynth render-extras assets/hdris assets/sprites_v2 --kind nearfield  --per-hdri 9  --seed 6 --size 768 --samples 32
aerosynth build-detect configs/detect_v2.yaml out/detect_v2 --sprites assets/sprites_v2

(assets/sprites_v2 is first a plain copy of assets/sprites_v1; the sprites are unchanged between the two v2 builds — dropping helicopter is a filter at index time. See docs/datasets/detect_v2.md for the sprite sha256 check and the build's self-checks.)

7.2 The three experiments
python scripts/make_yolo_split.py out/detect_v2 out/splits/syn_v2 --val-frac 0.1 --seed 0 --single-class
python scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2 --dry-run
python scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2
python scripts/build_site.py

E3 and E4 are deliberately not in that --only list: they contain no synthetic data, and their rows are reused from the plan-3 run. Budget ≈ 1.6 h of training on an RTX 3070 Ti plus inference over 2 200 images twice.

On the resume. E0v2's training was interrupted after epoch 14 and restarted from out/models/syn_v2/weights/last.pt; the driver re-invokes train_yolo.py with --resume, appends a ===== RESUME … from epoch 14 ===== marker to out/eval/logs/E0v2.log and records resumed_from_epoch in summary.json. A clean run needs no resume and reports the whole 30 epochs in one timer; see the ‡ footnote in §2 for how to read the split figure when it happens.

7.3 The v2-rc artifacts

The first build's runs were not deleted. They live beside the current ones as out/eval/e0v2_sanity_rc, out/eval/e1v2_syn_only_rc, out/eval/e2v2_syn_then_real10_rc and out/models/syn_v2_rc, out/models/syn_v2_then_dut10_rc (all git-ignored). The two ablation arms of §8 were run through train_yolo.py/predict_coco.py directly rather than the driver, so their numbers exist only as out/eval/e1v2a_syn_only/report.json and out/eval/e1v2b_syn_only/report.json; their configs are committed as configs/detect_v2a.yaml and configs/detect_v2b.yaml.

7.4 The counting script

The conf-0.25 precision/recall table, the per-bucket recall table, the localized-vs-suppressed table and the two figures are not produced by any committed tool — they are short one-off scripts over the prediction JSONs, the same approach as plan 3 §4. Each loads assets/real/dut/test_coco.json plus an out/eval/*/preds.json, does greedy highest-score-first matching at IoU 0.5, and prints counts; the localized/suppressed table instead takes, per GT box, the highest-scoring prediction at IoU ≥ 0.5. There is no hidden filtering: every prediction in the file is considered, and the "boxes at conf 0.001" column is simply len(preds).

7.5 Where the numbers live

out/** is git-ignored; nothing but this write-up and its two figures is committed. The driver writes one row per experiment to out/eval/summary.json (keyed by experiment, replacing that experiment's earlier row on a re-run — the E0v2/E1v2/E2v2 rows there are the rebuilt runs, and the v2-rc numbers survive only in the _rc directories above and in §8 of this document).

8. History: v2-rc, the first build

The first v2 build (09d522d, 2026-09-04) was trained and evaluated before review found two defects in it:

Both are fixed at 6cf56d0. The rc results are kept because §5 rests on them:

arm what it is mAP50 mAP50-95 tiny small medium large
v1 (E1) reference 0.375 0.195 0.038 0.201 0.381 0.458
v2a v2-rc without clutter (near field + capture realism + 20 % negatives) 0.334 0.164 0.077 0.177 0.327 0.420
v2b v2-rc clutter only (v1 size mix, v1 capture defaults, 20 % negatives) 0.231 0.093 0.067 0.193 0.306 0.148
v2-rc both 0.154 0.059 0.109 0.309 0.286 0.064
v2 (rebuilt) both, C1/C2 fixed 0.159 0.063 0.071 0.203 0.262 0.118

E0v2-rc scored 0.825 on its own synthetic val (sanity gate passed) and E2v2-rc reached 0.768 / 0.459 on DUT test — i.e. the fine-tuned arm was as indistinguishable from E2/E3 then as it is now.

Caveats that apply to the four rc rows, and therefore to §5:

Итог для основателя

Главное: вторая версия синтетического набора не заработала, и починка её дефектов ничего не изменила. В первой сборке v2 нашли два дефекта — рамки на 10,7 % объектов были смещены относительно дрона, а отвлекающие объекты были слишком крупными. Оба исправили, набор пересобрали и обучили заново. Результат на реальном тесте DUT Anti-UAV: mAP50 = 0.159 против 0.154 у сломанной сборки и 0.375 у первой версии набора. Порог спецификации — 0.5. То есть починка данных дала +0.005, это шум, а отставание от v1 никуда не делось. Порог не взят второй раз подряд.

Дело не в качестве сборки, а в самой идее v2. Набор проигрывает v1 во всех размерных группах, кроме «крошечной» (а она посчитана по 15 рамкам во всём тесте и ничего не доказывает). На крупных дронах — 758 рамок, самая важная часть теста — метрика 0.118 против 0.458 у v1.

Наша прошлая объяснительная версия оказалась в лучшем случае частичной. В прошлый раз мы решили, что виноваты слишком крупные неразмеченные объекты: их было почти три на каждый размеченный крупный дрон, и модель научилась «большое в небе обычно не цель». В пересобранном наборе весь мусор строго мельче порога «крупного» — таких объектов ровно ноль. Проверка провалилась: провал на крупных объектах восстановился с 0.064 только до 0.118 при 0.458 у v1, то есть примерно на седьмую часть. Значит, размер мусора — не главная причина, и прошлый отчёт был слишком уверен. Мы это записали прямо в документ, а не переформулировали задним числом.

Что подтверждается по-прежнему. Модель дроны видит: она ставит хорошую рамку на 72 % крупных реальных дронов — и присваивает ей уверенность 0.086 при рабочем пороге 0.25, то есть сама же выбрасывает. Причём теперь это подавление распространилось на все размеры, а не только на крупные. Из всех вариантов, что мы обучили, ближе всего к v1 (0.334 против 0.375) оказался тот, где отвлекающих объектов не было вообще, — так что подозрение на мусор остаётся, но механизм мы назвали неправильно. Ближняя зона и реализм съёмки стоят немного (−0.041) и на одном кадре дали единственную настоящую победу: дрон, заполняющий кадр, который раньше разваливался на куски, теперь уверенно распознаётся целиком.

Одна вещь всё-таки улучшилась. Точность при рабочем пороге выросла с 0.189 у v1 до 0.252 — ложных срабатываний на перилах и крышах стало меньше, поток мусорных рамок сократился на четверть. Но платой стала полнота: 0.280 против 0.529. Модель не научилась «перила — это не дрон», она научилась «реже нажимать».

Предобучение: разницы нет никакой. Если дообучить на 10 % реальных данных, новая синтетика даёт 0.750, старая 0.761, а вообще без синтетики — 0.759. Всё это внутри шума. 520 реальных кадров стирают разрыв в 0.216, который был без них. Полный реальный набор по-прежнему бьёт всё: 0.912. Вывод плана 3 не изменился — синтетика это экономия разметки, а не замена ей.

Решение. Релизным синтетическим набором остаётся v1. v2 сохраняем как задокументированный отрицательный результат вместе с историей его дефектов — он не лучше ни без дообучения, ни как предобучение.

Что делать дальше — и это пока гипотезы, ни одна не проверена. Ошибка плана 5 в том, что мы поменяли четыре вещи сразу и теперь не можем сказать, какая именно стоила нам 0.216. Следующий раунд надо начинать от v1 и добавлять по одной вещи за прогон, меряя одним и тем же числом (E1 на DUT, примерно полтора часа обучения на арм): (1) v1 плюс только удвоенная доля пустых кадров — это единственное, что мы ни разу не изолировали, и это лучше всего объясняет «всеобщее» подавление уверенности; (2) v1 плюс только реализм съёмки; (3) v1 плюс только мелкий наземный мусор без самолётов. Ближнюю зону вернуть после того, как один из этих трёх прогонов объяснит потерю. И только потом — две гипотезы, которых мы вообще не касались: реальные CAD-модели дронов вместо процедурных и реальные фотографии вместо панорам. Четыре изменения данных подряд не сдвинули E1 к 0.5, так что эти две могут значить больше, чем всё, что мы сделали в этом плане.

Оговорка о доверии к цифрам. На каждый вариант — один прогон обучения без доверительного интервала. Разница меньше ~0.02 на этом тесте — шум; под это попадают и +0.005 от починки дефектов, и все сравнения с дообучением. Колонка «крошечные объекты» посчитана по 15 рамкам и является направлением, а не измерением.

Требуется ваше действие — то же, что и в прошлый раз. Набор Drone-vs-Bird — единственный реальный тест, где вообще есть птицы, то есть единственный способ честно проверить, помогают ли наши синтетические птицы. Скачать его нельзя: нужно письмо на wosdetc@googlegroups.com и подписанное соглашение об использовании данных от вашего имени (https://github.com/wosdetc/challenge). В DUT птиц нет вовсе, поэтому все выводы про отвлекающие объекты проверены только на нём.

Real-imagery gate plan for AeroBench Nav

docs/experiments/2026-09-nav-real-gate-plan.md

Survey done 2026-09-05 on branch plan6-nav at 92143a9. This is a plan, not a run — no dataset has been downloaded. Every fact below about a third-party dataset is cited to the page it was read from; anything not confirmed on a primary source is marked as such rather than guessed.

Headline: unlike Detect (which already has a real-data gate result — DUT Anti-UAV, see docs/experiments/2026-09-transfer-dut.md), Nav has never been run against a single real photograph. docs/datasets/nav_v0.md is entirely synthetic: rendered Blender frames over a Copernicus DEM + PNOA texture, evaluated against a same-source PNOA reference map from a different year. The spec's own Nav acceptance test (§8) — "a baseline solution gives meaningful errors, metrics stable across runs" — has only ever been checked against ground truth that the renderer itself generated. This plan proposes the smallest real-data experiment that would actually test it, using datasets that already exist and are (mostly) downloadable without a data-use agreement.

1. What the synthetic baseline needs from a real dataset

Recapping the actual artifacts (not the spec's original proposal) so the fit column in §2 means something concrete:

So a real dataset is a good fit for this exact gate if it gives: (a) real nadir(-ish) UAV photos, (b) per-frame altitude and heading (or enough to derive them), (c) absolute position ground truth (GPS, convertible to a local metric frame), and (d) a georeferenced reference image of the area at a known ground sample distance (GSD) — ideally from a different capture date than the query photos, to keep testing the same synthetic→real question nav_v0.md already poses (map-vs-observation gap), not a new one.

2. Candidate datasets

Dataset Region Images Altitude Pitch Resolution/GSD GT fields Reference map Licence Download Verdict
UAV-VisLoc (Xu et al., arXiv 2405.11936, 2024) [1][2] 11 sites, China (villages, towns, farms, cities, rivers, hills) 6,742 drone images + 11 satellite maps Varies by site, not tabulated in abstract/README Nadir (downward-facing; no oblique mentioned) [1] Drone imagery ≈0.1–0.2 m/px; satellite maps 0.3 m/px [1] GPS lat/lon, altitude, heading (Phi), capture date [1] Packaged per-site satellite map (source not stated — likely a Chinese commercial provider) Not explicitly stated in repo/paper; arXiv's own CC0 badge is for the paper listing, not necessarily the imagery — unverified, flag for founder Google Drive / Baidu, full set 16.4 GB, official "sample" subset 2.04 GB [1] — both over our 200 MB cap Best fit for the baseline as built — has exactly the altitude+heading fields baseline_nav.py needs, and a matching reference map already packaged
DenseUAV / UL14 (Dai et al., IEEE TIP 2023; Hangzhou, Zhejiang, China) [3][4][5] 14 university campuses, Hangzhou 27k+ total; UL14 subset: 6,768 train + 2,331 test UAV images, 13,536/9,099 satellite images 80 m / 90 m / 100 m (three fixed heights, encoded in filenames H80/H90/H100) [3][4] Nadir (low-altitude urban self-positioning task) Not stated in repo README GPS lat/lon/height in per-flight files [3] Two dated satellite mosaics per campus per height (e.g. H100.tif vs H100_old.tif) — years not disclosed in README, but this is a built-in multi-year map-gap test, the closest real-world analogue to our map_gap_years tag Apache-2.0 stated on the code repo — not confirmed this covers the imagery itself, flag for founder Hugging Face, no registration found; total size not confirmed, almost certainly >200 MB across all campuses Second-best fit — the only candidate with an explicit multi-vintage reference map, directly testing the same map-vs-observation gap nav_v0.md measures, but no confirmed camera heading per frame (fixed-heading flights, unclear if yaw varies)
SUES-200 (Zhu et al., IEEE TCSVT 2023; arXiv 2204.10704) [6][7] Not disclosed in README/abstract (China; university-affiliated) 24,120 images total across drone + satellite views, 4 altitudes 150 / 200 / 250 / 300 m (four fixed heights) [6] Not stated Not stated Not stated in README/abstract Per-location satellite reference image README states data is "ONLY available to academic research" [7] — a use restriction, not a copyright licence; would need to check the actual EULA before any commercial-adjacent use Google Drive / Baidu / Tianyi Cloud; size not disclosed Weaker fit — altitudes are higher than our nadir sequences' mid/high buckets (good for stress-testing scale), but GT fields and licence are too thin to plan around without opening the actual download
AerialVL (He et al., IEEE RA-L 2024) [8] Shandong Province, China (~120.42–120.48°E, 36.57–36.60°N) 18,361 aerial images + 14,096 cropped map patches, 11 sequences, 3.7–11 km each (~70 km total) [8] Not stated (fixed-wing/rotor mix implied by "aerial-based vehicle" framing) Not stated Not stated; reference maps are Google Earth / USGS satellite imagery [8] GNSS position (NovAtel OEM718D, ~1.5 m RMS), filename-encoded UTC timestamp + lat/lon [8] Google Earth and USGS satellite imagery, cropped per sequence MIT [8] — the most permissive licence found in this survey Hugging Face / Tsinghua Cloud / Baidu; size not disclosed Good licence, but the reference map is itself a scrape of Google Earth/USGS — redistributing our evaluation output is fine, but we'd be depending on someone else's re-use of Google's tiles, which is a licence smell worth independent checking before citing publicly
ALTO (Cisneros et al., arXiv 2207.12317, 2022) [9][10] Ohio & Pennsylvania, USA (helicopter, Pittsburgh corridor) Not disclosed (two trajectories, ~150 km and ~260 km) [9][10] Not stated (helicopter cruise altitude, not UAV) Nadir, "RGB downward-facing camera" [9][10] Not stated High-precision GPS-INS position, IMU, laser altimeter [9][10] States reference imagery is provided over the flight paths, but exact source/GSD not found in this survey BSD-3-Clause [9] Round 1/2 competition subsets via Dropbox; "full dataset coming soon" as of the page read [9] — sizes not disclosed Manned helicopter, not a drone — altitude and camera model likely far outside our low/mid/high AGL buckets. Useful only as a stretch/robustness check, not a like-for-like Nav validation
VPAIR (Schleiss et al., arXiv 2205.11567, 2022) [11][12] Bonn–Eifel corridor, Germany, light aircraft 2,706 query/reference pairs + 10,000 distractors [12] >300 m AGL [12] Nadir (downward-facing) [12] Not stated GPS-derived 6-DoF pose (NED), dense depth [12] High-res render from public geodata, includes depth — closer to a simulated reference than a plain orthophoto "Academic use only, commercial use prohibited" [12] — explicit no-commercial clause Sample 0.86 GB via Zenodo; full set requires a request form [12] Fails our commercial-use constraint outright given AeroSynth is a paid-product play — usable for a pure internal sanity check but not for anything the founder might later want to reference commercially
OrthoLoC (Dhaouadi et al., NeurIPS 2025 D&B, oral) [13] 47 locations, 19 regions, Germany and the United States [13] 16,425 real UAV images [13] Not stated in the page fetched Not stated Reference: official national/regional DOPs (digital orthophotos) + DSMs from European open geoportals, licensed CC BY 4.0 [13] Orthophotos + digital surface models from multiple European open geoportals (CC BY 4.0) [13] Dataset itself: CC BY-NC-SA 4.0; underlying geodata CC BY 4.0 [13] Direct HTTP from TUM CVG webshare, 287.3 GB total [13] Exactly the non-Spain, real-national-orthophoto case the brief asked to discuss (see §3) — best licence clarity of any candidate, but far too large to survey-download and would need per-region subsetting before it is even a candidate
Aerial Template Matching dataset (Mughal, GitHub) [14] NUST Islamabad, DHA Islamabad, Gujar Khan District, Pakistan 2,052 images across the 3 areas [14] Not stated Not stated Not stated Point correspondences (labels.csv) between images and orthomosaics, not GPS pose [14] Orthomosaics built from the image set itself, plus a separate satellite-derived orthomosaic (no correspondences given for that one) [14] BSD-2-Clause [14] git clone + download.sh, size not disclosed but the image count (2,052) suggests it is small — the only candidate plausibly under our 200 MB cap Wrong ground-truth shape for our metrics (point correspondences, not x,y position + altitude/heading) — would need a bespoke adapter and gives no absolute-position error metric; useful only as a cheap template-matching smoke test, not a metrics_nav gate
University-1652 (Zheng et al., ACM MM 2020) [15] 72 universities worldwide 50,218 train + ~90k query/gallery N/A N/A Real satellite + street view; drone-view images are simulated flights rendered in Google Earth, not real UAV photography [15] Real satellite imagery (source unstated) MIT (code); imagery on request, "usually reply in 5 minutes" [15] Manual request to the maintainer [15] Excluded per task scope — drone-view frames are synthetic renders of a 3D reconstruction, exactly the kind of synthetic-only set the brief said to exclude, even though the paper doesn't market it that way

Sources fetched for this table: [1] https://github.com/IntelliSensing/UAV-VisLoc, [2] https://arxiv.org/abs/2405.11936, [3] https://github.com/Dmmm1997/DenseUAV, [4] web search result quoting the DenseUAV/UL14 TIP 2023 paper abstract (Hangzhou, 14 campuses, 80/90/100 m, 2020/2022 satellite imagery) — no single fetchable URL returned the primary PDF within size limits, so this fact is cross-referenced across two independent search snippets rather than one primary fetch, [5] https://arxiv.org/pdf/2208.06561 (UL14 as an FPI-benchmark reconstruction of DenseUAV), [6] https://arxiv.org/abs/2204.10704, [7] https://github.com/Reza-Zhu/SUES-200-Benchmark, [8] https://github.com/hmf21/AerialVL, [9] https://github.com/MetaSLAM/ALTO, [10] https://metaslam.github.io/datasets/alto/, [11] https://arxiv.org/abs/2205.11567 (via search summary), [12] https://github.com/AerVisLoc/vpair, [13] https://deepscenario.github.io/OrthoLoC/, [14] https://github.com/m-hamza-mughal/aerial-template-matching-dataset, [15] https://github.com/layumi/University1652-Baseline and search summary of Zheng et al. 2020 confirming the Google Earth simulated-flight origin of drone-view frames.

Also surfaced but not deep-dived (recency noted for completeness, not verified beyond a search snippet — do not cite these as fact without a follow-up fetch): SatLoc (MDPI Remote Sensing, Sept 2025, doi:10.3390/rs17173048 — rotorcraft UAV-to-satellite matching, 100–300 m AGL, multi-terrain/weather; page returned HTTP 403 to WebFetch, only a search snippet was read), AnyVisLoc (2025, 18,000 images, 2.5D reference maps combining aerial photogrammetry and historical satellite maps — another multi-vintage-map candidate worth a follow-up look), and VNS (UAV video + synced GPS + satellite imagery, no further detail found). None of these three should be treated as confirmed until fetched directly.

3. Recommendation

Primary: UAV-VisLoc. It is the only candidate that hands our existing baseline_nav.py exactly the inputs it already consumes — per-image altitude and heading, plus a packaged, georeferenced satellite reference image at a stated GSD — without inventing new fields or re-deriving intrinsics from something the dataset doesn't provide. Diverse Chinese terrain (villages/farms/hills/cities) gives us tag-like slicing similar to our own altitude/view buckets even though the dataset doesn't label them that way. The catch: the dataset's own licence for the imagery (as opposed to the arXiv listing) is not stated anywhere found in this survey — see open questions below — and even the official "sample" subset (2.04 GB) is ~10x our 200 MB survey cap, so pulling even one location needs an explicit go-ahead.

Secondary: DenseUAV / UL14. Weaker on per-frame heading (not confirmed to vary — the flights may hold a fixed heading, which would silently degrade our aided baseline to "yaw ≈ 0 always," a real limitation to test for before trusting any number from it), but it is the only candidate in this survey with an explicit two-vintage reference map per site (H100.tif vs H100_old.tif) — the single closest real-world analogue to the map_gap_years axis nav_v0.md already reports on. If UAV-VisLoc's licence turns out to be unusable, this is the fallback with the most direct relevance to the specific claim we're testing (does the aided-NCC baseline survive a real map-capture-date gap, not just a synthetic one).

On the non-Spain reference-map question the brief asked about: none of our top two picks need us to source a reference map ourselves — both ship one. But if a future real-flight dataset (e.g. ALTO, AerialVL) lacks a redistributable map, the options and their tradeoffs are:

4. Gate protocol

Mirrors the Detect acceptance test's structure (train/fit only on our own artifact, evaluate unmodified against real ground truth, report pass/fail against a stated number) as closely as the two tracks' shapes allow — Detect trains a model; Nav's "model" is the deterministic aided-NCC baseline itself, so here "gate" means "run the unmodified algorithm against real frames it has never seen, using the dataset's own ground truth."

  1. Adapter, not baseline change. Write src/aerosynth/realdata/uav_visloc.py (mirrors the existing realdata/dut_antiuav.py pattern) that converts one UAV-VisLoc location into our Nav schema: sequences.json, poses.csv (frame,t_s,x,y,z,yaw_deg, pitch_deg,roll_deg,agl_m), intrinsics.json, reference_map.png + reference_map.json. baseline_nav.py and metrics_nav.py themselves must not change — the whole point is testing them unmodified against a new ground truth source, the same way the Detect gate reused the existing eval code against DUT/Anti-UAV rather than writing new metrics for real data.
  2. GPS → local metric frame: reproject lat/lon to the same UTM zone/plane the packaged satellite map is gridded in (need to confirm the map's georeferencing format on inspection — GeoTIFF with an embedded CRS would be ideal; a plain PNG/JPG would need a companion world file or explicit corner coordinates, unconfirmed from the README).
  3. GSD: the dataset states resolution directly ("≈0.1–0.2 m/px") rather than giving a focal length — baseline_nav.py's footprint_gsd(agl_m, fx_px) assumes we derive GSD from altitude and focal length, so the adapter should bypass that helper and set the matcher's scale directly from the stated per-image GSD (or a derived intrinsic if focal length turns out to be in the dataset's own metadata columns — not confirmed either way here).
  4. pitch_deg and roll_deg: UAV-VisLoc's README only confirms nadir shooting and heading — pitch/roll are presumably ≈ −90°/0°, but this needs confirming on the actual metadata files (only the paper abstract/README was read for this survey, not the CSV schema itself).
  5. Pick a small slice first. Convert one location (whichever Drive subfolder looks smallest on inspection) rather than all 11, to keep the founder-approval ask small and to get a fast first read on whether the adapter even produces sane crops (repeat the nav_v0.md "visual sanity check": manually compare 2–3 frames against their reference-map crop before trusting any number).
  6. Run exactly the existing commands: ``` aerobench baseline nav assets/real/nav/uav_visloc/ --out out/real_nav//preds_gtseed.csv aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_gtseed.csv --out out/real_nav//report_gtseed

aerobench baseline nav assets/real/nav/uav_visloc/ --out out/real_nav//preds_cold.csv --no-init-from-gt aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_cold.csv --out out/real_nav//report_cold

aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_gtseed.csv --out out/real_nav//report_gtseed_aligned --align-first aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_cold.csv --out out/real_nav//report_cold_aligned --align-first `` Same four-way split nav_v0.md already reports (GT-seeded / cold-start × with/without--align-first), same metrics (p50/p90/p99,success_10m,success_50m,coverage,init_success,mean_drift_pct). 4. **Repeat the run twice** on the same converted data with no code changes — the aided-NCC matcher andmetrics_nav` are deterministic, so "stable across runs" (spec §8) for this baseline should mean bit-identical CSVs, not just similar numbers. If two runs differ at all, that is itself a bug to fix before any pass/fail number is trusted, distinct from whether the number is good. 5. Compare against the synthetic baseline as a reference point, not because the two are expected to match, but because a wildly different order of magnitude is the signal to look at:

nav_v0 (synthetic), nadir, GT-seeded nav_v0 (synthetic), nadir, cold real-data target
p50 4.7 m 6.8 m see thresholds below
p90 10.8 m 844.1 m —
success_10m 0.872 0.628 —
coverage 1.000 1.000 —

5. Pass/fail threshold

The spec (§8) only says the Nav acceptance test should give "meaningful errors, [with] metrics stable across runs" — no number, unlike Detect's explicit mAP@0.5 ≥ 0.5. Proposing a concrete number here rather than leaving it as vague as the spec:

6. Steps, disk, and time

Step Path/command Est. disk Est. time
1. Inspect UAV-VisLoc Drive folder structure (no download) manual, or gdown --folder --dry-run equivalent if available 0 15 min
2. Founder go/no-go on downloading (exceeds 200 MB cap; licence unconfirmed) — — —
3. Download one location subset assets/real/nav/uav_visloc/<site>/ (git-ignored, matches existing assets/real/ convention) unknown per-site, sample subset is 2.04 GB total across 11 sites → guess 100–300 MB/site 5–15 min depending on link
4. Write + unit-test the adapter (no network, no Blender) src/aerosynth/realdata/uav_visloc.py, tests/test_realdata_uav_visloc.py 0 0.5–1 day
5. Convert one site, visual sanity check (2–3 frames vs. reference crop, as in nav_v0.md) out/real_nav/<site>/ <50 MB 1–2 hrs
6. Run baseline + eval, both start modes, twice each commands in §4.3 <10 MB (CSVs + HTML reports) minutes (NCC is fast; nav_v0.md's 320-frame synthetic run had no reported bottleneck)
7. Write up results in this same doc (append a "Results" section) or a new docs/experiments/2026-09-nav-real-gate-results.md — — 1–2 hrs

Total new disk for a one-site pilot: roughly 150–350 MB, once the founder approves exceeding the 200 MB survey cap. Total elapsed effort: about 1–2 days of focused work, dominated by the adapter (step 4) and by however long the licence question (§7) takes to resolve, which is not on our clock.

7. Open questions for the founder

  1. UAV-VisLoc's imagery licence is not stated anywhere found in this survey (GitHub README, arXiv abstract). Before we download and build a public-facing number from it, someone should either find an explicit licence in the actual Drive package (a LICENSE/readme.txt inside the zip, not visible from outside) or email the authors. This is a "read the fine print" ask, not a signed-DUA ask like Drone-vs-Bird was for Detect (2026-09-transfer-dut.md §5) — no registration or agreement is mentioned anywhere, but "not mentioned" isn't "confirmed clear."
  2. Same open question, one notch more serious, for DenseUAV/UL14 — the Apache-2.0 badge found on the GitHub repo almost certainly covers the code, not necessarily the imagery (this is a common and easy-to-miss conflation); worth a direct look at the Hugging Face dataset card's own licence field before relying on it.
  3. SUES-200 is explicitly "academic research only" — fine for our own internal validation, but should not be cited in anything shown to investors or customers without checking whether that phrase is a EULA clause or just a README aside.
  4. Do we want to spend the ~1–2 days in §6 now, ahead of other stream work, or park this plan until the founder has bandwidth to clear the licence questions above? Nothing here is blocked on Blender, GPU time, or anything this worktree can't do — the only blocker is the licence read and the go-ahead to exceed the 200 MB survey cap.
  5. If UAV-VisLoc's licence turns out to be unusable, is DenseUAV/UL14 (open question 2) an acceptable fallback, or should we instead scope a small, from-scratch real-flight capture (a DJI consumer drone doing a short GPS-logged nadir flight over a spot with existing national-orthophoto coverage) as the "definitely-ours, definitely-licensed" alternative? That is a materially bigger effort (a physical flight) but sidesteps every licence question in this document.

Итог для основателя

У Nav никогда не было проверки на настоящих фотографиях. Всё, что описано в docs/datasets/nav_v0.md, — это синтетика: отрендеренные Blender-кадры поверх реального рельефа и реального ортофото PNOA, но и «наблюдение», и «эталонная карта» построены из одного и того же источника, просто за разные годы. Приёмочный тест Nav из спецификации (§8) — «базовое решение даёт осмысленные ошибки, метрики стабильны между прогонами» — ни разу не проверялся на кадре, который не сгенерировал наш собственный рендерер.

Нашёл подходящий набор. UAV-VisLoc (Китай, 6742 кадра с 11 точек, статья 2024 года) — единственный из проверенных наборов, который отдаёт ровно то, что нужно нашему baseline (baseline_nav.py): высоту и курс на каждый кадр плюс готовую привязанную к координатам спутниковую карту. Второй кандидат — DenseUAV/UL14 (14 университетских кампусов под Ханчжоу) — слабее по курсу камеры (не подтверждено, меняется ли он вообще), зато это единственный набор с картой за два разных года на одну и ту же точку — ровно то же явление «карта устарела», которое мы уже меряем на Теруэле.

Есть загвоздка с лицензией. Ни для UAV-VisLoc, ни для DenseUAV нигде не нашлось явно прописанной лицензии именно на снимки (не на код). Это не история про подписание соглашения, как с Drone-vs-Bird для Detect (там нужно письмо и подпись основателя) — скорее «нужно заглянуть внутрь архива или написать авторам письмо и получить подтверждение». Пока это не сделано, публично ссылаться на цифры с этих наборов рискованно.

План проверки. Не переписывать baseline_nav.py/metrics_nav.py — взять их как есть и написать конвертер одного участка UAV-VisLoc в наш формат (по образцу уже существующего конвертера DUT Anti-UAV для Detect). Прогнать те же четыре варианта, что уже есть в nav_v0.md (с подсказкой GT-старта и без неё, с выравниванием и без), и сравнить: если ошибка на реальных кадрах в пределах ~25 м (мягче синтетических 4.7 м, потому что настоящая линза, сжатие JPEG и чужой спутниковый снимок — это честные усложнения) — считаем, что подход переживает переход на реальные данные. Если ошибка соскакивает на сотни-тысячи метров, как в худшем случае синтетического «холодного старта» — это не баг, а настоящий, стоящий доклада результат: значит, наш упрощённый сопоставитель (не тот, что был в изначальном плане — SuperPoint/LightGlue/ORB так и не написан) не переживает встречу с реальным сенсором.

Что нужно от вас. (1) Разрешить скачать один участок UAV-VisLoc — это больше 200 МБ, лимита на этот этап, официальный «сэмпл»-архив уже 2 ГБ. (2) Либо заглянуть внутрь архива на предмет файла лицензии, либо написать авторам письмо с вопросом про условия использования снимков — и то же самое для DenseUAV/UL14. Технической части (конвертер + прогон) — на день-два работы, и она ни от чего не зависит, кроме этого разрешения.

Synthetic→real transfer on DUT Anti-UAV (E0–E4)

docs/experiments/2026-09-transfer-dut.md

Run 2026-09-04 on branch plan3-transfer: E0 at c401cbc, E1–E4 at e05a131 (identical predict/eval code). Every number below is read from out/eval/summary.json and the per-experiment out/eval/*/report.json files (both git-ignored); nothing is typed from memory or rounded up.

Headline: the synthetic-only model does not transfer. Trained on AeroBench Detect v1 alone it scores mAP50 0.375 on the DUT Anti-UAV test set against a spec §8 threshold of 0.5 — a clear fail, not a near miss. Using the same synthetic model as a starting point before fine-tuning on 10 % of DUT looks like it helps box quality (mAP50-95 +0.034) rather than detection rate (mAP50 +0.0015, i.e. noise) — but that is one seed per arm with no error bar, so treat it as suggestive, not established.

1. Setup
Detector yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
Training imgsz 640, batch 32, seed 0, --workers 0 (Windows DataLoader hangs), default augmentation
Epochs 30 (E0, synthetic) / 20 (E2, E3, E4, real)
Inference scripts/predict_coco.py, conf 0.001, imgsz 640 (the standard low-threshold mAP protocol), Ultralytics' default max_det of 300 ‡
Metrics aerobench eval --merge-classes — the three synthetic classes are merged to one drone class so the synthetic model and DUT's single class are comparable. For E1–E4 the flag is a no-op safety net: the model was already trained single-class and DUT's GT has one category, so there is nothing to merge
Driver scripts/run_transfer_experiments.py (one entry point for all five runs)

‡ max_det 300 at predict time does not mean 300 boxes per image are scored. COCOeval's standard maxDets for the summary metrics is 100, so only the top-100 predictions per image ever count towards mAP. That bites hardest on E1, whose predictions are diffuse: 963 of the 2 200 test images carry more than 100 boxes at conf 0.001 (see §4). The numbers in this write-up are unaffected — they are all measured under that same standard protocol, which is the point of using it — but a reader comparing them against a differently configured evaluator should know which cap applied.

Splits
Split Contents Images
syn_v1 train / val Detect v1 (out/detect_v1), single-class, 90/10 split 4 500 / 500
dut10 train 10 % subsample of DUT train, --seed 0 520
dut_full train full DUT train 5 200
DUT val (E2/E3/E4 training val) full DUT val 2 600
DUT test (all real evaluations) held out end-to-end 2 200

Synthetic source set (see docs/datasets/detect_v1.md): 5 000 frames, 500 of them negatives, 6 760 boxes, 40 CC0 Poly Haven panoramas, procedurally modelled drones.

Data attribution

Every real image and label in this write-up comes from a third-party dataset. The same text is written to assets/real/dut/ATTRIBUTION.md (git-ignored, along with the data itself) by aerosynth.realdata.dut_antiuav.write_attribution:

Images and annotations: DUT Anti-UAV Detection dataset.

Jie Zhao, Jingshu Zhang, Dongdong Li, Dong Wang. "Vision-based Anti-UAV Detection and Tracking." IEEE Transactions on Intelligent Transportation Systems, 2022.

https://github.com/wangdongdut/DUT-Anti-UAV

No LICENSE/README found inside the dut_{train,val,test}.zip archives or their extracted folders (checked via unzip -l on all three archives and a filesystem search of the extracted train/val/test directories). The paper's companion code repository (github.com/wangdongdut/DUT-Anti-UAV) does carry a root LICENSE file that GitHub's API identifies as Apache-2.0, but that repository holds only code, docs and download links (~6 MB total) — the dataset itself (hundreds of MB per split) is hosted separately on Google Drive / Baidu Cloud and is not confirmed to fall under that grant. Treated as: license not stated for the dataset distribution itself; used here for evaluation only, not redistributed.

Size buckets

Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS): tiny < 16, small 16–32, medium 32–96, large ≥ 96.

The DUT test set is heavily weighted to big objects, and this shapes how much any per-bucket number is worth:

bucket tiny small medium large total
GT boxes in DUT test 15 385 1 087 758 2 245

The tiny column in every table below is computed over 15 boxes. One object moves it by several points. Read it as a direction, never as a measurement.

2. Results

Exp Train data Init weights Eval set mAP50 mAP50-95 tiny small medium large train time
E0 syn_v1 (4 500 syn imgs) yolov8n.pt (COCO) synthetic val (500 imgs) 0.905 0.697 0.706 0.929 0.996 0.999 928.1 s †
E1 — (no training) E0 best.pt DUT test (2 200 imgs) 0.375 0.195 0.038 0.201 0.381 0.458 n/a
E2 dut10 (520 real imgs) E0 best.pt DUT test (2 200 imgs) 0.761 0.468 0.242 0.744 0.717 0.848 1 550.1 s
E3 dut10 (520 real imgs) yolov8n.pt (COCO) DUT test (2 200 imgs) 0.759 0.434 0.156 0.713 0.743 0.824 1 525.8 s
E4 dut_full (5 200 real imgs) yolov8n.pt (COCO) DUT test (2 200 imgs) 0.912 0.580 0.535 0.816 0.913 0.968 5 286.2 s

Per-bucket columns are mAP50. The same buckets at mAP50-95:

Exp tiny small medium large
E0 0.384 0.688 0.867 0.915
E1 0.007 0.043 0.174 0.314
E2 0.024 0.355 0.430 0.591
E3 0.025 0.301 0.405 0.549
E4 0.134 0.373 0.554 0.723

E0 is on a different evaluation set (held-out synthetic val, not DUT). It is a pipeline sanity check — "does training work at all" — not a comparable score. It passes its own gate comfortably (0.905 vs the > 0.8 expectation).

That 0.905 is a validation number, not a held-out test number: those 500 images are out/splits/syn_v1/val.txt, i.e. exactly the set Ultralytics validated against after every epoch and used to pick which checkpoint became best.pt. Selecting on a set and then reporting on it is mildly optimistic by construction. It does not affect anything E0 is used for here — a pipeline sanity check, and weights for E1/E2 — and every real-data number in this write-up comes from the DUT test split, which no training run ever saw.

† E0's 928.1 s is not the cost of 30 epochs. E0's training was interrupted and restarted: out/models/syn_v1/results.csv shows the epoch timer resetting at epoch 25, and out/eval/logs/E0.log records Resuming training … from epoch 25 to 30 total epochs, i.e. the run was resumed from last.pt. The driver measures only the subprocess it launched, so the 928.1 s recorded in summary.json covers the final resumed segment (epochs 25–30, 6 epochs completed in 0.251 hours in the log). The full picture from results.csv is 3 510.2 s for epochs 1–24 plus 897.7 s for epochs 25–30 ≈ 4 408 s of training time in total. These are training timers (Ultralytics' own per-epoch clock), not wall-clock from first command to last: whatever elapsed between the two segments — the interruption itself — is in neither figure. E2/E3/E4 ran straight through with no resume, and for them the driver's number and Ultralytics' own (1 492.7 / 1 466.5 / 5 209.7 s) differ only by model load plus final validation.

One metric in the table means nothing here

report.json reports fp_per_image: 0.000 for E1–E4. That is an artifact, not a result: it is measured on negative (drone-free) frames, and DUT test contains zero negative frames (negatives.images: 0 in all four reports). E0's fp_per_image: 0.000 over 54 synthetic negatives is real; the real-data ones should be ignored entirely. The false-positive story on DUT has to be told a different way — see §4.

3. Verdict against spec §8

Gate 1 — E1 ≥ 0.5 mAP50 on real data: FAILED. 0.375 against a 0.5 threshold (Anti-UAV only — the Drone-vs-Bird half of spec §8's real-data test is pending the founder's access request, §5). This is not a rounding problem or a threshold-tuning problem; at mAP50-95 the synthetic-only model scores 0.195, and on everything below ~32 px long side it is effectively blind (small 0.201, tiny 0.038). A detector trained purely on AeroBench Detect v1 is not usable on this real dataset.

Gate 2 — E2 > E3: the strict metric points that way, the loose one does not.

E2 (syn → 10 % real) E3 (COCO → 10 % real) Δ (E2 − E3)
mAP50 0.7605 0.7590 +0.0015
mAP50-95 0.4682 0.4340 +0.0342
tiny mAP50 0.2416 0.1559 +0.0856
tiny mAP50-95 0.0242 0.0248 −0.0007
small mAP50-95 0.3548 0.3007 +0.0542
medium mAP50-95 0.4301 0.4053 +0.0249
large mAP50-95 0.5908 0.5485 +0.0423

Read honestly:

The ceiling matters more than either. E4 — the same architecture on the full 5 200-image DUT train set, no synthetic data at all — reaches 0.912 / 0.580, i.e. +0.152 mAP50 and +0.112 mAP50-95 over E2, and +0.293 on tiny mAP50. Ten times the real labels beats anything the synthetic set contributed. Synthetic pre-training is not a substitute for real data; on this evidence it is a modest label-efficiency aid that helps most when labels are scarce and boxes need to be tight.

4. Where E1 breaks down

By size

E1's mAP50 falls off a cliff as objects shrink: large 0.458 → medium 0.381 → small 0.201 → tiny 0.038. At mAP50-95 the small bucket is 0.043 and tiny is 0.007 — which means that even where a small object is found, the box is nowhere near tight. This is the opposite of E0's profile on synthetic val (tiny 0.706, small 0.929), so it is not that the model cannot detect small objects in principle; it cannot detect these small objects, in real imagery.

Confidence is spread across the frame

At the evaluation threshold (conf 0.001), the synthetic-only model emits 235 584 boxes for 2 245 real objects — 107 per image. The same protocol gives E4 25 670 boxes (11.7 per image), E2 69 961 and E3 89 529. Low-confidence boxes are normal and expected under this protocol; a 9× spread between models under an identical protocol is not. The syn-only model is diffusely uncertain about real scenes in a way the real-trained models are not.

At a usable operating point (conf 0.25, IoU 0.5, greedy matching) E1 produces 6 297 boxes across the test set: TP 1 187, FP 5 110, FN 1 058 → precision 0.189, recall 0.529. Roughly four out of five confident detections are wrong, and roughly half the drones are missed.

Three examples

Green = ground truth, red = E1 prediction at conf ≥ 0.25. Drawn with OpenCV from out/eval/e1_syn_only/preds.json against assets/real/dut/test_coco.json.

a) Small drone, nothing fires at all — img/00890.jpg

Small drone missed entirely

A 23 × 10 px quadcopter against bright sky, framed by trees and a building edge. Zero predictions clear conf 0.25 anywhere in the frame; the highest-scoring box in the whole image is 0.19. This is the small/tiny collapse in its plainest form — not a mislocalization, an absence.

b) The drone is missed, the architecture is detected — img/01986.jpg

Building clutter detected as drones, real drone missed

Hazy overcast, red-brick campus building. The real drone (green, 25 × 11 px, low against the facade) draws no confident prediction. Eleven boxes at conf ≥ 0.25 land on balcony railings, roof structures and — at 0.58 — on a 408 × 301 px slab of building. Detect v1's backgrounds are HDRI panoramas with drones composited over them and no distractor objects at all; the model has never had to learn that a railing is not a multirotor.

c) Near-field drone boxed as fragments — img/01124.jpg

Near-field drone detected as disconnected fragments

A large white multirotor filling most of the frame: GT is 865 × 305 px. The model's best box (0.67) covers a 350 × 82 px slice of it — IoU 0.109, so it counts as both a miss and a false positive. Seven more boxes sit on rotor booms, landing-gear legs and the payload pod. The model has learned "a drone is a small compact blob" and, faced with one at close range, finds drone-sized pieces of it instead of the aircraft.

Hypotheses

Ordered by how much of the gap we think each explains, all of them untested individually — this experiment separates synthetic-vs-real, not cause-by-cause:

  1. No distractors. Detect v1 has zero birds, aircraft, poles, cables, railings or rooftop hardware. DUT is shot over real ground clutter. Example (b) is a direct illustration; the 4:1 false-positive rate at conf 0.25 is the aggregate version. We consider this the largest single contributor.
  2. Scale and distance range mismatch. Detect v1's boxes are spread across all four buckets by construction (tiny 1 691 / small 2 353 / medium 2 048 / large 668), but "large" in the synthetic set means a composited sprite, not a drone at two metres filling the frame. Example (c) shows the model has no representation of a near-field airframe. In the other direction, real small drones sit in sensor noise and JPEG blocking that a clean composite never reproduces.
  3. Procedural drone models vs real silhouettes. Detect v1's drones are procedurally generated, not CAD or photogrammetry: arm/prop geometry, body proportions, the payload gimbal, landing gear are all approximations. A model fitted to those shapes has learned a silhouette family that the real commercial airframes in DUT do not belong to.
  4. Backgrounds. HDRI panoramas are smooth, low-frequency and captured for lighting, not for surveillance geometry. DUT frames have real camera optics and depth of field, and high-frequency vegetation and architecture behind the target.
  5. Capture realism. No motion blur, no rolling shutter, no per-frame JPEG artifacts, no sensor noise in the synthetic set; real footage carries all of these to some degree, and DUT's frames are JPEGs.

The exposure fix already shipped in v1 (see docs/datasets/detect_v1.md) removed one obvious domain gap — overbright pasted-on sprites — and E1 is still 0.375. That is useful negative evidence: photometric matching alone is not the bottleneck.

What changes in v2

In descending expected value:

  1. Distractors as a first-class part of the generator. Birds, fixed-wing aircraft, helicopters, poles, cranes, cables, rooftop hardware — rendered as unlabelled objects, plus far more negative frames containing them. Hypothesis 1 predicts this alone should move E1's precision the most.
  2. Real drone geometry. Replace or supplement procedural models with real CAD / photogrammetric airframes across the classes DUT actually contains.
  3. Explicit near-field coverage. Extend the scale schedule so drones from ~2 m (filling the frame) to long range are all represented, with correct perspective — not just sprite scaling.
  4. Capture-realism augmentation at build time. Motion blur along the drone's own velocity vector, rolling-shutter skew, per-frame JPEG requantization, sensor noise matched to a real camera's ISO curve.
  5. Real-photo backgrounds alongside HDRIs, for the high-frequency ground clutter panoramas do not provide.

The measurement to re-run after each of those is exactly E1 (mAP50 on DUT test, zero real training) with E2/E3 as the label-efficiency check. E1 is the number that tells us whether the synthetic set is getting closer to reality; E2 − E3 tells us whether it is worth pre-training on.

Built and measured: 2026-09-detect-v2.md — items 1, 3 and 4 of the list above; items 2 and 5 were not done. E1v2 came out at 0.159, i.e. worse than the 0.375 below, so v1 remains the recommended synthetic set. An ablation there points at item 1 (the distractors) as the largest suspect, but the rebuild that removed the size-matched clutter recovered almost none of the loss, so that write-up's §5 marks the mechanism as unexplained. Nothing in the present write-up is restated or revised there; its numbers stand as measured.

5. Drone-vs-Bird: founder action required

The obvious next real-data benchmark is the WOSDETC Drone-vs-Bird Detection Challenge dataset, because it contains exactly the distractor class (birds) that hypothesis 1 says we are missing. It cannot be downloaded — access requires a signed data usage agreement, and the founder has to request it personally.

Verified on 2026-09-04: the organisers' repository at https://github.com/wosdetc/challenge states that a request should be sent to wosdetc@googlegroups.com, that the requester "will be asked to sign a data usage agreement", and that the data may then be used for research purposes. The challenge homepage https://wosdetc2024.wordpress.com/ also resolves (WOSDETC Drone-vs-Bird Detection Grand Challenge @ MLSP24).

The URL named in the original plan, https://wosdetc2024.wordpress.com/drone-vs-bird-detection-challenge/, returns HTTP 404 as of 2026-09-04 — the two links above are the verified replacements.

Nothing in that process can be automated on our side: it needs a named person, an institution, and a signature.

6. Reproducing this

Everything below assumes the repo root and the project venv with the training extra installed (uv pip install -e ".[dev,train]" — gdown and ultralytics live there; see the README for the torch/CUDA install order, which matters).

6.1 Fetch and convert DUT Anti-UAV

assets/real/dut/ is git-ignored and holds ~10 000 images across the three splits. There is no CLI wrapper for this step; it is three library calls per split, run from a Python shell at the repo root:

from pathlib import Path
from aerosynth.realdata import dut_antiuav

cache = Path("assets/real/dut")
for split in ("train", "val", "test"):
    split_dir = dut_antiuav.download_split(split, cache)   # gdown; no-op if already fetched
    pairs = dut_antiuav.find_pairs(split_dir)              # (image, VOC xml) matched by stem
    dut_antiuav.to_coco(pairs, cache / f"{split}_coco.json",
                        source_tag=f"dut_antiuav_{split}")  # tags["source"] per split
    dut_antiuav.to_yolo(pairs, cache / f"{split}_yolo")     # labels/*.txt + images.txt
dut_antiuav.write_attribution(cache, None)

That produces assets/real/dut/{train,val,test}_coco.json (the GT the evaluations read) and assets/real/dut/{train,val,test}_yolo/ (labels plus an images.txt of absolute image paths — no symlinks, which need privileges on Windows). The explicit per-split source_tag is what makes tags["source"] read dut_antiuav_test rather than a bare default, so merged splits stay distinguishable.

6.2 Build the three training splits

Deterministic given --seed 0; reproduces the image counts in §1's Splits table (4 500 / 500, 520 and 5 200 train images, all real runs validating on the same 2 600-image DUT val set):

python scripts/make_yolo_split.py out/detect_v1 out/splits/syn_v1 --val-frac 0.1 --seed 0 --single-class
python scripts/make_yolo_split.py assets/real/dut/train_yolo out/splits/dut10 --val-frac 0.0 --subsample 0.10 --seed 0 --val-list assets/real/dut/val_yolo/images.txt
python scripts/make_yolo_split.py assets/real/dut/train_yolo out/splits/dut_full --val-frac 0.0 --seed 0 --val-list assets/real/dut/val_yolo/images.txt

--single-class on the synthetic split rewrites Detect v1's three classes to one drone class; DUT's own labels are already single-class. --val-frac 0 plus --val-list is what makes the two real splits hold nothing back from their own pool and validate against the independent DUT val split instead.

6.3 Run the experiments and rebuild the page
python scripts/run_transfer_experiments.py            # all five
python scripts/run_transfer_experiments.py --only E1  # just the headline number
python scripts/build_site.py                          # local results page (site/, git-ignored)

Budget roughly 3.5 h of training on an RTX 3070 Ti (the times in §2 sum to ~12 770 s once E0's full 30 epochs are counted), plus inference over 2 200 images four times.

Outputs land in out/models/** and out/eval/**, both git-ignored: the model weights, prediction files and reports are not committed, only this write-up and its three example images. The driver writes one row per experiment to out/eval/summary.json, replacing that experiment's earlier row on a re-run. A summary produced before that behaviour existed can still hold duplicates — the current file has E1 twice, identically — so a reader should keep the last row per experiment.

Итог для основателя

Главное: модель, обученная только на синтетике, на реальных данных не работает. На тесте DUT Anti-UAV она даёт mAP50 = 0.375 при пороге 0.5 из спецификации — это не «чуть-чуть не дотянули», а провал с запасом. На объектах мельче ~32 пикселей она практически слепа (0.201 и ниже), а при рабочем пороге уверенности примерно четыре из пяти её срабатываний — ложные: она принимает за дроны перила балконов, крыши и куски зданий, потому что в нашем синтетическом наборе нет ни одного отвлекающего объекта.

Синтетика, похоже, не бесполезна, но пользы меньше, чем хотелось. Если взять эту модель как стартовую точку и дообучить на 10 % реальных данных (E2), она обгоняет такую же модель без синтетики (E3) на строгой метрике mAP50-95: 0.468 против 0.434. Выигрыш небольшой, и на каждую сторону у нас всего один прогон без доверительного интервала: похоже, что синтетика улучшает точность рамок, а не способность находить дроны, — но это указание, а не доказанный факт. По мягкой метрике mAP50 разница 0.7605 против 0.7590, то есть шум; выдавать её за результат нельзя. По «крошечным» объектам разрыв выглядит крупным (0.242 против 0.156), но он посчитан всего по 15 рамкам во всём тесте — это направление, а не измерение.

Реальные данные бьют всё. Та же архитектура на полном реальном наборе (E4, в 10 раз больше размеченных кадров) даёт 0.912 / 0.580 — на 0.152 mAP50 выше, чем синтетика плюс 10 % реальных. Вывод трезвый: синтетический набор — это не замена разметке, а инструмент экономии разметки, и пока экономия скромная.

Что делать дальше. Главная гипотеза — в наборе нет отвлекающих объектов (птиц, самолётов, столбов, перил) и нет дронов на близкой дистанции. Это и есть план v2 в порядке ожидаемой отдачи: отвлекающие объекты и негативные кадры → реальная геометрия дронов вместо процедурной → близкая дистанция → смаз, шум и артефакты сжатия → реальные фотофоны. Проверять после каждого шага одним и тем же числом — E1 на DUT.

Требуется ваше действие. Набор Drone-vs-Bird (тот самый, где есть птицы) скачать нельзя: нужно письмо на wosdetc@googlegroups.com и подписанное соглашение об использовании данных от вашего имени — см. https://github.com/wosdetc/challenge. Автоматизировать это невозможно.

About & contact

AeroBench is built by Alexey Dyachenko (founder & technical lead). Co-founder & administrator (Spain): TBD.

Send predictions (COCO results format) or a model/container we can call. We’ll return a per-slice report — where your model holds, where it breaks, with example frames — in 5 working days. Free, no strings, your data stays private unless you opt into a public leaderboard.

alexdyachenko784@gmail.com

Code and benchmark: https://github.com/AlexOpasnost/aerobench. Submission format and rules: SUBMIT.md.

Data & licences

Real evaluation data: DUT Anti-UAV Detection dataset — Jie Zhao, Jingshu Zhang, Dongdong Li, Dong Wang, “Vision-based Anti-UAV Detection and Tracking”, IEEE Transactions on Intelligent Transportation Systems, 2022 (github.com/wangdongdut/DUT-Anti-UAV). No licence is stated for the dataset distribution itself; used here for evaluation only, not redistributed. Panoramas: Poly Haven HDRIs, CC0. Our own datasets: CC BY-NC 4.0. Code: Apache-2.0.