Built by scripts/build_site.py from the report files and write-ups in the repository. Every number is measured, none is typed in by hand.
Leaderboard
AeroBench Detect, one row per evaluated model. Every number is produced by the same
command — aerobench eval --track detect — over the submitted predictions;
per-bucket columns are mAP50 by long side in pixels (tiny < 16,
small 16–32, medium 32–96, large ≥ 96). Read the “real frames” column before
the score column: a model trained on thousands of real labelled frames and one
trained on none are not competing at the same thing. Rows are sorted by mAP50 across
the whole table, but a score is only comparable within one eval set —
check the eval-set column before reading two rows against each other. Rows marked
internal baseline are our own runs, not submissions; rows whose
eval set is — did not declare what they were scored on. See docs/benchmark/SUBMIT.md for how
to submit a model.
model
eval set
training data
real frames
mAP50
mAP50-95
tiny
small
medium
large
date
report
COCO → full real DUT train
internal baseline
AeroSynth
· yolov8n
· e4_real_full
DUT Anti-UAV test (2 200 images, 2 245 boxes)
COCO weights then full DUT train, 5 200 real images
real
Known limitations: procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video); sprites in v0 were rendered before exposure matching (world strength 1.0); drones are brighter than their background by a factor ~2–5 — fixed in code, v1 will re-render
Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image; size slices use the long-side buckets (matching annotate.size_bucket), not raw w*h area; repeat build (--limit 300) -> identical annotations for the shared image ids
Per class: multirotor_small 2237, multirotor_medium 2249, fixed_wing 2274
Per size: tiny 1691, small 2353, medium 2048, large 668
Per time of day: dusk 1643, day 1596, night 1761; per weather: clear 2846, haze 1181, rain 479, fog 494; backlit: 557
Panoramas: same 40 Poly Haven HDRIs (CC0) as v0 (configs/detect_v1.yaml pins the identical hdri_ids list); see assets/hdris/catalog.json for the id/source list (ATTRIBUTION.md in the dataset covers full attribution)
Sprites: assets/sprites_v1, rendered with aerosynth render-sprites assets/hdris assets/sprites_v1 --per-hdri 30 --seed 1 --size 256 --samples 32 (40 HDRIs x 30 sprites); --strength is passed per HDRI as hdri.pano_exposure(hdr), the same exposure the background panorama is tonemapped to (see "What changed" below)
Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image; size slices use the long-side buckets (matching annotate.size_bucket), not raw w*h area; repeat build (--limit 300) -> identical annotations for the shared image ids; per-class/size/time-of-day/weather/backlit counts and all 1508 box pixel coordinates in the first 1000 positive images are bit-identical to v0 (v0 and v1 replay the exact same seeded job list; only sprite RGB brightness differs, never the alpha silhouette that boxes are derived from)
Known limitations: procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video)
What changed vs v0
v0's sprites (assets/sprites) were rendered at a fixed Blender world Strength of 1.0, independent of each HDRI's own exposure. The background panorama, however, is tonemapped by load_pano_ldr to a fixed target of 0.18 mean luminance (hdri.pano_exposure, commit 676c6ee) — for these 40 HDRIs that means scaling the raw HDR by 0.19x-2.72x (median 0.34x) before compositing. v0 lit its drones at the raw, un-scaled strength, so most were rendered several times brighter than the exposure their background was tonemapped to. render-sprites now passes --strength <pano_exposure(hdr)> per HDRI, so sprite lighting matches the background's tonemap exactly. assets/sprites_v1 is a full re-render with this fix: same seed (1), same 40 HDRIs, same per-HDRI count (30); the drone silhouettes (alpha, hence bounding boxes) are unaffected, only RGB brightness changes.
Exposure and mean opaque-pixel brightness (grayscale, alpha>0 pixels only, averaged over each HDRI's 30 sprites), v0 vs v1, all 40 HDRIs:
HDRI
exposure
v0 opaque-mean
v1 opaque-mean
v1/v0
sprites.json sha256 (v1, 12c)
aarfontein_dirt_road
0.255
112.3
60.0
0.53
583130adbeb3
aarfontein_dusk
0.272
115.8
61.8
0.53
f77f07b0fa3b
abandoned_church
0.416
88.1
57.0
0.65
0ee8cdd8d0df
abandoned_hopper_terminal_01
0.338
112.7
66.9
0.59
fd37e4e87bb8
abandoned_hopper_terminal_02
0.397
101.2
64.7
0.64
2d792230cfc4
abandoned_hopper_terminal_03
0.274
118.9
64.0
0.54
0106229612c9
abandoned_hopper_terminal_04
0.284
121.9
67.6
0.55
d01d9f852fc3
abandoned_parking
0.296
113.8
63.2
0.56
0deb72c780f0
abandoned_pathway
0.478
84.4
59.0
0.70
4f1965764a09
abandoned_slipway
0.350
112.2
67.6
0.60
4d28a53ec78a
abandoned_tank_farm_01
0.420
91.1
59.4
0.65
22e39f674ca1
abandoned_tank_farm_02
0.437
90.0
59.9
0.67
c0569dec6e2f
abandoned_tank_farm_03
0.339
113.8
68.2
0.60
a2851b3e88f9
abandoned_tank_farm_04
0.392
93.5
59.3
0.63
9f9ddfa9f35d
abandoned_tank_farm_05
0.302
124.6
70.5
0.57
ee9dca80dcd6
altanka
0.263
128.0
68.9
0.54
2634f94a808f
arboretum
0.335
115.2
68.2
0.59
f3775e4e9b88
autumn_forest_04
0.262
114.8
61.8
0.54
ee9f0dab10ca
aviation_museum
0.249
130.7
68.3
0.52
9e19caab7324
bambanani_sunset
0.275
111.6
59.8
0.54
e7fa0d57dd48
beach_parking
0.500
86.9
61.7
0.71
ac004c4056a8
belfast_sunset
0.277
106.7
57.1
0.54
3baac07acc34
belfast_sunset_puresky
0.192
136.4
62.8
0.46
d4bfddbbe134
bell_park_dawn
0.477
98.6
68.9
0.70
cb76cd6b2535
bell_park_pier
0.435
97.4
65.0
0.67
b5ee0375ea54
between_bridges
0.209
145.1
73.9
0.51
a0ecb1a6542c
blaubeuren_church_square
0.574
93.2
71.2
0.76
77ac68eb864e
blaubeuren_night
0.464
101.3
70.5
0.70
25799956eed7
blue_lagoon_night
0.286
128.2
71.2
0.56
3bc8503d7417
clarens_night_01
0.285
116.6
65.0
0.56
dcb667ffe565
clarens_night_02
0.288
100.5
54.7
0.54
68dfbb511e74
cobblestone_street_night
0.318
112.1
66.3
0.59
dba6cee8fbd2
courtyard_night
0.905
67.8
64.5
0.95
3517e8304e49
dikhololo_night
2.722
44.6
74.7
1.67
1c2b388c0458
donkin_reserve
0.246
129.1
67.0
0.52
97fdefa65e77
golden_bay
0.348
111.3
67.5
0.61
717a3e58fe1b
hansaplatz
0.279
120.4
66.3
0.55
b734086d41cc
kloppenheim_02
0.775
76.5
67.5
0.88
1311e14ab2ad
kloppenheim_02_puresky
0.788
79.3
70.7
0.89
aac319021b1d
rooftop_night
0.401
97.6
62.6
0.64
2699a3e7cc00
40/40 HDRIs, mean opaque-pixel brightness (0-255) averaged across all 30 sprites/HDRI: v0 106.1 -> v1 65.1 overall (per-HDRI v1/v0 ratio mean 0.64, range 0.46-1.67). The three HDRIs spot-checked at render time (rooftop_night, abandoned_church, golden_bay) land at 88-111 (v0) -> 57-68 (v1), matching exposures 0.35-0.42 as expected. Not every HDRI gets darker: courtyard_night (exposure 0.91) and dikhololo_night (exposure 2.72, the one HDRI in this set whose raw panorama is under-exposed) end up roughly the same or brighter in v1 — the fix follows each panorama's own exposure rather than uniformly dimming every sprite.
For the first 1000 positive images (by image id), for every box: mean grayscale brightness inside the box vs. an 8px ring immediately outside it (clipped to the frame edges).
Same 1000 images and the same 1508 boxes at the same pixel coordinates (see the self-check bullet above), so this is a paired before/after comparison, not two independent samples. The share of egregiously-overbright boxes (inside/ring > 2) roughly halves, 3.6% -> 1.9% — that is the tail the root-cause commit (676c6ee) describes, and it moves the direction the fix predicts.
The median moving from 0.984 to 0.907 is not closer to 1, and is worth calling out rather than glossing over: a box's rectangle is mostly transparent alpha for a multirotor's spread-out arms/props, so a whole-box mean is diluted by however much background shows through inside that rectangle, in both v0 and v1 — it mixes drone-pixel and background-pixel brightness rather than isolating the drone. That dilutes the metric everywhere, so the aggregate median is a blunt read on a fix that mainly removes an overbright peak (see the per-box example below), not a uniform brightness shift. scripts/brightness_check.py is left as specified (whole box vs. ring, no alpha) because the composited JPEGs it reads don't carry an alpha channel to isolate opaque pixels with — the raw-sprite table above is the cleaner signal for the underlying fix; this check is the closest thing derivable from the shipped dataset files alone, and it does move in the right direction on the metric that most directly targets "egregiously bright."
Per-box detail for the two examples used in the visual check below (both [x,y,w,h] in pixels, ratio = inside/ring, max = brightest pixel in the box):
image
box
v0 inside/ring/ratio/max
v1 inside/ring/ratio/max
000006.jpg (quadcopter, clarens_night_02)
[74,66,86,41]
107.1 / 99.3 / 1.08 / 241
88.6 / 99.3 / 0.89 / 146
000016.jpg (fixed-wing, golden_bay)
[181,285,68,62]
86.4 / 74.1 / 1.17 / 191
71.2 / 74.1 / 0.96 / 132
Both boxes cross from ratio > 1 (v0, drone brighter than its ring) to ratio < 1 (v1); the brightest pixel in each box (the part most likely to look like an overbright artifact) drops by 95 and 59 (out of 255) respectively.
Visual check
Read on out/detect_v0/images/000006.jpg vs out/detect_v1/images/000006.jpg (same box, a quadcopter over a rain-streaked dusk scene, clarens_night_02): in v0 the hub is a distinct, near-white glowing patch that reads as pasted onto the muted, rain-hazed backdrop; in v1 the same hub is a soft pale blue-grey that sits inside the scene's tonal range — still the lightest part of the drone, but no longer an obvious sprite seam. 000016.jpg (golden_bay, dusk, fixed-wing): v0's fuselage is a saturated, almost glowing orange that pops out of the dim building silhouette behind it; v1's is a darker, more brownish orange that recedes into the same silhouette. 000000.jpg (daytime, two boxes, both small/thin drones): visually near-identical between v0 and v1 — daytime HDRIs in this catalog sit closer to pano_exposure ~1.0 (less scaling needed), so the fix has little to correct there, consistent with the inside-mean deltas being only 3-6 (daytime) vs 15-19 (the two night/dusk examples above).
Remaining limitations
Procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video). The whole-box brightness-consistency check above is a blunt instrument for sparse-silhouette boxes (transparent gaps inside the rectangle dilute the mean toward background) — it understates the fix's effect on the aggregate median while still correctly showing the overbright tail shrink; the raw-sprite opaque-pixel table is the more direct evidence for the underlying exposure fix.
AeroBench Detect v2 (distractors, near field, capture realism)
Per class: multirotor_small 1940, multirotor_medium 1979, fixed_wing 2053
Per size: tiny 1345, small 1742, medium 1566, large 1319
Per time of day: day 1629, night 1717, dusk 1654; per weather: clear 2863, haze 1180, rain 460, fog 497; backlit: 522
New in v2: frames_with_distractors 2876 (0.575 of frames), negatives_with_distractors 841 (0.841 of the 1000 negatives), near_field_boxes 615 (long side >= 200 px), boxes_touching_border 653
Panoramas: the same 40 Poly Haven HDRIs (CC0) as v0/v1 — configs/detect_v2.yaml pins a byte-identical hdri_ids list; see assets/hdris/catalog.json for the id/source list (ATTRIBUTION.md in the dataset covers full attribution)
Sprites (drones): assets/sprites_v2 is a straight copy of assets/sprites_v1 (40 HDRIs x 30 sprites). The drone sprites were not re-rendered, so "we added clutter" is not mixed with "we changed the drones": all 40/40sprites.json files match the sprites[].sha256 recorded in out/detect_v1/manifest.json (re-verified after the eXIf strip described under "Sprites" below)
Sprites (new families): 40 x 24 distractor renders = 960 and 40 x 9 near-field drones = 360, in assets/sprites_v2/<hdri_id>/{distractors,nearfield}/, pinned in out/detect_v2/manifest.json as 80 extras records (hdri_id, kind, count, sha256 of <kind>.json). 800 of the 960 distractor renders are used: the 160 helicopter sprites stay on disk and are skipped by sprites.load_extra_index, see "What changed vs v2-rc"
Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image on 1000 negatives; repeat build (--limit 300) -> identical annotations for the shared image ids; scripts/detect_qa.py hard checks 0 problems and 0 coverage warnings; boxes in frames that carry distractors = drones planned for those frames, exactly (3023 = 3023)
Licence: CC BY-NC 4.0 (as v0/v1). All artifacts (assets/sprites_v2/**, out/detect_v2/**) are git-ignored; only this card and the five QA images below are committed
Known limitations: procedural drones (no real CAD models), HDRI panoramas rather than real photographs as backgrounds, RGB only, single frames (no video); the motion-blur smear is inside the box (see "What is still wrong")
For reference, v1 on the same 40 panoramas: 5000 frames, 500 negatives, 6760 boxes, zero distractors, no near field, wall time 7m47s.
History: this is the second v2 build
The first v2 build (2026-09-04, commit 09d522d) shipped with two defects, both
found by review after it had already been trained on:
C1 — displaced boxes.render_frame applied the rolling-shutter skew to
the pixels afterpaste had measured the boxes, so on the 25 % of frames
that carry a skew every annotation sat up to rolling_shutter_px * y / H to
the left of its drone: 636 of that build's 5972 boxes (10.7 %) had IoU < 0.9
with their own pixels, 51 below 0.5.
C2 — clutter that dwarfed the targets.distractor_px: [12, 260], drawn
uniformly, put 3824 unlabelled objects at >= 96 px against 1318 labelled
large drones — 2.9 unlabelled large objects per labelled one.
That build's transfer numbers (E1v2 mAP50 0.154 and the ablation arms) are
recorded in docs/experiments/2026-09-detect-v2.md
as the v2-rc result and are kept there unchanged: they are what that data
did, and §5 of that write-up is what identified C2. This card describes the
rebuild at 6cf56d0, which fixes both. The rebuilt set has since been trained
and evaluated, and it did not help: E1v2 mAP50 0.159 against v2-rc's 0.154 and
v1's 0.375. The card's claim is therefore only that two measured data defects
are gone — not that the data is better. v1 remains the recommended set; see
that write-up's §3 and §5.
What changed vs v2-rc
v2-rc (09d522d)
v2 (6cf56d0)
box vs pixels
636/5972 at IoU < 0.9, 51 < 0.5
5893/5972 at IoU >= 0.9, 0 < 0.5 (below)
distractor_px
[12, 260], median 136
[10, 96], median 53, max 95
clutter instances >= 96 px
3824
0
distractor kinds
6
5 (helicopter dropped)
distractor instances
5787
5787 (unchanged count)
boxes / frames / negatives
5972 / 5000 / 1000
5972 / 5000 / 1000
The job stream is otherwise the same: the two builds plan the same 5972 targets
in the same frames, so the difference is exactly "the clutter got smaller, the
helicopters left, and the boxes now follow the skew".
Why the clutter is capped at 96 px.annotate.SIZE_BUCKETS["large"] starts
at 96 px, and distractor_px: [10, 96) is now strictly below it, so no
unlabelled object lands in the same size bucket as a labelled large drone
(0 instances vs 1319 labelled large boxes, against 3824 vs 1318 in v2-rc). This
is v3 hypothesis 1 from the experiment write-up made real — and E1 has now
falsified it as the main cause: with zero size-matched clutter, large mAP50
recovered only 0.064 → 0.118 against v1's 0.458. The clutter cap is still the
right thing to do; it is not what was breaking the large bucket.
Why helicopter was dropped. Its main-rotor disc is rendered with the same
translucent-disc trick as the drone propellers, so under --merge-classes it is
an unlabelled object built out of the labelled class's most distinctive feature.
Its sprites are still on disk (160 of the 960 distractor renders); they are
filtered out by sprites.load_extra_index, which skips any distractor record
whose kind is no longer in scenario.DISTRACTOR_KINDS.
What changed vs v1
v2 turns three of the five failure hypotheses from docs/experiments/2026-09-transfer-dut.md §4 into data.
Hypotheses 3 (real CAD airframes) and 4 (real photographs as backgrounds) are not addressed — see the
last section. annotate.SIZE_BUCKETS was deliberately left untouched, so the v1 and v2 size slices stay
comparable.
1. Unlabelled distractors (hypothesis 1 — the big one)
A Blender sprite family rendered by render_extras.py --kind distractors with the same procedural style,
the same per-HDRI seed and the same per-HDRI world strength (hdri.pano_exposure) as the drones.
Five kinds, cycled so each is guaranteed to appear:
kind
what it is
why it is here
bird_soaring
elongated dark body, near-horizontal wings
the classic sky false positive
bird_flapping
same body, wings raised 20-55 degrees
a second bird silhouette, not one pose
airliner_far
fuselage + swept wing + fin + two nacelles, long lens at 8-20x distance
"aircraft far away", named in §4
mast_with_cables
vertical pole, 2-4 cross arms, 2-5 thin guy cables, camera below the horizon
thin man-made structure
rooftop_hardware
2-4 AC boxes, 1-2 antennas, a parapet slab
exactly what E1 fired on at conf 0.25
They are composited before the drones and are never annotated, so a drone is never occluded by an
unlabelled object and its alpha-derived box stays exact. Config knobs: distractor_prob: 0.50 (half of the
positive frames), negative_distractor_prob: 0.85 (almost every negative), distractors_per_frame: [1, 3],
distractor_px: [10, 96]. Expected share of frames with clutter: 0.20*0.85 + 0.80*0.50 = 0.57; measured
0.575 (2876/5000). Negatives are up from 10 % in v1 to 20 % (negative_fraction: 0.20), and 841 of the
1000 carry clutter (target 0.85).
5787 distractor instances were composited in total: airliner_far 1179, bird_flapping 1174,
bird_soaring 1119, mast_with_cables 1136, rooftop_hardware 1179 (counted from the planned job list; the
per-frame tags["distractors"] in annotations.json sum to the same 5787, and distractors_missing is 0,
so every planned instance found a sprite). Their planned long side is min 10 / median 53 / max 95 px.
2. Near field (part of hypothesis 2)
scenario.SIZE_RANGES_PX gained a fifth bucket near: (200, 1100) — from "fills a third of the frame" up to
"wider than the 640 px frame", so the sprite is genuinely clipped by the frame border. size_mix gives it
0.10; 621 of the 5972 targets (0.104) were planned as near.
These are not upscaled v1 sprites. render_extras.py --kind nearfield re-renders the same procedural
drones (build_multirotor / build_fixed_wing, class cycled CLASSES[k % 3]) at 768 px with a close camera
(dist_mult=(1.05, 1.5)) and a wide lens (24-35 mm), i.e. real near-field perspective. render_frame prefers
this pool whenever size_bucket == "near".
Rendered result: 615 boxes with a long side >= 200 px (near_field_boxes), long side min 200 / median 492 /
max 640 px. 8 planned near targets came out under 200 px, cut down by the frame border, so 613 of the 615
come from the near bucket; the other two are large targets planned at exactly 199 px whose box crossed
200 px when the rolling shutter sheared it (1.2 px of skew, one pixel of width).
653 boxes touch a frame edge, and they come from where you would expect: 512 near, 111 large, 30 medium.
Note that annotate.SIZE_BUCKETS still ends at large: (96, inf), so per_size.large (1319) contains both
the v1-style large drones and the near-field ones; near_field_boxes is the separate key.
In v1 this whole slice was empty, and §4 of the plan-3 report showed a real DUT near-field drone
(GT 865 x 305 px) breaking into pieces (best box IoU 0.109).
3. Capture realism (hypothesis 5)
Applied per frame, after compositing, in this order: atmosphere -> rolling shutter -> sensor noise ->
shot noise -> JPEG requantisation. The rolling shutter also shears the boxes (camera.
rolling_shutter_bbox), which is the C1 fix.
effect
config
measured on this build
motion blur along the drone's own velocity vector
motion_blur_velocity: true, blur_prob: 0.40
all 5972 targets carry a vel_deg (v1 drew a fresh random angle inside the compositor instead); blur is actually applied on the 1985 frames (0.397) where blur_px > 0, min 1.00 / median 2.47 / max 4.00 px
Gaussian sensor noise, sigma drawn per frame
noise_sigma_range: [2.0, 9.0]
all 5000 frames, sigma min 2.00 / median 5.42 / max 9.00 (v1 used a fixed 3.0 day / 6.0 night)
The in-pipeline requantisation is deliberately a second compression: the file is still written at quality 92
by build_detect, so every frame goes through two JPEG passes, like a real frame recompressed on export.
The copy was cp -r assets/sprites_v1 assets/sprites_v2 (1200 PNGs + 40 sprites.json), then the two
render-extras runs wrote the new families beside the drone sprites. 40/40 HDRIs ok, zero FAIL, on both
families; total extra-sprite render time 35m04s (2104 s) on the founder's RTX 3070 Ti. The sprites were
not re-rendered for this build — the helicopter change is a filter at index time, not a re-render, so
the two builds composite bit-identical sprite pixels for every kind they share.
Indexed for this build: 800 distractor sprites (160 per kind x 5) and 360 near-field sprites. The 160
helicopter renders remain on disk and in distractors.json (which is what the manifest's count: 24 and
its sha256 cover), and are skipped when the index is built.
One byte-level change to the PNGs. Blender 4.2 writes two identical eXIf chunks into every PNG it
renders, so cv2.imread printed libpng warning: eXIf: duplicate on each of the ~12000 sprite reads a build
does. sprites.strip_duplicate_exif was run once over all 2520 sprite PNGs (and both render-* commands now
do it automatically for what they produce): the repeated chunk is dropped, the decoded pixels are unchanged
(verified on a random sample of 12), and the sprites.json sha256 in the v1 manifest still matches 40/40,
because the manifest hashes the JSON, not the PNG bytes. The build log for this dataset contains zero libpng
warnings.
Boxes vs pixels (the C1 check). For every one of the 5972 boxes, the target's sprite alpha was placed on
an empty frame with placement arithmetic written independently of composite.py, sheared by that frame's own
rolling shutter, thresholded at the same alpha > 8, and its tight box compared with the shipped annotation:
boxes checked: 5972 (frames whose box count did not replay: 0)
min IoU: 0.7778 mean IoU: 0.9928
boxes with IoU < 0.9: 79 < 0.5: 0
low-IoU boxes by size bucket: {'tiny': 79}
self-check, unskewed frames: 4476/4476 boxes identical to the pixels
counterfactual on the 1496 boxes of skewed frames: the pre-fix (unsheared) box
would score IoU < 0.9 on 625 of them
Read that in order. The self-check line is what makes the rest trustworthy: on the 4476 boxes of frames
with no skew, the independent re-derivation lands on exactly the same box, to the pixel, so it is measuring
the compositor and not itself. 5008 of the 5972 boxes are pixel-identical, 5893 (98.7 %) are at IoU >= 0.9,
and none is below 0.5 (v2-rc: 636 below 0.9, 51 below 0.5). The 79 exceptions are all 6-8 px boxes
(worst 0.778): the shear is a closed form on integer pixel coordinates, so on a box a handful of pixels wide a
sub-pixel displacement rounds to a one-pixel difference against the interpolated pixels, and one pixel on a
7 px box is 0.12 of IoU. That residual is a property of representing an interpolated shear with an integer
box, not a displacement — the v2-rc defect was the box sitting up to 6 px away from its drone.
Dataset QA (hard checks).
$ python scripts/detect_qa.py out/detect_v2 --sample 12 --img-out docs/datasets/img
image_size: (640, 640)
...
negative_share: 0.200
frames_with_distractors: 2876 (0.575 of frames)
negatives_with_distractors: 841 (0.841 of negatives)
near_field_boxes: 615 (>= 200 px on the long side)
boxes_touching_border: 653
zero_area_boxes: 0
hard checks: 0 problem(s)
OK
Exit code 0, and no check_coverage warning at all. The hard checks that would have returned a non-zero
exit code: no box with w<=0 or h<=0, no box outside the frame, COCO and YOLO labels agree row by row in both
directions, every file_name exists, every annotation points at an existing image. The coverage warnings are
now measured against this build's own config, read from manifest.json (negative fraction, whether the near
bucket was planned), so silence here means "it delivered what it asked for", not "the check was skipped".
Perfect predictions.annotations.json fed back in as COCO results with score 1.0
([{"image_id", "category_id", "bbox", "score": 1.0}, ...], one record per annotation):
Comparing the two annotations.json on the 300 shared image ids (images + annotations + categories,
json.dumps(..., sort_keys=True)): same annotations: True, sha256 315f0e54062a8e24... on both sides.
This covers the new JPEG requantisation, both noise terms and the box shear, since all of them derive from the
frame's seeded RNG.
Unlabelledness (the central invariant). Distractors must never produce a box. Replaying the job list
(make_jobs on the same config and the same HDRI order build_detect uses) and comparing against the
shipped annotations:
planned jobs: 5000, images: 5000, annotations: 5972
frames with boxes > planned drones (a distractor would have been labelled): 0
negative frames (0 planned drones) carrying a box: 0
frames with distractors: 2876
boxes in those frames: 3023; planned drones in those frames: 3023
frames tagged distractors_missing: 0
total planned drones: 5972; total annotations: 5972; dropped: 0
So in the 2876 frames that carry clutter, the box count equals the drone count exactly, and no negative frame
got a box. distractors_missing: 0 also means every planned distractor found a sprite pool — in particular,
dropping helicopter from the kind list did not silently leave planned clutter undrawn.
Brightness consistency (scripts/brightness_check.py, first 1000 positive images, box mean vs an 8 px
ring outside it). Exposure handling did not change between v1 and v2, so this is a "did anything drift"
check, not a fix:
v1 on the same script: 1508 boxes, median_ratio 0.907, share_ratio_gt_2 0.019 (v0: 0.984 / 0.036). v2 sits
right next to v1 on both numbers, which is what "same sprites, same exposure" predicts. This is not the
paired comparison the v1 card ran against v0: v2's first 1000 positive images are a different set of frames
with different boxes (1494 vs 1508), because the job stream changed (more negatives, a fifth size bucket), so
treat it as two samples from the same generator, not a before/after on identical boxes.
Visual check
The QA sampler above wrote 12 annotated frames; five are kept in docs/datasets/img/ (boxes drawn from
annotations.json; the caption strip is the frame's own tags).
image
what to look at
qa_negative_000926.jpg
Negative frame, 2 distractors, 0 boxes. An airliner_far (68 px, lower left) and a bird_soaring (55 px, right) against dusk cloud. Nothing is boxed, and this frame counts towards the false-positive metric. jpeg=75, sigma 7.86.
qa_random_002919.jpg
The C2 fix, in one frame. Two multirotor_medium drones (75 and 81 px) are boxed; a bird_soaring (45 px), an airliner_far (79 px, right) and a mast_with_cables (46 px, left of centre) are not. All three unlabelled objects are now the same scale as the labelled ones — in v2-rc that airliner could have been 260 px wide next to a 12 px drone. boxes=2 distractors=3.
qa_near_000291.jpg
Near field on a rolling-shutter frame — the C1 fix. A near-field fixed_wing (planned 491 px) with rolling_shutter_px=2.88: the box [71, 163, 492, 273] sits on the airframe rather than 2-3 px to its left, which is what v2-rc would have shipped. A 9 px tiny multirotor is boxed at the right edge; the frame's two distractors (airliner_far 90 px, bird_soaring 36 px) are not.
qa_near_001759.jpg
Frame-filling near field, in rain, backlit, skewed 5.17 px. A near-field multirotor planned at 959 px: the box [3, 315, 637, 325] is clipped by the frame at the right and the bottom, starts 3 px from the left edge, and its top edge lands on the rotor discs. This is the perspective the near-field sprite family exists for — not an upscaled 256 px sprite. jpeg=55.
qa_tiny_001001.jpg
Tiny drones next to bigger clutter. A 28 px multirotor_small and a 9 px multirotor_medium are boxed; three unlabelled birds (94, 83, 67 px) share the sky. The birds are still larger than the drones — the cap is at 96 px, not at "smaller than every target" — but none of them reaches the large bucket any more. boxes=2 distractors=3.
What is still wrong
Nothing here says v2 is better. The rebuilt set has since been trained and evaluated (E1v2 = 0.159
mAP50 on DUT test vs v1's 0.375): removing the two data defects (C1, C2) did not recover the large
bucket materially (0.064 → 0.118 against v1's 0.458). See docs/experiments/2026-09-detect-v2.md §5.
79 boxes are 1 px wider or narrower than their pixels. All of them 6-8 px, all on rolling-shutter
frames; see the C1 check above. It is a rounding residual, not a displacement, but it is not zero.
Hypothesis 3 is not closed. The drones are still procedural primitives assembled in Blender, not real
CAD airframes. Nothing in v2 changes the geometry of a drone.
Hypothesis 4 is not closed. Backgrounds are still the same 40 HDRI panoramas, tonemapped and reprojected,
not real photographs. v2 adds objects in front of those panoramas; it does not make the panoramas real.
Boxes include the motion smear. Blur is applied to the RGBA sprite before compositing, so the alpha is
smeared too and the box grows to cover the streak. This is deliberate — the object really does cover that
area during the exposure — but it means a v2 box for a fast target is slightly larger than the drone's
instantaneous silhouette, and a detector trained on v2 will be nudged towards predicting the streak.
Distractors are placed anywhere in the frame, sky included. The compositor puts a distractor at the
planned (cx, cy) with no scene understanding, so a mast or a rooftop AC unit can end up floating in open
sky. As unlabelled negative texture that is still useful, but it is not a physically plausible scene.
The clutter cap is a guess with a number on it. 96 px is the labelled-large boundary, so the rule
"no unlabelled object in the large bucket" is exact — but that the bucket is the thing that mattered is
an inference from one ablation with one seed per arm.
Still RGB, still single frames. No IR, no video, so nothing here helps temporal or thermal detection.
Five distractor kinds, all procedural. They cover most of the categories §4 named (the rotorcraft is
gone), but a real bird is not a squashed sphere with two flat wings; expect these to be easier to reject
than the real thing.
Location: Teruel, Spain — single 2000 m x 2000 m patch centred on lat 40.3456, lon -1.1065 (EPSG:25830 origin 659815, 4466837), margin_m: 300
Terrain: Copernicus DEM GLO-30 (dem_res_m: 30), interpolated centre elevation z0 = 870.5 m
Texture (rendered world): PNOA orthomosaic, year 2024 (texture_year: 2024), tex_gsd_m: 1.0. Probed PNOA-histórico coverage at this bbox newest-first via geo.fetch_orthophoto(..., year=Y, cache_dir=assets/geo/pnoa): PNOA2025 returned no image, PNOA2024 has real coverage (std ≈ 45.5-47.9 depending on the exact crop), PNOA2023 is a blank tile — 2024 is the newest year with real imagery at this bbox, so texture_year is pinned to it rather than left null/treated as "the current mosaic". sequence_tags/navexport.py fall back to the literal string map_gap_years: "unknown" (never a guessed number) when texture_year is null.
Reference map (reference_map.png): PNOA historical, year 2021, ref_gsd_m: 1.0. refmap_year was originally 2020 per the task plan, but IGN's PNOA-histórico WMS has no imagery coverage for this patch in 2020 (nor in 2017/2019/2022/2023 — verified directly against the WMS: those years return valid, well-formed, uniformly-blank JPEG tiles at HTTP 200, not an error). Only 2018, 2021, and 2024 carry real imagery for this location; 2021 was chosen as the nearest year with coverage for the reference map, giving an exact texture/map gap of 3 years (|2024-2021|, bucketed map_gap_years: "1-3" — see Tag distribution below; this replaces the pre-rebuild card's "4+" estimate, which assumed the render used a 2026 "current mosaic"). src/aerosynth/geo.py::fetch_orthophoto raises ValueError("...is blank...") before caching if a fetched tile is uniform (std < 2.0), so this class of silent-blank-map failure can't recur unnoticed.
Rendering: Blender 4.2.16 LTS, Cycles, device: GPU:OPTIX, samples: 16, mesh_grid: 256; wall time ≈ 26 min for the full 12-sequence build. exposure is computed per sequence from that sequence's own sun elevation (-4.0 + log2(sin(45°)/sin(elev))) and now lives in each sequence's sequences.json tags (exposure: -4.28 .. -3.08 across the 12 sequences here) rather than as a single dataset-wide value in manifest["blender"] (that key now holds only device/blender_version/samples/mesh_grid, which really are shared across every sequence's render).
Rebuild note (2026-09-03): this is a full rebuild of v0 from the same seed/config after the
final-review fix wave — the SUN lamp's azimuth now matches the Nishita sky's visible sun (previously
every frame's shadows fell on the opposite side of the sun from where they should), texture_year is
pinned to 2024 (previously an assumed-current mosaic), exposure moved to per-sequence tags, and
manifest["conventions"] documents the pose/frame convention (all described above and below). Poses,
tags, and per-sequence sun/haze/exposure values are byte-identical to the pre-rebuild build (seeded
generation does not depend on rendering — verified directly, see Determinism below); only the rendered
pixels and the manifest/config metadata changed. The previous build is kept at out/nav_v0_old for
comparison.
Dataset layout: what's publishable
aerosynth build-nav writes only dataset files under OUT — as of the F2 fix, the Blender render
inputs (heights.npy, texture.png, world.json: the exact orthophoto pixels and terrain each
frame was rendered from, i.e. the localization answer key) are exported under cache_dir/_world_<cfg.name>/
instead, and never appear under OUT at all (build_nav used to write them to OUT/_world/).
The publishable file set — everything a build-nav run leaves under OUT — is:
seq_XXXX/frames/*.jpg — the rendered, post-processed frames
seq_XXXX/poses.csv — per-frame ground-truth pose (see POSES_HEADER, and Conventions below)
intrinsics.json, sequences.json, manifest.json — camera model, per-sequence tags, and full build metadata
reference_map.png / reference_map.json — the historical PNOA reference map used for localization
ATTRIBUTION.md — source/licence attribution
Anything else found under a working out/nav_v0 directory in this repo's history (_sanity_*.png,
*_preds.csv, report_*/) is not written by build-nav — those are ad hoc self-check/debug
artifacts from validating this card (baseline predictions, eval reports, visual spot-checks) and
should be stripped before publishing the dataset; they were never part of the tracked file set and
carry no answer-key risk of their own (predictions/reports derived from the public data, not
inputs used to render it), but they are also not part of what a consumer of the dataset needs.
Confirmed directly on the 2026-09-03 rebuild: out/nav_v0/ has no _world/ (verified with ls);
the pre-rebuild _world/, _sanity_*.png, and *_preds.csv are preserved for comparison under
out/nav_v0_old/, which predates the F2 fix.
Conventions
Also written verbatim as manifest["conventions"] (kept in sync with navexport.CONVENTIONS) so a
consumer of poses.csv/manifest.json can interpret x/y/yaw/pitch/roll without reading the
render code:
field
meaning
origin
SW corner of patch in EPSG:25830 (origin_e, origin_n)
axes
x east, y north, z up (metres above z0)
yaw_deg
compass heading, clockwise from north; image top points along the heading
pitch_deg
-90 nadir, -60 oblique
roll_deg
right-wing-down positive
blender_euler
(90+pitch, roll, -yaw) XYZ degrees — the Euler angles render_nav.py::camera_rotation feeds to the Blender camera
poses_csv_columns
frame,t_s,x,y,z,yaw_deg,pitch_deg,roll_deg,agl_m
Tag distribution (frame counts; each sequence contributes all 80 of its frames to one bucket per tag)
altitude (AGL at t=0)
sequences
frames
low (< 120 m)
2
160
mid (120-200 m)
7
560
high (> 200 m)
3
240
view (pitch)
sequences
frames
nadir (pitch <= -80°)
4
320
oblique
8
640
time_of_day
sequences
frames
day (sun elevation > 15°)
12
960
map_gap_years
sequences
frames
1-3
12
960
pitch_choices: [-90, -90, -60] in configs/nav_v0.yaml samples nadir with probability 2/3 per sequence on average (rng.integers(0, len(pitch_choices)), two of three indices give -90°) — for 12 sequences the expected split is ~8 nadir/4 oblique, but seed 20260904's 12 draws landed on 4 nadir / 8 oblique (checked directly in src/aerosynth/navscenario.py:96; this is sampling variance for n=12, not a bug). This matters for reading the baseline table below: its coverage figures should be compared against the actual nadir share (1/3), not the config's long-run average (~2/3).
Sun elevation across the 12 sequences ranges ~22-59° (config range [10, 65]); haze fired on 5/12 sequences (haze_prob: 0.3 config, magnitudes 0.13-0.32 where present) — again plausible variance for n=12 Bernoulli(0.3) draws (expected ~3.6).
Self-check 1: perfect predictions
Ground-truth poses.csv fed straight back in as predictions (sequence,frame,x,y), evaluated with aerobench eval --track nav. perfect_preds.csv isn't written by any CLI command (it's the self-check's own input, not a dataset file — see "what's publishable" above) — generate it by concatenating every sequence's poses.csv into sequence,frame,x,y rows, e.g.:
$ python -c "
import csv, glob, os
rows = ['sequence,frame,x,y']
for d in sorted(glob.glob('out/nav_v0/seq_*')):
with open(os.path.join(d, 'poses.csv'), newline='') as f:
for r in csv.DictReader(f):
rows.append(f\"{os.path.basename(d)},{r['frame']},{r['x']},{r['y']}\")
open('out/nav_v0/perfect_preds.csv', 'w', newline='').write('\n'.join(rows) + '\n')
"
$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/perfect_preds.csv --out out/nav_v0/report_perfect
p50=0.00 m success<50m=1.000 coverage=1.000
p50
p90
p99
success<10m
success<50m
coverage
frames
overall
0.0 m
0.0 m
0.0 m
1.000
1.000
1.000
960
Exactly the expected result (identical for every tag slice too) — confirms the eval pipeline itself introduces zero error when predictions equal ground truth.
Self-check 2: aided NCC baseline
aerobench baseline nav is a nadir-only aided template matcher: it knows AGL and yaw (compass heading) for each frame, rescales the nadir frame to the reference map's 1 m GSD (footprint_gsd = agl_m / fx_px), derotates it to north-up using the known yaw, and NCC-matches it against a search window of the reference map. Oblique frames have no well-defined north-up rectangular footprint, so the baseline does not attempt them at all — it predicted exactly 320/960 frames, i.e. all and only the nadir frames, under every variant below.
Correction on the search window: by default (--init-from-gt, the CLI's default) the very first frame of each sequence is seeded at the true ground-truth position ± a 150 m search window, not matched against the whole map — only later frames search around the previous fix. This means init_success and the nadir p50/p90/p99 below are not a measurement of the baseline finding its position from nothing; they measure how well it tracks once handed a correct starting fix. --no-init-from-gt runs the honest cold-start version, where frame 0 of every sequence is matched against the entire reference map with no positional hint at all — that is the fair number for "no external aiding of position". Both are reported below.
$ aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_preds.csv
baseline nav: 320 predictions -> out\nav_v0\baseline_preds.csv
$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline
p50=inf m success<50m=0.333 coverage=0.333
p50
p90
p99
success<10m
success<50m
coverage
frames
predicted
overall
inf
inf
inf
0.291
0.333
0.333
960
320
nadir
4.7 m
10.8 m
18.0 m
0.872
1.000
1.000
320
320
oblique
inf
inf
inf
0.000
0.000
0.000
640
0
init_success: 0.333 (4/12 sequences — exactly the 4 nadir sequences: only they get a first-frame lock), mean_drift_pct: 1.79%, unmatched_predictions: 0 (every prediction row matched a GT sequence,frame — the F6 sanity check has nothing to flag here).
Per nadir sequence (the 8 oblique sequences all show p50: null, init_ok: false):
sequence
p50 (m)
final error (m)
path length (m)
drift %
seq_0003
3.8
2.4
433
0.56
seq_0004
4.5
2.5
622
0.41
seq_0007
4.1
6.6
601
1.10
seq_0009
9.4
21.5
423
5.08
--align-first variant (still GT-seeded start)
Aligning on the first predicted frame before computing errors (removes any constant first-fix offset, so what's left is drift rather than absolute bias):
$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline_align --align-first
p50=inf m success<50m=0.333 coverage=0.333
p50
p90
p99
success<10m
success<50m
coverage
overall
inf
inf
inf
0.256
0.333
0.333
nadir
5.1 m
16.1 m
31.6 m
0.769
1.000
1.000
init_success: 0.333, mean_drift_pct: 1.17% (lower than the un-aligned 1.79% — removing the fixed first-frame offset reduces path-relative drift even though the raw p50/p90/p99 land slightly higher than without alignment, since the aligning transform itself carries some of the first fix's own error into every later frame). init_success here is computed from the un-aligned first-frame error (the F4 fix — see below), so it correctly still reads 0.333 rather than being inflated by the alignment transform's own trivially-zero first-frame residual.
Cold start (--no-init-from-gt) — the fair "no positional aiding" number
Same aided NCC matcher (still given AGL and yaw), but frame 0 of every sequence is now matched against the entire reference map with no ground-truth position hint — this is what "the baseline localizes itself" actually means, as opposed to the GT-seeded numbers above which measure tracking-after-a-correct-fix.
$ aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_cold_preds.csv --no-init-from-gt
baseline nav: 320 predictions -> out\nav_v0\baseline_cold_preds.csv
$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold
p50=inf m success<50m=0.250 coverage=0.333
$ aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold_align --align-first
p50=inf m success<50m=0.256 coverage=0.333
p50
p90
p99
success<10m
success<50m
coverage
nadir, cold start
6.8 m
844.1 m
1367.9 m
0.628
0.750
1.000
nadir, cold start, align-first
8.8 m
824.5 m
1232.9 m
0.538
0.769
1.000
init_success drops from 0.333 (GT-seeded) to 0.250 (cold, no --align-first) — one of the four nadir sequences, seq_0007, fails to lock on at all without the GT seed: p50=756.3 m, final_error=564.1 m, drift_pct=93.9%, i.e. the matcher locks onto the wrong part of the map on frame 0 and never recovers. Under --align-first this sequence's aligned errors change (p50=757.7 m, final_error=1122.9 m, drift_pct=187.0%) but init_ok is still correctly False — this is the F4 fix: init_ok is now judged on the un-aligned first-frame error (which is what actually matters — "did the baseline find itself" — not on the post-alignment residual, which is trivially ~0 for whichever frame the alignment is anchored to regardless of how wrong that anchor was). Pre-rebuild, the same align-first case incorrectly showed init_ok: True for seq_0007 (the bug this card is now rebuilt without) and init_success: 0.333, one sequence too high; the correct value in both cold-start variants is 0.250 (3 of 4 nadir sequences acquire correctly; seq_0007 fails in both). The other three nadir sequences (seq_0003, seq_0004, seq_0009) localize correctly from a cold start with numbers close to the GT-seeded run (e.g. seq_0003: p50=3.8 m cold vs. 3.8 m GT-seeded). mean_drift_pct jumps to 24.99% (cold) / 47.76% (cold, aligned) — both dominated by seq_0007's divergence; excluding it, the other three sequences drift at roughly the same ~0.3-2% seen in the GT-seeded run.
Comparison across all four variants (nadir frames only — oblique is always coverage: 0 regardless of start mode)
variant
p50
p90
p99
success<10m
success<50m
init_success
mean drift %
aided, GT-seeded start (CLI default)
4.7 m
10.8 m
18.0 m
0.872
1.000
0.333
1.79
aided, GT-seeded start, align-first
5.1 m
16.1 m
31.6 m
0.769
1.000
0.333
1.17
aided, cold start (--no-init-from-gt)
6.8 m
844.1 m
1367.9 m
0.628
0.750
0.250
24.99
aided, cold start, align-first
8.8 m
824.5 m
1232.9 m
0.538
0.769
0.250
47.76
(init_success for cold-start align-first reads 0.250 here, not the pre-rebuild card's 0.333 — see the F4 note above; that column changed because a metrics bug was fixed, not because the baseline's actual behaviour changed.)
Is this a working pipeline or a broken one?
Nadir p50 under every variant (4.7-8.8 m) is far below the "investigate before writing the card" threshold (300 m) the task brief flagged, so no template-scale/yaw/exposure debugging was needed. But p50 alone is a misleading headline once the GT-seeded start is understood for what it is: the GT-seeded numbers describe tracking quality given a correct starting fix (a plausible real scenario — e.g. a last-known GPS position before entering a GNSS-denied zone), while the cold-start numbers describe genuine from-scratch localization, and there the tail is dramatically worse (p90/p99 in the hundreds to over a thousand metres, driven by seq_0007 failing to acquire at all). Read together, this is a real, if imperfect, aided tracker (87% of nadir frames within 10 m when correctly seeded) whose initial acquisition is unreliable roughly 1 in 4 times among nadir sequences in this build — both facts are true and neither should be reported without the other. Error otherwise grows roughly with path length/altitude rather than being uniformly random. seq_0009, the sequence with the largest GT-seeded drift, is altitude: high (agl ≈ 262 m) with roll_deg up to ~5.7° (mean ~1.7° across its 80 frames) — the dominant cause is roll, not the haze: 0.266 also present on this sequence: the baseline corrects for yaw but treats every nadir frame as perfectly nadir, so a rolled frame's true ground footprint centre is offset from the camera's (x, y) by roughly agl·tan|roll| (a purely geometric parallax effect, independent of image content). That formula gives ~19.5 m at the last frame's roll (4.25°) and a ~7.8 m mean across the sequence — closely tracking the observed final_error=21.5 m and p50=9.4 m respectively (checked directly against out/nav_v0/seq_0009/poses.csv). Haze reduces template contrast/matching confidence but has no comparable direct positional-bias mechanism, so it is at most a secondary contributor here.
Before/after the rebuild: the GT-seeded aided-baseline nadir p50 is 4.7 m both before and after this rebuild (10.78 m vs. the pre-rebuild card's 10.8 m on p90, 18.03 m vs. 18.0 m on p99 — the same to the precision reported). Two independent things changed the render between builds — the SUN lamp azimuth fix (F1) and pinning texture_year from an assumed "current mosaic" to the explicitly-probed 2024 flight (F3) — and either could plausibly have shifted NCC-vs-2021-map matching quality, yet the aided baseline's numbers are essentially unchanged, sequence-by-sequence (see the per-nadir-sequence table above, unchanged to 1 decimal place for all four sequences). The most likely explanation: PNOA's PNOA_CURRENT WMS layer (what the pre-rebuild config's texture_year: null fetched) already reflected 2024-era imagery at this bbox — i.e. pinning texture_year: 2024 mostly corrected the dataset's metadata (map_gap_years bucket, the manifest's stated year) rather than changing which pixels got rendered. The larger movements are in the cold-start tail (p50 9.3→8.8 m, p90/p99 shift by single-digit percentages) and are consistent with ordinary Cycles sampling noise on a chaotic, already-diverging case (seq_0007) rather than a systematic effect of either fix — see the frame-lighting comparison below for the actual, isolated visual effect of the F1 fix on this build.
Visual sanity check (2 nadir frames from seq_0003, cross-referenced against reference_map.png cropped at the frame's ground-truth (x, y) — row = 2000 - y, col = x):
- Frame 0 (x=960.5, y=1544.7): the frame shows a curving road/railway above a badlands scrub patch, with a cluster of terracotta-roofed houses below-left. The matching 400x400 crop of the reference map at that location shows the same curving road, the same scrub patch, and the same house cluster in the same relative arrangement (rotated to account for the frame's yaw, ~83-89°).
- Frame 40 (x=1179.1, y=1562.5): the frame shows a straight road/bridge crossing diagonally through reddish eroded terrain with scattered trees. The corresponding reference-map crop shows the same diagonal road/bridge crossing the same reddish terrain, with a building complex and a second road visible at the same relative offset.
Both crops are recognizably the same ground truth location as their frames — the NCC localizer is matching real content, not degenerating to a random/constant guess.
Lighting: out/nav_v0 (rebuilt, F1-fixed sun lamp) vs. out/nav_v0_old (pre-fix). Compared seq_0003 frames 000000.jpg and 000040.jpg directly (Read tool) between the two build directories. seq_0003's sun (sun_elevation_deg: 48.51, sun_azimuth_deg: 211.71) is identical in both builds — poses, tags, and sun/haze parameters are seeded and don't depend on rendering (confirmed byte-identical poses.csv, see Determinism below) — so any visual difference is purely the F1 lamp fix, not a different sun setting. Framing/content is pixel-for-pixel the same scene in both (same camera pose over the same baked PNOA texture): a curving road/railway, a scrub hillside, and a cluster of terracotta-roofed houses, immediately recognizable as the same location in both builds.
The lighting difference itself is real but subtle at this location, not a dramatic shadow flip: a Gaussian-smoothed (σ=9 px) per-pixel diff (new minus old, greyscale) over the full 640x480 frame has range roughly -3 to +4 (out of 0-255) for both frames 0 and 40, with a small positive mean (+0.45, i.e. the corrected render is very slightly brighter here on average) — a few-percent effect, not a reversal. The diff pattern is mottled and follows scene content (tree-crown and roof edges) rather than forming one clean directional gradient across the frame. This is expected given how render_nav.py::build_terrain works: building and tree shadows are baked into the flat orthophoto texture itself and are identical in both builds (the mesh has no separate building/tree geometry to shadow); only shading from the DEM's terrain relief (a 256x256 mesh over the 2000 m patch, ~7.8 m/vertex) responds to the corrected sun direction, and the relief within this particular nadir frame's ~300-450 m footprint is gentle — Teruel's terrain here doesn't provide the kind of tall, isolated obstruction that would cast an unambiguous, dramatic cast shadow within one frame. The F1 smoke test (tests/test_render_nav_smoke.py::test_shadow_falls_away_from_sun, a synthetic 60 m bump on flat ground) is the clean, isolated demonstration of the actual fixed mechanism (128-point west/east contrast) — real orthophoto terrain at this site simply doesn't hand the fix as dramatic a natural showcase within a single frame.
seq_0000/poses.csv is byte-identical between the full 12-sequence build and an independent single-sequence rebuild from the same seed and config — the seeded pose generation is deterministic and does not depend on how many other sequences are built alongside it. Re-verified on the 2026-09-03 rebuild (same command/output as above). Also checked directly across builds: out/nav_v0/seq_0009/poses.csv is byte-identical to out/nav_v0_old/seq_0009/poses.csv — poses, tags, and sun/haze/exposure parameters are entirely unaffected by the render-side fixes (F1 sun lamp, F3 texture year), exactly as expected since pose/scenario generation never touches Blender.
Limitations
Single patch: one 2000 m x 2000 m location (Teruel town and immediate surroundings). No generalization test across terrain types, latitudes, or urban/rural mix.
PNOA-only imagery: both the rendered-world texture and the reference map come from the same source (IGN PNOA), just different years. No Sentinel-2 or other independent sensor is mixed in (deferred to plan 2b) — real cross-sensor domain gap (resolution, spectral response, processing pipeline) is not represented, only a same-sensor multi-year gap.
2020 coverage gap: the original plan called for a 2020 reference map; IGN's historical WMS has no real imagery for this location in 2020 (see above), so the reference map is 2021 instead — giving an exact 3-year gap vs. the pinned 2024 texture year (|2024-2021|, updated by the F3 fix; the original plan's own target was a ~6-year gap vs. an assumed-current mosaic). fetch_orthophoto now fails loudly (ValueError, no cache write) rather than silently caching a blank tile if this recurs at another location/year.
Aided baseline, not blind localization: the baseline solver is given ground-truth AGL and yaw (compass heading) for every frame; it only has to solve 2-DoF (x, y) via template matching, not full 6-DoF pose recovery. It also only works on nadir frames — it made no attempt at all on the 640 oblique frames (2/3 of this build), which all show up as coverage: 0/infinite error by design, not because the baseline failed to localize them. It is also, by default (--init-from-gt), given the ground-truth position of the first frame of every sequence (a ±150 m search window seeded at the true location) — so the headline nadir p50/p90/p99 and init_success numbers measure tracking quality after a correct fix, not the baseline's ability to localize itself from nothing. The --no-init-from-gt "cold start" numbers above are the fair comparison for "no external aiding of position", and they are substantially worse in the tail (p90/p99 in the hundreds of metres, init_success drops from 0.333 to 0.250) — one of the four nadir sequences fails to acquire at all without the seed.
No clouds, no seasonal variation: the only "domain gap" between the rendered scene and the reference map is the multi-year PNOA capture difference (season/lighting/minor construction changes baked into the two orthophotos); there is no synthetic weather, cloud shadow, or seasonal vegetation change applied on top.
Nadir/oblique split for this build is 1/3 vs. 2/3, not the config's long-run ~2/3 vs. ~1/3 average (see Tag distribution above) — a re-seed or larger sequences count would be needed to hit the config's intended average split more closely.
Blank-tile guard is a whole-mosaic std test: fetch_orthophoto's new guard (out.std() < 2.0 raises ValueError) catches the uniform-white "no coverage" tiles seen for this patch's 2020 imagery, but it is a blunt instrument — a genuinely uniform real scene (e.g. a large flat snowfield, a calm water body, or a very small/low-contrast patch) could in principle trip the same threshold and be misdiagnosed as a coverage gap rather than real imagery. It has not misfired on any tile fetched for this dataset (the pinned 2024 texture and the 2021 reference map both have std in the 45-55 range), but it is a heuristic, not a proof of coverage.
No dusk sequence was drawn in v0: every one of the 12 sequences landed time_of_day: day (sun elevation > 15°, actual range ~22-59° — see Tag distribution above), even though sun_elevation_deg: [10, 65] in the config allows the low end. render_nav.py's low-sun exposure compensation (-4.0 + log2(sin(45°)/sin(elev)), raised/less-negative as elevation drops) is therefore unverified by any frame in this build — it has never been exercised near the 10-15° end of the configured range, so whether it actually keeps a genuinely low-sun frame from clipping/under-exposing is untested here, not just unmeasured.
Reproduce
export AEROSYNTH_BLENDER=/path/to/blender.exe # or put blender on PATH
source .venv/Scripts/activate
aerosynth fetch-geo configs/nav_v0.yaml --cache assets/geo
aerosynth build-nav configs/nav_v0.yaml out/nav_v0 --cache assets/geo --samples 16
aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_preds.csv # GT-seeded start (default)
aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_preds.csv --out out/nav_v0/report_baseline
aerobench baseline nav out/nav_v0 --out out/nav_v0/baseline_cold_preds.csv --no-init-from-gt # cold start, no positional aiding
aerobench eval --track nav --gt out/nav_v0 --pred out/nav_v0/baseline_cold_preds.csv --out out/nav_v0/report_baseline_cold
Detect v2 vs v1: synthetic→real transfer on DUT Anti-UAV (E0v2, E1v2, E2v2)
docs/experiments/2026-09-detect-v2.md
Run 2026-09-05 on branch plan5-detect-v2 at 8e4a015, on the rebuilt Detect
v2 (dataset built at 6cf56d0), same predict/eval code as every earlier arm.
Every number below is read from out/eval/summary.json, the per-experiment
out/eval/*/report.json, the out/models/*/results.csv epoch timers and the
training logs under out/eval/logs/ — all git-ignored — or computed by the
counting script described in §7.4. Nothing is typed from memory and nothing is
rounded up.
Headline: fixing the data defects did not fix the transfer, and v2 is still
worse than v1. The synthetic-only model trained on the rebuilt AeroBench Detect
v2 scores mAP50 0.159 on the DUT Anti-UAV test set against the spec §8
threshold of 0.5. The first v2 build — which carried a 10.7 % label-displacement
defect and clutter up to 260 px — scored 0.154. v1 scores 0.375. Repairing both
defects moved the headline number by +0.005, inside noise. This is a negative
result about the v2 design, not about the v2 build quality: unlabelled clutter
plus near field plus capture realism transfers worse than v1 on every size bucket
except tiny, and the large-object collapse (v1 0.458 → v2 0.118) survives
capping every unlabelled object below 96 px. v1 remains the recommended
synthetic set.
1. Setup
Everything about the experiment is unchanged from
2026-09-transfer-dut.md. What changed is the
synthetic training set.
Detector
yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime
torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
scripts/predict_coco.py, conf 0.001, imgsz 640, Ultralytics' default max_det 300 (COCOeval still scores only the top 100 per image — the same caveat as in the plan-3 write-up)
DUT Anti-UAV test, 2 200 images, 2 245 boxes — the identical file used in plan 3
What is the same as v1, deliberately: the same 40 Poly Haven panoramas in the
same order, the same drone sprites (assets/sprites_v2 is a byte copy of
assets/sprites_v1, 40/40 sprites.json sha256 match — see the card), the same
detector, the same hyperparameters, the same splits recipe (4 500 / 500), the same
DUT test set and the same evaluator. So a v1↔v2 difference is a difference in the
generator, not in the experiment.
What v2 changed — full details and the build's self-checks in the dataset card
docs/datasets/detect_v2.md:
Unlabelled distractors (hypothesis 1 of the plan-3 §4 list, the one we
rated largest): five procedural kinds — bird_soaring, bird_flapping,
airliner_far, mast_with_cables, rooftop_hardware — composited before the
drones and never annotated. 5 787 instances across 2 876 frames (0.575 of the
set); negatives doubled from 10 % to 20 %, and 841 of the 1 000 negatives carry
clutter. Their long side is drawn from distractor_px: [10, 96] — min 10 /
median 53 / max 95 px — so zero unlabelled objects land in the large
bucket against 1 319 labelled large boxes.
Near field (part of hypothesis 2): a fifth size bucket near: (200, 1100)
at 0.10 of the mix, fed by a separate 768 px sprite family rendered with a
close camera (1.05–1.5× object distance, 24–35 mm lens) — real perspective, not
an upscaled 256 px sprite. 615 boxes with a long side ≥ 200 px; 653 boxes touch
a frame edge.
Capture realism (hypothesis 5): motion blur along each drone's own velocity
vector, per-frame Gaussian sensor noise (σ 2–9), signal-dependent shot noise,
rolling-shutter skew on 25 % of frames — shearing the boxes with the pixels
— and in-pipeline JPEG requantisation at quality 55–95 on every frame.
What v2 did NOT change, and the report must not pretend otherwise:
hypothesis 3 (real CAD airframes — the drones are still procedural primitives) and
hypothesis 4 (real photographs as backgrounds — still HDRI panoramas). Those were
a controller decision, on licence and effort grounds, before any of this ran.
One structural difference beyond the three: v2 has 20 % negative frames where v1
had 10 %. That change rides along with the distractors in every v2 arm here and
in §5, and is not isolated by any of them.
This is the second v2 build. The first (09d522d, trained 2026-09-04) shipped
with two defects — displaced boxes on skewed frames and clutter up to 260 px — and
its runs are kept in §8 as v2-rc, because they are what that data did and
because §5's ablation was measured on it. Everything in §1–§4 and §6–§7 is the
rebuilt set.
Size buckets and what they are worth
Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS, untouched
by plan 5 exactly so these slices stay comparable): tiny < 16, small 16–32,
medium 32–96, large ≥ 96.
bucket
tiny
small
medium
large
total
GT boxes in DUT test
15
385
1 087
758
2 245
The tiny column is 15 boxes. It is a direction, never a measurement. The
large column is 758 boxes and is the one that carries this write-up's finding.
2. Results
Exp
Train data
Init weights
Eval set
mAP50
mAP50-95
tiny
small
medium
large
train time
E0
syn_v1 (4 500)
yolov8n.pt
synthetic v1 val (500)
0.905
0.697
0.706
0.929
0.996
0.999
≈ 4 408 s ‡
E0v2
syn_v2 (4 500)
yolov8n.pt
synthetic v2 val (500)
0.826
0.645
0.466
0.820
0.931
0.980
≈ 4 608 s ‡
E1
— (no training)
E0 best.pt
DUT test (2 200)
0.375
0.195
0.038
0.201
0.381
0.458
n/a
E1v2
— (no training)
E0v2 best.pt
DUT test (2 200)
0.159
0.063
0.071
0.203
0.262
0.118
n/a
E2
dut10 (520 real)
E0 best.pt
DUT test (2 200)
0.761
0.468
0.242
0.744
0.717
0.848
1 550.1 s
E2v2
dut10 (520 real)
E0v2 best.pt
DUT test (2 200)
0.750
0.448
0.242
0.703
0.733
0.837
1 245.1 s
E3 †
dut10 (520 real)
yolov8n.pt
DUT test (2 200)
0.759
0.434
0.156
0.713
0.743
0.824
1 525.8 s
E4 †
dut_full (5 200 real)
yolov8n.pt
DUT test (2 200)
0.912
0.580
0.535
0.816
0.913
0.968
5 286.2 s
Per-bucket columns are mAP50. The same buckets at mAP50-95:
Exp
tiny
small
medium
large
E0v2
0.242
0.563
0.787
0.889
E1
0.007
0.043
0.174
0.314
E1v2
0.023
0.042
0.097
0.072
E2
0.024
0.355
0.430
0.591
E2v2
0.024
0.301
0.426
0.584
E3 †
0.025
0.301
0.405
0.549
E4 †
0.134
0.373
0.554
0.723
† E3 and E4 were not retrained for this report. Their rows are reused verbatim
from the plan-3 run (out/eval/summary.json, rows E3/E4, produced at
e05a131). They are control arms that contain no synthetic data at all, so
nothing in plan 5 could change them; they were computed with the same
predict_coco.py / aerobench eval code path that E0v2–E2v2 use here.
‡ Neither E0 nor E0v2 ran straight through, and neither driver figure is the
training cost. E0's 928.1 s in summary.json covers only a resumed segment;
the plan-3 write-up reconstructs ≈ 4 408 s from results.csv. E0v2 was resumed
too: summary.json records resumed_from_epoch: 14,
out/eval/logs/E0v2.log line 2408 carries a ===== RESUME … from epoch 14 =====
marker followed by Resuming training … from epoch 15 to 30 total epochs, and
out/models/syn_v2/results.csv shows the epoch timer resetting between epoch 14
(2 134.66 s) and epoch 15 (167.80 s). The driver's 2 511.4 s therefore covers
epochs 15–30 only (16 epochs completed in 0.687 hours = 2 473.2 s in the log);
adding the first segment's 2 134.7 s gives ≈ 4 608 s of training time, in line
with v1. Whatever wall-clock elapsed during the interruption itself is in neither
figure. E2v2 ran straight through (20 epochs completed in 0.332 hours,
resumed_from_epoch: null), so its 1 245.1 s is the whole run; the gap to E2's
1 550.1 s for the same 20 epochs on the same 520 images is machine load, not data.
E0v2 is on its own evaluation set (held-out v2 synthetic val), so it is not
comparable to E0 as a score — both are pipeline sanity checks. The gate the plan
set was E0v2 mAP50 ≥ 0.8 or stop and debug the training: 0.826, passed.
Ultralytics' own best-epoch val number in the log agrees (MAP50_BEST 0.82758).
Training is not broken; the data is what changed. That 0.826 is still a
validation number picked on the same 500 images Ultralytics selected best.pt
with — mildly optimistic by construction, exactly as in v1, and irrelevant to
every DUT number here.
One metric that is meaningful — and two that are not
fp_per_image is measured on drone-free frames. DUT test has none
(negatives.images: 0 in every real report), so fp_per_image: 0.000 for E1v2 and
E2v2 is an artifact and must be ignored, as in plan 3.
For E0v2 it is real: the v2 synthetic val split carries 100 negative frames
(v2 is 20 % negatives), and out/eval/e0v2_sanity/report.json records 14 false
positives over 100 negatives = 0.14 per image. v1's E0 saw 54 negatives and 0
false positives. On its own domain, the v2-trained model still fires on empty
frames more than the v1-trained one did.
3. Verdict against spec §8
Gate 1 — E1v2 ≥ 0.5 mAP50 on real data with no real training images: FAILED.
0.159 against 0.5. It is not a near miss and not progress towards the threshold;
it is worse than v1's 0.375 by 0.216, far outside the ~0.02 noise band this
test set was shown to have in plan 3. Two failures in a row, and the second is a
regression that survived a full dataset rebuild. (The Drone-vs-Bird half of the
spec's real-data gate remains untested and still needs the founder's signature —
see §6.)
At mAP50-95 the picture is the same, 0.063 against v1's 0.195. Per bucket, v2 is
behind v1 everywhere except tiny — and tiny is 15 boxes:
bucket (mAP50-95)
E1 (v1)
E1v2 (v2)
Δ
tiny (15 boxes)
0.007
0.023
+0.016
small (385)
0.043
0.042
−0.001
medium (1 087)
0.174
0.097
−0.077
large (758)
0.314
0.072
−0.242
Gate 2 — E2v2 > E3 (is it worth pre-training on this synthetic set at all): no
signal in either direction.
E2v2 (syn_v2 → 10 % real)
E2 (syn_v1 → 10 % real)
E3 (COCO → 10 % real)
mAP50
0.7498
0.7605
0.7590
mAP50-95
0.4480
0.4682
0.4340
Δ mAP50 vs E3
−0.0092
+0.0015
—
Δ mAP50-95 vs E3
+0.0140
+0.0342
—
Read honestly:
E2v2 − E3 is −0.0092 mAP50 and +0.0140 mAP50-95. Both are inside the ~0.02
noise band, and they have opposite signs. Unlike v1 — whose +0.0342 at
mAP50-95 was at least outside the band — the rebuilt v2 buys nothing measurable
as pre-training.
E2v2 − E2 is −0.0107 mAP50 / −0.0202 mAP50-95. Also at or inside the band:
after fine-tuning on 520 real images the two synthetic sets are
indistinguishable, even though one of them is 2.4× worse zero-shot. 520
real images wash out a 0.216 zero-shot gap. That is the single most useful
practical fact in this write-up, and it repeats what v2-rc showed (§8).
E2v2's tiny 0.242 equals E2's exactly — 15 boxes, and both arms find the same
4 of them. Direction, not measurement.
E4 — the same architecture on 5 200 real images, no synthetic — is still
0.912 / 0.580, +0.162 mAP50 over E2v2. Nothing in v2 changes plan 3's
conclusion that real labels dominate.
Release decision: v1 stays the recommended synthetic set. v2 is kept as a
documented negative result together with its defect history; nothing in it beats
v1 zero-shot, and it does not beat v1 as pre-training either.
4. What actually changed in the errors
All counts below are our own greedy highest-score-first matching at IoU 0.5 over
the prediction JSONs (§7.4), never the evaluator.
The box flood got better; the detections did not
arm
boxes at conf 0.001
per image
images with > 100 boxes
boxes at conf 0.25
TP
FP
FN
precision
recall
E1 (v1)
235 584
107.1
963
6 297
1 187
5 110
1 058
0.189
0.529
E1v2 (v2)
174 433
79.3
592
2 500
629
1 871
1 616
0.252
0.280
E1v2-rc (§8)
290 192
131.9
1 333
4 585
879
3 706
1 366
0.192
0.392
E2v2
59 196
26.9
63
1 949
1 585
364
660
0.813
0.706
E4
25 670
11.7
9
2 231
1 999
232
246
0.896
0.890
This is the one place the rebuilt clutter did what hypothesis 1 promised.
Precision at the working threshold rose 0.189 → 0.252 (v2-rc, with its
oversized clutter, had managed 0.192), the diffuse box count fell by a quarter
against v1 and by 40 % against v2-rc, and the number of images drowning in
low-confidence boxes fell from 963 to 592. Recall fell from 0.529 to 0.280 to
pay for it. The model did not learn "a railing is not a drone"; it learned
"fire less", and it fires less at drones too.
The model still localizes, and still refuses to commit
For every GT box: is there any prediction over it at IoU ≥ 0.5, at any score; if
so, does the highest-scoring such prediction clear conf 0.25; and what is the
median of those scores.
arm
bucket
localized
of those, clear 0.25
median score
E1 (v1)
small
260/385 = 0.675
0.642
0.479
medium
808/1 087 = 0.743
0.681
0.703
large
594/758 = 0.784
0.785
0.772
E1v2 (v2)
small
270/385 = 0.701
0.559
0.308
medium
768/1 087 = 0.707
0.397
0.135
large
543/758 = 0.716
0.311
0.086
Localization is essentially intact — v2 puts a well-fitted box on 72 % of the
large real drones, slightly fewer than v1's 78 % but nowhere near a collapse —
and then scores it at a median of 0.086 against a 0.25 threshold. The failure
is confidence, not sight, exactly as in v2-rc. The difference from v2-rc is where
it now falls: v2-rc's suppression was concentrated at large (its small/medium
medians were 0.451 / 0.360, close to v1's 0.479 / 0.703); the rebuilt v2 has
dragged every bucket down (medium 0.703 → 0.135). Capping the clutter below
96 px did not lift the large bucket back to v1; it lowered the middle to match.
Recall at conf 0.25 by bucket, same matching:
arm
tiny
small
medium
large
E1 (v1)
4/15 = 0.267
167/385 = 0.434
550/1 087 = 0.506
466/758 = 0.615
E1v2 (v2)
4/15 = 0.267
151/385 = 0.392
305/1 087 = 0.281
169/758 = 0.223
The two frames from the v2-rc report, re-measured
The two figures below were drawn from the v2-rc prediction files and are kept
as they were; the rebuilt model's numbers on the same two frames are given in the
text. (Left panel = E1 on v1, right = E1v2 on v2-rc; green = ground truth, red =
prediction at conf ≥ 0.25, yellow = the model's best-IoU box when it falls below
0.25.)
a) The large-object collapse — img/00103.jpg, GT 185 × 102 px
v1: one box, IoU 0.979, score 0.904. v2-rc: zero boxes clear 0.25; best-IoU
box 0.991 at score 0.0038. The rebuilt v2 is no better and slightly
worse: still zero boxes over 0.25 (its highest-scoring prediction anywhere in
the frame is 0.010), best-IoU box 0.694 at score 0.0012. n = 1, but it is
the §3 table in one image.
b) The near-field case from plan 3 — img/01124.jpg, GT 865 × 305 px
This is the one frame where the rebuilt v2 wins outright. v1 fragments the
aircraft (best-IoU box anywhere 0.445, at score 0.153 — below threshold; its best
scoring box, 0.672, covers a 432 px slice). v2-rc gets the extent right —
IoU 0.877 — but scores it 0.062, below threshold. The rebuilt v2 gets IoU 0.877
at score 0.488, i.e. a confident, correct detection of a frame-filling drone,
which neither of the other two models produced. The near-field sprites work; the
suppression that hid this in v2-rc is gone here, and not in (a).
5. The causal reading, revised
The v2-rc write-up named a cause: unlabelled clutter, and specifically clutter in
the same size bucket as the targets. The rebuild was that hypothesis made real —
distractor_px: [10, 96], 0 unlabelled instances ≥ 96 px against 1 319
labelled large boxes, and the two large aerial kinds (helicopter, plus the
already-narrow airliner_far shrunk with the rest) no longer able to appear at
drone scale. It was a direct test with a stated falsifier: large mAP50 returns
towards v1's 0.458 while precision at conf 0.25 keeps some of v2b's 0.259.
Half of that came true and the half that mattered did not. Precision went to
0.252. large mAP50 went 0.064 → 0.118, against v1's 0.458: about 14 % of the
gap, on one seed. So:
The "large unlabelled clutter outnumbers large labelled drones" explanation is
at most partial. It cannot be the main cause of the large-object collapse,
because removing it entirely left 86 % of the collapse in place. The v2-rc
report should be read as having over-committed to it.
The label defect is refuted as a driver too: C1 displaced 10.7 % of v2-rc's boxes,
the rebuild has none below IoU 0.5 and 98.7 % above 0.9, and the headline moved
0.154 → 0.159.
What the evidence still supports, all of it one seed per arm:
Clutter of any kind is the largest single suspect, but by arm-level evidence,
not by the size mechanism. In the rc-era 2 × 2 (§8) the arm with no clutter
at all — v2a, which kept the near field, the capture realism and the 20 %
negatives — scored 0.334, the closest any v2 variant has come to v1's 0.375.
Every arm that contains unlabelled clutter, at any size, scores 0.154–0.231.
What the rebuild rules out is the size-matching explanation of how clutter
hurts, not clutter itself.
Near field + capture realism costs small and medium, and buys the
near-field frame. The clean rc-era comparison (v2a − v1, both without clutter
on one side only) was tiny +0.039, small −0.024, medium −0.054, large −0.038,
overall −0.041 — small, above the noise band but the smallest effect measured.
§4b is the qualitative other side of it.
The suppression mechanism is confirmed and is now global. Whatever v2 teaches,
it is "do not commit", and in the rebuilt set that applies to every size bucket
rather than only to large objects.
A confound we never removed: 20 % negatives. v1 has 10 %; every v2 arm ever
trained, including v2a, has 20 %. No arm isolates it, so "more empty frames make
the model reticent" remains a live and completely untested alternative to
"clutter makes the model reticent" — and it would explain the global
suppression better than the size story does.
6. Honesty rules and what is next
Rules applied throughout, same as the plan-3 write-up:
One seed per arm. Every comparison here is one training run against one
training run. Words like "suggestive" are load-bearing; nothing in §5 is an
established causal result.
A mAP50 difference below ~0.02 on this test set is noise (established in plan
3 from E2 vs E3). That covers E1v2 − E1v2-rc (+0.005), E2v2 − E3 (−0.0092),
E2v2 − E2 (−0.0107) and E2v2 − E3 at mAP50-95 (+0.0140). It does not cover
E1v2 − E1 (−0.216).
A slice with fewer than ~50 boxes is a direction, not a measurement. That is
the tiny column everywhere: 15 boxes in DUT test.
fp_per_image on DUT test is an artifact (zero negative frames). Only
E0v2's 0.14 over 100 synthetic negatives is a real false-positive rate, and it is
on synthetic data.
Every number traces to out/eval/summary.json, an out/eval/*/report.json, an
out/models/*/results.csv, a log under out/eval/logs/, or the counting script
in §7.4.
v3 hypotheses — none of these has been tested
v2 changed four things at once (clutter, near field, capture realism, negative
fraction) and the rebuild changed a fifth. That is why we cannot say which one
costs the 0.216. The next round should start from v1 and add exactly one thing
per arm, each measured by the same E1 — train on synthetic only, evaluate on DUT
test, report per-bucket — with nothing else touched. Each is a guess consistent
with §4 and §5, and each is one 30-epoch run plus one inference pass (≈ 1.5 h on
an RTX 3070 Ti):
v1 + 20 % negatives only. No clutter, no near field, no capture realism —
only negative_fraction: 0.20. This removes the confound §5 names last and is
the cheapest of the three. If E1 drops materially below 0.375, the reticence is
about empty frames and none of the rest of v2 needs explaining.
v1 + capture realism only. Motion blur, sensor/shot noise, rolling shutter
(boxes sheared), JPEG requantisation, at the v1 size mix and 10 % negatives.
Isolates hypothesis 5 for the first time.
v1 + small ground clutter only.bird_soaring, bird_flapping,
mast_with_cables, rooftop_hardware at distractor_px: [10, 96], no
airliner_far, v1 negatives, v1 size mix. Isolates the thing §5 still suspects
most, without the negative-fraction change riding along.
The near field is deliberately not on that list: §4b says it works, and it is the
only change with a positive result to its name. Add it back after one of 1–3
explains the loss.
Then the two hypotheses v2 never touched, in expected-value order: hypothesis 3
(real CAD / photogrammetric airframes instead of procedural primitives) and
hypothesis 4 (real photographs as backgrounds alongside HDRIs). Both were deferred
on licence and effort grounds and both remain completely open. Given that four
data changes in a row have not moved E1 towards 0.5, they may matter more than
anything in this plan did.
Founder action still required — unchanged from plan 3 §5. The WOSDETC
Drone-vs-Bird dataset — the one real benchmark that actually contains birds, and
therefore the honest test of whether bird distractors help — cannot be downloaded.
Access needs an email to wosdetc@googlegroups.com and a signed data usage
agreement from a named person (https://github.com/wosdetc/challenge). Nothing in
that can be automated. Until it exists, "our synthetic birds teach the model to
reject real birds" is untestable, and every clutter conclusion here is measured
only against DUT, which contains no birds at all.
7. Reproducing this
Repo root, project venv with the training extra (uv pip install -e ".[dev,train]").
assets/real/dut/** is fetched exactly as in 2026-09-transfer-dut.md §6.1 — that
step is unchanged and not repeated here.
(assets/sprites_v2 is first a plain copy of assets/sprites_v1; the sprites are
unchanged between the two v2 builds — dropping helicopter is a filter at index
time. See docs/datasets/detect_v2.md for the sprite
sha256 check and the build's self-checks.)
E3 and E4 are deliberately not in that --only list: they contain no synthetic
data, and their rows are reused from the plan-3 run. Budget ≈ 1.6 h of training on
an RTX 3070 Ti plus inference over 2 200 images twice.
On the resume. E0v2's training was interrupted after epoch 14 and restarted
from out/models/syn_v2/weights/last.pt; the driver re-invokes train_yolo.py
with --resume, appends a ===== RESUME … from epoch 14 ===== marker to
out/eval/logs/E0v2.log and records resumed_from_epoch in summary.json. A
clean run needs no resume and reports the whole 30 epochs in one timer; see the
‡ footnote in §2 for how to read the split figure when it happens.
7.3 The v2-rc artifacts
The first build's runs were not deleted. They live beside the current ones as
out/eval/e0v2_sanity_rc, out/eval/e1v2_syn_only_rc,
out/eval/e2v2_syn_then_real10_rc and out/models/syn_v2_rc,
out/models/syn_v2_then_dut10_rc (all git-ignored). The two ablation arms of §8
were run through train_yolo.py/predict_coco.py directly rather than the
driver, so their numbers exist only as out/eval/e1v2a_syn_only/report.json and
out/eval/e1v2b_syn_only/report.json; their configs are committed as
configs/detect_v2a.yaml and configs/detect_v2b.yaml.
7.4 The counting script
The conf-0.25 precision/recall table, the per-bucket recall table, the
localized-vs-suppressed table and the two figures are not produced by any committed
tool — they are short one-off scripts over the prediction JSONs, the same approach
as plan 3 §4. Each loads assets/real/dut/test_coco.json plus an
out/eval/*/preds.json, does greedy highest-score-first matching at IoU 0.5, and
prints counts; the localized/suppressed table instead takes, per GT box, the
highest-scoring prediction at IoU ≥ 0.5. There is no hidden filtering: every
prediction in the file is considered, and the "boxes at conf 0.001" column is
simply len(preds).
7.5 Where the numbers live
out/** is git-ignored; nothing but this write-up and its two figures is
committed. The driver writes one row per experiment to out/eval/summary.json
(keyed by experiment, replacing that experiment's earlier row on a re-run — the
E0v2/E1v2/E2v2 rows there are the rebuilt runs, and the v2-rc numbers survive only
in the _rc directories above and in §8 of this document).
8. History: v2-rc, the first build
The first v2 build (09d522d, 2026-09-04) was trained and evaluated before review
found two defects in it:
C1 — displaced boxes. The rolling-shutter skew was applied to the pixels
after the boxes had been measured: 636 of its 5 972 boxes (10.7 %) sat off their
drone, 51 at IoU < 0.5.
C2 — oversized clutter.distractor_px: [12, 260] put 3 824 unlabelled
objects ≥ 96 px against 1 318 labelled large boxes, 2.9:1, including a
helicopter whose rotor disc is rendered with the same translucent-disc trick as
the drone propellers.
Both are fixed at 6cf56d0. The rc results are kept because §5 rests on them:
arm
what it is
mAP50
mAP50-95
tiny
small
medium
large
v1 (E1)
reference
0.375
0.195
0.038
0.201
0.381
0.458
v2a
v2-rc without clutter (near field + capture realism + 20 % negatives)
E0v2-rc scored 0.825 on its own synthetic val (sanity gate passed) and
E2v2-rc reached 0.768 / 0.459 on DUT test — i.e. the fine-tuned arm was as
indistinguishable from E2/E3 then as it is now.
Caveats that apply to the four rc rows, and therefore to §5:
One seed per cell, four single runs; the 2 × 2 is suggestive, not
established.
All three v2 arms carry 20 % negatives and v1 carries 10 %, so every delta
measured against v1 also contains that change.
v2a and v2b were trained on data with the C1 label defect (v2b also with
C2's oversized clutter). The rebuild shows C1 is worth ≈ 0.005 on the headline,
which is why we still quote them — but they are not clean arms.
The two arms' synthetic sanity numbers come from Ultralytics
(MAP50_BEST 0.85305 and 0.89097 on each arm's own val split), not from
aerobench eval; they are used only to confirm the ≥ 0.8 training gate.
v2a's wall time is not a training cost.out/models/syn_v2a/results.csv ends
at 36 360.4 s because one epoch absorbed 32 433.8 s when the machine slept; the
other 29 total 3 926.6 s, so the real cost was ≈ 4 060 s, in line with v2b's
3 820.2 s.
Итог для основателя
Главное: вторая версия синтетического набора не заработала, и починка её
дефектов ничего не изменила. В первой сборке v2 нашли два дефекта — рамки на
10,7 % объектов были смещены относительно дрона, а отвлекающие объекты были
слишком крупными. Оба исправили, набор пересобрали и обучили заново. Результат на
реальном тесте DUT Anti-UAV: mAP50 = 0.159 против 0.154 у сломанной сборки и
0.375 у первой версии набора. Порог спецификации — 0.5. То есть починка данных
дала +0.005, это шум, а отставание от v1 никуда не делось. Порог не взят второй
раз подряд.
Дело не в качестве сборки, а в самой идее v2. Набор проигрывает v1 во всех
размерных группах, кроме «крошечной» (а она посчитана по 15 рамкам во всём тесте
и ничего не доказывает). На крупных дронах — 758 рамок, самая важная часть теста —
метрика 0.118 против 0.458 у v1.
Наша прошлая объяснительная версия оказалась в лучшем случае частичной. В
прошлый раз мы решили, что виноваты слишком крупные неразмеченные объекты: их было
почти три на каждый размеченный крупный дрон, и модель научилась «большое в небе
обычно не цель». В пересобранном наборе весь мусор строго мельче порога «крупного»
— таких объектов ровно ноль. Проверка провалилась: провал на крупных объектах
восстановился с 0.064 только до 0.118 при 0.458 у v1, то есть примерно на седьмую
часть. Значит, размер мусора — не главная причина, и прошлый отчёт был слишком
уверен. Мы это записали прямо в документ, а не переформулировали задним числом.
Что подтверждается по-прежнему. Модель дроны видит: она ставит хорошую
рамку на 72 % крупных реальных дронов — и присваивает ей уверенность 0.086 при
рабочем пороге 0.25, то есть сама же выбрасывает. Причём теперь это подавление
распространилось на все размеры, а не только на крупные. Из всех вариантов, что мы
обучили, ближе всего к v1 (0.334 против 0.375) оказался тот, где отвлекающих
объектов не было вообще, — так что подозрение на мусор остаётся, но механизм
мы назвали неправильно. Ближняя зона и реализм съёмки стоят немного (−0.041) и на
одном кадре дали единственную настоящую победу: дрон, заполняющий кадр, который
раньше разваливался на куски, теперь уверенно распознаётся целиком.
Одна вещь всё-таки улучшилась. Точность при рабочем пороге выросла с 0.189 у
v1 до 0.252 — ложных срабатываний на перилах и крышах стало меньше, поток мусорных
рамок сократился на четверть. Но платой стала полнота: 0.280 против 0.529. Модель
не научилась «перила — это не дрон», она научилась «реже нажимать».
Предобучение: разницы нет никакой. Если дообучить на 10 % реальных данных,
новая синтетика даёт 0.750, старая 0.761, а вообще без синтетики — 0.759. Всё это
внутри шума. 520 реальных кадров стирают разрыв в 0.216, который был без них.
Полный реальный набор по-прежнему бьёт всё: 0.912. Вывод плана 3 не изменился —
синтетика это экономия разметки, а не замена ей.
Решение. Релизным синтетическим набором остаётся v1. v2 сохраняем как
задокументированный отрицательный результат вместе с историей его дефектов — он
не лучше ни без дообучения, ни как предобучение.
Что делать дальше — и это пока гипотезы, ни одна не проверена. Ошибка плана 5
в том, что мы поменяли четыре вещи сразу и теперь не можем сказать, какая именно
стоила нам 0.216. Следующий раунд надо начинать от v1 и добавлять по одной вещи за
прогон, меряя одним и тем же числом (E1 на DUT, примерно полтора часа обучения на
арм): (1) v1 плюс только удвоенная доля пустых кадров — это единственное, что мы
ни разу не изолировали, и это лучше всего объясняет «всеобщее» подавление
уверенности; (2) v1 плюс только реализм съёмки; (3) v1 плюс только мелкий наземный
мусор без самолётов. Ближнюю зону вернуть после того, как один из этих трёх
прогонов объяснит потерю. И только потом — две гипотезы, которых мы вообще не
касались: реальные CAD-модели дронов вместо процедурных и реальные фотографии
вместо панорам. Четыре изменения данных подряд не сдвинули E1 к 0.5, так что эти
две могут значить больше, чем всё, что мы сделали в этом плане.
Оговорка о доверии к цифрам. На каждый вариант — один прогон обучения без
доверительного интервала. Разница меньше ~0.02 на этом тесте — шум; под это
попадают и +0.005 от починки дефектов, и все сравнения с дообучением. Колонка
«крошечные объекты» посчитана по 15 рамкам и является направлением, а не
измерением.
Требуется ваше действие — то же, что и в прошлый раз. Набор Drone-vs-Bird —
единственный реальный тест, где вообще есть птицы, то есть единственный способ
честно проверить, помогают ли наши синтетические птицы. Скачать его нельзя: нужно
письмо на wosdetc@googlegroups.com и подписанное соглашение об использовании
данных от вашего имени (https://github.com/wosdetc/challenge). В DUT птиц нет
вовсе, поэтому все выводы про отвлекающие объекты проверены только на нём.
Real-imagery gate plan for AeroBench Nav
docs/experiments/2026-09-nav-real-gate-plan.md
Survey done 2026-09-05 on branch plan6-nav at 92143a9. This is a plan, not a
run — no dataset has been downloaded. Every fact below about a third-party
dataset is cited to the page it was read from; anything not confirmed on a primary
source is marked as such rather than guessed.
Headline: unlike Detect (which already has a real-data gate result — DUT
Anti-UAV, see docs/experiments/2026-09-transfer-dut.md), Nav has never been run
against a single real photograph.docs/datasets/nav_v0.md is entirely
synthetic: rendered Blender frames over a Copernicus DEM + PNOA texture, evaluated
against a same-source PNOA reference map from a different year. The spec's own
Nav acceptance test (§8) — "a baseline solution gives meaningful errors, metrics
stable across runs" — has only ever been checked against ground truth that the
renderer itself generated. This plan proposes the smallest real-data experiment
that would actually test it, using datasets that already exist and are (mostly)
downloadable without a data-use agreement.
1. What the synthetic baseline needs from a real dataset
Recapping the actual artifacts (not the spec's original proposal) so the fit
column in §2 means something concrete:
src/aerobench/baseline_nav.py is an aided normalized cross-correlation
template matcher, not the SuperPoint/LightGlue/ORB feature matcher the spec
(§8) names — that was never built. It needs, per frame: agl_m (altitude above
ground) and yaw_deg (compass heading) to rescale and de-rotate the frame to
the reference map's north-up grid (footprint_gsd(agl_m, fx_px) = agl_m / fx_px,
then NCC-match against a search window of a single reference image at a known
ref_gsd_m). It only attempts nadir frames — oblique frames get no prediction
at all (out/nav_v0 report: coverage: 0 on 640/960 oblique frames).
src/aerobench/metrics_nav.py needs per-sequence pose ground truth in a metric
(x, y) frame matching the reference map's pixel grid, keyed as
sequence,frame,x,y — see schema_nav.REQUIRED. It computes p50/p90/p99
horizontal error, success_10m/success_50m, coverage (fraction of frames
with any prediction), init_success (first-frame error < 50 m), and
mean_drift_pct.
The CLI is aerobench baseline nav DATASET_DIR --out preds.csv
[--no-init-from-gt] [--no-aided] and aerobench eval --track nav --gt
DATASET_DIR --pred preds.csv --out DIR [--align-first] (src/aerobench/cli.py).
So a real dataset is a good fit for this exact gate if it gives: (a) real
nadir(-ish) UAV photos, (b) per-frame altitude and heading (or enough to derive
them), (c) absolute position ground truth (GPS, convertible to a local metric
frame), and (d) a georeferenced reference image of the area at a known ground
sample distance (GSD) — ideally from a different capture date than the query
photos, to keep testing the same synthetic→real question nav_v0.md already poses
(map-vs-observation gap), not a new one.
2. Candidate datasets
Dataset
Region
Images
Altitude
Pitch
Resolution/GSD
GT fields
Reference map
Licence
Download
Verdict
UAV-VisLoc (Xu et al., arXiv 2405.11936, 2024) [1][2]
11 sites, China (villages, towns, farms, cities, rivers, hills)
80 m / 90 m / 100 m (three fixed heights, encoded in filenames H80/H90/H100) [3][4]
Nadir (low-altitude urban self-positioning task)
Not stated in repo README
GPS lat/lon/height in per-flight files [3]
Two dated satellite mosaics per campus per height (e.g. H100.tif vs H100_old.tif) — years not disclosed in README, but this is a built-in multi-year map-gap test, the closest real-world analogue to our map_gap_years tag
Apache-2.0 stated on the code repo — not confirmed this covers the imagery itself, flag for founder
Hugging Face, no registration found; total size not confirmed, almost certainly >200 MB across all campuses
Second-best fit — the only candidate with an explicit multi-vintage reference map, directly testing the same map-vs-observation gap nav_v0.md measures, but no confirmed camera heading per frame (fixed-heading flights, unclear if yaw varies)
SUES-200 (Zhu et al., IEEE TCSVT 2023; arXiv 2204.10704) [6][7]
Not disclosed in README/abstract (China; university-affiliated)
24,120 images total across drone + satellite views, 4 altitudes
150 / 200 / 250 / 300 m (four fixed heights) [6]
Not stated
Not stated
Not stated in README/abstract
Per-location satellite reference image
README states data is "ONLY available to academic research" [7] — a use restriction, not a copyright licence; would need to check the actual EULA before any commercial-adjacent use
Google Drive / Baidu / Tianyi Cloud; size not disclosed
Weaker fit — altitudes are higher than our nadir sequences' mid/high buckets (good for stress-testing scale), but GT fields and licence are too thin to plan around without opening the actual download
AerialVL (He et al., IEEE RA-L 2024) [8]
Shandong Province, China (~120.42–120.48°E, 36.57–36.60°N)
18,361 aerial images + 14,096 cropped map patches, 11 sequences, 3.7–11 km each (~70 km total) [8]
Not stated (fixed-wing/rotor mix implied by "aerial-based vehicle" framing)
Not stated
Not stated; reference maps are Google Earth / USGS satellite imagery [8]
GNSS position (NovAtel OEM718D, ~1.5 m RMS), filename-encoded UTC timestamp + lat/lon [8]
Google Earth and USGS satellite imagery, cropped per sequence
MIT [8] — the most permissive licence found in this survey
Hugging Face / Tsinghua Cloud / Baidu; size not disclosed
Good licence, but the reference map is itself a scrape of Google Earth/USGS — redistributing our evaluation output is fine, but we'd be depending on someone else's re-use of Google's tiles, which is a licence smell worth independent checking before citing publicly
ALTO (Cisneros et al., arXiv 2207.12317, 2022) [9][10]
Ohio & Pennsylvania, USA (helicopter, Pittsburgh corridor)
Not disclosed (two trajectories, ~150 km and ~260 km) [9][10]
States reference imagery is provided over the flight paths, but exact source/GSD not found in this survey
BSD-3-Clause [9]
Round 1/2 competition subsets via Dropbox; "full dataset coming soon" as of the page read [9] — sizes not disclosed
Manned helicopter, not a drone — altitude and camera model likely far outside our low/mid/high AGL buckets. Useful only as a stretch/robustness check, not a like-for-like Nav validation
VPAIR (Schleiss et al., arXiv 2205.11567, 2022) [11][12]
High-res render from public geodata, includes depth — closer to a simulated reference than a plain orthophoto
"Academic use only, commercial use prohibited" [12] — explicit no-commercial clause
Sample 0.86 GB via Zenodo; full set requires a request form [12]
Fails our commercial-use constraint outright given AeroSynth is a paid-product play — usable for a pure internal sanity check but not for anything the founder might later want to reference commercially
OrthoLoC (Dhaouadi et al., NeurIPS 2025 D&B, oral) [13]
47 locations, 19 regions, Germany and the United States [13]
16,425 real UAV images [13]
Not stated in the page fetched
Not stated
Reference: official national/regional DOPs (digital orthophotos) + DSMs from European open geoportals, licensed CC BY 4.0 [13]
Orthophotos + digital surface models from multiple European open geoportals (CC BY 4.0) [13]
Dataset itself: CC BY-NC-SA 4.0; underlying geodata CC BY 4.0 [13]
Direct HTTP from TUM CVG webshare, 287.3 GB total [13]
Exactly the non-Spain, real-national-orthophoto case the brief asked to discuss (see §3) — best licence clarity of any candidate, but far too large to survey-download and would need per-region subsetting before it is even a candidate
NUST Islamabad, DHA Islamabad, Gujar Khan District, Pakistan
2,052 images across the 3 areas [14]
Not stated
Not stated
Not stated
Point correspondences (labels.csv) between images and orthomosaics, not GPS pose [14]
Orthomosaics built from the image set itself, plus a separate satellite-derived orthomosaic (no correspondences given for that one) [14]
BSD-2-Clause [14]
git clone + download.sh, size not disclosed but the image count (2,052) suggests it is small — the only candidate plausibly under our 200 MB cap
Wrong ground-truth shape for our metrics (point correspondences, not x,y position + altitude/heading) — would need a bespoke adapter and gives no absolute-position error metric; useful only as a cheap template-matching smoke test, not a metrics_nav gate
University-1652 (Zheng et al., ACM MM 2020) [15]
72 universities worldwide
50,218 train + ~90k query/gallery
N/A
N/A
Real satellite + street view; drone-view images are simulated flights rendered in Google Earth, not real UAV photography [15]
Real satellite imagery (source unstated)
MIT (code); imagery on request, "usually reply in 5 minutes" [15]
Manual request to the maintainer [15]
Excluded per task scope — drone-view frames are synthetic renders of a 3D reconstruction, exactly the kind of synthetic-only set the brief said to exclude, even though the paper doesn't market it that way
Sources fetched for this table: [1] https://github.com/IntelliSensing/UAV-VisLoc, [2]
https://arxiv.org/abs/2405.11936, [3] https://github.com/Dmmm1997/DenseUAV, [4]
web search result quoting the DenseUAV/UL14 TIP 2023 paper abstract (Hangzhou, 14
campuses, 80/90/100 m, 2020/2022 satellite imagery) — no single fetchable URL
returned the primary PDF within size limits, so this fact is cross-referenced
across two independent search snippets rather than one primary fetch, [5]
https://arxiv.org/pdf/2208.06561 (UL14 as an FPI-benchmark reconstruction of
DenseUAV), [6] https://arxiv.org/abs/2204.10704, [7]
https://github.com/Reza-Zhu/SUES-200-Benchmark, [8] https://github.com/hmf21/AerialVL,
[9] https://github.com/MetaSLAM/ALTO, [10] https://metaslam.github.io/datasets/alto/,
[11] https://arxiv.org/abs/2205.11567 (via search summary), [12]
https://github.com/AerVisLoc/vpair, [13] https://deepscenario.github.io/OrthoLoC/,
[14] https://github.com/m-hamza-mughal/aerial-template-matching-dataset, [15]
https://github.com/layumi/University1652-Baseline and search summary of Zheng et
al. 2020 confirming the Google Earth simulated-flight origin of drone-view frames.
Also surfaced but not deep-dived (recency noted for completeness, not verified
beyond a search snippet — do not cite these as fact without a follow-up fetch):
SatLoc (MDPI Remote Sensing, Sept 2025, doi:10.3390/rs17173048 — rotorcraft
UAV-to-satellite matching, 100–300 m AGL, multi-terrain/weather; page returned
HTTP 403 to WebFetch, only a search snippet was read), AnyVisLoc (2025,
18,000 images, 2.5D reference maps combining aerial photogrammetry and
historical satellite maps — another multi-vintage-map candidate worth a follow-up
look), and VNS (UAV video + synced GPS + satellite imagery, no further detail
found). None of these three should be treated as confirmed until fetched directly.
3. Recommendation
Primary: UAV-VisLoc. It is the only candidate that hands our existing
baseline_nav.py exactly the inputs it already consumes — per-image altitude and
heading, plus a packaged, georeferenced satellite reference image at a stated GSD
— without inventing new fields or re-deriving intrinsics from something the
dataset doesn't provide. Diverse Chinese terrain (villages/farms/hills/cities)
gives us tag-like slicing similar to our own altitude/view buckets even
though the dataset doesn't label them that way. The catch: the dataset's own
licence for the imagery (as opposed to the arXiv listing) is not stated
anywhere found in this survey — see open questions below — and even the official
"sample" subset (2.04 GB) is ~10x our 200 MB survey cap, so pulling even one
location needs an explicit go-ahead.
Secondary: DenseUAV / UL14. Weaker on per-frame heading (not confirmed to
vary — the flights may hold a fixed heading, which would silently degrade our
aided baseline to "yaw ≈ 0 always," a real limitation to test for before trusting
any number from it), but it is the only candidate in this survey with an
explicit two-vintage reference map per site (H100.tif vs H100_old.tif) — the
single closest real-world analogue to the map_gap_years axis nav_v0.md already
reports on. If UAV-VisLoc's licence turns out to be unusable, this is the
fallback with the most direct relevance to the specific claim we're testing
(does the aided-NCC baseline survive a real map-capture-date gap, not just a
synthetic one).
On the non-Spain reference-map question the brief asked about: none of our
top two picks need us to source a reference map ourselves — both ship one. But
if a future real-flight dataset (e.g. ALTO, AerialVL) lacks a redistributable
map, the options and their tradeoffs are:
Sentinel-2 (ESA/Copernicus free, full and open data policy; attribution required; 10 m GSD) — freely usable anywhere on Earth, no
registration friction, but 10 m GSD is coarser than our own success_10m
threshold itself; a template match against a 10 m/px map cannot in principle
distinguish positions closer than roughly one pixel of that map, so it is not
fit for validating a <10 m-error claim, only for a much looser "found the right
square kilometer" check.
National mapping-agency orthophotos (PNOA-equivalent programs — e.g.
Germany's state DOP series, used directly by OrthoLoC [13]; the US's NAIP,
public-domain, ~0.6–1 m GSD, distributed by USGS/AWS) — same shape as what we
already do for Teruel, typically CC BY or public-domain, and at a GSD that
actually supports a <10 m claim. This is the right long-term answer for any
new region, but means writing one geo.py-style fetcher per country/agency —
real but bounded engineering work, not a licence problem.
Commercial tiles (Google/Bing/Maxar/Airbus) — best GSD, but ToS for these
almost universally forbid bulk scraping and redistribution outside the
vendor's own SDK/map widget; several of the papers above (AerialVL, [8];
University-1652 satellite-view, unstated) appear to depend on Google Earth
imagery without a clear redistribution licence, which is exactly the kind of
provenance gap we should not repeat in anything AeroSynth republishes. Treat
as internal-eyes-only if used at all, never as a citable public number.
4. Gate protocol
Mirrors the Detect acceptance test's structure (train/fit only on our own
artifact, evaluate unmodified against real ground truth, report pass/fail against
a stated number) as closely as the two tracks' shapes allow — Detect trains a
model; Nav's "model" is the deterministic aided-NCC baseline itself, so here
"gate" means "run the unmodified algorithm against real frames it has never seen,
using the dataset's own ground truth."
Adapter, not baseline change. Write
src/aerosynth/realdata/uav_visloc.py (mirrors the existing
realdata/dut_antiuav.py pattern) that converts one UAV-VisLoc location into
our Nav schema: sequences.json, poses.csv (frame,t_s,x,y,z,yaw_deg,
pitch_deg,roll_deg,agl_m), intrinsics.json, reference_map.png +
reference_map.json. baseline_nav.py and metrics_nav.py themselves must
not change — the whole point is testing them unmodified against a new
ground truth source, the same way the Detect gate reused the existing eval
code against DUT/Anti-UAV rather than writing new metrics for real data.
GPS → local metric frame: reproject lat/lon to the same UTM zone/plane the
packaged satellite map is gridded in (need to confirm the map's
georeferencing format on inspection — GeoTIFF with an embedded CRS would be
ideal; a plain PNG/JPG would need a companion world file or explicit corner
coordinates, unconfirmed from the README).
GSD: the dataset states resolution directly ("≈0.1–0.2 m/px") rather than
giving a focal length — baseline_nav.py's footprint_gsd(agl_m, fx_px)
assumes we derive GSD from altitude and focal length, so the adapter should
bypass that helper and set the matcher's scale directly from the stated
per-image GSD (or a derived intrinsic if focal length turns out to be in
the dataset's own metadata columns — not confirmed either way here).
pitch_deg and roll_deg: UAV-VisLoc's README only confirms nadir shooting
and heading — pitch/roll are presumably ≈ −90°/0°, but this needs
confirming on the actual metadata files (only the paper abstract/README was
read for this survey, not the CSV schema itself).
Pick a small slice first. Convert one location (whichever Drive
subfolder looks smallest on inspection) rather than all 11, to keep the
founder-approval ask small and to get a fast first read on whether the
adapter even produces sane crops (repeat the nav_v0.md "visual sanity check":
manually compare 2–3 frames against their reference-map crop before trusting
any number).
Run exactly the existing commands:
```
aerobench baseline nav assets/real/nav/uav_visloc/ --out out/real_nav//preds_gtseed.csv
aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_gtseed.csv --out out/real_nav//report_gtseed
aerobench baseline nav assets/real/nav/uav_visloc/ --out out/real_nav//preds_cold.csv --no-init-from-gt
aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_cold.csv --out out/real_nav//report_cold
aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_gtseed.csv --out out/real_nav//report_gtseed_aligned --align-first
aerobench eval --track nav --gt assets/real/nav/uav_visloc/ --pred out/real_nav//preds_cold.csv --out out/real_nav//report_cold_aligned --align-first
``
Same four-way split nav_v0.md already reports (GT-seeded / cold-start ×
with/without--align-first), same metrics (p50/p90/p99,success_10m,success_50m,coverage,init_success,mean_drift_pct).
4. **Repeat the run twice** on the same converted data with no code changes —
the aided-NCC matcher andmetrics_nav` are deterministic, so "stable across
runs" (spec §8) for this baseline should mean bit-identical CSVs, not just
similar numbers. If two runs differ at all, that is itself a bug to fix before
any pass/fail number is trusted, distinct from whether the number is good.
5. Compare against the synthetic baseline as a reference point, not because
the two are expected to match, but because a wildly different order of
magnitude is the signal to look at:
nav_v0 (synthetic), nadir, GT-seeded
nav_v0 (synthetic), nadir, cold
real-data target
p50
4.7 m
6.8 m
see thresholds below
p90
10.8 m
844.1 m
—
success_10m
0.872
0.628
—
coverage
1.000
1.000
—
5. Pass/fail threshold
The spec (§8) only says the Nav acceptance test should give "meaningful errors,
[with] metrics stable across runs" — no number, unlike Detect's explicit mAP@0.5
≥ 0.5. Proposing a concrete number here rather than leaving it as vague as the
spec:
PASS: on the nadir subset of the converted real slice, GT-seeded start,
p50 < 25 mandcoverage > 0.8and the two repeated runs are
bit-identical. The threshold is deliberately looser than the synthetic 4.7 m —
real lens distortion, unconfirmed intrinsics, JPEG compression, and a genuine
cross-sensor gap (drone camera vs. whatever produced the packaged satellite
tile) are all real degradations our synthetic-only build never has to survive.
25 m is still two-and-a-half success_10m-widths, i.e. "clearly finding the
right neighbourhood," not "clearly finding the right building" — a real pass
here is evidence the algorithm generalizes past its own renderer, not evidence
it is production-ready.
FAIL: p50 lands at hundreds/thousands of metres — comparable to the
synthetic cold-start tail's p90/p99 (844 m/1368 m) or worse, i.e.
indistinguishable from a matcher that locked onto the wrong part of the map —
orcoverage collapses because most real frames error out in the adapter
(format/scale mismatches, not an NCC failure). These two failure modes must be
told apart before reporting anything: the first is a genuine
synthetic-training/real-world domain-gap finding (parallel to Detect's E1
result, docs/experiments/2026-09-transfer-dut.md — a real, reportable
negative result, not an embarrassment to hide); the second is a bug in this
plan's own adapter code and says nothing about the Nav product.
A fail under the first reading would mean: our aided-NCC baseline's
assumptions (known-good GSD, near-perfect nadir with no roll, a same-sensor
same-processing-pipeline reference map) don't survive contact with a real
drone/satellite sensor pair, and the Nav acceptance test in the spec should be
rewritten around a more robust baseline (the SuperPoint/LightGlue/ORB feature
matcher §8 originally named but that was never built) before anyone claims
Nav "works" beyond the Teruel synthetic set.
6. Steps, disk, and time
Step
Path/command
Est. disk
Est. time
1. Inspect UAV-VisLoc Drive folder structure (no download)
manual, or gdown --folder --dry-run equivalent if available
5. Convert one site, visual sanity check (2–3 frames vs. reference crop, as in nav_v0.md)
out/real_nav/<site>/
<50 MB
1–2 hrs
6. Run baseline + eval, both start modes, twice each
commands in §4.3
<10 MB (CSVs + HTML reports)
minutes (NCC is fast; nav_v0.md's 320-frame synthetic run had no reported bottleneck)
7. Write up results in this same doc (append a "Results" section) or a new docs/experiments/2026-09-nav-real-gate-results.md
—
—
1–2 hrs
Total new disk for a one-site pilot: roughly 150–350 MB, once the founder
approves exceeding the 200 MB survey cap. Total elapsed effort: about 1–2 days
of focused work, dominated by the adapter (step 4) and by however long the
licence question (§7) takes to resolve, which is not on our clock.
7. Open questions for the founder
UAV-VisLoc's imagery licence is not stated anywhere found in this survey
(GitHub README, arXiv abstract). Before we download and build a public-facing
number from it, someone should either find an explicit licence in the actual
Drive package (a LICENSE/readme.txt inside the zip, not visible from
outside) or email the authors. This is a "read the fine print" ask, not a
signed-DUA ask like Drone-vs-Bird was for Detect (2026-09-transfer-dut.md
§5) — no registration or agreement is mentioned anywhere, but "not mentioned"
isn't "confirmed clear."
Same open question, one notch more serious, for DenseUAV/UL14 — the
Apache-2.0 badge found on the GitHub repo almost certainly covers the code,
not necessarily the imagery (this is a common and easy-to-miss conflation);
worth a direct look at the Hugging Face dataset card's own licence field
before relying on it.
SUES-200 is explicitly "academic research only" — fine for our own
internal validation, but should not be cited in anything shown to investors
or customers without checking whether that phrase is a EULA clause or just a
README aside.
Do we want to spend the ~1–2 days in §6 now, ahead of other stream work,
or park this plan until the founder has bandwidth to clear the licence
questions above? Nothing here is blocked on Blender, GPU time, or anything
this worktree can't do — the only blocker is the licence read and the
go-ahead to exceed the 200 MB survey cap.
If UAV-VisLoc's licence turns out to be unusable, is DenseUAV/UL14 (open
question 2) an acceptable fallback, or should we instead scope a small,
from-scratch real-flight capture (a DJI consumer drone doing a short
GPS-logged nadir flight over a spot with existing national-orthophoto
coverage) as the "definitely-ours, definitely-licensed" alternative? That is
a materially bigger effort (a physical flight) but sidesteps every licence
question in this document.
Итог для основателя
У Nav никогда не было проверки на настоящих фотографиях. Всё, что описано в
docs/datasets/nav_v0.md, — это синтетика: отрендеренные Blender-кадры поверх
реального рельефа и реального ортофото PNOA, но и «наблюдение», и «эталонная
карта» построены из одного и того же источника, просто за разные годы. Приёмочный
тест Nav из спецификации (§8) — «базовое решение даёт осмысленные ошибки, метрики
стабильны между прогонами» — ни разу не проверялся на кадре, который не сгенерировал
наш собственный рендерер.
Нашёл подходящий набор. UAV-VisLoc (Китай, 6742 кадра с 11 точек, статья 2024
года) — единственный из проверенных наборов, который отдаёт ровно то, что нужно
нашему baseline (baseline_nav.py): высоту и курс на каждый кадр плюс готовую
привязанную к координатам спутниковую карту. Второй кандидат — DenseUAV/UL14
(14 университетских кампусов под Ханчжоу) — слабее по курсу камеры (не подтверждено,
меняется ли он вообще), зато это единственный набор с картой за два разных года на
одну и ту же точку — ровно то же явление «карта устарела», которое мы уже меряем
на Теруэле.
Есть загвоздка с лицензией. Ни для UAV-VisLoc, ни для DenseUAV нигде не нашлось
явно прописанной лицензии именно на снимки (не на код). Это не история про
подписание соглашения, как с Drone-vs-Bird для Detect (там нужно письмо и подпись
основателя) — скорее «нужно заглянуть внутрь архива или написать авторам письмо и
получить подтверждение». Пока это не сделано, публично ссылаться на цифры с этих
наборов рискованно.
План проверки. Не переписывать baseline_nav.py/metrics_nav.py — взять их
как есть и написать конвертер одного участка UAV-VisLoc в наш формат (по образцу
уже существующего конвертера DUT Anti-UAV для Detect). Прогнать те же четыре
варианта, что уже есть в nav_v0.md (с подсказкой GT-старта и без неё, с
выравниванием и без), и сравнить: если ошибка на реальных кадрах в пределах ~25 м
(мягче синтетических 4.7 м, потому что настоящая линза, сжатие JPEG и чужой
спутниковый снимок — это честные усложнения) — считаем, что подход переживает
переход на реальные данные. Если ошибка соскакивает на сотни-тысячи метров, как в
худшем случае синтетического «холодного старта» — это не баг, а настоящий, стоящий
доклада результат: значит, наш упрощённый сопоставитель (не тот, что был в
изначальном плане — SuperPoint/LightGlue/ORB так и не написан) не переживает
встречу с реальным сенсором.
Что нужно от вас. (1) Разрешить скачать один участок UAV-VisLoc — это больше
200 МБ, лимита на этот этап, официальный «сэмпл»-архив уже 2 ГБ. (2) Либо
заглянуть внутрь архива на предмет файла лицензии, либо написать авторам письмо
с вопросом про условия использования снимков — и то же самое для DenseUAV/UL14.
Технической части (конвертер + прогон) — на день-два работы, и она ни от чего не
зависит, кроме этого разрешения.
Synthetic→real transfer on DUT Anti-UAV (E0–E4)
docs/experiments/2026-09-transfer-dut.md
Run 2026-09-04 on branch plan3-transfer: E0 at c401cbc, E1–E4 at e05a131
(identical predict/eval code). Every number below is read from
out/eval/summary.json and the per-experiment out/eval/*/report.json files (both
git-ignored); nothing is typed from memory or rounded up.
Headline: the synthetic-only model does not transfer. Trained on AeroBench
Detect v1 alone it scores mAP50 0.375 on the DUT Anti-UAV test set against a
spec §8 threshold of 0.5 — a clear fail, not a near miss. Using the same synthetic
model as a starting point before fine-tuning on 10 % of DUT looks like it helps box
quality (mAP50-95 +0.034) rather than detection rate (mAP50 +0.0015, i.e. noise) —
but that is one seed per arm with no error bar, so treat it as suggestive, not
established.
1. Setup
Detector
yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime
torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
scripts/predict_coco.py, conf 0.001, imgsz 640 (the standard low-threshold mAP protocol), Ultralytics' default max_det of 300 ‡
Metrics
aerobench eval --merge-classes — the three synthetic classes are merged to one drone class so the synthetic model and DUT's single class are comparable. For E1–E4 the flag is a no-op safety net: the model was already trained single-class and DUT's GT has one category, so there is nothing to merge
Driver
scripts/run_transfer_experiments.py (one entry point for all five runs)
‡ max_det 300 at predict time does not mean 300 boxes per image are scored.
COCOeval's standard maxDets for the summary metrics is 100, so only the
top-100 predictions per image ever count towards mAP. That bites hardest on E1,
whose predictions are diffuse: 963 of the 2 200 test images carry more than 100
boxes at conf 0.001 (see §4). The numbers in this write-up are unaffected — they
are all measured under that same standard protocol, which is the point of using it
— but a reader comparing them against a differently configured evaluator should
know which cap applied.
Synthetic source set (see docs/datasets/detect_v1.md): 5 000 frames, 500 of them
negatives, 6 760 boxes, 40 CC0 Poly Haven panoramas, procedurally modelled drones.
Data attribution
Every real image and label in this write-up comes from a third-party dataset. The
same text is written to assets/real/dut/ATTRIBUTION.md (git-ignored, along with
the data itself) by aerosynth.realdata.dut_antiuav.write_attribution:
Images and annotations: DUT Anti-UAV Detection dataset.
Jie Zhao, Jingshu Zhang, Dongdong Li, Dong Wang. "Vision-based Anti-UAV Detection
and Tracking." IEEE Transactions on Intelligent Transportation Systems, 2022.
https://github.com/wangdongdut/DUT-Anti-UAV
No LICENSE/README found inside the dut_{train,val,test}.zip archives or their
extracted folders (checked via unzip -l on all three archives and a filesystem
search of the extracted train/val/test directories). The paper's companion code
repository (github.com/wangdongdut/DUT-Anti-UAV) does carry a root LICENSE file
that GitHub's API identifies as Apache-2.0, but that repository holds only code,
docs and download links (~6 MB total) — the dataset itself (hundreds of MB per
split) is hosted separately on Google Drive / Baidu Cloud and is not confirmed to
fall under that grant. Treated as: license not stated for the dataset distribution
itself; used here for evaluation only, not redistributed.
Size buckets
Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS):
tiny < 16, small 16–32, medium 32–96, large ≥ 96.
The DUT test set is heavily weighted to big objects, and this shapes how much
any per-bucket number is worth:
bucket
tiny
small
medium
large
total
GT boxes in DUT test
15
385
1 087
758
2 245
The tiny column in every table below is computed over 15 boxes. One object
moves it by several points. Read it as a direction, never as a measurement.
2. Results
Exp
Train data
Init weights
Eval set
mAP50
mAP50-95
tiny
small
medium
large
train time
E0
syn_v1 (4 500 syn imgs)
yolov8n.pt (COCO)
synthetic val (500 imgs)
0.905
0.697
0.706
0.929
0.996
0.999
928.1 s †
E1
— (no training)
E0 best.pt
DUT test (2 200 imgs)
0.375
0.195
0.038
0.201
0.381
0.458
n/a
E2
dut10 (520 real imgs)
E0 best.pt
DUT test (2 200 imgs)
0.761
0.468
0.242
0.744
0.717
0.848
1 550.1 s
E3
dut10 (520 real imgs)
yolov8n.pt (COCO)
DUT test (2 200 imgs)
0.759
0.434
0.156
0.713
0.743
0.824
1 525.8 s
E4
dut_full (5 200 real imgs)
yolov8n.pt (COCO)
DUT test (2 200 imgs)
0.912
0.580
0.535
0.816
0.913
0.968
5 286.2 s
Per-bucket columns are mAP50. The same buckets at mAP50-95:
Exp
tiny
small
medium
large
E0
0.384
0.688
0.867
0.915
E1
0.007
0.043
0.174
0.314
E2
0.024
0.355
0.430
0.591
E3
0.025
0.301
0.405
0.549
E4
0.134
0.373
0.554
0.723
E0 is on a different evaluation set (held-out synthetic val, not DUT). It is a
pipeline sanity check — "does training work at all" — not a comparable score. It
passes its own gate comfortably (0.905 vs the > 0.8 expectation).
That 0.905 is a validation number, not a held-out test number: those 500 images
are out/splits/syn_v1/val.txt, i.e. exactly the set Ultralytics validated against
after every epoch and used to pick which checkpoint became best.pt. Selecting on a
set and then reporting on it is mildly optimistic by construction. It does not
affect anything E0 is used for here — a pipeline sanity check, and weights for E1/E2
— and every real-data number in this write-up comes from the DUT test split,
which no training run ever saw.
† E0's 928.1 s is not the cost of 30 epochs. E0's training was interrupted and
restarted: out/models/syn_v1/results.csv shows the epoch timer resetting at epoch
25, and out/eval/logs/E0.log records Resuming training … from epoch 25 to 30
total epochs, i.e. the run was resumed from last.pt. The driver measures only the
subprocess it launched, so the 928.1 s recorded in summary.json covers the final
resumed segment (epochs 25–30, 6 epochs completed in 0.251 hours in the log). The
full picture from results.csv is 3 510.2 s for epochs 1–24 plus 897.7 s for epochs
25–30 ≈ 4 408 s of training time in total. These are training timers (Ultralytics'
own per-epoch clock), not wall-clock from first command to last: whatever elapsed
between the two segments — the interruption itself — is in neither figure. E2/E3/E4 ran
straight through with no resume, and for them the driver's number and Ultralytics'
own (1 492.7 / 1 466.5 / 5 209.7 s) differ only by model load plus final validation.
One metric in the table means nothing here
report.json reports fp_per_image: 0.000 for E1–E4. That is an artifact, not a
result: it is measured on negative (drone-free) frames, and DUT test contains
zero negative frames (negatives.images: 0 in all four reports). E0's
fp_per_image: 0.000 over 54 synthetic negatives is real; the real-data ones should
be ignored entirely. The false-positive story on DUT has to be told a different way
— see §4.
3. Verdict against spec §8
Gate 1 — E1 ≥ 0.5 mAP50 on real data: FAILED. 0.375 against a 0.5 threshold
(Anti-UAV only — the Drone-vs-Bird half of spec §8's real-data test is pending the
founder's access request, §5).
This is not a rounding problem or a threshold-tuning problem; at mAP50-95 the
synthetic-only model scores 0.195, and on everything below ~32 px long side it is
effectively blind (small 0.201, tiny 0.038). A detector trained purely on AeroBench
Detect v1 is not usable on this real dataset.
Gate 2 — E2 > E3: the strict metric points that way, the loose one does not.
E2 (syn → 10 % real)
E3 (COCO → 10 % real)
Δ (E2 − E3)
mAP50
0.7605
0.7590
+0.0015
mAP50-95
0.4682
0.4340
+0.0342
tiny mAP50
0.2416
0.1559
+0.0856
tiny mAP50-95
0.0242
0.0248
−0.0007
small mAP50-95
0.3548
0.3007
+0.0542
medium mAP50-95
0.4301
0.4053
+0.0249
large mAP50-95
0.5908
0.5485
+0.0423
Read honestly:
The mAP50 difference (+0.0015) is noise. These are two single-seed 20-epoch
runs; a gap of 0.0015 on a 2 245-box test set is not a result. Anyone quoting
"synthetic pre-training raises mAP50" from this table is misreading it.
The mAP50-95 difference (+0.034) is the larger of the two gaps, and it keeps
the same sign in three of four size buckets (+0.025 to +0.054), reversing only in
the 15-box tiny bucket (−0.0007). That is one seed per arm and therefore no
error bar on either number: suggestive, not established. mAP50-95 rewards tight
boxes, so the reading it suggests is that synthetic pre-training improved
localization quality — plausible, since synthetic boxes are derived from
sprite alpha and are pixel-exact — rather than improving what gets found at all.
Confirming it needs several seeds per arm, which this experiment did not run.
The tiny mAP50 gap (+0.086) is the weakest evidence in the table, not the
strongest. Fifteen boxes. It also reverses sign at mAP50-95. It is consistent with
the "helps most where objects are small" story, but it does not establish it.
At an operating threshold of conf 0.25 (greedy matching, IoU 0.5, our own count
over the prediction files) the two models trade off rather than one dominating:
E2 precision 0.757 / recall 0.729; E3 precision 0.830 / recall 0.694. E2 finds
more and is wrong more often.
The ceiling matters more than either. E4 — the same architecture on the full
5 200-image DUT train set, no synthetic data at all — reaches 0.912 / 0.580, i.e.
+0.152 mAP50 and +0.112 mAP50-95 over E2, and +0.293 on tiny mAP50. Ten times
the real labels beats anything the synthetic set contributed. Synthetic
pre-training is not a substitute for real data; on this evidence it is a modest
label-efficiency aid that helps most when labels are scarce and boxes need to be
tight.
4. Where E1 breaks down
By size
E1's mAP50 falls off a cliff as objects shrink: large 0.458 → medium 0.381 →
small 0.201 → tiny 0.038. At mAP50-95 the small bucket is 0.043 and tiny is 0.007
— which means that even where a small object is found, the box is nowhere near
tight. This is the opposite of E0's profile on synthetic val (tiny 0.706, small
0.929), so it is not that the model cannot detect small objects in principle; it
cannot detect these small objects, in real imagery.
Confidence is spread across the frame
At the evaluation threshold (conf 0.001), the synthetic-only model emits
235 584 boxes for 2 245 real objects — 107 per image. The same protocol gives
E4 25 670 boxes (11.7 per image), E2 69 961 and E3 89 529. Low-confidence boxes
are normal and expected under this protocol; a 9× spread between models under an
identical protocol is not. The syn-only model is diffusely uncertain about real
scenes in a way the real-trained models are not.
At a usable operating point (conf 0.25, IoU 0.5, greedy matching) E1 produces
6 297 boxes across the test set: TP 1 187, FP 5 110, FN 1 058 → precision 0.189,
recall 0.529. Roughly four out of five confident detections are wrong, and
roughly half the drones are missed.
Three examples
Green = ground truth, red = E1 prediction at conf ≥ 0.25. Drawn with OpenCV from
out/eval/e1_syn_only/preds.json against assets/real/dut/test_coco.json.
a) Small drone, nothing fires at all — img/00890.jpg
A 23 × 10 px quadcopter against bright sky, framed by trees and a building edge.
Zero predictions clear conf 0.25 anywhere in the frame; the highest-scoring box
in the whole image is 0.19. This is the small/tiny collapse in its plainest
form — not a mislocalization, an absence.
b) The drone is missed, the architecture is detected — img/01986.jpg
Hazy overcast, red-brick campus building. The real drone (green, 25 × 11 px, low
against the facade) draws no confident prediction. Eleven boxes at conf ≥ 0.25 land
on balcony railings, roof structures and — at 0.58 — on a 408 × 301 px slab of
building. Detect v1's backgrounds are HDRI panoramas with drones composited over
them and no distractor objects at all; the model has never had to learn that a
railing is not a multirotor.
c) Near-field drone boxed as fragments — img/01124.jpg
A large white multirotor filling most of the frame: GT is 865 × 305 px. The model's best
box (0.67) covers a 350 × 82 px slice of it — IoU 0.109, so it counts as both a
miss and a false positive. Seven more boxes sit on rotor booms, landing-gear legs
and the payload pod. The model has learned "a drone is a small compact blob" and,
faced with one at close range, finds drone-sized pieces of it instead of the
aircraft.
Hypotheses
Ordered by how much of the gap we think each explains, all of them untested
individually — this experiment separates synthetic-vs-real, not cause-by-cause:
No distractors. Detect v1 has zero birds, aircraft, poles, cables, railings
or rooftop hardware. DUT is shot over real ground clutter. Example (b) is a
direct illustration; the 4:1 false-positive rate at conf 0.25 is the aggregate
version. We consider this the largest single contributor.
Scale and distance range mismatch. Detect v1's boxes are spread across all
four buckets by construction (tiny 1 691 / small 2 353 / medium 2 048 / large
668), but "large" in the synthetic set means a composited sprite, not a drone at
two metres filling the frame. Example (c) shows the model has no representation
of a near-field airframe. In the other direction, real small drones sit in
sensor noise and JPEG blocking that a clean composite never reproduces.
Procedural drone models vs real silhouettes. Detect v1's drones are
procedurally generated, not CAD or photogrammetry: arm/prop geometry, body
proportions, the payload gimbal, landing gear are all approximations. A model
fitted to those shapes has learned a silhouette family that the real commercial
airframes in DUT do not belong to.
Backgrounds. HDRI panoramas are smooth, low-frequency and captured for
lighting, not for surveillance geometry. DUT frames have real camera optics and
depth of field, and high-frequency vegetation and architecture behind the target.
Capture realism. No motion blur, no rolling shutter, no per-frame JPEG
artifacts, no sensor noise in the synthetic set; real footage carries all of
these to some degree, and DUT's frames are JPEGs.
The exposure fix already shipped in v1 (see docs/datasets/detect_v1.md) removed
one obvious domain gap — overbright pasted-on sprites — and E1 is still 0.375. That
is useful negative evidence: photometric matching alone is not the bottleneck.
What changes in v2
In descending expected value:
Distractors as a first-class part of the generator. Birds, fixed-wing
aircraft, helicopters, poles, cranes, cables, rooftop hardware — rendered as
unlabelled objects, plus far more negative frames containing them. Hypothesis 1
predicts this alone should move E1's precision the most.
Real drone geometry. Replace or supplement procedural models with real CAD /
photogrammetric airframes across the classes DUT actually contains.
Explicit near-field coverage. Extend the scale schedule so drones from ~2 m
(filling the frame) to long range are all represented, with correct
perspective — not just sprite scaling.
Capture-realism augmentation at build time. Motion blur along the drone's own
velocity vector, rolling-shutter skew, per-frame JPEG requantization, sensor
noise matched to a real camera's ISO curve.
Real-photo backgrounds alongside HDRIs, for the high-frequency ground clutter
panoramas do not provide.
The measurement to re-run after each of those is exactly E1 (mAP50 on DUT test,
zero real training) with E2/E3 as the label-efficiency check. E1 is the number that
tells us whether the synthetic set is getting closer to reality; E2 − E3 tells us
whether it is worth pre-training on.
Built and measured:2026-09-detect-v2.md — items 1,
3 and 4 of the list above; items 2 and 5 were not done. E1v2 came out at 0.159,
i.e. worse than the 0.375 below, so v1 remains the recommended synthetic set. An
ablation there points at item 1 (the distractors) as the largest suspect, but the
rebuild that removed the size-matched clutter recovered almost none of the loss,
so that write-up's §5 marks the mechanism as unexplained. Nothing in the present
write-up is restated or revised there; its numbers stand as measured.
5. Drone-vs-Bird: founder action required
The obvious next real-data benchmark is the WOSDETC Drone-vs-Bird Detection
Challenge dataset, because it contains exactly the distractor class (birds) that
hypothesis 1 says we are missing. It cannot be downloaded — access requires a
signed data usage agreement, and the founder has to request it personally.
Verified on 2026-09-04: the organisers' repository at
https://github.com/wosdetc/challenge states that a request should be sent to
wosdetc@googlegroups.com, that the requester "will be asked to sign a data usage
agreement", and that the data may then be used for research purposes. The challenge
homepage https://wosdetc2024.wordpress.com/ also resolves (WOSDETC Drone-vs-Bird
Detection Grand Challenge @ MLSP24).
The URL named in the original plan,
https://wosdetc2024.wordpress.com/drone-vs-bird-detection-challenge/, returns
HTTP 404 as of 2026-09-04 — the two links above are the verified replacements.
Nothing in that process can be automated on our side: it needs a named person, an
institution, and a signature.
6. Reproducing this
Everything below assumes the repo root and the project venv with the training extra
installed (uv pip install -e ".[dev,train]" — gdown and ultralytics live
there; see the README for the torch/CUDA install order, which matters).
6.1 Fetch and convert DUT Anti-UAV
assets/real/dut/ is git-ignored and holds ~10 000 images across the three splits.
There is no CLI wrapper for this step; it is three library calls per split, run from
a Python shell at the repo root:
from pathlib import Path
from aerosynth.realdata import dut_antiuav
cache = Path("assets/real/dut")
for split in ("train", "val", "test"):
split_dir = dut_antiuav.download_split(split, cache) # gdown; no-op if already fetched
pairs = dut_antiuav.find_pairs(split_dir) # (image, VOC xml) matched by stem
dut_antiuav.to_coco(pairs, cache / f"{split}_coco.json",
source_tag=f"dut_antiuav_{split}") # tags["source"] per split
dut_antiuav.to_yolo(pairs, cache / f"{split}_yolo") # labels/*.txt + images.txt
dut_antiuav.write_attribution(cache, None)
That produces assets/real/dut/{train,val,test}_coco.json (the GT the evaluations
read) and assets/real/dut/{train,val,test}_yolo/ (labels plus an images.txt of
absolute image paths — no symlinks, which need privileges on Windows). The explicit
per-split source_tag is what makes tags["source"] read dut_antiuav_test rather
than a bare default, so merged splits stay distinguishable.
6.2 Build the three training splits
Deterministic given --seed 0; reproduces the image counts in §1's Splits table
(4 500 / 500, 520 and 5 200 train images, all real runs validating on the same
2 600-image DUT val set):
--single-class on the synthetic split rewrites Detect v1's three classes to one
drone class; DUT's own labels are already single-class. --val-frac 0 plus
--val-list is what makes the two real splits hold nothing back from their own pool
and validate against the independent DUT val split instead.
6.3 Run the experiments and rebuild the page
python scripts/run_transfer_experiments.py # all five
python scripts/run_transfer_experiments.py --only E1 # just the headline number
python scripts/build_site.py # local results page (site/, git-ignored)
Budget roughly 3.5 h of training on an RTX 3070 Ti (the times in §2 sum to
~12 770 s once E0's full 30 epochs are counted), plus inference over 2 200 images
four times.
Outputs land in out/models/** and out/eval/**, both git-ignored: the model
weights, prediction files and reports are not committed, only this write-up and its
three example images. The driver writes one row per experiment to
out/eval/summary.json, replacing that experiment's earlier row on a re-run. A
summary produced before that behaviour existed can still hold duplicates — the
current file has E1 twice, identically — so a reader should keep the last row per
experiment.
Итог для основателя
Главное: модель, обученная только на синтетике, на реальных данных не работает.
На тесте DUT Anti-UAV она даёт mAP50 = 0.375 при пороге 0.5 из спецификации — это
не «чуть-чуть не дотянули», а провал с запасом. На объектах мельче ~32 пикселей она
практически слепа (0.201 и ниже), а при рабочем пороге уверенности примерно четыре
из пяти её срабатываний — ложные: она принимает за дроны перила балконов, крыши и
куски зданий, потому что в нашем синтетическом наборе нет ни одного отвлекающего
объекта.
Синтетика, похоже, не бесполезна, но пользы меньше, чем хотелось. Если взять эту
модель как стартовую точку и дообучить на 10 % реальных данных (E2), она обгоняет
такую же модель без синтетики (E3) на строгой метрике mAP50-95: 0.468 против 0.434.
Выигрыш небольшой, и на каждую сторону у нас всего один прогон без доверительного
интервала: похоже, что синтетика улучшает точность рамок, а не способность находить
дроны, — но это указание, а не доказанный факт. По мягкой метрике mAP50 разница
0.7605 против 0.7590, то есть шум; выдавать её за результат нельзя. По «крошечным»
объектам разрыв выглядит крупным (0.242 против 0.156), но он посчитан всего по 15
рамкам во всём тесте — это направление, а не измерение.
Реальные данные бьют всё. Та же архитектура на полном реальном наборе (E4, в 10
раз больше размеченных кадров) даёт 0.912 / 0.580 — на 0.152 mAP50 выше, чем
синтетика плюс 10 % реальных. Вывод трезвый: синтетический набор — это не замена
разметке, а инструмент экономии разметки, и пока экономия скромная.
Что делать дальше. Главная гипотеза — в наборе нет отвлекающих объектов
(птиц, самолётов, столбов, перил) и нет дронов на близкой дистанции. Это и есть
план v2 в порядке ожидаемой отдачи: отвлекающие объекты и негативные кадры →
реальная геометрия дронов вместо процедурной → близкая дистанция → смаз, шум и
артефакты сжатия → реальные фотофоны. Проверять после каждого шага одним и тем же
числом — E1 на DUT.
Требуется ваше действие. Набор Drone-vs-Bird (тот самый, где есть птицы)
скачать нельзя: нужно письмо на wosdetc@googlegroups.com и подписанное
соглашение об использовании данных от вашего имени — см.
https://github.com/wosdetc/challenge. Автоматизировать это невозможно.
