AeroBench

AeroBench Detect v0 (composite fast path)

docs/datasets/detect_v0.md

• Frames: 5000, negatives: 500, boxes (annotations): 6760
• Per class: multirotor_small 2237, multirotor_medium 2249, fixed_wing 2274
• Per size: tiny 1691, small 2353, medium 2048, large 668
• Per time of day: dusk 1643, day 1596, night 1761; per weather: clear 2846, haze 1181, rain 479, fog 494; backlit: 557
• Panoramas: 40 Poly Haven HDRIs (CC0), see assets/hdris/catalog.json for the id/source list (ATTRIBUTION.md covers full attribution for the dataset)
• Generation: aerosynth build-detect configs/detect_v0.yaml out/detect_v0 (seed 20260903, git cccfa76a6d52e2f5b64a9be8071e6be552b538e1), wall time 6m46s, failed_frames: 0
• Known limitations: procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video); sprites in v0 were rendered before exposure matching (world strength 1.0); drones are brighter than their background by a factor ~2–5 — fixed in code, v1 will re-render
• Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image; size slices use the long-side buckets (matching annotate.size_bucket), not raw w*h area; repeat build (--limit 300) -> identical annotations for the shared image ids

AeroBench Detect v1 (exposure-matched sprites)

docs/datasets/detect_v1.md

• Frames: 5000, negatives: 500, boxes (annotations): 6760
• Per class: multirotor_small 2237, multirotor_medium 2249, fixed_wing 2274
• Per size: tiny 1691, small 2353, medium 2048, large 668
• Per time of day: dusk 1643, day 1596, night 1761; per weather: clear 2846, haze 1181, rain 479, fog 494; backlit: 557
• Panoramas: same 40 Poly Haven HDRIs (CC0) as v0 (configs/detect_v1.yaml pins the identical hdri_ids list); see assets/hdris/catalog.json for the id/source list (ATTRIBUTION.md in the dataset covers full attribution)
• Sprites: assets/sprites_v1, rendered with aerosynth render-sprites assets/hdris assets/sprites_v1 --per-hdri 30 --seed 1 --size 256 --samples 32 (40 HDRIs x 30 sprites); --strength is passed per HDRI as hdri.pano_exposure(hdr), the same exposure the background panorama is tonemapped to (see "What changed" below)
• Generation: aerosynth build-detect configs/detect_v1.yaml out/detect_v1 --sprites assets/sprites_v1 (seed 20260903, git 7e2d99c372b51f4dd32a27d5b7af56f87aa5bc21), wall time 7m47s, failed_frames: 0
• Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image; size slices use the long-side buckets (matching annotate.size_bucket), not raw w*h area; repeat build (--limit 300) -> identical annotations for the shared image ids; per-class/size/time-of-day/weather/backlit counts and all 1508 box pixel coordinates in the first 1000 positive images are bit-identical to v0 (v0 and v1 replay the exact same seeded job list; only sprite RGB brightness differs, never the alpha silhouette that boxes are derived from)
• Known limitations: procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video)

What changed vs v0

v0's sprites (assets/sprites) were rendered at a fixed Blender world Strength of 1.0, independent of each HDRI's own exposure. The background panorama, however, is tonemapped by load_pano_ldr to a fixed target of 0.18 mean luminance (hdri.pano_exposure, commit 676c6ee) — for these 40 HDRIs that means scaling the raw HDR by 0.19x-2.72x (median 0.34x) before compositing. v0 lit its drones at the raw, un-scaled strength, so most were rendered several times brighter than the exposure their background was tonemapped to. render-sprites now passes --strength <pano_exposure(hdr)> per HDRI, so sprite lighting matches the background's tonemap exactly. assets/sprites_v1 is a full re-render with this fix: same seed (1), same 40 HDRIs, same per-HDRI count (30); the drone silhouettes (alpha, hence bounding boxes) are unaffected, only RGB brightness changes.

Exposure and mean opaque-pixel brightness (grayscale, alpha>0 pixels only, averaged over each HDRI's 30 sprites), v0 vs v1, all 40 HDRIs:

HDRI	exposure	v0 opaque-mean	v1 opaque-mean	v1/v0	sprites.json sha256 (v1, 12c)
aarfontein_dirt_road	0.255	112.3	60.0	0.53	583130adbeb3
aarfontein_dusk	0.272	115.8	61.8	0.53	f77f07b0fa3b
abandoned_church	0.416	88.1	57.0	0.65	0ee8cdd8d0df
abandoned_hopper_terminal_01	0.338	112.7	66.9	0.59	fd37e4e87bb8
abandoned_hopper_terminal_02	0.397	101.2	64.7	0.64	2d792230cfc4
abandoned_hopper_terminal_03	0.274	118.9	64.0	0.54	0106229612c9
abandoned_hopper_terminal_04	0.284	121.9	67.6	0.55	d01d9f852fc3
abandoned_parking	0.296	113.8	63.2	0.56	0deb72c780f0
abandoned_pathway	0.478	84.4	59.0	0.70	4f1965764a09
abandoned_slipway	0.350	112.2	67.6	0.60	4d28a53ec78a
abandoned_tank_farm_01	0.420	91.1	59.4	0.65	22e39f674ca1
abandoned_tank_farm_02	0.437	90.0	59.9	0.67	c0569dec6e2f
abandoned_tank_farm_03	0.339	113.8	68.2	0.60	a2851b3e88f9
abandoned_tank_farm_04	0.392	93.5	59.3	0.63	9f9ddfa9f35d
abandoned_tank_farm_05	0.302	124.6	70.5	0.57	ee9dca80dcd6
altanka	0.263	128.0	68.9	0.54	2634f94a808f
arboretum	0.335	115.2	68.2	0.59	f3775e4e9b88
autumn_forest_04	0.262	114.8	61.8	0.54	ee9f0dab10ca
aviation_museum	0.249	130.7	68.3	0.52	9e19caab7324
bambanani_sunset	0.275	111.6	59.8	0.54	e7fa0d57dd48
beach_parking	0.500	86.9	61.7	0.71	ac004c4056a8
belfast_sunset	0.277	106.7	57.1	0.54	3baac07acc34
belfast_sunset_puresky	0.192	136.4	62.8	0.46	d4bfddbbe134
bell_park_dawn	0.477	98.6	68.9	0.70	cb76cd6b2535
bell_park_pier	0.435	97.4	65.0	0.67	b5ee0375ea54
between_bridges	0.209	145.1	73.9	0.51	a0ecb1a6542c
blaubeuren_church_square	0.574	93.2	71.2	0.76	77ac68eb864e
blaubeuren_night	0.464	101.3	70.5	0.70	25799956eed7
blue_lagoon_night	0.286	128.2	71.2	0.56	3bc8503d7417
clarens_night_01	0.285	116.6	65.0	0.56	dcb667ffe565
clarens_night_02	0.288	100.5	54.7	0.54	68dfbb511e74
cobblestone_street_night	0.318	112.1	66.3	0.59	dba6cee8fbd2
courtyard_night	0.905	67.8	64.5	0.95	3517e8304e49
dikhololo_night	2.722	44.6	74.7	1.67	1c2b388c0458
donkin_reserve	0.246	129.1	67.0	0.52	97fdefa65e77
golden_bay	0.348	111.3	67.5	0.61	717a3e58fe1b
hansaplatz	0.279	120.4	66.3	0.55	b734086d41cc
kloppenheim_02	0.775	76.5	67.5	0.88	1311e14ab2ad
kloppenheim_02_puresky	0.788	79.3	70.7	0.89	aac319021b1d
rooftop_night	0.401	97.6	62.6	0.64	2699a3e7cc00

40/40 HDRIs, mean opaque-pixel brightness (0-255) averaged across all 30 sprites/HDRI: v0 106.1 -> v1 65.1 overall (per-HDRI v1/v0 ratio mean 0.64, range 0.46-1.67). The three HDRIs spot-checked at render time (rooftop_night, abandoned_church, golden_bay) land at 88-111 (v0) -> 57-68 (v1), matching exposures 0.35-0.42 as expected. Not every HDRI gets darker: courtyard_night (exposure 0.91) and dikhololo_night (exposure 2.72, the one HDRI in this set whose raw panorama is under-exposed) end up roughly the same or brighter in v1 — the fix follows each panorama's own exposure rather than uniformly dimming every sprite.

Brightness-consistency check (scripts/brightness_check.py)

For the first 1000 positive images (by image id), for every box: mean grayscale brightness inside the box vs. an 8px ring immediately outside it (clipped to the frame edges).

$ python scripts/brightness_check.py out/detect_v0
images: 1000
boxes: 1508
median_ratio: 0.984
share_ratio_gt_2: 0.036

$ python scripts/brightness_check.py out/detect_v1
images: 1000
boxes: 1508
median_ratio: 0.907
share_ratio_gt_2: 0.019

Same 1000 images and the same 1508 boxes at the same pixel coordinates (see the self-check bullet above), so this is a paired before/after comparison, not two independent samples. The share of egregiously-overbright boxes (inside/ring > 2) roughly halves, 3.6% -> 1.9% — that is the tail the root-cause commit (676c6ee) describes, and it moves the direction the fix predicts.

The median moving from 0.984 to 0.907 is not closer to 1, and is worth calling out rather than glossing over: a box's rectangle is mostly transparent alpha for a multirotor's spread-out arms/props, so a whole-box mean is diluted by however much background shows through inside that rectangle, in both v0 and v1 — it mixes drone-pixel and background-pixel brightness rather than isolating the drone. That dilutes the metric everywhere, so the aggregate median is a blunt read on a fix that mainly removes an overbright peak (see the per-box example below), not a uniform brightness shift. scripts/brightness_check.py is left as specified (whole box vs. ring, no alpha) because the composited JPEGs it reads don't carry an alpha channel to isolate opaque pixels with — the raw-sprite table above is the cleaner signal for the underlying fix; this check is the closest thing derivable from the shipped dataset files alone, and it does move in the right direction on the metric that most directly targets "egregiously bright."

Per-box detail for the two examples used in the visual check below (both [x,y,w,h] in pixels, ratio = inside/ring, max = brightest pixel in the box):

image	box	v0 inside/ring/ratio/max	v1 inside/ring/ratio/max
000006.jpg (quadcopter, clarens_night_02)	[74,66,86,41]	107.1 / 99.3 / 1.08 / 241	88.6 / 99.3 / 0.89 / 146
000016.jpg (fixed-wing, golden_bay)	[181,285,68,62]	86.4 / 74.1 / 1.17 / 191	71.2 / 74.1 / 0.96 / 132

Both boxes cross from ratio > 1 (v0, drone brighter than its ring) to ratio < 1 (v1); the brightest pixel in each box (the part most likely to look like an overbright artifact) drops by 95 and 59 (out of 255) respectively.

Visual check

Read on out/detect_v0/images/000006.jpg vs out/detect_v1/images/000006.jpg (same box, a quadcopter over a rain-streaked dusk scene, clarens_night_02): in v0 the hub is a distinct, near-white glowing patch that reads as pasted onto the muted, rain-hazed backdrop; in v1 the same hub is a soft pale blue-grey that sits inside the scene's tonal range — still the lightest part of the drone, but no longer an obvious sprite seam. 000016.jpg (golden_bay, dusk, fixed-wing): v0's fuselage is a saturated, almost glowing orange that pops out of the dim building silhouette behind it; v1's is a darker, more brownish orange that recedes into the same silhouette. 000000.jpg (daytime, two boxes, both small/thin drones): visually near-identical between v0 and v1 — daytime HDRIs in this catalog sit closer to pano_exposure ~1.0 (less scaling needed), so the fix has little to correct there, consistent with the inside-mean deltas being only 3-6 (daytime) vs 15-19 (the two night/dusk examples above).

Remaining limitations

Procedural drones (no real CAD models), no bird/aircraft distractors, RGB only, single frames (no video). The whole-box brightness-consistency check above is a blunt instrument for sparse-silhouette boxes (transparent gaps inside the rectangle dilute the mean toward background) — it understates the fix's effect on the aggregate median while still correctly showing the overbright tail shrink; the raw-sprite opaque-pixel table is the more direct evidence for the underlying exposure fix.

AeroBench Detect v2 (distractors, near field, capture realism)

docs/datasets/detect_v2.md

• Frames: 5000, negatives: 1000, boxes (annotations): 5972
• Per class: multirotor_small 1940, multirotor_medium 1979, fixed_wing 2053
• Per size: tiny 1345, small 1742, medium 1566, large 1319
• Per time of day: day 1629, night 1717, dusk 1654; per weather: clear 2863, haze 1180, rain 460, fog 497; backlit: 522
• New in v2: frames_with_distractors 2876 (0.575 of frames), negatives_with_distractors 841 (0.841 of the 1000 negatives), near_field_boxes 615 (long side >= 200 px), boxes_touching_border 653
• Panoramas: the same 40 Poly Haven HDRIs (CC0) as v0/v1 — configs/detect_v2.yaml pins a byte-identical hdri_ids list; see assets/hdris/catalog.json for the id/source list (ATTRIBUTION.md in the dataset covers full attribution)
• Sprites (drones): assets/sprites_v2 is a straight copy of assets/sprites_v1 (40 HDRIs x 30 sprites). The drone sprites were not re-rendered, so "we added clutter" is not mixed with "we changed the drones": all 40/40 sprites.json files match the sprites[].sha256 recorded in out/detect_v1/manifest.json (re-verified after the eXIf strip described under "Sprites" below)
• Sprites (new families): 40 x 24 distractor renders = 960 and 40 x 9 near-field drones = 360, in assets/sprites_v2/<hdri_id>/{distractors,nearfield}/, pinned in out/detect_v2/manifest.json as 80 extras records (hdri_id, kind, count, sha256 of <kind>.json). 800 of the 960 distractor renders are used: the 160 helicopter sprites stay on disk and are skipped by sprites.load_extra_index, see "What changed vs v2-rc"
• Generation: aerosynth build-detect configs/detect_v2.yaml out/detect_v2 --sprites assets/sprites_v2 (seed 20260904, git 6cf56d0cfee1f10cb53eee1a87f0d186a0cece35, Blender 4.2.16 LTS), wall time 9m19s (559 s), failed_frames: 0
• Self-check: perfect predictions -> mAP50 = 1.000, mAP50-95 = 1.000, all slices (size/time-of-day/weather/backlit) 1.000, false positives 0.000/image on 1000 negatives; repeat build (--limit 300) -> identical annotations for the shared image ids; scripts/detect_qa.py hard checks 0 problems and 0 coverage warnings; boxes in frames that carry distractors = drones planned for those frames, exactly (3023 = 3023)
• Licence: CC BY-NC 4.0 (as v0/v1). All artifacts (assets/sprites_v2/**, out/detect_v2/**) are git-ignored; only this card and the five QA images below are committed
• Known limitations: procedural drones (no real CAD models), HDRI panoramas rather than real photographs as backgrounds, RGB only, single frames (no video); the motion-blur smear is inside the box (see "What is still wrong")

For reference, v1 on the same 40 panoramas: 5000 frames, 500 negatives, 6760 boxes, zero distractors, no near field, wall time 7m47s.

History: this is the second v2 build

The first v2 build (2026-09-04, commit 09d522d) shipped with two defects, both found by review after it had already been trained on:

• C1 — displaced boxes. render_frame applied the rolling-shutter skew to the pixels after paste had measured the boxes, so on the 25 % of frames that carry a skew every annotation sat up to rolling_shutter_px * y / H to the left of its drone: 636 of that build's 5972 boxes (10.7 %) had IoU < 0.9 with their own pixels, 51 below 0.5.
• C2 — clutter that dwarfed the targets. distractor_px: [12, 260], drawn uniformly, put 3824 unlabelled objects at >= 96 px against 1318 labelled large drones — 2.9 unlabelled large objects per labelled one.

That build's transfer numbers (E1v2 mAP50 0.154 and the ablation arms) are recorded in docs/experiments/2026-09-detect-v2.md as the v2-rc result and are kept there unchanged: they are what that data did, and §5 of that write-up is what identified C2. This card describes the rebuild at 6cf56d0, which fixes both. The rebuilt set has since been trained and evaluated, and it did not help: E1v2 mAP50 0.159 against v2-rc's 0.154 and v1's 0.375. The card's claim is therefore only that two measured data defects are gone — not that the data is better. v1 remains the recommended set; see that write-up's §3 and §5.

What changed vs v2-rc

	v2-rc (09d522d)	v2 (6cf56d0)
box vs pixels	636/5972 at IoU < 0.9, 51 < 0.5	5893/5972 at IoU >= 0.9, 0 < 0.5 (below)
distractor_px	[12, 260], median 136	[10, 96], median 53, max 95
clutter instances >= 96 px	3824	0
distractor kinds	6	5 (helicopter dropped)
distractor instances	5787	5787 (unchanged count)
boxes / frames / negatives	5972 / 5000 / 1000	5972 / 5000 / 1000

The job stream is otherwise the same: the two builds plan the same 5972 targets in the same frames, so the difference is exactly "the clutter got smaller, the helicopters left, and the boxes now follow the skew".

Why the clutter is capped at 96 px. annotate.SIZE_BUCKETS["large"] starts at 96 px, and distractor_px: [10, 96) is now strictly below it, so no unlabelled object lands in the same size bucket as a labelled large drone (0 instances vs 1319 labelled large boxes, against 3824 vs 1318 in v2-rc). This is v3 hypothesis 1 from the experiment write-up made real — and E1 has now falsified it as the main cause: with zero size-matched clutter, large mAP50 recovered only 0.064 → 0.118 against v1's 0.458. The clutter cap is still the right thing to do; it is not what was breaking the large bucket.

Why helicopter was dropped. Its main-rotor disc is rendered with the same translucent-disc trick as the drone propellers, so under --merge-classes it is an unlabelled object built out of the labelled class's most distinctive feature. Its sprites are still on disk (160 of the 960 distractor renders); they are filtered out by sprites.load_extra_index, which skips any distractor record whose kind is no longer in scenario.DISTRACTOR_KINDS.

What changed vs v1

v2 turns three of the five failure hypotheses from docs/experiments/2026-09-transfer-dut.md §4 into data. Hypotheses 3 (real CAD airframes) and 4 (real photographs as backgrounds) are not addressed — see the last section. annotate.SIZE_BUCKETS was deliberately left untouched, so the v1 and v2 size slices stay comparable.

1. Unlabelled distractors (hypothesis 1 — the big one)

A Blender sprite family rendered by render_extras.py --kind distractors with the same procedural style, the same per-HDRI seed and the same per-HDRI world strength (hdri.pano_exposure) as the drones. Five kinds, cycled so each is guaranteed to appear:

kind	what it is	why it is here
bird_soaring	elongated dark body, near-horizontal wings	the classic sky false positive
bird_flapping	same body, wings raised 20-55 degrees	a second bird silhouette, not one pose
airliner_far	fuselage + swept wing + fin + two nacelles, long lens at 8-20x distance	"aircraft far away", named in §4
mast_with_cables	vertical pole, 2-4 cross arms, 2-5 thin guy cables, camera below the horizon	thin man-made structure
rooftop_hardware	2-4 AC boxes, 1-2 antennas, a parapet slab	exactly what E1 fired on at conf 0.25

They are composited before the drones and are never annotated, so a drone is never occluded by an unlabelled object and its alpha-derived box stays exact. Config knobs: distractor_prob: 0.50 (half of the positive frames), negative_distractor_prob: 0.85 (almost every negative), distractors_per_frame: [1, 3], distractor_px: [10, 96]. Expected share of frames with clutter: 0.20*0.85 + 0.80*0.50 = 0.57; measured 0.575 (2876/5000). Negatives are up from 10 % in v1 to 20 % (negative_fraction: 0.20), and 841 of the 1000 carry clutter (target 0.85).

5787 distractor instances were composited in total: airliner_far 1179, bird_flapping 1174, bird_soaring 1119, mast_with_cables 1136, rooftop_hardware 1179 (counted from the planned job list; the per-frame tags["distractors"] in annotations.json sum to the same 5787, and distractors_missing is 0, so every planned instance found a sprite). Their planned long side is min 10 / median 53 / max 95 px.

2. Near field (part of hypothesis 2)

scenario.SIZE_RANGES_PX gained a fifth bucket near: (200, 1100) — from "fills a third of the frame" up to "wider than the 640 px frame", so the sprite is genuinely clipped by the frame border. size_mix gives it 0.10; 621 of the 5972 targets (0.104) were planned as near.

These are not upscaled v1 sprites. render_extras.py --kind nearfield re-renders the same procedural drones (build_multirotor / build_fixed_wing, class cycled CLASSES[k % 3]) at 768 px with a close camera (dist_mult=(1.05, 1.5)) and a wide lens (24-35 mm), i.e. real near-field perspective. render_frame prefers this pool whenever size_bucket == "near".

Rendered result: 615 boxes with a long side >= 200 px (near_field_boxes), long side min 200 / median 492 / max 640 px. 8 planned near targets came out under 200 px, cut down by the frame border, so 613 of the 615 come from the near bucket; the other two are large targets planned at exactly 199 px whose box crossed 200 px when the rolling shutter sheared it (1.2 px of skew, one pixel of width). 653 boxes touch a frame edge, and they come from where you would expect: 512 near, 111 large, 30 medium. Note that annotate.SIZE_BUCKETS still ends at large: (96, inf), so per_size.large (1319) contains both the v1-style large drones and the near-field ones; near_field_boxes is the separate key.

In v1 this whole slice was empty, and §4 of the plan-3 report showed a real DUT near-field drone (GT 865 x 305 px) breaking into pieces (best box IoU 0.109).

3. Capture realism (hypothesis 5)

Applied per frame, after compositing, in this order: atmosphere -> rolling shutter -> sensor noise -> shot noise -> JPEG requantisation. The rolling shutter also shears the boxes (camera. rolling_shutter_bbox), which is the C1 fix.

effect	config	measured on this build
motion blur along the drone's own velocity vector	motion_blur_velocity: true, blur_prob: 0.40	all 5972 targets carry a vel_deg (v1 drew a fresh random angle inside the compositor instead); blur is actually applied on the 1985 frames (0.397) where blur_px > 0, min 1.00 / median 2.47 / max 4.00 px
Gaussian sensor noise, sigma drawn per frame	noise_sigma_range: [2.0, 9.0]	all 5000 frames, sigma min 2.00 / median 5.42 / max 9.00 (v1 used a fixed 3.0 day / 6.0 night)
signal-dependent shot noise	shot_noise_k: 6.0	all 5000 frames
rolling-shutter skew (pixels and boxes)	rolling_shutter_prob: 0.25, rolling_shutter_px: [1.0, 6.0]	1258 frames (0.252), shift min 1.00 / median 3.50 / max 6.00 px, carrying 1496 boxes
in-pipeline JPEG requantisation	jpeg_quality_range: [55, 95]	all 5000 frames: 50-59 598, 60-69 1237, 70-79 1245, 80-89 1197, 90-99 723

The in-pipeline requantisation is deliberately a second compression: the file is still written at quality 92 by build_detect, so every frame goes through two JPEG passes, like a real frame recompressed on export.

Sprites

family	per HDRI	total	command	render wall time
drones (v1, reused)	30	1200	aerosynth render-sprites assets/hdris assets/sprites_v1 --per-hdri 30 --seed 1 --size 256 --samples 32 (v1; copied, not re-run)	— (copy only)
distractors	24	960 (800 used)	aerosynth render-extras assets/hdris assets/sprites_v2 --kind distractors --per-hdri 24 --seed 5 --size 256 --samples 32	19m18s (1158 s)
near field	9	360	aerosynth render-extras assets/hdris assets/sprites_v2 --kind nearfield --per-hdri 9 --seed 6 --size 768 --samples 32	15m46s (946 s)

The copy was cp -r assets/sprites_v1 assets/sprites_v2 (1200 PNGs + 40 sprites.json), then the two render-extras runs wrote the new families beside the drone sprites. 40/40 HDRIs ok, zero FAIL, on both families; total extra-sprite render time 35m04s (2104 s) on the founder's RTX 3070 Ti. The sprites were not re-rendered for this build — the helicopter change is a filter at index time, not a re-render, so the two builds composite bit-identical sprite pixels for every kind they share.

Indexed for this build: 800 distractor sprites (160 per kind x 5) and 360 near-field sprites. The 160 helicopter renders remain on disk and in distractors.json (which is what the manifest's count: 24 and its sha256 cover), and are skipped when the index is built.

One byte-level change to the PNGs. Blender 4.2 writes two identical eXIf chunks into every PNG it renders, so cv2.imread printed libpng warning: eXIf: duplicate on each of the ~12000 sprite reads a build does. sprites.strip_duplicate_exif was run once over all 2520 sprite PNGs (and both render-* commands now do it automatically for what they produce): the repeated chunk is dropped, the decoded pixels are unchanged (verified on a random sample of 12), and the sprites.json sha256 in the v1 manifest still matches 40/40, because the manifest hashes the JSON, not the PNG bytes. The build log for this dataset contains zero libpng warnings.

Build and self-checks

$ aerosynth build-detect configs/detect_v2.yaml out/detect_v2 --sprites assets/sprites_v2
frames: 5000
negatives: 1000
annotations: 5972
per_size: {'tiny': 1345, 'small': 1742, 'medium': 1566, 'large': 1319}
per_time_of_day: {'day': 1629, 'night': 1717, 'dusk': 1654}  per_weather: {'clear': 2863, 'rain': 460, 'haze': 1180, 'fog': 497}
frames_with_distractors: 2876
negatives_with_distractors: 841
near_field_boxes: 615
boxes_touching_border: 653
per_jpeg_quality: {'50-59': 598, '60-69': 1237, '70-79': 1245, '80-89': 1197, '90-99': 723}
# 559 s wall, manifest failed_frames: 0, no WARN line

Boxes vs pixels (the C1 check). For every one of the 5972 boxes, the target's sprite alpha was placed on an empty frame with placement arithmetic written independently of composite.py, sheared by that frame's own rolling shutter, thresholded at the same alpha > 8, and its tight box compared with the shipped annotation:

boxes checked: 5972  (frames whose box count did not replay: 0)
min IoU: 0.7778   mean IoU: 0.9928
boxes with IoU < 0.9: 79   < 0.5: 0
low-IoU boxes by size bucket: {'tiny': 79}
self-check, unskewed frames: 4476/4476 boxes identical to the pixels
counterfactual on the 1496 boxes of skewed frames: the pre-fix (unsheared) box
  would score IoU < 0.9 on 625 of them

Read that in order. The self-check line is what makes the rest trustworthy: on the 4476 boxes of frames with no skew, the independent re-derivation lands on exactly the same box, to the pixel, so it is measuring the compositor and not itself. 5008 of the 5972 boxes are pixel-identical, 5893 (98.7 %) are at IoU >= 0.9, and none is below 0.5 (v2-rc: 636 below 0.9, 51 below 0.5). The 79 exceptions are all 6-8 px boxes (worst 0.778): the shear is a closed form on integer pixel coordinates, so on a box a handful of pixels wide a sub-pixel displacement rounds to a one-pixel difference against the interpolated pixels, and one pixel on a 7 px box is 0.12 of IoU. That residual is a property of representing an interpolated shear with an integer box, not a displacement — the v2-rc defect was the box sitting up to 6 px away from its drone.

Dataset QA (hard checks).

$ python scripts/detect_qa.py out/detect_v2 --sample 12 --img-out docs/datasets/img
image_size: (640, 640)
...
negative_share: 0.200
frames_with_distractors: 2876 (0.575 of frames)
negatives_with_distractors: 841 (0.841 of negatives)
near_field_boxes: 615 (>= 200 px on the long side)
boxes_touching_border: 653
zero_area_boxes: 0
hard checks: 0 problem(s)
OK

Exit code 0, and no check_coverage warning at all. The hard checks that would have returned a non-zero exit code: no box with w<=0 or h<=0, no box outside the frame, COCO and YOLO labels agree row by row in both directions, every file_name exists, every annotation points at an existing image. The coverage warnings are now measured against this build's own config, read from manifest.json (negative fraction, whether the near bucket was planned), so silence here means "it delivered what it asked for", not "the check was skipped".

Perfect predictions. annotations.json fed back in as COCO results with score 1.0 ([{"image_id", "category_id", "bbox", "score": 1.0}, ...], one record per annotation):

$ aerobench eval --track detect --gt out/detect_v2/annotations.json \
    --pred out/detect_v2/perfect_preds.json --out out/detect_v2/report_perfect
mAP50=1.000 mAP50-95=1.000

out/detect_v2/report_perfect/report.json: overall mAP50 1.000 / mAP50-95 1.000; every size slice (tiny/small/medium/large) 1.000; every time-of-day, weather and backlit slice 1.000; negatives {"images": 1000, "false_positives": 0, "fp_per_image": 0.0}; counts images 5000, annotations 5972, predictions 5972.

Determinism.

$ aerosynth build-detect configs/detect_v2.yaml out/detect_v2_rep --sprites assets/sprites_v2 --limit 300

Comparing the two annotations.json on the 300 shared image ids (images + annotations + categories, json.dumps(..., sort_keys=True)): same annotations: True, sha256 315f0e54062a8e24... on both sides. This covers the new JPEG requantisation, both noise terms and the box shear, since all of them derive from the frame's seeded RNG.

Unlabelledness (the central invariant). Distractors must never produce a box. Replaying the job list (make_jobs on the same config and the same HDRI order build_detect uses) and comparing against the shipped annotations:

planned jobs: 5000, images: 5000, annotations: 5972
frames with boxes > planned drones (a distractor would have been labelled): 0
negative frames (0 planned drones) carrying a box: 0
frames with distractors: 2876
  boxes in those frames: 3023; planned drones in those frames: 3023
frames tagged distractors_missing: 0
total planned drones: 5972; total annotations: 5972; dropped: 0

So in the 2876 frames that carry clutter, the box count equals the drone count exactly, and no negative frame got a box. distractors_missing: 0 also means every planned distractor found a sprite pool — in particular, dropping helicopter from the kind list did not silently leave planned clutter undrawn.

Brightness consistency (scripts/brightness_check.py, first 1000 positive images, box mean vs an 8 px ring outside it). Exposure handling did not change between v1 and v2, so this is a "did anything drift" check, not a fix:

$ python scripts/brightness_check.py out/detect_v2
images: 1000
boxes: 1494
median_ratio: 0.913
share_ratio_gt_2: 0.017

v1 on the same script: 1508 boxes, median_ratio 0.907, share_ratio_gt_2 0.019 (v0: 0.984 / 0.036). v2 sits right next to v1 on both numbers, which is what "same sprites, same exposure" predicts. This is not the paired comparison the v1 card ran against v0: v2's first 1000 positive images are a different set of frames with different boxes (1494 vs 1508), because the job stream changed (more negatives, a fifth size bucket), so treat it as two samples from the same generator, not a before/after on identical boxes.

Visual check

The QA sampler above wrote 12 annotated frames; five are kept in docs/datasets/img/ (boxes drawn from annotations.json; the caption strip is the frame's own tags).

image	what to look at
qa_negative_000926.jpg	Negative frame, 2 distractors, 0 boxes. An airliner_far (68 px, lower left) and a bird_soaring (55 px, right) against dusk cloud. Nothing is boxed, and this frame counts towards the false-positive metric. jpeg=75, sigma 7.86.
qa_random_002919.jpg	The C2 fix, in one frame. Two multirotor_medium drones (75 and 81 px) are boxed; a bird_soaring (45 px), an airliner_far (79 px, right) and a mast_with_cables (46 px, left of centre) are not. All three unlabelled objects are now the same scale as the labelled ones — in v2-rc that airliner could have been 260 px wide next to a 12 px drone. boxes=2 distractors=3.
qa_near_000291.jpg	Near field on a rolling-shutter frame — the C1 fix. A near-field fixed_wing (planned 491 px) with rolling_shutter_px=2.88: the box [71, 163, 492, 273] sits on the airframe rather than 2-3 px to its left, which is what v2-rc would have shipped. A 9 px tiny multirotor is boxed at the right edge; the frame's two distractors (airliner_far 90 px, bird_soaring 36 px) are not.
qa_near_001759.jpg	Frame-filling near field, in rain, backlit, skewed 5.17 px. A near-field multirotor planned at 959 px: the box [3, 315, 637, 325] is clipped by the frame at the right and the bottom, starts 3 px from the left edge, and its top edge lands on the rotor discs. This is the perspective the near-field sprite family exists for — not an upscaled 256 px sprite. jpeg=55.
qa_tiny_001001.jpg	Tiny drones next to bigger clutter. A 28 px multirotor_small and a 9 px multirotor_medium are boxed; three unlabelled birds (94, 83, 67 px) share the sky. The birds are still larger than the drones — the cap is at 96 px, not at "smaller than every target" — but none of them reaches the large bucket any more. boxes=2 distractors=3.

What is still wrong

• Nothing here says v2 is better. The rebuilt set has since been trained and evaluated (E1v2 = 0.159 mAP50 on DUT test vs v1's 0.375): removing the two data defects (C1, C2) did not recover the large bucket materially (0.064 → 0.118 against v1's 0.458). See docs/experiments/2026-09-detect-v2.md §5.
• 79 boxes are 1 px wider or narrower than their pixels. All of them 6-8 px, all on rolling-shutter frames; see the C1 check above. It is a rounding residual, not a displacement, but it is not zero.
• Hypothesis 3 is not closed. The drones are still procedural primitives assembled in Blender, not real CAD airframes. Nothing in v2 changes the geometry of a drone.
• Hypothesis 4 is not closed. Backgrounds are still the same 40 HDRI panoramas, tonemapped and reprojected, not real photographs. v2 adds objects in front of those panoramas; it does not make the panoramas real.
• Boxes include the motion smear. Blur is applied to the RGBA sprite before compositing, so the alpha is smeared too and the box grows to cover the streak. This is deliberate — the object really does cover that area during the exposure — but it means a v2 box for a fast target is slightly larger than the drone's instantaneous silhouette, and a detector trained on v2 will be nudged towards predicting the streak.
• Distractors are placed anywhere in the frame, sky included. The compositor puts a distractor at the planned (cx, cy) with no scene understanding, so a mast or a rooftop AC unit can end up floating in open sky. As unlabelled negative texture that is still useful, but it is not a physically plausible scene.
• The clutter cap is a guess with a number on it. 96 px is the labelled-large boundary, so the rule "no unlabelled object in the large bucket" is exact — but that the bucket is the thing that mattered is an inference from one ablation with one seed per arm.
• Still RGB, still single frames. No IR, no video, so nothing here helps temporal or thermal detection.
• Five distractor kinds, all procedural. They cover most of the categories §4 named (the rotorcraft is gone), but a real bird is not a squashed sphere with two flat wings; expect these to be easier to reject than the real thing.

Detect v2 vs v1: synthetic→real transfer on DUT Anti-UAV (E0v2, E1v2, E2v2)

docs/experiments/2026-09-detect-v2.md

Run 2026-09-05 on branch plan5-detect-v2 at 8e4a015, on the rebuilt Detect v2 (dataset built at 6cf56d0), same predict/eval code as every earlier arm. Every number below is read from out/eval/summary.json, the per-experiment out/eval/*/report.json, the out/models/*/results.csv epoch timers and the training logs under out/eval/logs/ — all git-ignored — or computed by the counting script described in §7.4. Nothing is typed from memory and nothing is rounded up.

Headline: fixing the data defects did not fix the transfer, and v2 is still worse than v1. The synthetic-only model trained on the rebuilt AeroBench Detect v2 scores mAP50 0.159 on the DUT Anti-UAV test set against the spec §8 threshold of 0.5. The first v2 build — which carried a 10.7 % label-displacement defect and clutter up to 260 px — scored 0.154. v1 scores 0.375. Repairing both defects moved the headline number by +0.005, inside noise. This is a negative result about the v2 design, not about the v2 build quality: unlabelled clutter plus near field plus capture realism transfers worse than v1 on every size bucket except tiny, and the large-object collapse (v1 0.458 → v2 0.118) survives capping every unlabelled object below 96 px. v1 remains the recommended synthetic set.

1. Setup

Everything about the experiment is unchanged from 2026-09-transfer-dut.md. What changed is the synthetic training set.

Detector	yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime	torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
Training	imgsz 640, batch 32, seed 0, --workers 0, deterministic=True, default augmentation
Epochs	30 (E0v2, synthetic) / 20 (E2v2, real)
Inference	scripts/predict_coco.py, conf 0.001, imgsz 640, Ultralytics' default max_det 300 (COCOeval still scores only the top 100 per image — the same caveat as in the plan-3 write-up)
Metrics	aerobench eval --merge-classes
Driver	scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2
Real test set	DUT Anti-UAV test, 2 200 images, 2 245 boxes — the identical file used in plan 3

What is the same as v1, deliberately: the same 40 Poly Haven panoramas in the same order, the same drone sprites (assets/sprites_v2 is a byte copy of assets/sprites_v1, 40/40 sprites.json sha256 match — see the card), the same detector, the same hyperparameters, the same splits recipe (4 500 / 500), the same DUT test set and the same evaluator. So a v1↔v2 difference is a difference in the generator, not in the experiment.

What v2 changed — full details and the build's self-checks in the dataset card docs/datasets/detect_v2.md:

1. Unlabelled distractors (hypothesis 1 of the plan-3 §4 list, the one we rated largest): five procedural kinds — bird_soaring, bird_flapping, airliner_far, mast_with_cables, rooftop_hardware — composited before the drones and never annotated. 5 787 instances across 2 876 frames (0.575 of the set); negatives doubled from 10 % to 20 %, and 841 of the 1 000 negatives carry clutter. Their long side is drawn from distractor_px: [10, 96] — min 10 / median 53 / max 95 px — so zero unlabelled objects land in the large bucket against 1 319 labelled large boxes.
2. Near field (part of hypothesis 2): a fifth size bucket near: (200, 1100) at 0.10 of the mix, fed by a separate 768 px sprite family rendered with a close camera (1.05–1.5× object distance, 24–35 mm lens) — real perspective, not an upscaled 256 px sprite. 615 boxes with a long side ≥ 200 px; 653 boxes touch a frame edge.
3. Capture realism (hypothesis 5): motion blur along each drone's own velocity vector, per-frame Gaussian sensor noise (σ 2–9), signal-dependent shot noise, rolling-shutter skew on 25 % of frames — shearing the boxes with the pixels — and in-pipeline JPEG requantisation at quality 55–95 on every frame.

What v2 did NOT change, and the report must not pretend otherwise: hypothesis 3 (real CAD airframes — the drones are still procedural primitives) and hypothesis 4 (real photographs as backgrounds — still HDRI panoramas). Those were a controller decision, on licence and effort grounds, before any of this ran.

One structural difference beyond the three: v2 has 20 % negative frames where v1 had 10 %. That change rides along with the distractors in every v2 arm here and in §5, and is not isolated by any of them.

This is the second v2 build. The first (09d522d, trained 2026-09-04) shipped with two defects — displaced boxes on skewed frames and clutter up to 260 px — and its runs are kept in §8 as v2-rc, because they are what that data did and because §5's ablation was measured on it. Everything in §1–§4 and §6–§7 is the rebuilt set.

Size buckets and what they are worth

Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS, untouched by plan 5 exactly so these slices stay comparable): tiny < 16, small 16–32, medium 32–96, large ≥ 96.

bucket	tiny	small	medium	large	total
GT boxes in DUT test	15	385	1 087	758	2 245

The tiny column is 15 boxes. It is a direction, never a measurement. The large column is 758 boxes and is the one that carries this write-up's finding.

2. Results

Exp	Train data	Init weights	Eval set	mAP50	mAP50-95	tiny	small	medium	large	train time
E0	syn_v1 (4 500)	yolov8n.pt	synthetic v1 val (500)	0.905	0.697	0.706	0.929	0.996	0.999	≈ 4 408 s ‡
E0v2	syn_v2 (4 500)	yolov8n.pt	synthetic v2 val (500)	0.826	0.645	0.466	0.820	0.931	0.980	≈ 4 608 s ‡
E1	— (no training)	E0 best.pt	DUT test (2 200)	0.375	0.195	0.038	0.201	0.381	0.458	n/a
E1v2	— (no training)	E0v2 best.pt	DUT test (2 200)	0.159	0.063	0.071	0.203	0.262	0.118	n/a
E2	dut10 (520 real)	E0 best.pt	DUT test (2 200)	0.761	0.468	0.242	0.744	0.717	0.848	1 550.1 s
E2v2	dut10 (520 real)	E0v2 best.pt	DUT test (2 200)	0.750	0.448	0.242	0.703	0.733	0.837	1 245.1 s
E3 †	dut10 (520 real)	yolov8n.pt	DUT test (2 200)	0.759	0.434	0.156	0.713	0.743	0.824	1 525.8 s
E4 †	dut_full (5 200 real)	yolov8n.pt	DUT test (2 200)	0.912	0.580	0.535	0.816	0.913	0.968	5 286.2 s

Per-bucket columns are mAP50. The same buckets at mAP50-95:

Exp	tiny	small	medium	large
E0v2	0.242	0.563	0.787	0.889
E1	0.007	0.043	0.174	0.314
E1v2	0.023	0.042	0.097	0.072
E2	0.024	0.355	0.430	0.591
E2v2	0.024	0.301	0.426	0.584
E3 †	0.025	0.301	0.405	0.549
E4 †	0.134	0.373	0.554	0.723

† E3 and E4 were not retrained for this report. Their rows are reused verbatim from the plan-3 run (out/eval/summary.json, rows E3/E4, produced at e05a131). They are control arms that contain no synthetic data at all, so nothing in plan 5 could change them; they were computed with the same predict_coco.py / aerobench eval code path that E0v2–E2v2 use here.

‡ Neither E0 nor E0v2 ran straight through, and neither driver figure is the training cost. E0's 928.1 s in summary.json covers only a resumed segment; the plan-3 write-up reconstructs ≈ 4 408 s from results.csv. E0v2 was resumed too: summary.json records resumed_from_epoch: 14, out/eval/logs/E0v2.log line 2408 carries a ===== RESUME … from epoch 14 ===== marker followed by Resuming training … from epoch 15 to 30 total epochs, and out/models/syn_v2/results.csv shows the epoch timer resetting between epoch 14 (2 134.66 s) and epoch 15 (167.80 s). The driver's 2 511.4 s therefore covers epochs 15–30 only (16 epochs completed in 0.687 hours = 2 473.2 s in the log); adding the first segment's 2 134.7 s gives ≈ 4 608 s of training time, in line with v1. Whatever wall-clock elapsed during the interruption itself is in neither figure. E2v2 ran straight through (20 epochs completed in 0.332 hours, resumed_from_epoch: null), so its 1 245.1 s is the whole run; the gap to E2's 1 550.1 s for the same 20 epochs on the same 520 images is machine load, not data.

E0v2 is on its own evaluation set (held-out v2 synthetic val), so it is not comparable to E0 as a score — both are pipeline sanity checks. The gate the plan set was E0v2 mAP50 ≥ 0.8 or stop and debug the training: 0.826, passed. Ultralytics' own best-epoch val number in the log agrees (MAP50_BEST 0.82758). Training is not broken; the data is what changed. That 0.826 is still a validation number picked on the same 500 images Ultralytics selected best.pt with — mildly optimistic by construction, exactly as in v1, and irrelevant to every DUT number here.

One metric that is meaningful — and two that are not

fp_per_image is measured on drone-free frames. DUT test has none (negatives.images: 0 in every real report), so fp_per_image: 0.000 for E1v2 and E2v2 is an artifact and must be ignored, as in plan 3.

For E0v2 it is real: the v2 synthetic val split carries 100 negative frames (v2 is 20 % negatives), and out/eval/e0v2_sanity/report.json records 14 false positives over 100 negatives = 0.14 per image. v1's E0 saw 54 negatives and 0 false positives. On its own domain, the v2-trained model still fires on empty frames more than the v1-trained one did.

3. Verdict against spec §8

Gate 1 — E1v2 ≥ 0.5 mAP50 on real data with no real training images: FAILED. 0.159 against 0.5. It is not a near miss and not progress towards the threshold; it is worse than v1's 0.375 by 0.216, far outside the ~0.02 noise band this test set was shown to have in plan 3. Two failures in a row, and the second is a regression that survived a full dataset rebuild. (The Drone-vs-Bird half of the spec's real-data gate remains untested and still needs the founder's signature — see §6.)

At mAP50-95 the picture is the same, 0.063 against v1's 0.195. Per bucket, v2 is behind v1 everywhere except tiny — and tiny is 15 boxes:

bucket (mAP50-95)	E1 (v1)	E1v2 (v2)	Δ
tiny (15 boxes)	0.007	0.023	+0.016
small (385)	0.043	0.042	−0.001
medium (1 087)	0.174	0.097	−0.077
large (758)	0.314	0.072	−0.242

Gate 2 — E2v2 > E3 (is it worth pre-training on this synthetic set at all): no signal in either direction.

	E2v2 (syn_v2 → 10 % real)	E2 (syn_v1 → 10 % real)	E3 (COCO → 10 % real)
mAP50	0.7498	0.7605	0.7590
mAP50-95	0.4480	0.4682	0.4340
Δ mAP50 vs E3	−0.0092	+0.0015	—
Δ mAP50-95 vs E3	+0.0140	+0.0342	—

Read honestly:

• E2v2 − E3 is −0.0092 mAP50 and +0.0140 mAP50-95. Both are inside the ~0.02 noise band, and they have opposite signs. Unlike v1 — whose +0.0342 at mAP50-95 was at least outside the band — the rebuilt v2 buys nothing measurable as pre-training.
• E2v2 − E2 is −0.0107 mAP50 / −0.0202 mAP50-95. Also at or inside the band: after fine-tuning on 520 real images the two synthetic sets are indistinguishable, even though one of them is 2.4× worse zero-shot. 520 real images wash out a 0.216 zero-shot gap. That is the single most useful practical fact in this write-up, and it repeats what v2-rc showed (§8).
• E2v2's tiny 0.242 equals E2's exactly — 15 boxes, and both arms find the same 4 of them. Direction, not measurement.
• E4 — the same architecture on 5 200 real images, no synthetic — is still 0.912 / 0.580, +0.162 mAP50 over E2v2. Nothing in v2 changes plan 3's conclusion that real labels dominate.

Release decision: v1 stays the recommended synthetic set. v2 is kept as a documented negative result together with its defect history; nothing in it beats v1 zero-shot, and it does not beat v1 as pre-training either.

4. What actually changed in the errors

All counts below are our own greedy highest-score-first matching at IoU 0.5 over the prediction JSONs (§7.4), never the evaluator.

The box flood got better; the detections did not

arm	boxes at conf 0.001	per image	images with > 100 boxes	boxes at conf 0.25	TP	FP	FN	precision	recall
E1 (v1)	235 584	107.1	963	6 297	1 187	5 110	1 058	0.189	0.529
E1v2 (v2)	174 433	79.3	592	2 500	629	1 871	1 616	0.252	0.280
E1v2-rc (§8)	290 192	131.9	1 333	4 585	879	3 706	1 366	0.192	0.392
E2v2	59 196	26.9	63	1 949	1 585	364	660	0.813	0.706
E4	25 670	11.7	9	2 231	1 999	232	246	0.896	0.890

This is the one place the rebuilt clutter did what hypothesis 1 promised. Precision at the working threshold rose 0.189 → 0.252 (v2-rc, with its oversized clutter, had managed 0.192), the diffuse box count fell by a quarter against v1 and by 40 % against v2-rc, and the number of images drowning in low-confidence boxes fell from 963 to 592. Recall fell from 0.529 to 0.280 to pay for it. The model did not learn "a railing is not a drone"; it learned "fire less", and it fires less at drones too.

The model still localizes, and still refuses to commit

For every GT box: is there any prediction over it at IoU ≥ 0.5, at any score; if so, does the highest-scoring such prediction clear conf 0.25; and what is the median of those scores.

arm	bucket	localized	of those, clear 0.25	median score
E1 (v1)	small	260/385 = 0.675	0.642	0.479
	medium	808/1 087 = 0.743	0.681	0.703
	large	594/758 = 0.784	0.785	0.772
E1v2 (v2)	small	270/385 = 0.701	0.559	0.308
	medium	768/1 087 = 0.707	0.397	0.135
	large	543/758 = 0.716	0.311	0.086

Localization is essentially intact — v2 puts a well-fitted box on 72 % of the large real drones, slightly fewer than v1's 78 % but nowhere near a collapse — and then scores it at a median of 0.086 against a 0.25 threshold. The failure is confidence, not sight, exactly as in v2-rc. The difference from v2-rc is where it now falls: v2-rc's suppression was concentrated at large (its small/medium medians were 0.451 / 0.360, close to v1's 0.479 / 0.703); the rebuilt v2 has dragged every bucket down (medium 0.703 → 0.135). Capping the clutter below 96 px did not lift the large bucket back to v1; it lowered the middle to match.

Recall at conf 0.25 by bucket, same matching:

arm	tiny	small	medium	large
E1 (v1)	4/15 = 0.267	167/385 = 0.434	550/1 087 = 0.506	466/758 = 0.615
E1v2 (v2)	4/15 = 0.267	151/385 = 0.392	305/1 087 = 0.281	169/758 = 0.223

The two frames from the v2-rc report, re-measured

Both frames are DUT Anti-UAV test images. The annotated figures are not redistributed — the dataset states no licence for redistribution and we use it for evaluation only — so each case is given in numbers below. Our own annotated copies were drawn from the v2-rc prediction files and live under the git-ignored out/figures/; to reproduce them, fetch the DUT test split (§6.1 of 2026-09-transfer-dut.md), run the E1/E1v2 arms, and draw two side-by-side panels with cv2.rectangle — left = E1 on v1, right = E1v2 on v2-rc; green = ground truth, red = prediction at conf ≥ 0.25, yellow = the model's best-IoU box when it falls below 0.25. The rebuilt model's numbers on the same two frames are given in the text.

a) The large-object collapse — img/00103.jpg, GT 185 × 102 px

v1: one box, IoU 0.979, score 0.904. v2-rc: zero boxes clear 0.25; best-IoU box 0.991 at score 0.0038. The rebuilt v2 is no better and slightly worse: still zero boxes over 0.25 (its highest-scoring prediction anywhere in the frame is 0.010), best-IoU box 0.694 at score 0.0012. n = 1, but it is the §3 table in one image.

b) The near-field case from plan 3 — img/01124.jpg, GT 865 × 305 px

This is the one frame where the rebuilt v2 wins outright. v1 fragments the aircraft (best-IoU box anywhere 0.445, at score 0.153 — below threshold; its best scoring box, 0.672, covers a 432 px slice). v2-rc gets the extent right — IoU 0.877 — but scores it 0.062, below threshold. The rebuilt v2 gets IoU 0.877 at score 0.488, i.e. a confident, correct detection of a frame-filling drone, which neither of the other two models produced. The near-field sprites work; the suppression that hid this in v2-rc is gone here, and not in (a).

5. The causal reading, revised

The v2-rc write-up named a cause: unlabelled clutter, and specifically clutter in the same size bucket as the targets. The rebuild was that hypothesis made real — distractor_px: [10, 96], 0 unlabelled instances ≥ 96 px against 1 319 labelled large boxes, and the two large aerial kinds (helicopter, plus the already-narrow airliner_far shrunk with the rest) no longer able to appear at drone scale. It was a direct test with a stated falsifier: large mAP50 returns towards v1's 0.458 while precision at conf 0.25 keeps some of v2b's 0.259.

Half of that came true and the half that mattered did not. Precision went to 0.252. large mAP50 went 0.064 → 0.118, against v1's 0.458: about 14 % of the gap, on one seed. So:

The "large unlabelled clutter outnumbers large labelled drones" explanation is at most partial. It cannot be the main cause of the large-object collapse, because removing it entirely left 86 % of the collapse in place. The v2-rc report should be read as having over-committed to it.

The label defect is refuted as a driver too: C1 displaced 10.7 % of v2-rc's boxes, the rebuild has none below IoU 0.5 and 98.7 % above 0.9, and the headline moved 0.154 → 0.159.

What the evidence still supports, all of it one seed per arm:

• Clutter of any kind is the largest single suspect, but by arm-level evidence, not by the size mechanism. In the rc-era 2 × 2 (§8) the arm with no clutter at all — v2a, which kept the near field, the capture realism and the 20 % negatives — scored 0.334, the closest any v2 variant has come to v1's 0.375. Every arm that contains unlabelled clutter, at any size, scores 0.154–0.231. What the rebuild rules out is the size-matching explanation of how clutter hurts, not clutter itself.
• Near field + capture realism costs small and medium, and buys the near-field frame. The clean rc-era comparison (v2a − v1, both without clutter on one side only) was tiny +0.039, small −0.024, medium −0.054, large −0.038, overall −0.041 — small, above the noise band but the smallest effect measured. §4b is the qualitative other side of it.
• The suppression mechanism is confirmed and is now global. Whatever v2 teaches, it is "do not commit", and in the rebuilt set that applies to every size bucket rather than only to large objects.
• A confound we never removed: 20 % negatives. v1 has 10 %; every v2 arm ever trained, including v2a, has 20 %. No arm isolates it, so "more empty frames make the model reticent" remains a live and completely untested alternative to "clutter makes the model reticent" — and it would explain the global suppression better than the size story does.

6. Honesty rules and what is next

Rules applied throughout, same as the plan-3 write-up:

• One seed per arm. Every comparison here is one training run against one training run. Words like "suggestive" are load-bearing; nothing in §5 is an established causal result.
• A mAP50 difference below ~0.02 on this test set is noise (established in plan 3 from E2 vs E3). That covers E1v2 − E1v2-rc (+0.005), E2v2 − E3 (−0.0092), E2v2 − E2 (−0.0107) and E2v2 − E3 at mAP50-95 (+0.0140). It does not cover E1v2 − E1 (−0.216).
• A slice with fewer than ~50 boxes is a direction, not a measurement. That is the tiny column everywhere: 15 boxes in DUT test.
• fp_per_image on DUT test is an artifact (zero negative frames). Only E0v2's 0.14 over 100 synthetic negatives is a real false-positive rate, and it is on synthetic data.
• Every number traces to out/eval/summary.json, an out/eval/*/report.json, an out/models/*/results.csv, a log under out/eval/logs/, or the counting script in §7.4.

v3 hypotheses — none of these has been tested

v2 changed four things at once (clutter, near field, capture realism, negative fraction) and the rebuild changed a fifth. That is why we cannot say which one costs the 0.216. The next round should start from v1 and add exactly one thing per arm, each measured by the same E1 — train on synthetic only, evaluate on DUT test, report per-bucket — with nothing else touched. Each is a guess consistent with §4 and §5, and each is one 30-epoch run plus one inference pass (≈ 1.5 h on an RTX 3070 Ti):

1. v1 + 20 % negatives only. No clutter, no near field, no capture realism — only negative_fraction: 0.20. This removes the confound §5 names last and is the cheapest of the three. If E1 drops materially below 0.375, the reticence is about empty frames and none of the rest of v2 needs explaining.
2. v1 + capture realism only. Motion blur, sensor/shot noise, rolling shutter (boxes sheared), JPEG requantisation, at the v1 size mix and 10 % negatives. Isolates hypothesis 5 for the first time.
3. v1 + small ground clutter only. bird_soaring, bird_flapping, mast_with_cables, rooftop_hardware at distractor_px: [10, 96], no airliner_far, v1 negatives, v1 size mix. Isolates the thing §5 still suspects most, without the negative-fraction change riding along.

The near field is deliberately not on that list: §4b says it works, and it is the only change with a positive result to its name. Add it back after one of 1–3 explains the loss.

Then the two hypotheses v2 never touched, in expected-value order: hypothesis 3 (real CAD / photogrammetric airframes instead of procedural primitives) and hypothesis 4 (real photographs as backgrounds alongside HDRIs). Both were deferred on licence and effort grounds and both remain completely open. Given that four data changes in a row have not moved E1 towards 0.5, they may matter more than anything in this plan did.

Founder action still required — unchanged from plan 3 §5. The WOSDETC Drone-vs-Bird dataset — the one real benchmark that actually contains birds, and therefore the honest test of whether bird distractors help — cannot be downloaded. Access needs an email to wosdetc@googlegroups.com and a signed data usage agreement from a named person (https://github.com/wosdetc/challenge). Nothing in that can be automated. Until it exists, "our synthetic birds teach the model to reject real birds" is untestable, and every clutter conclusion here is measured only against DUT, which contains no birds at all.

7. Reproducing this

Repo root, project venv with the training extra (uv pip install -e ".[dev,train]"). assets/real/dut/** is fetched exactly as in 2026-09-transfer-dut.md §6.1 — that step is unchanged and not repeated here.

7.1 The dataset

aerosynth render-extras assets/hdris assets/sprites_v2 --kind distractors --per-hdri 24 --seed 5 --size 256 --samples 32
aerosynth render-extras assets/hdris assets/sprites_v2 --kind nearfield  --per-hdri 9  --seed 6 --size 768 --samples 32
aerosynth build-detect configs/detect_v2.yaml out/detect_v2 --sprites assets/sprites_v2

(assets/sprites_v2 is first a plain copy of assets/sprites_v1; the sprites are unchanged between the two v2 builds — dropping helicopter is a filter at index time. See docs/datasets/detect_v2.md for the sprite sha256 check and the build's self-checks.)

7.2 The three experiments

python scripts/make_yolo_split.py out/detect_v2 out/splits/syn_v2 --val-frac 0.1 --seed 0 --single-class
python scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2 --dry-run
python scripts/run_transfer_experiments.py --only E0v2,E1v2,E2v2
python scripts/build_site.py

E3 and E4 are deliberately not in that --only list: they contain no synthetic data, and their rows are reused from the plan-3 run. Budget ≈ 1.6 h of training on an RTX 3070 Ti plus inference over 2 200 images twice.

On the resume. E0v2's training was interrupted after epoch 14 and restarted from out/models/syn_v2/weights/last.pt; the driver re-invokes train_yolo.py with --resume, appends a ===== RESUME … from epoch 14 ===== marker to out/eval/logs/E0v2.log and records resumed_from_epoch in summary.json. A clean run needs no resume and reports the whole 30 epochs in one timer; see the ‡ footnote in §2 for how to read the split figure when it happens.

7.3 The v2-rc artifacts

The first build's runs were not deleted. They live beside the current ones as out/eval/e0v2_sanity_rc, out/eval/e1v2_syn_only_rc, out/eval/e2v2_syn_then_real10_rc and out/models/syn_v2_rc, out/models/syn_v2_then_dut10_rc (all git-ignored). The two ablation arms of §8 were run through train_yolo.py/predict_coco.py directly rather than the driver, so their numbers exist only as out/eval/e1v2a_syn_only/report.json and out/eval/e1v2b_syn_only/report.json; their configs are committed as configs/detect_v2a.yaml and configs/detect_v2b.yaml.

7.4 The counting script

The conf-0.25 precision/recall table, the per-bucket recall table, the localized-vs-suppressed table and the two figures are not produced by any committed tool — they are short one-off scripts over the prediction JSONs, the same approach as plan 3 §4. Each loads assets/real/dut/test_coco.json plus an out/eval/*/preds.json, does greedy highest-score-first matching at IoU 0.5, and prints counts; the localized/suppressed table instead takes, per GT box, the highest-scoring prediction at IoU ≥ 0.5. There is no hidden filtering: every prediction in the file is considered, and the "boxes at conf 0.001" column is simply len(preds).

7.5 Where the numbers live

out/** is git-ignored; nothing but this write-up and its two figures is committed. The driver writes one row per experiment to out/eval/summary.json (keyed by experiment, replacing that experiment's earlier row on a re-run — the E0v2/E1v2/E2v2 rows there are the rebuilt runs, and the v2-rc numbers survive only in the _rc directories above and in §8 of this document).

8. History: v2-rc, the first build

The first v2 build (09d522d, 2026-09-04) was trained and evaluated before review found two defects in it:

• C1 — displaced boxes. The rolling-shutter skew was applied to the pixels after the boxes had been measured: 636 of its 5 972 boxes (10.7 %) sat off their drone, 51 at IoU < 0.5.
• C2 — oversized clutter. distractor_px: [12, 260] put 3 824 unlabelled objects ≥ 96 px against 1 318 labelled large boxes, 2.9:1, including a helicopter whose rotor disc is rendered with the same translucent-disc trick as the drone propellers.

Both are fixed at 6cf56d0. The rc results are kept because §5 rests on them:

arm	what it is	mAP50	mAP50-95	tiny	small	medium	large
v1 (E1)	reference	0.375	0.195	0.038	0.201	0.381	0.458
v2a	v2-rc without clutter (near field + capture realism + 20 % negatives)	0.334	0.164	0.077	0.177	0.327	0.420
v2b	v2-rc clutter only (v1 size mix, v1 capture defaults, 20 % negatives)	0.231	0.093	0.067	0.193	0.306	0.148
v2-rc	both	0.154	0.059	0.109	0.309	0.286	0.064
v2 (rebuilt)	both, C1/C2 fixed	0.159	0.063	0.071	0.203	0.262	0.118

E0v2-rc scored 0.825 on its own synthetic val (sanity gate passed) and E2v2-rc reached 0.768 / 0.459 on DUT test — i.e. the fine-tuned arm was as indistinguishable from E2/E3 then as it is now.

Caveats that apply to the four rc rows, and therefore to §5:

• One seed per cell, four single runs; the 2 × 2 is suggestive, not established.
• All three v2 arms carry 20 % negatives and v1 carries 10 %, so every delta measured against v1 also contains that change.
• v2a and v2b were trained on data with the C1 label defect (v2b also with C2's oversized clutter). The rebuild shows C1 is worth ≈ 0.005 on the headline, which is why we still quote them — but they are not clean arms.
• The two arms' synthetic sanity numbers come from Ultralytics (MAP50_BEST 0.85305 and 0.89097 on each arm's own val split), not from aerobench eval; they are used only to confirm the ≥ 0.8 training gate.
• v2a's wall time is not a training cost. out/models/syn_v2a/results.csv ends at 36 360.4 s because one epoch absorbed 32 433.8 s when the machine slept; the other 29 total 3 926.6 s, so the real cost was ≈ 4 060 s, in line with v2b's 3 820.2 s.

Итог для основателя

Главное: вторая версия синтетического набора не заработала, и починка её дефектов ничего не изменила. В первой сборке v2 нашли два дефекта — рамки на 10,7 % объектов были смещены относительно дрона, а отвлекающие объекты были слишком крупными. Оба исправили, набор пересобрали и обучили заново. Результат на реальном тесте DUT Anti-UAV: mAP50 = 0.159 против 0.154 у сломанной сборки и 0.375 у первой версии набора. Порог спецификации — 0.5. То есть починка данных дала +0.005, это шум, а отставание от v1 никуда не делось. Порог не взят второй раз подряд.

Дело не в качестве сборки, а в самой идее v2. Набор проигрывает v1 во всех размерных группах, кроме «крошечной» (а она посчитана по 15 рамкам во всём тесте и ничего не доказывает). На крупных дронах — 758 рамок, самая важная часть теста — метрика 0.118 против 0.458 у v1.

Наша прошлая объяснительная версия оказалась в лучшем случае частичной. В прошлый раз мы решили, что виноваты слишком крупные неразмеченные объекты: их было почти три на каждый размеченный крупный дрон, и модель научилась «большое в небе обычно не цель». В пересобранном наборе весь мусор строго мельче порога «крупного» — таких объектов ровно ноль. Проверка провалилась: провал на крупных объектах восстановился с 0.064 только до 0.118 при 0.458 у v1, то есть примерно на седьмую часть. Значит, размер мусора — не главная причина, и прошлый отчёт был слишком уверен. Мы это записали прямо в документ, а не переформулировали задним числом.

Что подтверждается по-прежнему. Модель дроны видит: она ставит хорошую рамку на 72 % крупных реальных дронов — и присваивает ей уверенность 0.086 при рабочем пороге 0.25, то есть сама же выбрасывает. Причём теперь это подавление распространилось на все размеры, а не только на крупные. Из всех вариантов, что мы обучили, ближе всего к v1 (0.334 против 0.375) оказался тот, где отвлекающих объектов не было вообще, — так что подозрение на мусор остаётся, но механизм мы назвали неправильно. Ближняя зона и реализм съёмки стоят немного (−0.041) и на одном кадре дали единственную настоящую победу: дрон, заполняющий кадр, который раньше разваливался на куски, теперь уверенно распознаётся целиком.

Одна вещь всё-таки улучшилась. Точность при рабочем пороге выросла с 0.189 у v1 до 0.252 — ложных срабатываний на перилах и крышах стало меньше, поток мусорных рамок сократился на четверть. Но платой стала полнота: 0.280 против 0.529. Модель не научилась «перила — это не дрон», она научилась «реже нажимать».

Предобучение: разницы нет никакой. Если дообучить на 10 % реальных данных, новая синтетика даёт 0.750, старая 0.761, а вообще без синтетики — 0.759. Всё это внутри шума. 520 реальных кадров стирают разрыв в 0.216, который был без них. Полный реальный набор по-прежнему бьёт всё: 0.912. Вывод плана 3 не изменился — синтетика это экономия разметки, а не замена ей.

Решение. Релизным синтетическим набором остаётся v1. v2 сохраняем как задокументированный отрицательный результат вместе с историей его дефектов — он не лучше ни без дообучения, ни как предобучение.

Что делать дальше — и это пока гипотезы, ни одна не проверена. Ошибка плана 5 в том, что мы поменяли четыре вещи сразу и теперь не можем сказать, какая именно стоила нам 0.216. Следующий раунд надо начинать от v1 и добавлять по одной вещи за прогон, меряя одним и тем же числом (E1 на DUT, примерно полтора часа обучения на арм): (1) v1 плюс только удвоенная доля пустых кадров — это единственное, что мы ни разу не изолировали, и это лучше всего объясняет «всеобщее» подавление уверенности; (2) v1 плюс только реализм съёмки; (3) v1 плюс только мелкий наземный мусор без самолётов. Ближнюю зону вернуть после того, как один из этих трёх прогонов объяснит потерю. И только потом — две гипотезы, которых мы вообще не касались: реальные CAD-модели дронов вместо процедурных и реальные фотографии вместо панорам. Четыре изменения данных подряд не сдвинули E1 к 0.5, так что эти две могут значить больше, чем всё, что мы сделали в этом плане.

Оговорка о доверии к цифрам. На каждый вариант — один прогон обучения без доверительного интервала. Разница меньше ~0.02 на этом тесте — шум; под это попадают и +0.005 от починки дефектов, и все сравнения с дообучением. Колонка «крошечные объекты» посчитана по 15 рамкам и является направлением, а не измерением.

Требуется ваше действие — то же, что и в прошлый раз. Набор Drone-vs-Bird — единственный реальный тест, где вообще есть птицы, то есть единственный способ честно проверить, помогают ли наши синтетические птицы. Скачать его нельзя: нужно письмо на wosdetc@googlegroups.com и подписанное соглашение об использовании данных от вашего имени (https://github.com/wosdetc/challenge). В DUT птиц нет вовсе, поэтому все выводы про отвлекающие объекты проверены только на нём.

Update 2026-09-05. The single-change hypotheses above were tested: see 2026-09-v1-ablations.md. None moved the zero-shot number up (negatives ×2: 0.380; capture realism: 0.287; small clutter: 0.238); the 0.5 gate is closed for this generator family.

Synthetic→real transfer on DUT Anti-UAV (E0–E4)

docs/experiments/2026-09-transfer-dut.md

Run 2026-09-04 on branch plan3-transfer: E0 at c401cbc, E1–E4 at e05a131 (identical predict/eval code). Every number below is read from out/eval/summary.json and the per-experiment out/eval/*/report.json files (both git-ignored); nothing is typed from memory or rounded up.

Headline: the synthetic-only model does not transfer. Trained on AeroBench Detect v1 alone it scores mAP50 0.375 on the DUT Anti-UAV test set against a spec §8 threshold of 0.5 — a clear fail, not a near miss. Using the same synthetic model as a starting point before fine-tuning on 10 % of DUT looks like it helps box quality (mAP50-95 +0.034) rather than detection rate (mAP50 +0.0015, i.e. noise) — but that is one seed per arm with no error bar, so treat it as suggestive, not established.

1. Setup

Detector	yolov8n (3.0 M params, 8.1 GFLOPs), Ultralytics 8.4.138
Runtime	torch 2.14.0+cu126, Python 3.12.14, NVIDIA RTX 3070 Ti (8 GB)
Training	imgsz 640, batch 32, seed 0, --workers 0 (Windows DataLoader hangs), default augmentation
Epochs	30 (E0, synthetic) / 20 (E2, E3, E4, real)
Inference	scripts/predict_coco.py, conf 0.001, imgsz 640 (the standard low-threshold mAP protocol), Ultralytics' default max_det of 300 ‡
Metrics	aerobench eval --merge-classes — the three synthetic classes are merged to one drone class so the synthetic model and DUT's single class are comparable. For E1–E4 the flag is a no-op safety net: the model was already trained single-class and DUT's GT has one category, so there is nothing to merge
Driver	scripts/run_transfer_experiments.py (one entry point for all five runs)

‡ max_det 300 at predict time does not mean 300 boxes per image are scored. COCOeval's standard maxDets for the summary metrics is 100, so only the top-100 predictions per image ever count towards mAP. That bites hardest on E1, whose predictions are diffuse: 963 of the 2 200 test images carry more than 100 boxes at conf 0.001 (see §4). The numbers in this write-up are unaffected — they are all measured under that same standard protocol, which is the point of using it — but a reader comparing them against a differently configured evaluator should know which cap applied.

Splits

Split	Contents	Images
syn_v1 train / val	Detect v1 (out/detect_v1), single-class, 90/10 split	4 500 / 500
dut10 train	10 % subsample of DUT train, --seed 0	520
dut_full train	full DUT train	5 200
DUT val (E2/E3/E4 training val)	full DUT val	2 600
DUT test (all real evaluations)	held out end-to-end	2 200

Synthetic source set (see docs/datasets/detect_v1.md): 5 000 frames, 500 of them negatives, 6 760 boxes, 40 CC0 Poly Haven panoramas, procedurally modelled drones.

Data attribution

Every real image and label in this write-up comes from a third-party dataset. The same text is written to assets/real/dut/ATTRIBUTION.md (git-ignored, along with the data itself) by aerosynth.realdata.dut_antiuav.write_attribution:

Images and annotations: DUT Anti-UAV Detection dataset.

Jie Zhao, Jingshu Zhang, Dongdong Li, Dong Wang. "Vision-based Anti-UAV Detection and Tracking." IEEE Transactions on Intelligent Transportation Systems, 2022.

https://github.com/wangdongdut/DUT-Anti-UAV

No LICENSE/README found inside the dut_{train,val,test}.zip archives or their extracted folders (checked via unzip -l on all three archives and a filesystem search of the extracted train/val/test directories). The paper's companion code repository (github.com/wangdongdut/DUT-Anti-UAV) does carry a root LICENSE file that GitHub's API identifies as Apache-2.0, but that repository holds only code, docs and download links (~6 MB total) — the dataset itself (hundreds of MB per split) is hosted separately on Google Drive / Baidu Cloud and is not confirmed to fall under that grant. Treated as: license not stated for the dataset distribution itself; used here for evaluation only, not redistributed.

Size buckets

Buckets are by long side in pixels (aerosynth.annotate.SIZE_BUCKETS): tiny < 16, small 16–32, medium 32–96, large ≥ 96.

The DUT test set is heavily weighted to big objects, and this shapes how much any per-bucket number is worth:

bucket	tiny	small	medium	large	total
GT boxes in DUT test	15	385	1 087	758	2 245

The tiny column in every table below is computed over 15 boxes. One object moves it by several points. Read it as a direction, never as a measurement.

2. Results

Exp	Train data	Init weights	Eval set	mAP50	mAP50-95	tiny	small	medium	large	train time
E0	syn_v1 (4 500 syn imgs)	yolov8n.pt (COCO)	synthetic val (500 imgs)	0.905	0.697	0.706	0.929	0.996	0.999	928.1 s †
E1	— (no training)	E0 best.pt	DUT test (2 200 imgs)	0.375	0.195	0.038	0.201	0.381	0.458	n/a
E2	dut10 (520 real imgs)	E0 best.pt	DUT test (2 200 imgs)	0.761	0.468	0.242	0.744	0.717	0.848	1 550.1 s
E3	dut10 (520 real imgs)	yolov8n.pt (COCO)	DUT test (2 200 imgs)	0.759	0.434	0.156	0.713	0.743	0.824	1 525.8 s
E4	dut_full (5 200 real imgs)	yolov8n.pt (COCO)	DUT test (2 200 imgs)	0.912	0.580	0.535	0.816	0.913	0.968	5 286.2 s

Per-bucket columns are mAP50. The same buckets at mAP50-95:

Exp	tiny	small	medium	large
E0	0.384	0.688	0.867	0.915
E1	0.007	0.043	0.174	0.314
E2	0.024	0.355	0.430	0.591
E3	0.025	0.301	0.405	0.549
E4	0.134	0.373	0.554	0.723

E0 is on a different evaluation set (held-out synthetic val, not DUT). It is a pipeline sanity check — "does training work at all" — not a comparable score. It passes its own gate comfortably (0.905 vs the > 0.8 expectation).

That 0.905 is a validation number, not a held-out test number: those 500 images are out/splits/syn_v1/val.txt, i.e. exactly the set Ultralytics validated against after every epoch and used to pick which checkpoint became best.pt. Selecting on a set and then reporting on it is mildly optimistic by construction. It does not affect anything E0 is used for here — a pipeline sanity check, and weights for E1/E2 — and every real-data number in this write-up comes from the DUT test split, which no training run ever saw.

† E0's 928.1 s is not the cost of 30 epochs. E0's training was interrupted and restarted: out/models/syn_v1/results.csv shows the epoch timer resetting at epoch 25, and out/eval/logs/E0.log records Resuming training … from epoch 25 to 30 total epochs, i.e. the run was resumed from last.pt. The driver measures only the subprocess it launched, so the 928.1 s recorded in summary.json covers the final resumed segment (epochs 25–30, 6 epochs completed in 0.251 hours in the log). The full picture from results.csv is 3 510.2 s for epochs 1–24 plus 897.7 s for epochs 25–30 ≈ 4 408 s of training time in total. These are training timers (Ultralytics' own per-epoch clock), not wall-clock from first command to last: whatever elapsed between the two segments — the interruption itself — is in neither figure. E2/E3/E4 ran straight through with no resume, and for them the driver's number and Ultralytics' own (1 492.7 / 1 466.5 / 5 209.7 s) differ only by model load plus final validation.

One metric in the table means nothing here

report.json reports fp_per_image: 0.000 for E1–E4. That is an artifact, not a result: it is measured on negative (drone-free) frames, and DUT test contains zero negative frames (negatives.images: 0 in all four reports). E0's fp_per_image: 0.000 over 54 synthetic negatives is real; the real-data ones should be ignored entirely. The false-positive story on DUT has to be told a different way — see §4.

3. Verdict against spec §8

Gate 1 — E1 ≥ 0.5 mAP50 on real data: FAILED. 0.375 against a 0.5 threshold (Anti-UAV only — the Drone-vs-Bird half of spec §8's real-data test is pending the founder's access request, §5). This is not a rounding problem or a threshold-tuning problem; at mAP50-95 the synthetic-only model scores 0.195, and on everything below ~32 px long side it is effectively blind (small 0.201, tiny 0.038). A detector trained purely on AeroBench Detect v1 is not usable on this real dataset.

Gate 2 — E2 > E3: the strict metric points that way, the loose one does not.

	E2 (syn → 10 % real)	E3 (COCO → 10 % real)	Δ (E2 − E3)
mAP50	0.7605	0.7590	+0.0015
mAP50-95	0.4682	0.4340	+0.0342
tiny mAP50	0.2416	0.1559	+0.0856
tiny mAP50-95	0.0242	0.0248	−0.0007
small mAP50-95	0.3548	0.3007	+0.0542
medium mAP50-95	0.4301	0.4053	+0.0249
large mAP50-95	0.5908	0.5485	+0.0423

Read honestly:

• The mAP50 difference (+0.0015) is noise. These are two single-seed 20-epoch runs; a gap of 0.0015 on a 2 245-box test set is not a result. Anyone quoting "synthetic pre-training raises mAP50" from this table is misreading it.
• The mAP50-95 difference (+0.034) is the larger of the two gaps, and it keeps the same sign in three of four size buckets (+0.025 to +0.054), reversing only in the 15-box tiny bucket (−0.0007). That is one seed per arm and therefore no error bar on either number: suggestive, not established. mAP50-95 rewards tight boxes, so the reading it suggests is that synthetic pre-training improved localization quality — plausible, since synthetic boxes are derived from sprite alpha and are pixel-exact — rather than improving what gets found at all. Confirming it needs several seeds per arm, which this experiment did not run.
• The tiny mAP50 gap (+0.086) is the weakest evidence in the table, not the strongest. Fifteen boxes. It also reverses sign at mAP50-95. It is consistent with the "helps most where objects are small" story, but it does not establish it.
• At an operating threshold of conf 0.25 (greedy matching, IoU 0.5, our own count over the prediction files) the two models trade off rather than one dominating: E2 precision 0.757 / recall 0.729; E3 precision 0.830 / recall 0.694. E2 finds more and is wrong more often.

The ceiling matters more than either. E4 — the same architecture on the full 5 200-image DUT train set, no synthetic data at all — reaches 0.912 / 0.580, i.e. +0.152 mAP50 and +0.112 mAP50-95 over E2, and +0.293 on tiny mAP50. Ten times the real labels beats anything the synthetic set contributed. Synthetic pre-training is not a substitute for real data; on this evidence it is a modest label-efficiency aid that helps most when labels are scarce and boxes need to be tight.

4. Where E1 breaks down

By size

E1's mAP50 falls off a cliff as objects shrink: large 0.458 → medium 0.381 → small 0.201 → tiny 0.038. At mAP50-95 the small bucket is 0.043 and tiny is 0.007 — which means that even where a small object is found, the box is nowhere near tight. This is the opposite of E0's profile on synthetic val (tiny 0.706, small 0.929), so it is not that the model cannot detect small objects in principle; it cannot detect these small objects, in real imagery.

Confidence is spread across the frame

At the evaluation threshold (conf 0.001), the synthetic-only model emits 235 584 boxes for 2 245 real objects — 107 per image. The same protocol gives E4 25 670 boxes (11.7 per image), E2 69 961 and E3 89 529. Low-confidence boxes are normal and expected under this protocol; a 9× spread between models under an identical protocol is not. The syn-only model is diffusely uncertain about real scenes in a way the real-trained models are not.

At a usable operating point (conf 0.25, IoU 0.5, greedy matching) E1 produces 6 297 boxes across the test set: TP 1 187, FP 5 110, FN 1 058 → precision 0.189, recall 0.529. Roughly four out of five confident detections are wrong, and roughly half the drones are missed.

Three examples

The three frames below are DUT Anti-UAV test images. The annotated figures are not redistributed — the dataset states no licence for redistribution and we use it for evaluation only — so each case is described in words and in numbers instead. To reproduce them locally: fetch and convert the DUT test split with the snippet in §6.1 below, run E1 (python scripts/run_transfer_experiments.py --only E1), then draw out/eval/e1_syn_only/preds.json (red, conf ≥ 0.25) and assets/real/dut/test_coco.json (green) onto the named frame with cv2.rectangle. Our own copies live under the git-ignored out/figures/.

a) Small drone, nothing fires at all — img/00890.jpg

A 23 × 10 px quadcopter against bright sky, framed by trees and a building edge. Zero predictions clear conf 0.25 anywhere in the frame; the highest-scoring box in the whole image is 0.19. This is the small/tiny collapse in its plainest form — not a mislocalization, an absence.

b) The drone is missed, the architecture is detected — img/01986.jpg

Hazy overcast, red-brick campus building. The real drone (green, 25 × 11 px, low against the facade) draws no confident prediction. Eleven boxes at conf ≥ 0.25 land on balcony railings, roof structures and — at 0.58 — on a 408 × 301 px slab of building. Detect v1's backgrounds are HDRI panoramas with drones composited over them and no distractor objects at all; the model has never had to learn that a railing is not a multirotor.

c) Near-field drone boxed as fragments — img/01124.jpg

A large white multirotor filling most of the frame: GT is 865 × 305 px. The model's best box (0.67) covers a 350 × 82 px slice of it — IoU 0.109, so it counts as both a miss and a false positive. Seven more boxes sit on rotor booms, landing-gear legs and the payload pod. The model has learned "a drone is a small compact blob" and, faced with one at close range, finds drone-sized pieces of it instead of the aircraft.

Under backlight

Added after the fact, from photometric tags derived for this same test split (see docs/benchmark/real-data-slices.md): on the 465 frames tagged backlit — bright sky over a dark foreground — E1 scores 0.223 mAP50 against 0.412 on the other 1 735, a 46 % relative drop. E4 loses 0.021 on the same split (0.896 vs 0.917). The synthetic-only model is specifically weak at silhouettes against a blown-out sky, which Detect v1's exposure-matched HDRI composites barely contain. Those numbers come from re-evaluating the same prediction files against a tagged copy of the ground truth; the overall and per-size figures in §2 are unchanged and were verified identical. The same page reports that DUT test carries only five frames a photometric rule calls night, so no time-of-day claim can be made on it.

Hypotheses

Ordered by how much of the gap we think each explains, all of them untested individually — this experiment separates synthetic-vs-real, not cause-by-cause:

1. No distractors. Detect v1 has zero birds, aircraft, poles, cables, railings or rooftop hardware. DUT is shot over real ground clutter. Example (b) is a direct illustration; the 4:1 false-positive rate at conf 0.25 is the aggregate version. We consider this the largest single contributor.
2. Scale and distance range mismatch. Detect v1's boxes are spread across all four buckets by construction (tiny 1 691 / small 2 353 / medium 2 048 / large 668), but "large" in the synthetic set means a composited sprite, not a drone at two metres filling the frame. Example (c) shows the model has no representation of a near-field airframe. In the other direction, real small drones sit in sensor noise and JPEG blocking that a clean composite never reproduces.
3. Procedural drone models vs real silhouettes. Detect v1's drones are procedurally generated, not CAD or photogrammetry: arm/prop geometry, body proportions, the payload gimbal, landing gear are all approximations. A model fitted to those shapes has learned a silhouette family that the real commercial airframes in DUT do not belong to.
4. Backgrounds. HDRI panoramas are smooth, low-frequency and captured for lighting, not for surveillance geometry. DUT frames have real camera optics and depth of field, and high-frequency vegetation and architecture behind the target.
5. Capture realism. No motion blur, no rolling shutter, no per-frame JPEG artifacts, no sensor noise in the synthetic set; real footage carries all of these to some degree, and DUT's frames are JPEGs.

The exposure fix already shipped in v1 (see docs/datasets/detect_v1.md) removed one obvious domain gap — overbright pasted-on sprites — and E1 is still 0.375. That is useful negative evidence: photometric matching alone is not the bottleneck.

What changes in v2

In descending expected value:

1. Distractors as a first-class part of the generator. Birds, fixed-wing aircraft, helicopters, poles, cranes, cables, rooftop hardware — rendered as unlabelled objects, plus far more negative frames containing them. Hypothesis 1 predicts this alone should move E1's precision the most.
2. Real drone geometry. Replace or supplement procedural models with real CAD / photogrammetric airframes across the classes DUT actually contains.
3. Explicit near-field coverage. Extend the scale schedule so drones from ~2 m (filling the frame) to long range are all represented, with correct perspective — not just sprite scaling.
4. Capture-realism augmentation at build time. Motion blur along the drone's own velocity vector, rolling-shutter skew, per-frame JPEG requantization, sensor noise matched to a real camera's ISO curve.
5. Real-photo backgrounds alongside HDRIs, for the high-frequency ground clutter panoramas do not provide.

The measurement to re-run after each of those is exactly E1 (mAP50 on DUT test, zero real training) with E2/E3 as the label-efficiency check. E1 is the number that tells us whether the synthetic set is getting closer to reality; E2 − E3 tells us whether it is worth pre-training on.

Built and measured: 2026-09-detect-v2.md — items 1, 3 and 4 of the list above; items 2 and 5 were not done. E1v2 came out at 0.159, i.e. worse than the 0.375 below, so v1 remains the recommended synthetic set. An ablation there points at item 1 (the distractors) as the largest suspect, but the rebuild that removed the size-matched clutter recovered almost none of the loss, so that write-up's §5 marks the mechanism as unexplained. Nothing in the present write-up is restated or revised there; its numbers stand as measured.

5. Drone-vs-Bird: founder action required

The obvious next real-data benchmark is the WOSDETC Drone-vs-Bird Detection Challenge dataset, because it contains exactly the distractor class (birds) that hypothesis 1 says we are missing. It cannot be downloaded — access requires a signed data usage agreement, and the founder has to request it personally.

Verified on 2026-09-04: the organisers' repository at https://github.com/wosdetc/challenge states that a request should be sent to wosdetc@googlegroups.com, that the requester "will be asked to sign a data usage agreement", and that the data may then be used for research purposes. The challenge homepage https://wosdetc2024.wordpress.com/ also resolves (WOSDETC Drone-vs-Bird Detection Grand Challenge @ MLSP24).

The URL named in the original plan, https://wosdetc2024.wordpress.com/drone-vs-bird-detection-challenge/, returns HTTP 404 as of 2026-09-04 — the two links above are the verified replacements.

Nothing in that process can be automated on our side: it needs a named person, an institution, and a signature.

6. Reproducing this

Everything below assumes the repo root and the project venv with the training extra installed (uv pip install -e ".[dev,train]" — gdown and ultralytics live there; see the README for the torch/CUDA install order, which matters).

6.1 Fetch and convert DUT Anti-UAV

assets/real/dut/ is git-ignored and holds ~10 000 images across the three splits. There is no CLI wrapper for this step; it is three library calls per split, run from a Python shell at the repo root:

from pathlib import Path
from aerosynth.realdata import dut_antiuav

cache = Path("assets/real/dut")
for split in ("train", "val", "test"):
    split_dir = dut_antiuav.download_split(split, cache)   # gdown; no-op if already fetched
    pairs = dut_antiuav.find_pairs(split_dir)              # (image, VOC xml) matched by stem
    dut_antiuav.to_coco(pairs, cache / f"{split}_coco.json",
                        source_tag=f"dut_antiuav_{split}")  # tags["source"] per split
    dut_antiuav.to_yolo(pairs, cache / f"{split}_yolo")     # labels/*.txt + images.txt
dut_antiuav.write_attribution(cache, None)

That produces assets/real/dut/{train,val,test}_coco.json (the GT the evaluations read) and assets/real/dut/{train,val,test}_yolo/ (labels plus an images.txt of absolute image paths — no symlinks, which need privileges on Windows). The explicit per-split source_tag is what makes tags["source"] read dut_antiuav_test rather than a bare default, so merged splits stay distinguishable.

6.2 Build the three training splits

Deterministic given --seed 0; reproduces the image counts in §1's Splits table (4 500 / 500, 520 and 5 200 train images, all real runs validating on the same 2 600-image DUT val set):

python scripts/make_yolo_split.py out/detect_v1 out/splits/syn_v1 --val-frac 0.1 --seed 0 --single-class
python scripts/make_yolo_split.py assets/real/dut/train_yolo out/splits/dut10 --val-frac 0.0 --subsample 0.10 --seed 0 --val-list assets/real/dut/val_yolo/images.txt
python scripts/make_yolo_split.py assets/real/dut/train_yolo out/splits/dut_full --val-frac 0.0 --seed 0 --val-list assets/real/dut/val_yolo/images.txt

--single-class on the synthetic split rewrites Detect v1's three classes to one drone class; DUT's own labels are already single-class. --val-frac 0 plus --val-list is what makes the two real splits hold nothing back from their own pool and validate against the independent DUT val split instead.

6.3 Run the experiments and rebuild the page

python scripts/run_transfer_experiments.py            # all five
python scripts/run_transfer_experiments.py --only E1  # just the headline number
python scripts/build_site.py                          # local results page (site/, git-ignored)

Budget roughly 3.5 h of training on an RTX 3070 Ti (the times in §2 sum to ~12 770 s once E0's full 30 epochs are counted), plus inference over 2 200 images four times.

Outputs land in out/models/** and out/eval/**, both git-ignored: the model weights, prediction files and reports are not committed, only this write-up and its three example images. The driver writes one row per experiment to out/eval/summary.json, replacing that experiment's earlier row on a re-run. A summary produced before that behaviour existed can still hold duplicates — the current file has E1 twice, identically — so a reader should keep the last row per experiment.

Итог для основателя

Главное: модель, обученная только на синтетике, на реальных данных не работает. На тесте DUT Anti-UAV она даёт mAP50 = 0.375 при пороге 0.5 из спецификации — это не «чуть-чуть не дотянули», а провал с запасом. На объектах мельче ~32 пикселей она практически слепа (0.201 и ниже), а при рабочем пороге уверенности примерно четыре из пяти её срабатываний — ложные: она принимает за дроны перила балконов, крыши и куски зданий, потому что в нашем синтетическом наборе нет ни одного отвлекающего объекта.

Синтетика, похоже, не бесполезна, но пользы меньше, чем хотелось. Если взять эту модель как стартовую точку и дообучить на 10 % реальных данных (E2), она обгоняет такую же модель без синтетики (E3) на строгой метрике mAP50-95: 0.468 против 0.434. Выигрыш небольшой, и на каждую сторону у нас всего один прогон без доверительного интервала: похоже, что синтетика улучшает точность рамок, а не способность находить дроны, — но это указание, а не доказанный факт. По мягкой метрике mAP50 разница 0.7605 против 0.7590, то есть шум; выдавать её за результат нельзя. По «крошечным» объектам разрыв выглядит крупным (0.242 против 0.156), но он посчитан всего по 15 рамкам во всём тесте — это направление, а не измерение.

Реальные данные бьют всё. Та же архитектура на полном реальном наборе (E4, в 10 раз больше размеченных кадров) даёт 0.912 / 0.580 — на 0.152 mAP50 выше, чем синтетика плюс 10 % реальных. Вывод трезвый: синтетический набор — это не замена разметке, а инструмент экономии разметки, и пока экономия скромная.

Что делать дальше. Главная гипотеза — в наборе нет отвлекающих объектов (птиц, самолётов, столбов, перил) и нет дронов на близкой дистанции. Это и есть план v2 в порядке ожидаемой отдачи: отвлекающие объекты и негативные кадры → реальная геометрия дронов вместо процедурной → близкая дистанция → смаз, шум и артефакты сжатия → реальные фотофоны. Проверять после каждого шага одним и тем же числом — E1 на DUT.

Требуется ваше действие. Набор Drone-vs-Bird (тот самый, где есть птицы) скачать нельзя: нужно письмо на wosdetc@googlegroups.com и подписанное соглашение об использовании данных от вашего имени — см. https://github.com/wosdetc/challenge. Автоматизировать это невозможно.

Closing the synthetic-only question: three single-change ablations from Detect v1

docs/experiments/2026-09-v1-ablations.md

Date: 2026-09-05. Follows 2026-09-transfer-dut.md (v1: E1 = 0.375 mAP50 on DUT Anti-UAV test, zero real frames) and 2026-09-detect-v2.md (v2, all changes at once: 0.159). Question: does any single v2 ingredient, added alone to v1, move the zero-shot number towards the 0.5 gate?

Setup

Same protocol as E1 everywhere: build 5 000 frames from the v1 recipe with exactly one change, split 4 500 / 500 (make_yolo_split.py --val-frac 0.1 --seed 0 --single-class), train yolov8n 30 epochs (640 px, batch 32, seed 0, workers 0, yolov8n.pt init), predict on the DUT test split (2 200 images, conf 0.001), aerobench eval --merge-classes. One seed per arm. Configs are committed: configs/detect_v1n.yaml, configs/detect_v1c.yaml, configs/detect_v1g.yaml; the exact command sequence is run_arm in the ablation script quoted at the end.

Arm	The one change vs v1	Sprites
v1n	negative_fraction 0.10 → 0.20 (1 000 frames without drones)	assets/sprites_v1
v1c	capture realism on: velocity motion blur, sensor + shot noise, rolling shutter (p 0.25, 1–6 px), JPEG requantization (q 55–95)	assets/sprites_v1
v1g	small unlabelled clutter: distractor_prob 0.50, negative_distractor_prob 0.85, 1–3 per frame, 10–48 px (birds, far airliner, masts, rooftop hardware; no helicopter)	assets/sprites_v2 (drone sprites identical to v1)

v1g build: 5 000 frames, 500 negatives, 6 745 boxes, 2 665 frames carry clutter. Training times (Ultralytics timer): v1n 4 869 s, v1c 4 950 s, v1g 4 131 s.

Results — DUT Anti-UAV test, mAP50 (mAP50-95), per-bucket mAP50 by long side

Arm	overall	tiny (<16 px, 15 boxes)	small (16–32, 385)	medium (32–96, 1 087)	large (≥96, 758)	predictions
v1 (E1, reference)	0.375 (0.195)	0.038	0.201	0.381	0.458	235 584
v1n negatives ×2	0.380 (0.194)	0.030	0.243	0.385	0.449	189 794
v1c capture realism	0.287 (0.127)	0.099	0.180	0.280	0.375	382 089
v1g small clutter	0.238 (0.112)	0.143	0.264	0.265	0.277	139 515
v2 all changes (E1v2, for reference)	0.159 (0.063)	0.071	0.203	0.262	0.118	174 433

Sources: out/eval/e1v1n_syn_only/report.json, e1v1c_syn_only, e1v1g_syn_only; v1 and v2 rows from e1_syn_only and e1v2_syn_only.

Reading

1. No single ingredient moves the headline number up. Doubling negatives is neutral (+0.005, inside single-seed noise). Capture realism costs 0.088. Small clutter costs 0.137. v2, which combined them, lost 0.216. The ingredients are not merely "not helping": each of the two content changes hurts on its own, and the losses roughly add up.
2. Where the losses come from is consistent across arms: medium and large targets. Clutter alone takes large from 0.458 to 0.277 even though every clutter object is ≤ 48 px — so the v2 write-up's "oversized clutter" explanation is refuted a second time: clutter hurts large-target detection regardless of clutter size. The most likely mechanism is now a training-signal one (unlabelled objects that share texture and background statistics with drones push the detector towards conservatism on anything that is not a crisp silhouette), not a size-confusion one. This is a hypothesis; we did not test it.
3. Tiny and small targets improve under both content changes (tiny 0.038 → 0.099 / 0.143; small 0.201 → 0.264 with clutter). The DUT test split is dominated by medium and large boxes (1 845 of 2 245), so a trade that helps the 400 small boxes and hurts the 1 845 larger ones loses overall. On a test set dominated by tiny targets the ranking could differ; we have no such set.
4. The 0.5 zero-shot gate is closed for this generator family. Four builds (v1, v2, v2-rc, three v1 ablations) span 0.159–0.380; nothing approaches 0.5. Combined with 2026-09-transfer-dut.md (synthetic + 10 % real ≈ 10 % real alone), the product conclusion stands: synthetic data from this pipeline is not a substitute for real labels and is not, on this evidence, a measurable pre-training aid either. What the pipeline does deliver is a controlled, sliced test bench — which is what AeroBench now sells.
5. Caveats. One seed per arm; the tiny bucket has 15 boxes; the DUT split has no night/weather diversity to speak of (see docs/benchmark/real-data-slices.md); v1c doubled the number of low-confidence predictions (382 k vs 236 k), which is itself a sign the detector became less certain rather than more accurate.

What we would try next, if anyone asks for synthetic again

Only as paid, scoped experiments: (a) real drone geometry instead of procedural models; (b) real-photo backgrounds with the same clutter statistics as the target site; (c) mixing a small real labelled set into training and measuring the marginal value per 100 real frames. None of these is on the roadmap by default.

Reproduction

# configs: configs/detect_v1n.yaml, configs/detect_v1c.yaml, configs/detect_v1g.yaml
aerosynth build-detect configs/detect_v1n.yaml out/detect_v1n --sprites assets/sprites_v1 --no-extras
aerosynth build-detect configs/detect_v1c.yaml out/detect_v1c --sprites assets/sprites_v1 --no-extras
aerosynth build-detect configs/detect_v1g.yaml out/detect_v1g --sprites assets/sprites_v2
python scripts/make_yolo_split.py out/detect_<arm> out/splits/syn_<arm> --val-frac 0.1 --seed 0 --single-class
python scripts/train_yolo.py --data out/splits/syn_<arm>/data.yaml --epochs 30 --model yolov8n.pt --imgsz 640 --batch 32 --seed 0 --workers 0 --name syn_<arm> --project out/models
python scripts/predict_coco.py --weights out/models/syn_<arm>/weights/best.pt --gt assets/real/dut/test_coco.json --images-root assets/real/dut/test --out out/eval/e1<arm>_syn_only/preds.json --imgsz 640 --batch 32
python -m aerobench.cli eval --track detect --gt assets/real/dut/test_coco.json --pred out/eval/e1<arm>_syn_only/preds.json --out out/eval/e1<arm>_syn_only --merge-classes

Итог для основателя

Три проверки по одному изменению от v1 закрыли вопрос: больше негативных кадров — без эффекта (0,380 против 0,375); реализм съёмки — хуже (0,287); мелкий неразмеченный мусор — хуже (0,238), причём крупные цели страдают даже от мусора размером до 48 px, так что объяснение «слишком крупный мусор» из отчёта по v2 опровергнуто повторно. Мелкие цели при этом улучшаются, но их в тесте мало, и общий результат падает. Порог 0,5 без реальных данных для этого генератора недостижим; синтетика остаётся инструментом для контролируемого измерения, а не заменой разметки. Дальше синтетику делать только под оплаченный запрос клиента.