When the input queue is full, drop the frame's INFERENCE but keep the
1 submission : 1 result contract of AsyncDetectorRunner by emitting an
empty (20, 6) result for that connection_id (same pattern the producer
already uses when the buffer pool is starved). Raising here used to let a
burst of offered frames kill the detect worker and force a restart.
Test flipped: queue-full now asserts the zero-row drop instead of the raise.
Replace the pure-Python O(n^2) _nms_xyxy loop with the framework's own
C++ NMS (xyxy -> xywh first, same as frigate.util.model). Same contract:
kept indices in descending score order; empty-safe. Removes a per-frame
CPU hot spot on dense daytime frames without changing decode results.
Read the padded/original input shapes, the exact content offset inside the
padded input (n_padded_ch_inputs), per-output padded + real channel counts
and dequantize (scale, zero-point) from 1/manifest.json (fallback
quantized/manifest.json) instead of hardcoding YOLO-NAS constants, so the
decoder works for any compiled YOLO-family model shipping a postprocess
graph. Map each postprocess input to its model output by (C, H, W) shape
(the compile order of heads is not guaranteed to match the graph's input
order). Accept two postprocess output shapes: two outputs (scores + xyxy
boxes, YOLO-NAS) or one fused output [cx, cy, w, h, (objectness), C class
scores] in CENTER format (YOLOX/SDK DecodeYolo), converted to xyxy and
scaled by objectness. Verified on hardware against both YOLO-NAS M and
YOLOX-S: identical boxes on the same frames, YOLOX-S yields the same
bus/persons as the SDK stream reference.
Split the single worker into a producer/consumer pipeline: the producer
letterboxes + quantizes + runs the model into a pooled, ping-ponged buffer
set, the consumer dequantises + postprocesses the previous frame's outputs
while the NPU is already busy on the next one (all run() calls stay on the
one producer thread, so in-flight frames never execute the model in
parallel). Measured on the Metis: ~24.9 ms/frame (~40 fps) vs ~28.9 ms
serial, giving real headroom over the ~35 fps offered by 7 cameras.
resized - 128 in uint8 wraps mod 256, then astype(int8) yields exactly
px - 128 (verified for all 0..255, equivalent to the int16+clip path)
without the intermediate int16 array or clip pass.
The GStreamer-based create_inference_stream pipeline is depth-4 and clocked by
input arrival, so a frame's result round-trips in ~4x the input gap (~150 ms at
~35 fps) even though the accelerator computes a frame in ~21 ms back-to-back.
Switch to the vendor's direct ModelInstance.run() runtime API (the same
integration pattern as hailo8l/memryx): a synchronous run() with no video
pipeline, so per-frame latency is the flat model time (~21 ms measured for
YOLO-NAS M 704x576 at 1 instance), independent of scene activity.
- _AxeleraRuntimeInference: single ModelInstance plus a worker thread that pops
frames strictly FIFO, letterboxes + quantizes (int8, BGR->RGB), runs the
model, feeds the six int8 heads through the compiled model's own baked
postprocess_graph.onnx (dequant, crop per TensorInfo.padding[3], transpose)
and emits (class_id, score, box) rows after a conf threshold + NMS.
- Keeps the async contract (send_input/receive_output/shutdown); requests are
paired to results by construction through the FIFO queue, so no pixel tokens
and no keep-alives.
- Model resolution unchanged (preset under AXELERA_BUILD_ROOT, local path, or
URL); labels mapped by name via the merged labelmap.
AsyncDetectorRunner derived inference_speed from the send_input-to-receive_output
round-trip, which for stream-based detectors (memryx, axelera) is gated by the
SDK pipeline depth and the input cadence (latency ~= depth x input gap): at a
realistic ~35 fps aggregate it reads ~150-200 ms even though the accelerator
handles a frame in ~14 ms back-to-back, making a fast detector look ~10x slower
than sync detectors (which report a single detect_raw duration).
Derive inference_speed from result cadence instead: the rolling average of the
time between completed results equals 1000 / detection_fps, the per-frame
processing time, comparable across detectors. Idle gaps between results are not
folded in, so a quiet stream never inflates the reported speed. The per-send
timestamp deque is removed (it existed only for the old metric).
Replace the content-based pixel-token pairing with the reviewer's scheme:
the feeder registers every yielded frame on a pending deque (None for
primer/keep-alive synthetics, the connection id for real frames) and
collect() pops exactly one entry per result, so synthetics are dropped and
real results always reach the connection that owns them - without reading
the returned image at all.
Validated on hardware first: the SDK InferenceStream returns results
strictly in input order, one per yielded frame (200/200 frames in order,
no drops or duplicates, synthetics interleaved included), and the per-frame
image read (img.asarray + band decode) measured 0.58 ms against a 14.36 ms
back-to-back round-trip, i.e. not the driver of the System Metrics latency.
A desync guard raises (detector restarts) if the SDK ever emits a result
with no registered frame, instead of silently misattributing everything
after it (the P1 failure mode). The pixel-token path and its 200/200 FIFO
probe stay in git history for rollback.
next(stream) raises TypeError ('InferenceStream' object is not an
iterator), killing the result worker on the very first collect() (the
older sync pump used a for-loop). Grab iter(stream) once at init and
drive collect() through that iterator. Regression test: a stream that
is iterable-without-__next__ like the real SDK.
create_inference_stream consumes the source generator from the very
start; a blocking queue.get() in the feed left the source empty during
stream init, so the detector process hung before the async worker
threads started (no events). Re-add a sparse keep-alive zero (0.1 s)
that fires only while the incoming queue is momentarily empty and
carries no request token (dropped by collect); steady-state real
frames still clock the pipeline back-to-back via AsyncDetectorRunner.
The synchronous DetectorRunner path serialized run() and needed
20 Hz keep-alive zero frames just to keep the Voyager SDK pipeline
rotating: the SDK pipeline is clocked by the input arrival interval
(measured: 1 s gaps stretch a real frame's latency to ~4 s on the
depth-4 pipeline), and one-frame-in-flight caps multi-camera
throughput. AsyncDetectorRunner (as memryx) sends from the detect
thread and receives on the result thread, and closes the input SHM
buffer on the result side, so the frame copy in the plugin goes away.
- plugin implements send_input()/receive_output()/shutdown() and a
detect_raw() stub (memryx precedent); content-based pixel-token
pairing is preserved (primer results carry no token and are dropped).
- object_detection/base.py: route axelera to AsyncDetectorRunner.
With real frames clocking the pipeline, no keep-alives are needed;
the throughput ceiling imposed by one frame in flight is gone.
is_url() required an http(s) scheme, so a bare host like
www.example.com/model.axm fell through to the filesystem branch and
raised FileNotFoundError, blocking detector init on a valid remote
model (reviewer P1). A value matching a leading dotted host (or
localhost) with a path is now recognised as a URL and http:// is
prepended at fetch time; relative paths and preset names behave as
before.
The SDK pipeline is clocked by the input arrival interval: measured ~4 s
real-frame latency at 1 s keep-alive spacing vs ~58 ms back-to-back
(depth-4 pipeline x arrival gap). Gating keep-alives while a request is
outstanding also starved the SDK input source (it waited instead of
yielding) and broke the stream. Short 50 ms keep-alives keep the ring
rotating, so real frames get results in tens of ms; the zeros carry no
request token and are dropped by the content-based pump.
Keep-alive zeros injected during a pending request queue in the SDK
pipeline ahead of the real frame and delay its result by one slot each:
observed 3 s real-frame latency with 1 s keep-alive spacing vs 14 ms
for back-to-back real frames (and 10 ms delivered by the FIFO path).
The feeder now skips keep-alives whenever a request id is still
outstanding, so real frames are processed back-to-back and true-idle
behaviour (stream kept alive) is preserved.
The SDK does not tag results with a per-frame id (image_id is always
empty, src_timestamp is a pipeline-clock value), so pairing relied on
FIFO ordering, which a buffered primer/keep-alive result could break:
the stale result would pop the pending id and be delivered to the
wrong camera/crop (synthetic results consuming real IDs).
Every real frame now carries a 32-bit request token written into a
small top-left pixel band; the SDK returns the source frame back 1:1
(verified on hardware: BGR pass-through exact, BGRA), so the pump
decodes the token from the result image and delivers the outcome only
to the frame that carries it (content-based, no ordering assumption).
Primer/keep-alive frames carry no token; their results are dropped
even when they surface while a request is pending.
The token-based 1:1 pairing (None token per synthetic frame) is correct in
theory but regressed in production: inference latency went to ~3s and
detections dropped to ~0. Revert to the stable approach that has been
running fine (lock + FIFO + keep-alive): ~7ms inference, detections flowing.
P1 (synthetic results stealing real ids) remains a theoretical concern
under this scheme, but the observed production behavior with it was far
worse. Investigate the regression with per-frame diagnostics before
re-attempting.
Misattribution reproduced at night: when Frigate stops sending frames
during quiet periods the queue empties and keep-alive frames are emitted;
a pending real-frame result then races the keep-alive's (empty) result,
which steals the real id (frame/detections shown under the WRONG camera).
Fix: synthesize keep-alive frames ONLY until the first stream result
(_primed), then never again - no unregistered result can exist in the
pipeline afterwards. Queue-empty after init just waits; under Frigate's
continuous feed this branch is only hit during true quiet periods, where
there is no pending request to misattribute anyway.
The primer 'drain by run index' was too fragile under Frigate's real
workload: with 7 cameras starting simultaneously the first few real frames
arrive before the SDK finishes emitting primer results, their ids were
drained without a result, run() hit the timeout and Frigate restarted the
detect process every ~15s (zero detections).
Revert to the stable approach: keep-alive frames sustain the SDK during
quiet/init, _pump never consumes a pending id while the queue is empty, and
run() is serialized (lock) so at most one real frame is in flight. The
remaining theoretical primer/keep-alive misattribution window only exists
when the queue is empty mid-run, which does not happen under Frigate's
continuous feed.
The SDK can buffer primer results and emit them AFTER the first real frame
is already pending; the FIFO pump then popped a real id and stored the
skipped/buffered (empty) result under it, dropping the real one. This
misattributes detections across cameras even with run() serialized.
- eliminate keep-alive frames entirely (Frigate feeds the queue
continuously; nothing to keep alive) - a synthetic frame's result can
arrive during a pending real id,
- drain the primer results by index BEFORE any pending-id bookkeeping, so
they can never consume a request id; the first run() waits up to 0.5s
for the primer drain.
Voyager SDK does not tag results with a per-frame id and can reorder or
buffer stream results under sustained load (observed on live with many
thousands of frames: camera A started receiving boxes/paths from camera B).
With several frames in flight, the FIFO pairing can store a result under a
different frame's id.
run() now holds a lock so at most one frame is in flight; the next stream
result can only belong to the frame just pushed, making pairing exact
regardless of SDK reordering. Pipeline throughput stays far below limits
(~9ms per inference vs 28ms available per frame at 7x5fps).
- strict per-frame request/result pairing: the feeder registers a queue of
pending frame ids, the pump pops one per stream result and stores the
outcome under it, and run() waits for its own frame's id. Primer and
keep-alive results are dropped instead of being served to callers.
- copy frames out of shared memory before queuing them (detect_raw returns
before the feeder consumes the frame, unmapping the camera buffer).
- raise on inference timeout so the watchdog can restart a stalled stream
instead of silently serving zeros.
- use ModelDownloader.download_from_url for URL models (.part file, atomic
rename, progress events); drop the unused AX_DEFAULT_URL constant.
- is_url() requires a scheme; treat www.-style values as paths and fail with
a clear FileNotFoundError instead of a deep urlretrieve error, mirroring
hailo8l behavior for missing model files.
- guard detections against numpy arrays, drop the dead score initialization,
floor normalized box coords at 0, move the model-required check before
super().__init__(), and document detect_raw/close.
- deduplicate the model-compilation walkthrough in docs (keep installation.md
as the single source; object_detectors.md links to it).
The Metis has no utilization readout without the axsystemserver broker,
which cannot run alongside a live detector. Report the worker busy time
(fps * inference time) as a real load indicator instead, plus the board
temperature, under npu_usages[axelera].
Exposes the Metis temperature read (axcmd --board-temp) under
npu_usages so the /api/stats endpoint lists the NPU instead of
returning null. There is no utilization readout without the
axsystemserver broker, which cannot run alongside a live detector.
Use the SDK's axcmd --board-temp through the board controller, which is
safe to call while an inference stream is active. Matches the hailo8l
pattern of exposing temperature via detectors.*.temperature.
Memory audit: all hot-path structures are bounded (frame queue maxsize 64,
latest result is a single 20x6 buffer, rows capped at MAX_ROWS), so there is
no unbounded growth in steady state.
One resource hazard: run() used a blocking queue.put() with no guard. If the
pump thread dies (SDK stream exception, e.g. DEVICE_LOST), the SDK side stops
draining the queue, the bounded queue fills, and every subsequent run() blocks
forever on put, permanently wedging the detector (Frigate does not restart the
detector process on its own).
Fix: check thread liveness before put and raise immediately, and use put_nowait
(backpressure: drop the frame on a full queue) so run() always falls through to
the latest-result wait / timeout instead of blocking.
* installation.md: replace the hand-waved 'custom image' note with a
reference Dockerfile that installs the Voyager SDK into the image
(apt deps, axelera-rt/devkit from the Axelera index, make operators)
and keeps the /dev/metis passthrough instructions.
* object_detectors.md: document axelera model sourcing under
'Downloading YOLO Models' (model-zoo preset, axzoo/axcompile, HF
hosting, default model auto-download).
Other detector plugins shipping in this tree do not carry unit tests, and
the fixed-sleep timing in these tests (primer, keepalive, pump teardown)
is prone to flaking on CI runners. The regression coverage stays out of
tree beside the detector host worker.
Frigate detect() interprets detector boxes as fractions of the region and
discards any box resolving outside the frame. The plugin returned pixel
coordinates, so every detection was silently filtered out and no events
were ever created despite a working inference chain. Normalize by model
width/height (same contract as memryx/onnx). Live: 0 events -> 27 events
in ~70 s after the fix.
Per-frame request/result pairing (RequestStore/ResponseStore FIFO)
desynchronises on the live chain once keep-alive results interleave with
real frames: a request could receive a zero-frame result, and real results
got attributed to the wrong frame (recurring identical scores across
different frames in production). The pump now keeps the most recent
completed result and run() waits for a result newer than the sequence at
call time, then returns it. Frigate correlates objects across frames, so
per-frame identity is not needed. Worker engine mirrored.
The SDK returns detection boxes as numpy arrays on some frames; the
'list(box or [])' idiom evaluated array truthiness and raised ValueError,
killing the pump thread mid-detection. Extract boxes defensively. Live
repro: worker crashed the moment a real detection fired, so no event was
ever recorded.
Frigate feeds detect_raw() a batched NHWC frame (1, H, W, C); the plugin
raised ValueError for ndim != 3, so every real inference failed. Squeeze
the leading batch dim when present (same contract as other IPU plugins,
e.g. memryx). Regression test added.
Community plugin for the Axelera AI Metis PCIe/M.2 NPU, following the same
conventions as axengine.py / hailo8l.py (lazy SDK import,
RequestStore/ResponseStore, model path as URL/local-file/model-zoo preset,
type_key auto-registration).
Inference runs through a single Voyager SDK stream
(create_inference_stream(network=..., sources=[...])) fed with numpy frames;
detection labels are mapped to class ids by name via the merged labelmap.
Compiled .axm models can be downloaded into the model cache from a URL, used
from a local path, or omitted to fall back to the yolov8n-coco zoo preset.
Includes unit tests (unittest, no SDK/hardware required) covering the engine
round-trip, timeout, dead-stream and orphan behaviour plus model resolution,
and docs for the object detector page and the "Extra steps for specific
hardware" installation page.
* subscribe to add in webpush
* add docs for detector cpu usage
* rebuild notification camera access when a camera is added at runtime
* document how frigate shows CPU usage metrics
* add faq about version key in config
sanitize_filename leaves ".." intact and collapses variants like "..:" and "..*" to "..", so filesystem paths built from face names, classification model/category names, image ids, and trigger data could escape their base directory. Route every such site through new frigate/util/path.py helpers (safe_join, sanitize_path_component, sanitize_contained_path), which reject traversal and verify containment.
Worst case was DELETE /classification/{name}, which rmtree'd /media/frigate and /config while returning 200.
Important to note that all affected endpoints already require admin permission, so this sould be considered hardening rather than fixing exploitable code.
<camera>/notifications/suspended arrives as a string over the live connection but as a number in the camera_activity snapshot, and the truthiness guard dropped the numeric 0, so a camera with notifications off rendered as active after a reload. Normalize to a string and derive isSuspended instead of storing it.
* fix(audio): correct sodeling typo to yodeling
Fixes a typo in audio-labelmap.txt where the yodeling class was
misspelled as "sodeling".
* fix(i18n): remove duplicate sodeling key in en audio.json
The en audio.json already contains a correct "yodeling" key. Remove
the duplicate/misspelled "sodeling" entry to avoid ambiguity.
* add host npu requirements to docs
* allow toggling live audio transcription via mqtt
* improve spacing consistency on mobile drawers
* fix clearing the region grid not surviving a restart
