nik/frigate
1
0
Fork 0
mirror of https://github.com/blakeblackshear/frigate.git synced 2026-10-18 07:29:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Expanding PPU support for DEEPX and set yolo-generic as default.

Browse Source
This commit is contained in:
aahmetdurmaz 2026-09-10 17:25:51 +03:00
parent ca85445111
commit 91f7af8ab8
5 changed files with 468 additions and 85 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
docs/data/object_detectors_models.yaml
View File

@ -988,9 +988,37 @@ memryx:
deepx:
title: DEEPX NPU
models:
- key: yolo
label: YOLO
recommended: true
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration.
ui: |-
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------- |
| **Custom object detector model path** | `/config/model_cache/deepx/model.dxnn` |
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
| **Object detection model input width** | `640` |
| **Object detection model input height** | `640` |
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
| **Model Input D Type** | `int` (Frigate's default value) |
| **Object Detection Model Type** | `yolo-generic` |
The width and height must match the resolution the `.dxnn` file was compiled for. Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
yaml: |-
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/model.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolo-generic
width: 640
height: 640
- key: damoyolo
label: DAMO-YOLO (TinyNAS-L20T / S / M / L)
recommended: true
recommended: false
download: No model is bundled with Frigate. Download a pre-compiled DAMO-YOLO `.dxnn` model (TinyNAS-L20T, S, M, or L size) from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. All four sizes share one detection head, so nothing else has to be told apart.
ui: |-
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
@ -1016,34 +1044,6 @@ deepx:
model_type: damo-yolo
width: 640
height: 640
- key: yolo
label: YOLO
recommended: false
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and anchor-free PPU models need no extra configuration. PPU models compiled from an anchor-based head are not supported.
ui: |-
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
| Field | Value |
| ---------------------------------------- | -------------------------------------- |
| **Custom object detector model path** | `/config/model_cache/deepx/model.dxnn` |
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
| **Object detection model input width** | `640` |
| **Object detection model input height** | `640` |
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
| **Model Input D Type** | `int` (Frigate's default value) |
| **Object Detection Model Type** | `yolo-generic` |
The width and height must match the resolution the `.dxnn` file was compiled for. Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
yaml: |-
models:
- devices:
- deepx:PCIe:0
path: /config/model_cache/deepx/model.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: yolo-generic
width: 640
height: 640
tensorrt:
title: TensorRT
models:

8
docs/docs/configuration/object_detectors.md
View File

@ -657,8 +657,8 @@ The DX-RT Python bindings are not part of the Frigate image. They are downloaded
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
- `damo-yolo` for DAMO-YOLO models, the recommended choice. All four sizes (TinyNAS-L20T, S, M, L) are supported.
- `yolo-generic` for YOLO object detection models. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do anchor-free models compiled with DEEPX's Post-Processing Unit (PPU) support. PPU models compiled from an anchor-based head are not supported; compile those without PPU support.
- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
- `damo-yolo` for DAMO-YOLO models. All four sizes (TinyNAS-L20T, S, M, L) are supported.
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO and DAMO-YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler.
@ -668,12 +668,12 @@ models:
- deepx:PCIe:0
path: /config/model_cache/deepx/model.dxnn
labelmap_path: /labelmap/coco-80.txt
model_type: damo-yolo
model_type: yolo-generic
width: 640
height: 640
```
`model_type` must be set to `damo-yolo` or `yolo-generic` to match the model. Frigate defaults it to `ssd`, which this detector does not support, so a model that leaves it unset is rejected at startup.
`model_type` must be set to `yolo-generic` or `damo-yolo` to match the model; `yolo-generic` is the recommended default unless the model is specifically a DAMO-YOLO export. Frigate defaults it to `ssd`, which this detector does not support, so a model that leaves it unset is rejected at startup.
`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.

21
docs/docs/frigate/hardware.md
View File

@ -267,7 +267,9 @@ Frigate supports the DEEPX NPU in both of its form factors: the **DX-M1** M.2 mo
DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
The NPU runs models compiled to the `.dxnn` format with DEEPX's DX-COM compiler. Pre-compiled YOLO and DAMO-YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Anchor-free YOLO models compiled with Post-Processing Unit (PPU) support move candidate selection onto the NPU, which reduces host CPU usage; PPU models compiled from an anchor-based head are not supported.
The NPU runs models compiled to the `.dxnn` format with DEEPX's DX-COM compiler. Pre-compiled YOLO and DAMO-YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Models compiled with Post-Processing Unit (PPU) support move candidate selection onto the NPU, which reduces host CPU usage; both anchor-based and anchor-free PPU heads are supported.
Exception: **YOLOv7 PPU models are not supported**. A PPU record has no anchor sizes, only a grid position, so Frigate decodes anchor-based PPU heads against a fixed anchor table; YOLOv7's anchors differ, so it decodes to wrong box sizes. Non-PPU YOLOv7 is unaffected (its raw head already outputs pixel-space boxes).
The DEEPX kernel driver, the DX-RT runtime, and the `dxrtd` daemon all run on the Docker host rather than inside the Frigate container, and have to be installed there before the NPU can be used. Frigate connects to the daemon over its socket, so the NPU stays available to other programs on the host at the same time. See the [installation docs](installation.md#deepx-npu) for the setup steps.
@ -285,7 +287,22 @@ The DEEPX ModelZoo publishes pre-compiled `.dxnn` files for many YOLO variants a
These are DEEPX's own figures for the DX-M1, derived from the frames per second published in the ModelZoo. They cover the NPU alone and exclude the pre- and post-processing Frigate does on the host, so the inference speed Frigate reports will be higher, and noticeably so on a slower host such as a Raspberry Pi 5.
Besides the DAMO-YOLO family shown above, other models from the wider YOLO family published in the ModelZoo are also supported by the `deepx` detector, though inference-time measurements have not been carried out as thoroughly across all of them as for the models in the table. These YOLO models can be run in Frigate by setting `model_type: yolo-generic`.
Besides the DAMO-YOLO family shown above, other models from the wider YOLO family published in the ModelZoo are also supported by the `deepx` detector by setting `model_type: yolo-generic`. Inference times for these, measured through Frigate's own stats API on a DX-M1 rather than taken from DEEPX's published figures, are below:
| Model | Input Size | DX-M1 Inference Time (via Frigate) |
| ------------------ | ---------- | ----------------------------------- |
| YOLOv3-tiny (PPU) | 416 | ~ 6 ms |
| YOLOX-S (PPU) | 640 | ~ 13 ms |
| YOLOv3 (PPU) | 416 | ~ 16 ms |
| YOLOv9-t (PPU) | 640 | ~ 18 ms |
| YOLOv4 (PPU) | 512 | ~ 20 ms |
| YOLOv3 (PPU) | 608 | ~ 28 ms |
| YOLOX-S | 640 | ~ 34 ms |
| YOLOv9-s | 640 | ~ 39 ms |
Unlike the DAMO-YOLO table above, these figures include Frigate's own pipeline overhead (host-side pre/post-processing and the PCIe/IPC round trip), not just raw NPU compute time, so they are not directly comparable to DEEPX's published numbers. Other variants from the ModelZoo not listed here are also supported but have not been measured.
**PPU support is strongly recommended whenever a model offers it.** Moving candidate/box selection onto the NPU's Post-Processing Unit substantially cuts host-side CPU work and inference time for the same architecture and input size: YOLOX-S measured ~34 ms without PPU versus ~13 ms with PPU above, roughly a 2.5x difference from PPU alone. Prefer a `_ppu`-suffixed model whenever one is available for the architecture you want.
:::note
SSD models in the ModelZoo, trained on Pascal VOC rather than COCO, so their labels do not match the object vocabulary used by `objects.track` and the rest of Frigate. The `deepx` detector has no SSD decoder and rejects `model_type: ssd` at startup. Use a DAMO-YOLO or YOLO model instead.

209
frigate/detectors/plugins/deepx.py
View File

@ -40,9 +40,34 @@ SUPPORTED_MODEL_TYPES = (ModelTypeEnum.yologeneric, ModelTypeEnum.damoyolo)
# (uint8), score (float32), label (uint32), 4 bytes of padding.
PPU_RECORD_SIZE = 32
PPU_BOX_BYTES = (0, 16)
PPU_GRID_BYTES = (16, 20)
PPU_SCORE_BYTES = (20, 24)
PPU_LABEL_BYTES = (24, 28)
PPU_ANCHORS_BY_SCALES = {
2: np.array(
[
[[23, 27], [37, 58], [81, 82]],
[[81, 82], [135, 169], [344, 319]],
],
dtype=np.float32,
),
3: np.array(
[
[[10, 13], [16, 30], [33, 23]],
[[30, 61], [62, 45], [59, 119]],
[[116, 90], [156, 198], [373, 326]],
],
dtype=np.float32,
),
}
PPU_MAX_STRIDE = 32
# the largest scale count we have a table for; below this, a resolved
# scale_count is under-evidenced rather than disproven, so it still gets
# the biggest table as a best guess
PPU_MAX_KNOWN_SCALES = max(PPU_ANCHORS_BY_SCALES)
# A nms-in-head output is (N, 6) rows of x_min, y_min, x_max, y_max, score,
# class, with N decided per frame and capped in the head (300 for the
# ModelZoo exports). A raw head has thousands of rows, which is what tells
@ -368,9 +393,9 @@ def ppu_records(outputs: list[np.ndarray]) -> np.ndarray | None:
def ppu_boxes_are_anchor_based(boxes: np.ndarray) -> bool | None:
"""An anchor-based head leaves raw sigmoids in the box fields, an
anchor-free head pixels. A value above 1 settles anchor-free, a frame
strictly inside 0..1 settles anchor-based, anything else is None."""
"""A value above 1 settles anchor-free (raw sigmoids stay inside 0..1),
a frame strictly inside 0..1 settles anchor-based, anything else is
None."""
if len(boxes) == 0:
return None
@ -396,6 +421,88 @@ def ppu_layout_is_anchor_based(outputs: list[np.ndarray]) -> bool | None:
return ppu_boxes_are_anchor_based(boxes)
def ppu_needs_grid_decode(records: np.ndarray) -> bool:
"""Whether the layer field is real pyramid-level info (YOLOX-style,
still grid-relative) rather than a constant, unused value."""
layer_idx = reinterpret(records, PPU_GRID_BYTES, np.uint8)[:, 3]
return len(np.unique(layer_idx)) > 1
def resolve_scale_count(records: np.ndarray, scale_count: int | None) -> int:
"""`scale_count` if given, else the layer count this frame alone proves,
defaulting to 3 when the frame has no records to read it from."""
if scale_count is not None:
return scale_count
layer_idx = reinterpret(records, PPU_GRID_BYTES, np.uint8)[:, 3]
return int(layer_idx.max()) + 1 if len(layer_idx) else 3
def ppu_grid_regression_geometry(
records: np.ndarray, boxes: np.ndarray, scale_count: int
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""Pixel centre and size for a YOLOX-style grid-relative box: centre
offset from the grid cell, size a log-scale multiple of the stride.
`scale_count` picks the stride each layer_idx was measured in."""
grid = reinterpret(records, PPU_GRID_BYTES, np.uint8)
grid_y = grid[:, 0].astype(np.float32)
grid_x = grid[:, 1].astype(np.float32)
layer_idx = grid[:, 3].astype(np.int32)
known = layer_idx < scale_count
layer = np.where(known, layer_idx, 0)
stride = (PPU_MAX_STRIDE >> (scale_count - 1 - layer)).astype(np.float32)
centre_x = (boxes[:, 0] + grid_x) * stride
centre_y = (boxes[:, 1] + grid_y) * stride
box_w = np.exp(boxes[:, 2]) * stride
box_h = np.exp(boxes[:, 3]) * stride
return centre_x, centre_y, box_w, box_h, known
def ppu_anchor_geometry(
records: np.ndarray, boxes: np.ndarray, scale_count: int
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""Pixel centre and size for anchor-based PPU records, with the mask of
records the anchor table covers. `scale_count` picks the table and
stride; below the largest table it is under-evidenced, not disproven,
so it still gets that table as a best guess, but above it every record
is left unplaced instead of guessing at the wrong table."""
grid = reinterpret(records, PPU_GRID_BYTES, np.uint8)
grid_y = grid[:, 0].astype(np.float32)
grid_x = grid[:, 1].astype(np.float32)
box_idx = grid[:, 2].astype(np.int32)
layer_idx = grid[:, 3].astype(np.int32)
anchor_table = PPU_ANCHORS_BY_SCALES.get(scale_count)
if anchor_table is None:
if scale_count > PPU_MAX_KNOWN_SCALES:
# a real head proven to have more scales than any table covers
zeros = np.zeros(len(layer_idx), np.float32)
return zeros, zeros, zeros, zeros, np.zeros(len(layer_idx), dtype=bool)
anchor_table = PPU_ANCHORS_BY_SCALES[PPU_MAX_KNOWN_SCALES]
levels, per_level = anchor_table.shape[:2]
known = (layer_idx < levels) & (box_idx < per_level)
# index the table with the out-of-range rows folded onto a real entry,
# then let the mask drop them rather than raising here
layer = np.where(known, layer_idx, 0)
anchors = anchor_table[layer, np.where(known, box_idx, 0)]
stride = (PPU_MAX_STRIDE >> (levels - 1 - layer)).astype(np.float32)
centre_x = (boxes[:, 0] * 2.0 - 0.5 + grid_x) * stride
centre_y = (boxes[:, 1] * 2.0 - 0.5 + grid_y) * stride
box_w = (boxes[:, 2] * 2.0) ** 2 * anchors[:, 0]
box_h = (boxes[:, 3] * 2.0) ** 2 * anchors[:, 1]
return centre_x, centre_y, box_w, box_h, known
def decode_ppu(
outputs: list[np.ndarray],
width: int,
@ -403,11 +510,17 @@ def decode_ppu(
score_threshold: float,
nms_threshold: float,
anchor_based: bool | None = None,
) -> np.ndarray | None:
"""Decode anchor-free PPU records (pixel cx, cy, w, h, score, label),
leaving only NMS. None for anchor-based records, whose anchor table is
not in the model; `anchor_based` skips the heuristic once settled."""
records = ppu_records(outputs)
scale_count: int | None = None,
needs_grid_decode: bool | None = None,
records: np.ndarray | None = None,
) -> np.ndarray:
"""Decode PPU records, leaving only NMS. `anchor_based` and
`needs_grid_decode` skip their heuristics once settled by a caller
tracking them across frames; `scale_count` picks the anchor table or
stride, defaulting to what this frame alone shows; `records` reuses an
already-parsed buffer instead of re-parsing `outputs`."""
if records is None:
records = ppu_records(outputs)
if records is None:
return np.zeros((20, 6), np.float32)
@ -417,9 +530,6 @@ def decode_ppu(
if anchor_based is None:
anchor_based = ppu_boxes_are_anchor_based(boxes)
if anchor_based:
return None
if anchor_based is None:
# nothing conclusive in this frame; no pixel box to draw either
return np.zeros((20, 6), np.float32)
@ -427,10 +537,29 @@ def decode_ppu(
scores = reinterpret(records, PPU_SCORE_BYTES, np.float32).flatten()
labels = reinterpret(records, PPU_LABEL_BYTES, np.uint32).flatten()
box_w = boxes[:, 2]
box_h = boxes[:, 3]
x_min = boxes[:, 0] - box_w * 0.5
y_min = boxes[:, 1] - box_h * 0.5
if anchor_based:
scale_count = resolve_scale_count(records, scale_count)
centre_x, centre_y, box_w, box_h, known = ppu_anchor_geometry(
records, boxes, scale_count
)
# a record the table cannot place is dropped by the score filter
scores = np.where(known, scores, 0.0)
else:
if needs_grid_decode is None:
needs_grid_decode = ppu_needs_grid_decode(records)
if needs_grid_decode:
scale_count = resolve_scale_count(records, scale_count)
centre_x, centre_y, box_w, box_h, known = ppu_grid_regression_geometry(
records, boxes, scale_count
)
scores = np.where(known, scores, 0.0)
else:
centre_x, centre_y = boxes[:, 0], boxes[:, 1]
box_w, box_h = boxes[:, 2], boxes[:, 3]
x_min = centre_x - box_w * 0.5
y_min = centre_y - box_h * 0.5
order = run_nms(x_min, y_min, box_w, box_h, scores, score_threshold, nms_threshold)
@ -686,7 +815,10 @@ class DeepxDetector(DetectionApi):
self.logged_layout = False
# set once a frame proves the PPU head anchor-free, then kept
self.ppu_anchor_free = False
self.ppu_anchor_based_reported = False
# set once proven needed, then kept
self.ppu_needs_grid_decode = False
self.ppu_scale_count = 0
self.ppu_unsupported_scale_reported = False
self.damoyolo_outputs_reported = False
def inspect_model(self, config: DeepxDetectorConfig) -> YoloOutput | None:
@ -757,28 +889,40 @@ class DeepxDetector(DetectionApi):
return post_process_yolo(outputs, self.width, self.height)
def decode_ppu(self, outputs: list[np.ndarray]) -> np.ndarray:
"""Decode PPU records. A box value above 1 proves the head is
anchor-free and that is kept; the anchor-based verdict is re-read on
every frame, so a sub-pixel box cannot lock the model out."""
"""Decode PPU records, keeping each heuristic once a frame settles
it: anchor-free, needs-grid-decode, and scale count all only
move toward more certainty, never back."""
if self.ppu_anchor_free:
anchor_based = False
else:
anchor_based = ppu_layout_is_anchor_based(outputs)
self.ppu_anchor_free = anchor_based is False
if anchor_based:
if not self.ppu_anchor_based_reported:
self.ppu_anchor_based_reported = True
logger.error(
"This PPU model was compiled from an anchor-based head, whose "
"records cannot be decoded without the model's anchor table. "
"Compile it without PPU support, or use an anchor-free model"
)
records = ppu_records(outputs)
if records is not None and len(records):
layer_idx = reinterpret(records, PPU_GRID_BYTES, np.uint8)[:, 3]
self.ppu_scale_count = max(self.ppu_scale_count, int(layer_idx.max()) + 1)
return np.zeros((20, 6), np.float32)
if not anchor_based and not self.ppu_needs_grid_decode:
self.ppu_needs_grid_decode = ppu_needs_grid_decode(records)
# None here means nothing conclusive in this frame, and no pixel
# box to draw either
if (
anchor_based
and self.ppu_scale_count
and self.ppu_scale_count not in PPU_ANCHORS_BY_SCALES
and not self.ppu_unsupported_scale_reported
):
self.ppu_unsupported_scale_reported = True
logger.error(
"This PPU model's anchor-based head reports %d detection "
"scales, which Frigate has no anchor table for (supported: "
"%s); its detections cannot be decoded",
self.ppu_scale_count,
", ".join(str(n) for n in sorted(PPU_ANCHORS_BY_SCALES)),
)
# None here means nothing conclusive in this frame, and no box to
# draw either
return decode_ppu(
outputs,
self.width,
@ -786,6 +930,9 @@ class DeepxDetector(DetectionApi):
SCORE_THRESHOLD,
NMS_THRESHOLD,
anchor_based=anchor_based,
scale_count=self.ppu_scale_count or None,
needs_grid_decode=True if self.ppu_needs_grid_decode else None,
records=records,
)
def decode_damoyolo(self, outputs: list[np.ndarray]) -> np.ndarray:
@ -850,4 +997,4 @@ class DeepxDetector(DetectionApi):
array.dtype,
array.min() if array.size else "n/a",
array.max() if array.size else "n/a",
)
)

257
frigate/test/test_deepx.py
View File

@ -49,6 +49,12 @@ def build_ppu_record(box, score=0.9, label=0, grid=(7, 9, 2, 2)) -> np.ndarray:
return record.reshape(1, 1, PPU_RECORD_SIZE)
def build_ppu_records(*records: np.ndarray) -> np.ndarray:
"""Several single records, as build_ppu_record makes them, stacked into
one (1, N, 32) output."""
return np.concatenate(records, axis=1)
class TestDeepxPpuDecode(unittest.TestCase):
def test_reads_anchor_free_box_bytes_as_pixel_geometry(self):
"""The grid columns carry no meaning for an anchor-free head and must
@ -69,13 +75,150 @@ class TestDeepxPpuDecode(unittest.TestCase):
self.assertAlmostEqual(detections[0][4], 176 / 640, places=5)
self.assertAlmostEqual(detections[0][5], 352 / 640, places=5)
def test_anchor_based_records_are_reported_not_misdecoded(self):
"""An anchor-based head leaves raw sigmoids in the box fields; read
as pixels they would give a sub-pixel box in the top-left corner."""
self.assertIsNone(
decode_ppu([build_ppu_record((0.6, 0.4, 0.3, 0.7))], 640, 640, 0.25, 0.45)
def test_reads_anchor_based_box_bytes_through_the_grid_and_anchors(self):
"""An anchor-based head leaves ratios in the box fields, which the
grid columns and the anchor table turn back into pixels."""
detections = decode_ppu(
[build_ppu_record((0.6, 0.4, 0.3, 0.7), label=5)], 640, 640, 0.25, 0.45
)
# layer 2 is stride 32 and box 2 is the 373x326 anchor, so the centre
# is (9.7, 7.3) cells out and the box is 134.28 x 638.96
self.assertEqual(detections[0][0], 5)
self.assertAlmostEqual(detections[0][2], 0.0, places=5)
self.assertAlmostEqual(detections[0][3], 0.380094, places=5)
self.assertAlmostEqual(detections[0][4], 0.864187, places=5)
self.assertAlmostEqual(detections[0][5], 0.589906, places=5)
def test_reads_anchor_free_grid_regression_bytes_through_the_grid_and_stride(
self,
):
"""YOLOX's classic anchor-free head leaves its box regression
relative to the grid cell it was predicted at, unlike anchor-based
heads, which the PPU decodes to pixels outright; more than
one layer present in the frame is what tells the two apart. A
second, low-scoring record on another layer is what proves that."""
detections = decode_ppu(
[
build_ppu_records(
build_ppu_record((1.2, 0.5, 1.0, 0.5), grid=(9, 10, 0, 2), label=7),
build_ppu_record(
(0.2, 0.3, 0.1, 0.4), grid=(3, 4, 0, 0), score=0.1
),
)
],
640,
640,
0.25,
0.45,
)
# layer 2 of 3 is stride 32: centre (10+1.2, 9+0.5)*32 = (358.4, 304),
# size exp(1.0)*32 x exp(0.5)*32 = 86.985 x 52.759
self.assertEqual(detections[0][0], 7)
self.assertAlmostEqual(detections[0][1], 0.9, places=5)
self.assertAlmostEqual(detections[0][2], 0.433782, places=5)
self.assertAlmostEqual(detections[0][3], 0.492043, places=5)
self.assertAlmostEqual(detections[0][4], 0.516218, places=5)
self.assertAlmostEqual(detections[0][5], 0.627957, places=5)
def test_a_single_layer_anchor_free_frame_still_reads_as_pixel_geometry(self):
"""A frame that only ever shows one layer cannot prove the grid and
layer fields are meaningful, so it falls back to the pixel-geometry
reading, same as an actual pixel-decoded head would need."""
detections = decode_ppu(
[build_ppu_record((320.0, 160.0, 64.0, 32.0), grid=(7, 9, 0, 1))],
640,
640,
0.25,
0.45,
)
self.assertAlmostEqual(detections[0][2], 144 / 640, places=5)
self.assertAlmostEqual(detections[0][3], 288 / 640, places=5)
def test_drops_records_the_anchor_table_cannot_place(self):
"""A record naming a level or box the table does not carry is left
out rather than decoded against the wrong anchor."""
detections = decode_ppu(
[build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 2, 5))],
640,
640,
0.25,
0.45,
)
self.assertTrue(np.all(detections == 0))
def test_a_two_scale_head_reads_the_tiny_anchor_table_and_stride(self):
"""A head with only two scales numbers its layers 0
and 1, but layer 0 is stride 16 there, not stride 8 as it would be
in a three-scale head; scale_count picks the matching table."""
detections = decode_ppu(
[build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 1, 0))],
640,
640,
0.25,
0.45,
scale_count=2,
)
# layer 0 of 2 is stride 16 and box 1 is the 37x58 anchor, so the
# centre is (9.7, 7.3) cells out and the box is 13.32 x 113.68
self.assertAlmostEqual(detections[0][2], 0.093688, places=5)
self.assertAlmostEqual(detections[0][3], 0.232094, places=5)
self.assertAlmostEqual(detections[0][4], 0.271313, places=5)
self.assertAlmostEqual(detections[0][5], 0.252906, places=5)
def test_scale_count_defaults_to_the_three_scale_table(self):
"""With no scale_count given, a frame that only ever shows layer 0
cannot tell a two-scale head from the first layer of a three-scale
one, so it falls back to the three-scale table."""
detections = decode_ppu(
[build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 1, 0))],
640,
640,
0.25,
0.45,
)
# layer 0 of 3 is stride 8 and box 1 is the 16x30 anchor, so the
# centre is (9.7, 7.3) cells out and the box is 5.76 x 58.8
self.assertAlmostEqual(detections[0][2], 0.045313, places=5)
self.assertAlmostEqual(detections[0][3], 0.116750, places=5)
self.assertAlmostEqual(detections[0][4], 0.137188, places=5)
self.assertAlmostEqual(detections[0][5], 0.125750, places=5)
def test_anchor_based_records_with_an_unsupported_scale_count_are_dropped(self):
"""A scale_count with no anchor table must drop the records rather
than guess against the wrong table."""
detections = decode_ppu(
[build_ppu_record((0.6, 0.4, 0.3, 0.7))],
640,
640,
0.25,
0.45,
scale_count=4,
)
self.assertTrue(np.all(detections == 0))
def test_needs_grid_decode_true_is_kept_for_a_single_layer_frame(self):
"""A caller that already proved the head needs grid decoding must
not have that overridden by a frame that, alone, looks ambiguous."""
detections = decode_ppu(
[build_ppu_record((1.2, 0.5, 1.0, 0.5), grid=(9, 10, 0, 2))],
640,
640,
0.25,
0.45,
needs_grid_decode=True,
)
# layer 2 of 3 is stride 32: centre (10+1.2, 9+0.5)*32 = (358.4, 304),
# size exp(1.0)*32 x exp(0.5)*32 = 86.985 x 52.759
self.assertAlmostEqual(detections[0][3], 0.492043, places=5)
def test_drops_records_below_the_score_threshold(self):
detections = decode_ppu(
[build_ppu_record((320.0, 160.0, 64.0, 32.0), score=0.1)],
@ -693,19 +836,20 @@ class TestDeepxDetectorLayout(unittest.TestCase):
self.assertIs(detector.output.layout, YoloLayout.ppu)
def test_an_anchor_based_ppu_model_is_reported_once(self):
def test_an_anchor_based_ppu_model_decodes(self):
detector, _ = self._detector(
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True
)
detector.session.run.return_value = [build_ppu_record((0.6, 0.4, 0.3, 0.7))]
detector.session.run.return_value = [
build_ppu_record((0.6, 0.4, 0.3, 0.7), label=5)
]
detector.width, detector.height = 640, 640
with self.assertLogs("frigate.detectors.plugins.deepx", level="ERROR") as logs:
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
detections = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertTrue(np.all(first == 0) and np.all(second == 0))
self.assertEqual(len([r for r in logs.records if r.levelname == "ERROR"]), 1)
self.assertEqual(detections[0][0], 5)
self.assertAlmostEqual(detections[0][3], 0.380094, places=5)
self.assertAlmostEqual(detections[0][5], 0.589906, places=5)
def test_anchor_free_is_kept_once_proven(self):
"""A later frame whose boxes all happen to fall within 1.0 must not
@ -725,8 +869,7 @@ class TestDeepxDetectorLayout(unittest.TestCase):
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertEqual(first[0][0], 3)
self.assertFalse(detector.ppu_anchor_based_reported)
# decoded as pixels, which is a sub-pixel box in the corner
# still read as pixels, which is a sub-pixel box in the corner
self.assertAlmostEqual(second[0][1], 0.9, places=5)
def test_an_anchor_based_verdict_is_not_kept(self):
@ -739,9 +882,8 @@ class TestDeepxDetectorLayout(unittest.TestCase):
detector.width, detector.height = 640, 640
detector.session.run.return_value = [build_ppu_record((0.6, 0.4, 0.3, 0.7))]
with self.assertLogs("frigate.detectors.plugins.deepx", level="ERROR"):
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertTrue(np.all(first == 0))
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertAlmostEqual(first[0][3], 0.380094, places=5)
self.assertFalse(detector.ppu_anchor_free)
detector.session.run.return_value = [
@ -752,6 +894,84 @@ class TestDeepxDetectorLayout(unittest.TestCase):
self.assertEqual(second[0][0], 3)
self.assertTrue(detector.ppu_anchor_free)
def test_ppu_scale_count_is_kept_once_a_coarser_layer_is_seen(self):
"""A frame that only shows layer 0 cannot tell a two-scale head from
a three-scale one and falls back to three; once a frame proves
layer 1 exists, that head has only two scales and every later
layer-0-only frame must use the two-scale table instead."""
detector, _ = self._detector(
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True
)
detector.width, detector.height = 640, 640
detector.session.run.return_value = [
build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 1, 0))
]
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
# layer 0 read against the three-scale default: stride 8, anchor 16x30
self.assertAlmostEqual(first[0][3], 0.116750, places=5)
detector.session.run.return_value = [
build_ppu_record((0.5, 0.5, 0.4, 0.4), grid=(3, 4, 0, 1))
]
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
# layer 1 of 2 is stride 32 and box 0 is the 81x82 anchor
self.assertAlmostEqual(second[0][3], 0.1845, places=5)
self.assertEqual(detector.ppu_scale_count, 2)
detector.session.run.return_value = [
build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 1, 0))
]
third = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
# the same layer-0 record as the first frame, now read against the
# two-scale table proven by the second frame: stride 16, anchor 37x58
self.assertAlmostEqual(third[0][3], 0.232094, places=5)
def test_needs_grid_decode_is_kept_once_proven(self):
"""A frame proving a YOLOX-style grid-relative head must not fall
back to reading raw pixels just because a later frame holds only
one detection at one layer."""
detector, _ = self._detector(
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True
)
detector.width, detector.height = 640, 640
detector.session.run.return_value = [
build_ppu_records(
build_ppu_record((1.2, 0.5, 1.0, 0.5), grid=(9, 10, 0, 2), label=7),
build_ppu_record((0.2, 0.3, 0.1, 0.4), grid=(3, 4, 0, 0), score=0.1),
)
]
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertAlmostEqual(first[0][3], 0.492043, places=5)
self.assertTrue(detector.ppu_needs_grid_decode)
detector.session.run.return_value = [
build_ppu_record((1.2, 0.5, 1.0, 0.5), grid=(9, 10, 0, 2), label=7)
]
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
# still decoded through the grid and stride, not as raw pixels
self.assertAlmostEqual(second[0][3], 0.492043, places=5)
def test_an_unsupported_ppu_scale_count_is_reported_once(self):
"""An anchor-based head with a scale count Frigate has no anchor
table for logs once and drops the frame instead of guessing."""
detector, _ = self._detector(
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True
)
detector.width, detector.height = 640, 640
detector.session.run.return_value = [
build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 2, 3))
]
with self.assertLogs("frigate.detectors.plugins.deepx", level="ERROR") as logs:
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
self.assertTrue(np.all(first == 0) and np.all(second == 0))
self.assertEqual(len([r for r in logs.records if r.levelname == "ERROR"]), 1)
def test_an_empty_or_all_zero_ppu_frame_proves_nothing(self):
"""Only a box value above 1 (anchor-free) or a frame of values
strictly inside 0..1 (anchor-based) is evidence; an empty frame or a
@ -770,7 +990,6 @@ class TestDeepxDetectorLayout(unittest.TestCase):
self.assertTrue(np.all(detections == 0))
self.assertFalse(detector.ppu_anchor_free)
self.assertFalse(detector.ppu_anchor_based_reported)
def test_damoyolo_is_checked_against_the_model_at_load(self):
"""A YOLO model or a wrong label map under damo-yolo fails at load
@ -892,4 +1111,4 @@ class TestDeepxDetectorLayout(unittest.TestCase):
self.assertEqual(detections[0][0], 5)
self.assertAlmostEqual(detections[0][1], 0.8, places=5)
self.assertAlmostEqual(detections[0][4], 300 / 640, places=5)
self.assertAlmostEqual(detections[0][4], 300 / 640, places=5)