mirror of
https://github.com/blakeblackshear/frigate.git
synced 2026-10-18 07:29:45 +00:00
Extend PPU layout handling and YOLOX support to DEEPX detector
This commit is contained in:
aahmetdurmaz
parent
9d7899bfeb
commit
1388305439
@ -991,31 +991,59 @@ deepx:
|
||||
- key: yolo
|
||||
label: YOLO
|
||||
recommended: true
|
||||
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration.
|
||||
download: No model is bundled with Frigate. Download a pre-compiled YOLO `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo) or compile your own with DX-COM, then bind-mount it into the container and point the model's `path` at it. The recommended model is `yolox-s_640x640_ppu.dxnn`. Its Post-Processing Unit (PPU) compile moves candidate selection onto the NPU, which makes it the fastest ModelZoo model measured through Frigate (about 13 ms on a DX-M1). The output layout is read from the compiled model, so anchor-based, anchor-free, NMS-in-head and PPU models (anchor-based or anchor-free) all need no extra configuration; prefer a `PPU` variant whenever the ModelZoo offers one.
|
||||
ui: |-
|
||||
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
||||
|
||||
| Field | Value |
|
||||
| ---------------------------------------- | -------------------------------------- |
|
||||
| **Custom object detector model path** | `/config/model_cache/deepx/model.dxnn` |
|
||||
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
||||
| **Object detection model input width** | `640` |
|
||||
| **Object detection model input height** | `640` |
|
||||
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
||||
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
||||
| **Model Input D Type** | `int` (Frigate's default value) |
|
||||
| **Object Detection Model Type** | `yolo-generic` |
|
||||
| Field | Value |
|
||||
| ---------------------------------------- | ---------------------------------------------------- |
|
||||
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640_ppu.dxnn` |
|
||||
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
||||
| **Object detection model input width** | `640` |
|
||||
| **Object detection model input height** | `640` |
|
||||
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
||||
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
||||
| **Model Input D Type** | `int` (Frigate's default value) |
|
||||
| **Object Detection Model Type** | `yolo-generic` |
|
||||
|
||||
The width and height must match the resolution the `.dxnn` file was compiled for. Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
|
||||
Quantization is baked into the compiled model, so no normalization is applied on the host and the input defaults do not need to be overridden.
|
||||
yaml: |-
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/model.dxnn
|
||||
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolo-generic
|
||||
width: 640
|
||||
height: 640
|
||||
- key: yolox
|
||||
label: YOLOX
|
||||
recommended: false
|
||||
download: No model is bundled with Frigate. Download a pre-compiled YOLOX `.dxnn` model from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), then bind-mount it into the container and point the model's `path` at it. The `_ppu` variant is faster and also works with the `yolo-generic` model type; the plain export needs `yolox` so its raw head is decoded.
|
||||
ui: |-
|
||||
Navigate to **Settings > System > Detection models** and select **DEEPX NPU** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure:
|
||||
|
||||
| Field | Value |
|
||||
| ---------------------------------------- | ---------------------------------------- |
|
||||
| **Custom object detector model path** | `/config/model_cache/deepx/yolox-s_640x640.dxnn`|
|
||||
| **Label map for custom object detector** | `/labelmap/coco-80.txt` |
|
||||
| **Object detection model input width** | `640` |
|
||||
| **Object detection model input height** | `640` |
|
||||
| **Model Input Pixel Color Format** | `rgb` (Frigate's default value) |
|
||||
| **Model Input Tensor Shape** | `nhwc` (Frigate's default value) |
|
||||
| **Model Input D Type** | `int` (Frigate's default value) |
|
||||
| **Object Detection Model Type** | `yolox` |
|
||||
|
||||
The width and height must match the resolution the `.dxnn` file was compiled for.
|
||||
yaml: |-
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/yolox-s_640x640.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolox
|
||||
width: 640
|
||||
height: 640
|
||||
- key: damoyolo
|
||||
label: DAMO-YOLO (TinyNAS-L20T / S / M / L)
|
||||
recommended: false
|
||||
|
||||
@ -655,25 +655,26 @@ The DX-RT Python bindings are not part of the Frigate image. They are downloaded
|
||||
|
||||
<ModelConfigDropdown detectorTitle="DEEPX" models={objectDetectorsModels.deepx.models} />
|
||||
|
||||
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Two model types are supported:
|
||||
Frigate does not bundle a model for this detector. Models must be compiled to DEEPX's `.dxnn` format. Three model types are supported:
|
||||
|
||||
- `yolo-generic` for YOLO object detection models, the recommended default. The detector reads the model's output layout from the compiled file, so anchor-based, anchor-free and NMS-in-head models all work with the same configuration, as do models compiled with DEEPX's Post-Processing Unit (PPU) support.
|
||||
- `yolox` for YOLOX models compiled without PPU support, whose raw head needs Frigate's YOLOX decoder. A YOLOX model compiled with PPU support works under either `yolox` or `yolo-generic`.
|
||||
- `damo-yolo` for DAMO-YOLO models. All four sizes (TinyNAS-L20T, S, M, L) are supported.
|
||||
|
||||
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO and DAMO-YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler.
|
||||
The quickest way to get one is the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo), which publishes pre-compiled `.dxnn` files for a range of YOLO and DAMO-YOLO object detection models. Download the `.dxnn`, bind-mount it into the container, and point the model's `path` at it. Alternatively, compile your own model with the DX-COM compiler. The recommended starting point is `yolox-s_640x640_ppu.dxnn`, the fastest ModelZoo model measured through Frigate:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- devices:
|
||||
- deepx:PCIe:0
|
||||
path: /config/model_cache/deepx/model.dxnn
|
||||
path: /config/model_cache/deepx/yolox-s_640x640_ppu.dxnn
|
||||
labelmap_path: /labelmap/coco-80.txt
|
||||
model_type: yolo-generic
|
||||
width: 640
|
||||
height: 640
|
||||
```
|
||||
|
||||
`model_type` must be set to `yolo-generic` or `damo-yolo` to match the model; `yolo-generic` is the recommended default unless the model is specifically a DAMO-YOLO export. Frigate defaults it to `ssd`, which this detector does not support, so a model that leaves it unset is rejected at startup.
|
||||
`model_type` must be set to `yolo-generic`, `yolox` or `damo-yolo` to match the model; `yolo-generic` is the recommended default unless the model is a raw YOLOX or DAMO-YOLO export. Frigate defaults it to `ssd`, which this detector does not support, so a model that leaves it unset is rejected at startup.
|
||||
|
||||
`width` and `height` must match the resolution the model was compiled for. Quantization parameters are baked into the `.dxnn` file at compile time, so no normalization is applied on the host and Frigate's default `input_tensor`, `input_pixel_format`, and `input_dtype` values do not need to be overridden.
|
||||
|
||||
|
||||
@ -267,9 +267,9 @@ Frigate supports the DEEPX NPU in both of its form factors: the **DX-M1** M.2 mo
|
||||
|
||||
DEEPX NPU support in Frigate is developed and maintained by [Sixfab](https://sixfab.com).
|
||||
|
||||
The NPU runs models compiled to the `.dxnn` format with DEEPX's DX-COM compiler. Pre-compiled YOLO and DAMO-YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Models compiled with Post-Processing Unit (PPU) support move candidate selection onto the NPU, which reduces host CPU usage; both anchor-based and anchor-free PPU heads are supported.
|
||||
The NPU runs models compiled to the `.dxnn` format with DEEPX's DX-COM compiler. Pre-compiled YOLO and DAMO-YOLO models can be downloaded from the [DEEPX ModelZoo](https://developer.deepx.ai/modelzoo). Models compiled with Post-Processing Unit (PPU) support move candidate selection onto the NPU, which reduces host CPU usage; both anchor-based and anchor-free PPU heads are supported. Frigate reads the PPU head's kind and number of detection scales from the `.dxnn` file when the model loads, so no configuration is needed for either.
|
||||
|
||||
Exception: **YOLOv7 PPU models are not supported**. A PPU record has no anchor sizes, only a grid position, so Frigate decodes anchor-based PPU heads against a fixed anchor table; YOLOv7's anchors differ, so it decodes to wrong box sizes. Non-PPU YOLOv7 is unaffected (its raw head already outputs pixel-space boxes).
|
||||
Exception: **YOLOv7 and YOLOv7-x PPU models are not supported**. A PPU record has no anchor sizes, only a grid position, and the `.dxnn` does not carry them either, so Frigate decodes anchor-based PPU heads against a fixed anchor table per scale count: the usual COCO anchor set for three-scale heads (the one DEEPX's own reference decoder uses for every other three-scale model it ships) and the usual tiny set for two-scale heads. YOLOv7's anchors differ and nothing in the model tells them apart, so it decodes to wrong box sizes. Non-PPU YOLOv7 is unaffected (its raw head already outputs pixel-space boxes). An end-to-end PPU head that emits corner boxes rather than a centre and size is told apart from the records themselves. **Face and pose PPU models (SCRFD and the pose export) are not supported**: their records carry landmarks or keypoints in a different layout, and Frigate only decodes object detection records.
|
||||
|
||||
The DEEPX kernel driver, the DX-RT runtime, and the `dxrtd` daemon all run on the Docker host rather than inside the Frigate container, and have to be installed there before the NPU can be used. Frigate connects to the daemon over its socket, so the NPU stays available to other programs on the host at the same time. See the [installation docs](installation.md#deepx-npu) for the setup steps.
|
||||
|
||||
@ -287,7 +287,7 @@ The DEEPX ModelZoo publishes pre-compiled `.dxnn` files for many YOLO variants a
|
||||
|
||||
These are DEEPX's own figures for the DX-M1, derived from the frames per second published in the ModelZoo. They cover the NPU alone and exclude the pre- and post-processing Frigate does on the host, so the inference speed Frigate reports will be higher, and noticeably so on a slower host such as a Raspberry Pi 5.
|
||||
|
||||
Besides the DAMO-YOLO family shown above, other models from the wider YOLO family published in the ModelZoo are also supported by the `deepx` detector by setting `model_type: yolo-generic`. Inference times for these, measured through Frigate's own stats API on a DX-M1 rather than taken from DEEPX's published figures, are below:
|
||||
Besides the DAMO-YOLO family shown above, other models from the wider YOLO family published in the ModelZoo are also supported by the `deepx` detector by setting `model_type: yolo-generic`. A YOLOX model compiled without PPU support is the one exception: its raw head needs `model_type: yolox` (the PPU variant works with either). Inference times for these, measured through Frigate's own stats API on a DX-M1 rather than taken from DEEPX's published figures, are below:
|
||||
|
||||
| Model | Input Size | DX-M1 Inference Time (via Frigate) |
|
||||
| ------------------ | ---------- | ----------------------------------- |
|
||||
|
||||
@ -1,7 +1,10 @@
|
||||
"""DEEPX NPU detector running compiled .dxnn models via the DX-RT runtime."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Literal
|
||||
@ -12,7 +15,7 @@ from pydantic import ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
from frigate.detectors.detection_api import DetectionApi
|
||||
from frigate.detectors.detector_config import BaseDetectorConfig, ModelTypeEnum
|
||||
from frigate.util.model import post_process_yolo
|
||||
from frigate.util.model import post_process_yolo, post_process_yolox
|
||||
from frigate.util.runtime_deps import Artifact, ArtifactKind, RuntimeManifest
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@ -33,7 +36,14 @@ DXRT_IPC_SOCKET = "/run/dxrt/dxrt_dynamic_ipc.sock"
|
||||
SCORE_THRESHOLD = 0.4
|
||||
NMS_THRESHOLD = 0.4
|
||||
|
||||
SUPPORTED_MODEL_TYPES = (ModelTypeEnum.yologeneric, ModelTypeEnum.damoyolo)
|
||||
SUPPORTED_MODEL_TYPES = (
|
||||
ModelTypeEnum.yologeneric,
|
||||
ModelTypeEnum.yolox,
|
||||
ModelTypeEnum.damoyolo,
|
||||
)
|
||||
|
||||
# YOLOX's raw head concatenates one cell per position of its three strides
|
||||
YOLOX_STRIDES = (8, 16, 32)
|
||||
|
||||
# Fixed-width record the PPU emits, DeviceBoundingBox_t in DX-RT's
|
||||
# datatype.h: x, y, w, h (float32), grid_y, grid_x, box_idx, layer_idx
|
||||
@ -44,10 +54,14 @@ PPU_GRID_BYTES = (16, 20)
|
||||
PPU_SCORE_BYTES = (20, 24)
|
||||
PPU_LABEL_BYTES = (24, 28)
|
||||
|
||||
# Anchor sizes are not in the PPU record or the .dxnn, so an anchor-based
|
||||
# head is read against the table for its scale count: DEEPX's own reference
|
||||
# decoder's tables, shared by every three- or two-scale model it ships
|
||||
# except YOLOv7, whose anchors differ.
|
||||
PPU_ANCHORS_BY_SCALES = {
|
||||
2: np.array(
|
||||
[
|
||||
[[23, 27], [37, 58], [81, 82]],
|
||||
[[10, 14], [23, 27], [37, 58]],
|
||||
[[81, 82], [135, 169], [344, 319]],
|
||||
],
|
||||
dtype=np.float32,
|
||||
@ -68,6 +82,18 @@ PPU_MAX_STRIDE = 32
|
||||
# gets the biggest table as a best guess
|
||||
PPU_MAX_KNOWN_SCALES = max(PPU_ANCHORS_BY_SCALES)
|
||||
|
||||
# .dxnn: "DXNN", a 4-byte LE version, then a JSON index padded to
|
||||
# DXNN_HEADER_SIZE; every offset in the index counts from there.
|
||||
DXNN_MAGIC = b"DXNN"
|
||||
DXNN_HEADER_SIZE = 8192
|
||||
# compile_config's ppu.type for the heads decoded as bounding boxes
|
||||
PPU_TYPE_ANCHOR_BASED = 0
|
||||
PPU_TYPE_ANCHOR_FREE = 1
|
||||
# The PPU tensor table, ppu_info_header_t then one ppu_info_t per output
|
||||
# tensor, as DX-RT's ppu_binary_parser.h defines them.
|
||||
PPU_TABLE_HEADER = struct.Struct("<BBBB")
|
||||
PPU_TABLE_ENTRY = struct.Struct("<HHfBBBBBBBB")
|
||||
|
||||
# A nms-in-head output is (N, 6) rows of x_min, y_min, x_max, y_max, score,
|
||||
# class, with N decided per frame and capped in the head (300 for the
|
||||
# ModelZoo exports). A raw head has thousands of rows, which is what tells
|
||||
@ -218,6 +244,7 @@ class YoloLayout(str, Enum):
|
||||
model's head, so the head decides and the model is what gets asked."""
|
||||
|
||||
ppu = "ppu" # fixed-width records from the NPU's post-processing unit
|
||||
yolox = "yolox" # (N, 5+C) grid-relative offsets and log sizes, model_type yolox
|
||||
anchor = "anchor" # (N, 5+C) with an objectness column, pixel boxes
|
||||
anchor_free = "anchor_free" # (N, 4+C) or (4+C, N), no objectness
|
||||
nms_in_head = "nms_in_head" # (N, 6) final corner boxes
|
||||
@ -363,6 +390,31 @@ def validate_damoyolo_outputs(shapes: list[tuple[int, ...]], num_classes: int) -
|
||||
)
|
||||
|
||||
|
||||
def validate_yolox_outputs(
|
||||
shapes: list[tuple[int, ...]], num_classes: int, width: int, height: int
|
||||
) -> int:
|
||||
"""Fail unless the shapes are YOLOX's raw (1, N, 5+C) head for this
|
||||
input size; returns the row width to orient a channel-major export."""
|
||||
columns = 5 + num_classes
|
||||
cells = sum((height // s) * (width // s) for s in YOLOX_STRIDES)
|
||||
dims = [_significant_dims(shape) for shape in shapes]
|
||||
|
||||
if (
|
||||
len(dims) == 1
|
||||
and len(dims[0]) == 2
|
||||
and sorted(dims[0]) == sorted((cells, columns))
|
||||
):
|
||||
return columns
|
||||
|
||||
raise ValueError(
|
||||
f"output shapes {[tuple(shape) for shape in shapes]} are not YOLOX's "
|
||||
f"(1, {cells}, {columns}) raw head for a {width}x{height} input. Check "
|
||||
"that width and height match the compiled model, that labelmap_path "
|
||||
"matches it, usually /labelmap/coco-80.txt, and that model_type "
|
||||
"matches it; any other YOLO head needs model_type yolo-generic."
|
||||
)
|
||||
|
||||
|
||||
def rows_with_columns(tensor: np.ndarray, columns: int) -> np.ndarray:
|
||||
"""(N, columns) rows, transposing a channel-major export first."""
|
||||
if tensor.ndim >= 2 and tensor.shape[-1] != columns and tensor.shape[-2] == columns:
|
||||
@ -408,17 +460,11 @@ def ppu_boxes_are_anchor_based(boxes: np.ndarray) -> bool | None:
|
||||
return None
|
||||
|
||||
|
||||
def ppu_layout_is_anchor_based(outputs: list[np.ndarray]) -> bool | None:
|
||||
"""Whether this PPU model's records are anchor-based, or None when the
|
||||
frame cannot tell; the caller keeps the first conclusive answer."""
|
||||
records = ppu_records(outputs)
|
||||
|
||||
if records is None:
|
||||
return None
|
||||
|
||||
boxes = reinterpret(records, PPU_BOX_BYTES, np.float32).reshape(-1, 4)
|
||||
|
||||
return ppu_boxes_are_anchor_based(boxes)
|
||||
def ppu_boxes_are_centres(boxes: np.ndarray) -> bool:
|
||||
"""Whether these pixel boxes hold a centre and size rather than two
|
||||
corners: a corner box always runs x1<=x2, y1<=y2, so only a centre box
|
||||
can show a width below its x or a height below its y."""
|
||||
return bool(np.any(boxes[:, 2] < boxes[:, 0]) or np.any(boxes[:, 3] < boxes[:, 1]))
|
||||
|
||||
|
||||
def ppu_needs_grid_decode(records: np.ndarray) -> bool:
|
||||
@ -429,6 +475,134 @@ def ppu_needs_grid_decode(records: np.ndarray) -> bool:
|
||||
return len(np.unique(layer_idx)) > 1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PpuLayout:
|
||||
"""The PPU head's kind (None if unnamed) and each scale's grid, finest
|
||||
first, as the compiled model states them."""
|
||||
|
||||
anchor_based: bool | None
|
||||
grids: tuple[tuple[int, int], ...]
|
||||
|
||||
@property
|
||||
def scale_count(self) -> int:
|
||||
return len(self.grids)
|
||||
|
||||
|
||||
def read_ppu_layout(path: str) -> PpuLayout | None:
|
||||
"""Read the PPU head layout the compiler wrote into a .dxnn, or None
|
||||
when there is none to read; only the index and metadata are read."""
|
||||
try:
|
||||
with open(path, "rb") as model:
|
||||
header = model.read(DXNN_HEADER_SIZE)
|
||||
if header[:4] != DXNN_MAGIC:
|
||||
return None
|
||||
|
||||
index = json.JSONDecoder().raw_decode(
|
||||
header[8:].decode("utf-8", "replace")
|
||||
)[0]
|
||||
data = index["data"]
|
||||
|
||||
def section(entry: dict) -> bytes:
|
||||
model.seek(DXNN_HEADER_SIZE + int(entry["offset"]))
|
||||
return model.read(int(entry["size"]))
|
||||
|
||||
anchor_based = None
|
||||
ppu = None
|
||||
if data.get("compile_config"):
|
||||
ppu = json.loads(section(data["compile_config"])).get("ppu")
|
||||
if ppu is None:
|
||||
return None
|
||||
|
||||
anchor_based = {
|
||||
PPU_TYPE_ANCHOR_BASED: True,
|
||||
PPU_TYPE_ANCHOR_FREE: False,
|
||||
}.get(ppu.get("type"))
|
||||
|
||||
for chip in data.get("compiled_data", {}).values():
|
||||
for npu in chip.values():
|
||||
# v8 has a PPU table; v7's output tensor shapes hold the same grids
|
||||
if npu.get("ppu", {}).get("size"):
|
||||
grids = _ppu_grids_from_table(section(npu["ppu"]))
|
||||
elif npu.get("rmap_info", {}).get("size"):
|
||||
outputs = json.loads(section(npu["rmap_info"])).get(
|
||||
"outputs", []
|
||||
)
|
||||
grids = _ppu_grids_from_outputs(outputs, ppu)
|
||||
if grids and anchor_based is None:
|
||||
anchor_based = _ppu_outputs_are_anchor_based(outputs, ppu)
|
||||
else:
|
||||
continue
|
||||
|
||||
if grids:
|
||||
return PpuLayout(
|
||||
anchor_based, tuple(grids[k] for k in sorted(grids))
|
||||
)
|
||||
|
||||
return None
|
||||
except (OSError, ValueError, KeyError, TypeError, AttributeError, struct.error):
|
||||
return None
|
||||
|
||||
|
||||
def _ppu_grids_from_table(table: bytes) -> dict[int, tuple[int, int]]:
|
||||
"""Grid per scale index from the PPU table: field 3 is the scale,
|
||||
fields 8 and 9 its grid, shared by every anchor of that scale."""
|
||||
_, tensor_count, _, _ = PPU_TABLE_HEADER.unpack_from(table)
|
||||
if len(table) < PPU_TABLE_HEADER.size + tensor_count * PPU_TABLE_ENTRY.size:
|
||||
raise ValueError("PPU table shorter than its tensor count")
|
||||
|
||||
grids: dict[int, tuple[int, int]] = {}
|
||||
for i in range(tensor_count):
|
||||
fields = PPU_TABLE_ENTRY.unpack_from(
|
||||
table, PPU_TABLE_HEADER.size + i * PPU_TABLE_ENTRY.size
|
||||
)
|
||||
grids.setdefault(fields[3], (fields[8], fields[9]))
|
||||
|
||||
return grids
|
||||
|
||||
|
||||
def _ppu_output_scale(name: str, ppu: dict | None) -> int | None:
|
||||
"""Which scale a PPU output tensor belongs to, from ppu.outputs by
|
||||
name or else the name's own PPU_..._Output_<scale>[_anchor_<n>] form."""
|
||||
entry = ((ppu or {}).get("outputs") or {}).get(name)
|
||||
if isinstance(entry, dict) and "conv_idx" in entry:
|
||||
return int(entry["conv_idx"])
|
||||
|
||||
match = re.fullmatch(r"PPU_\w*?Output_(\d+)(?:_anchor_\d+)?", name)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def _ppu_grids_from_outputs(
|
||||
outputs: list[dict], ppu: dict | None
|
||||
) -> dict[int, tuple[int, int]]:
|
||||
"""Grid per scale index from rmap_info's output tensor shapes."""
|
||||
grids: dict[int, tuple[int, int]] = {}
|
||||
for output in outputs:
|
||||
scale = _ppu_output_scale(str(output.get("name", "")), ppu)
|
||||
shape = output.get("shape") or []
|
||||
if scale is None or len(shape) not in (3, 4):
|
||||
continue
|
||||
|
||||
grid = (int(shape[2]), int(shape[1])) if len(shape) == 4 else (int(shape[1]), 1)
|
||||
grids.setdefault(scale, grid)
|
||||
|
||||
return grids
|
||||
|
||||
|
||||
def _ppu_outputs_are_anchor_based(outputs: list[dict], ppu: dict | None) -> bool | None:
|
||||
"""Anchor-based when the PPU outputs are split per anchor; None when
|
||||
nothing says either way."""
|
||||
mapping = (ppu or {}).get("outputs") or {}
|
||||
if any(
|
||||
isinstance(entry, dict) and "anchor_idx" in entry for entry in mapping.values()
|
||||
):
|
||||
return True
|
||||
|
||||
if any("_anchor_" in str(output.get("name", "")) for output in outputs):
|
||||
return True
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def ppu_scale_count_lower_bound(records: np.ndarray, width: int, height: int) -> int:
|
||||
"""The fewest detection scales these records can come from. Each proves
|
||||
one past its layer index, and its grid cell can prove more: cell g of
|
||||
@ -537,19 +711,21 @@ def decode_ppu(
|
||||
scale_count: int | None = None,
|
||||
needs_grid_decode: bool | None = None,
|
||||
records: np.ndarray | None = None,
|
||||
centre_boxes: bool | None = None,
|
||||
boxes: np.ndarray | None = None,
|
||||
) -> np.ndarray:
|
||||
"""Decode PPU records, leaving only NMS. `anchor_based` and
|
||||
`needs_grid_decode` skip their heuristics once settled by a caller
|
||||
tracking them across frames; `scale_count` picks the anchor table or
|
||||
stride, defaulting to what this frame alone shows; `records` reuses an
|
||||
already-parsed buffer instead of re-parsing `outputs`."""
|
||||
"""Decode PPU records, leaving only NMS. `anchor_based`,
|
||||
`needs_grid_decode` and `centre_boxes` skip their heuristics once a
|
||||
caller settles them; `records` and `boxes` reuse an already-parsed
|
||||
buffer instead of re-parsing `outputs`."""
|
||||
if records is None:
|
||||
records = ppu_records(outputs)
|
||||
|
||||
if records is None:
|
||||
return np.zeros((20, 6), np.float32)
|
||||
|
||||
boxes = reinterpret(records, PPU_BOX_BYTES, np.float32).reshape(-1, 4)
|
||||
if boxes is None:
|
||||
boxes = reinterpret(records, PPU_BOX_BYTES, np.float32).reshape(-1, 4)
|
||||
|
||||
if anchor_based is None:
|
||||
anchor_based = ppu_boxes_are_anchor_based(boxes)
|
||||
@ -579,8 +755,17 @@ def decode_ppu(
|
||||
)
|
||||
scores = np.where(known, scores, 0.0)
|
||||
else:
|
||||
centre_x, centre_y = boxes[:, 0], boxes[:, 1]
|
||||
box_w, box_h = boxes[:, 2], boxes[:, 3]
|
||||
if centre_boxes is None:
|
||||
centre_boxes = ppu_boxes_are_centres(boxes)
|
||||
|
||||
if centre_boxes:
|
||||
centre_x, centre_y = boxes[:, 0], boxes[:, 1]
|
||||
box_w, box_h = boxes[:, 2], boxes[:, 3]
|
||||
else:
|
||||
box_w = boxes[:, 2] - boxes[:, 0]
|
||||
box_h = boxes[:, 3] - boxes[:, 1]
|
||||
centre_x = boxes[:, 0] + box_w * 0.5
|
||||
centre_y = boxes[:, 1] + box_h * 0.5
|
||||
|
||||
x_min = centre_x - box_w * 0.5
|
||||
y_min = centre_y - box_h * 0.5
|
||||
@ -836,15 +1021,77 @@ class DeepxDetector(DetectionApi):
|
||||
|
||||
self.session = InferenceEngine(str(model_path), options)
|
||||
self.output = self.inspect_model(config)
|
||||
if self.output is not None and self.output.layout is YoloLayout.yolox:
|
||||
# the shared YOLOX decoder's cell grids and strides for this input
|
||||
self.calculate_grids_strides()
|
||||
self.logged_layout = False
|
||||
# set once a frame proves the PPU head anchor-free, then kept
|
||||
self.ppu_anchor_free = False
|
||||
# set once proven needed, then kept
|
||||
self.ppu_needs_grid_decode = False
|
||||
self.ppu_scale_count = 0
|
||||
# set once proven; a corner-format head never produces such a record
|
||||
self.ppu_centre_boxes = False
|
||||
self.ppu_unsupported_scale_reported = False
|
||||
self.damoyolo_outputs_reported = False
|
||||
|
||||
# the model's own head layout; None falls back to the fields above
|
||||
self.ppu_layout = None
|
||||
if self.output is not None and self.output.layout is YoloLayout.ppu:
|
||||
self.ppu_layout = self.inspect_ppu_head(model_path)
|
||||
|
||||
def inspect_ppu_head(self, model_path: str) -> PpuLayout | None:
|
||||
"""Settle the PPU head from the model file, so the anchor table and
|
||||
strides are right from the first frame."""
|
||||
layout = read_ppu_layout(model_path)
|
||||
|
||||
if layout is None:
|
||||
logger.warning(
|
||||
"Could not read the PPU head layout from %s; inferring its "
|
||||
"scale count from the detections instead, which can misplace "
|
||||
"boxes until a record proves every scale",
|
||||
model_path,
|
||||
)
|
||||
return None
|
||||
|
||||
self.ppu_scale_count = layout.scale_count
|
||||
if layout.anchor_based is not None:
|
||||
self.ppu_anchor_free = not layout.anchor_based
|
||||
# only a grid-relative head keeps one tensor per scale
|
||||
self.ppu_needs_grid_decode = layout.anchor_based is False and (
|
||||
layout.scale_count > 1
|
||||
)
|
||||
|
||||
# a head laid out differently than the decoders assume would
|
||||
# decode to wrong positions, so say so once here
|
||||
if layout.anchor_based or layout.scale_count > 1:
|
||||
strides = [
|
||||
PPU_MAX_STRIDE >> (layout.scale_count - 1 - i)
|
||||
for i in range(layout.scale_count)
|
||||
]
|
||||
expected = tuple(
|
||||
(-(-self.width // s), -(-self.height // s)) for s in strides
|
||||
)
|
||||
if layout.grids != expected:
|
||||
logger.warning(
|
||||
"PPU head grids %s do not match the %d-scale layout Frigate "
|
||||
"decodes (%s for %dx%d input); boxes may be misplaced",
|
||||
layout.grids,
|
||||
layout.scale_count,
|
||||
expected,
|
||||
self.width,
|
||||
self.height,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"DEEPX PPU head from the model: %s, %d scale(s)",
|
||||
{True: "anchor-based", False: "anchor-free"}.get(
|
||||
layout.anchor_based, "unknown kind"
|
||||
),
|
||||
layout.scale_count,
|
||||
)
|
||||
return layout
|
||||
|
||||
def inspect_model(self, config: DeepxDetectorConfig) -> YoloOutput | None:
|
||||
"""Pick the decoder from what the runtime reports, failing here
|
||||
rather than on the first frame."""
|
||||
@ -864,19 +1111,27 @@ class DeepxDetector(DetectionApi):
|
||||
validate_damoyolo_outputs(shapes, num_classes)
|
||||
return None
|
||||
|
||||
output = infer_yolo_layout(
|
||||
shapes,
|
||||
num_classes,
|
||||
self.session.is_ppu(),
|
||||
self.session.has_dynamic_output(),
|
||||
)
|
||||
if self.model_type == ModelTypeEnum.yolox and not self.session.is_ppu():
|
||||
# a YOLOX compiled with PPU support is read as PPU below
|
||||
columns = validate_yolox_outputs(
|
||||
shapes, num_classes, self.width, self.height
|
||||
)
|
||||
output = YoloOutput(YoloLayout.yolox, columns)
|
||||
else:
|
||||
output = infer_yolo_layout(
|
||||
shapes,
|
||||
num_classes,
|
||||
self.session.is_ppu(),
|
||||
self.session.has_dynamic_output(),
|
||||
)
|
||||
except ValueError as err:
|
||||
raise ValueError(
|
||||
f"Cannot decode DEEPX model '{config.model.path}': {err}"
|
||||
) from None
|
||||
|
||||
logger.info(
|
||||
"DEEPX decoding yolo-generic output as %s with %d classes",
|
||||
"DEEPX decoding %s output as %s with %d classes",
|
||||
self.model_type.value,
|
||||
output.layout.value,
|
||||
num_classes,
|
||||
)
|
||||
@ -890,6 +1145,8 @@ class DeepxDetector(DetectionApi):
|
||||
match self.output.layout:
|
||||
case YoloLayout.ppu:
|
||||
return self.decode_ppu(outputs)
|
||||
case YoloLayout.yolox:
|
||||
return self.decode_yolox(outputs)
|
||||
case YoloLayout.anchor:
|
||||
return decode_raw_anchor(
|
||||
outputs,
|
||||
@ -912,29 +1169,54 @@ class DeepxDetector(DetectionApi):
|
||||
# per-scale feature maps, same thresholds as above
|
||||
return post_process_yolo(outputs, self.width, self.height)
|
||||
|
||||
def decode_yolox(self, outputs: list[np.ndarray]) -> np.ndarray:
|
||||
"""Decode YOLOX's raw head with Frigate's shared decoder; it writes
|
||||
into its input, so a float copy is oriented as (1, N, 5+C) first."""
|
||||
rows = rows_with_columns(outputs[0], self.output.columns)
|
||||
predictions = np.array(rows, dtype=np.float32).reshape(1, -1, rows.shape[-1])
|
||||
|
||||
return post_process_yolox(
|
||||
predictions, self.width, self.height, self.grids, self.expanded_strides
|
||||
)
|
||||
|
||||
def decode_ppu(self, outputs: list[np.ndarray]) -> np.ndarray:
|
||||
"""Decode PPU records, keeping each heuristic once a frame settles
|
||||
it: anchor-free, needs-grid-decode, and scale count all only
|
||||
move toward more certainty, never back. The scale count is the
|
||||
largest lower bound any frame has proven, so a three-scale head
|
||||
whose frames so far only held its finer layers near the origin
|
||||
reads against the two-scale table until a record lands far enough
|
||||
right or down, or on the coarsest layer, to prove the third."""
|
||||
if self.ppu_anchor_free:
|
||||
"""Decode PPU records. The head layout read from the model fixes
|
||||
the kind, scale count and grid decode from the first frame; without
|
||||
it, each is settled from the records and kept once proven."""
|
||||
records = ppu_records(outputs)
|
||||
boxes = None
|
||||
if records is not None and len(records):
|
||||
boxes = reinterpret(records, PPU_BOX_BYTES, np.float32).reshape(-1, 4)
|
||||
|
||||
if self.ppu_layout is not None and self.ppu_layout.anchor_based is not None:
|
||||
anchor_based = self.ppu_layout.anchor_based
|
||||
elif self.ppu_anchor_free:
|
||||
anchor_based = False
|
||||
else:
|
||||
anchor_based = ppu_layout_is_anchor_based(outputs)
|
||||
anchor_based = (
|
||||
ppu_boxes_are_anchor_based(boxes) if boxes is not None else None
|
||||
)
|
||||
self.ppu_anchor_free = anchor_based is False
|
||||
|
||||
records = ppu_records(outputs)
|
||||
if records is not None and len(records):
|
||||
self.ppu_scale_count = max(
|
||||
self.ppu_scale_count,
|
||||
ppu_scale_count_lower_bound(records, self.width, self.height),
|
||||
)
|
||||
if boxes is not None:
|
||||
if self.ppu_layout is None:
|
||||
self.ppu_scale_count = max(
|
||||
self.ppu_scale_count,
|
||||
ppu_scale_count_lower_bound(records, self.width, self.height),
|
||||
)
|
||||
|
||||
if not anchor_based and not self.ppu_needs_grid_decode:
|
||||
self.ppu_needs_grid_decode = ppu_needs_grid_decode(records)
|
||||
if self.ppu_layout is None:
|
||||
self.ppu_needs_grid_decode = ppu_needs_grid_decode(records)
|
||||
else:
|
||||
self.ppu_needs_grid_decode = self.ppu_layout.scale_count > 1
|
||||
|
||||
if (
|
||||
anchor_based is False
|
||||
and not self.ppu_needs_grid_decode
|
||||
and not self.ppu_centre_boxes
|
||||
):
|
||||
self.ppu_centre_boxes = ppu_boxes_are_centres(boxes)
|
||||
|
||||
if (
|
||||
anchor_based
|
||||
@ -951,8 +1233,13 @@ class DeepxDetector(DetectionApi):
|
||||
", ".join(str(n) for n in sorted(PPU_ANCHORS_BY_SCALES)),
|
||||
)
|
||||
|
||||
# None here means nothing conclusive in this frame, and no box to
|
||||
# draw either
|
||||
if self.ppu_layout is not None:
|
||||
needs_grid_decode = self.ppu_needs_grid_decode
|
||||
else:
|
||||
# None leaves it to the frame; only a proven True is kept
|
||||
needs_grid_decode = True if self.ppu_needs_grid_decode else None
|
||||
|
||||
# anchor_based None here means nothing conclusive; no box to draw
|
||||
return decode_ppu(
|
||||
outputs,
|
||||
self.width,
|
||||
@ -961,8 +1248,11 @@ class DeepxDetector(DetectionApi):
|
||||
NMS_THRESHOLD,
|
||||
anchor_based=anchor_based,
|
||||
scale_count=self.ppu_scale_count or None,
|
||||
needs_grid_decode=True if self.ppu_needs_grid_decode else None,
|
||||
needs_grid_decode=needs_grid_decode,
|
||||
records=records,
|
||||
# None leaves it to the frame; only a proven True is kept
|
||||
centre_boxes=True if self.ppu_centre_boxes else None,
|
||||
boxes=boxes,
|
||||
)
|
||||
|
||||
def decode_damoyolo(self, outputs: list[np.ndarray]) -> np.ndarray:
|
||||
|
||||
@ -1,7 +1,10 @@
|
||||
"""Tests for the DEEPX detector."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@ -17,6 +20,7 @@ from frigate.detectors.plugins.deepx import (
|
||||
PPU_RECORD_SIZE,
|
||||
DeepxDetector,
|
||||
DeepxDetectorConfig,
|
||||
PpuLayout,
|
||||
YoloLayout,
|
||||
class_count,
|
||||
decode_damoyolo_raw,
|
||||
@ -25,9 +29,11 @@ from frigate.detectors.plugins.deepx import (
|
||||
decode_raw_nms_in_head,
|
||||
infer_yolo_layout,
|
||||
ppu_scale_count_lower_bound,
|
||||
read_ppu_layout,
|
||||
resolve_device,
|
||||
rows_with_columns,
|
||||
validate_damoyolo_outputs,
|
||||
validate_yolox_outputs,
|
||||
)
|
||||
from frigate.util.runtime_deps import ArtifactKind
|
||||
|
||||
@ -56,6 +62,172 @@ def build_ppu_records(*records: np.ndarray) -> np.ndarray:
|
||||
return np.concatenate(records, axis=1)
|
||||
|
||||
|
||||
# compile_config.ppu as DX-COM writes it for the two head kinds; the layer
|
||||
# names inside are irrelevant to the reader, only the type is
|
||||
ANCHOR_BASED_PPU = {"type": 0, "num_classes": 80, "activation": "Sigmoid"}
|
||||
ANCHOR_FREE_PPU = {"type": 1, "num_classes": 80}
|
||||
|
||||
|
||||
def write_dxnn(directory, ppu, layers, name="model.dxnn", table=True) -> str:
|
||||
"""A minimal .dxnn as DX-RT's parsers read it: the container header, a
|
||||
compile_config carrying `ppu`, and either the PPU tensor table with one
|
||||
entry per (layer, anchor) as a v8 file has, or with `table` False only
|
||||
the rmap_info listing of the PPU output tensors, as a v7 file has.
|
||||
`layers` is (grid_w, grid_h, entries) per scale, finest first; an
|
||||
anchor-free scale has one entry, and grid_h 1 means a flattened
|
||||
(1, cells, channels) tensor."""
|
||||
compile_config = json.dumps({"compile_version": "2.4.0", "ppu": ppu}).encode()
|
||||
|
||||
if table:
|
||||
part = bytearray(struct.pack("<BBBB", 1, sum(n for _, _, n in layers), 0, 0))
|
||||
for conv, (grid_w, grid_h, entries) in enumerate(layers):
|
||||
for anchor in range(entries):
|
||||
part += struct.pack(
|
||||
"<HHfBBBBBBBB",
|
||||
128,
|
||||
80,
|
||||
0.001,
|
||||
conv,
|
||||
anchor,
|
||||
0,
|
||||
1,
|
||||
0,
|
||||
grid_w,
|
||||
grid_h,
|
||||
0,
|
||||
)
|
||||
else:
|
||||
outputs = []
|
||||
for conv, (grid_w, grid_h, entries) in enumerate(layers):
|
||||
for anchor in range(entries):
|
||||
name_ = f"PPU_Transpose_Output_{conv}"
|
||||
if entries > 1:
|
||||
name_ += f"_anchor_{anchor}"
|
||||
shape = [1, grid_w, 127] if grid_h == 1 else [1, grid_h, grid_w, 128]
|
||||
outputs.append({"name": name_, "shape": shape, "layout": "PPU_YOLO"})
|
||||
part = json.dumps({"inputs": [], "outputs": outputs}).encode()
|
||||
|
||||
index = json.dumps(
|
||||
{
|
||||
"version": 8 if table else 7,
|
||||
"signature": "DXNN",
|
||||
"size": 8192,
|
||||
"data": {
|
||||
"compile_config": {
|
||||
"type": "str",
|
||||
"offset": 0,
|
||||
"size": len(compile_config),
|
||||
},
|
||||
"compiled_data": {
|
||||
"M1A_4K": {
|
||||
"npu_0": {
|
||||
"rmap": {"type": "bytes", "offset": 0, "size": 0},
|
||||
"ppu" if table else "rmap_info": {
|
||||
"type": "bytes" if table else "str",
|
||||
"offset": len(compile_config),
|
||||
"size": len(part),
|
||||
},
|
||||
}
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
).encode()
|
||||
|
||||
path = os.path.join(directory, name)
|
||||
with open(path, "wb") as model:
|
||||
model.write(b"DXNN" + struct.pack("<I", 8))
|
||||
model.write(index.ljust(8192 - 8, b"\0"))
|
||||
model.write(compile_config + part)
|
||||
|
||||
return path
|
||||
|
||||
|
||||
class TestDeepxPpuLayoutFile(unittest.TestCase):
|
||||
"""The PPU head's kind and scale count are written into the .dxnn by
|
||||
DX-COM, so they are read from there rather than guessed from records."""
|
||||
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
|
||||
def test_reads_the_head_kind_and_one_grid_per_scale(self):
|
||||
# anchor-based: three anchors per scale collapse to one grid each
|
||||
path = write_dxnn(
|
||||
self.tmp.name, ANCHOR_BASED_PPU, [(80, 80, 3), (40, 40, 3), (20, 20, 3)]
|
||||
)
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path),
|
||||
PpuLayout(anchor_based=True, grids=((80, 80), (40, 40), (20, 20))),
|
||||
)
|
||||
|
||||
# anchor-free with every scale flattened into one tensor
|
||||
path = write_dxnn(self.tmp.name, ANCHOR_FREE_PPU, [(100, 84, 1)])
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path), PpuLayout(anchor_based=False, grids=((100, 84),))
|
||||
)
|
||||
|
||||
# a head kind the compiler does not name leaves the kind open but
|
||||
# still yields the scales
|
||||
path = write_dxnn(self.tmp.name, {"type": 7}, [(80, 80, 3), (40, 40, 3)])
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path),
|
||||
PpuLayout(anchor_based=None, grids=((80, 80), (40, 40))),
|
||||
)
|
||||
|
||||
def test_a_file_without_the_ppu_table_reads_its_output_tensors_instead(self):
|
||||
"""A v7-era compile lists its PPU output tensors in rmap_info but
|
||||
has no PPU table; their shapes hold the grids and their names the
|
||||
scale, and a per-anchor split says anchor-based when compile_config
|
||||
names no kind."""
|
||||
path = write_dxnn(
|
||||
self.tmp.name,
|
||||
{"num_classes": 80},
|
||||
[(80, 80, 3), (40, 40, 3), (20, 20, 3)],
|
||||
table=False,
|
||||
)
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path),
|
||||
PpuLayout(anchor_based=True, grids=((80, 80), (40, 40), (20, 20))),
|
||||
)
|
||||
|
||||
# compile_config's own output map places the scales ahead of the names
|
||||
outputs = {f"PPU_Transpose_Output_{i}": {"conv_idx": 2 - i} for i in range(3)}
|
||||
path = write_dxnn(
|
||||
self.tmp.name,
|
||||
{"type": 1, "outputs": outputs},
|
||||
[(20, 20, 1), (40, 40, 1), (80, 80, 1)],
|
||||
table=False,
|
||||
)
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path),
|
||||
PpuLayout(anchor_based=False, grids=((80, 80), (40, 40), (20, 20))),
|
||||
)
|
||||
|
||||
# every scale flattened into one (1, cells, channels) tensor
|
||||
path = write_dxnn(self.tmp.name, ANCHOR_FREE_PPU, [(8400, 1, 1)], table=False)
|
||||
self.assertEqual(
|
||||
read_ppu_layout(path), PpuLayout(anchor_based=False, grids=((8400, 1),))
|
||||
)
|
||||
|
||||
def test_nothing_is_read_from_a_model_without_ppu_metadata(self):
|
||||
self.assertIsNone(read_ppu_layout(os.path.join(self.tmp.name, "missing")))
|
||||
|
||||
# a non-PPU compile
|
||||
path = write_dxnn(self.tmp.name, None, [(80, 80, 3)])
|
||||
self.assertIsNone(read_ppu_layout(path))
|
||||
|
||||
# not a .dxnn at all
|
||||
with open(path, "wb") as model:
|
||||
model.write(b"ONNX" + b"\0" * 100)
|
||||
self.assertIsNone(read_ppu_layout(path))
|
||||
|
||||
# a table cut short of the entries it announces
|
||||
path = write_dxnn(self.tmp.name, ANCHOR_BASED_PPU, [(80, 80, 3), (40, 40, 3)])
|
||||
os.truncate(path, os.path.getsize(path) - 40)
|
||||
self.assertIsNone(read_ppu_layout(path))
|
||||
|
||||
|
||||
class TestDeepxPpuDecode(unittest.TestCase):
|
||||
def test_reads_anchor_free_box_bytes_as_pixel_geometry(self):
|
||||
"""The grid columns carry no meaning for an anchor-free head and must
|
||||
@ -164,12 +336,12 @@ class TestDeepxPpuDecode(unittest.TestCase):
|
||||
scale_count=2,
|
||||
)
|
||||
|
||||
# layer 0 of 2 is stride 16 and box 1 is the 37x58 anchor, so the
|
||||
# centre is (9.7, 7.3) cells out and the box is 13.32 x 113.68
|
||||
self.assertAlmostEqual(detections[0][2], 0.093688, places=5)
|
||||
self.assertAlmostEqual(detections[0][3], 0.232094, places=5)
|
||||
self.assertAlmostEqual(detections[0][4], 0.271313, places=5)
|
||||
self.assertAlmostEqual(detections[0][5], 0.252906, places=5)
|
||||
# layer 0 of 2 is stride 16 and box 1 is the 23x27 anchor, so the
|
||||
# centre is (9.7, 7.3) cells out and the box is 8.28 x 52.92
|
||||
self.assertAlmostEqual(detections[0][2], 0.141156, places=5)
|
||||
self.assertAlmostEqual(detections[0][3], 0.236031, places=5)
|
||||
self.assertAlmostEqual(detections[0][4], 0.223844, places=5)
|
||||
self.assertAlmostEqual(detections[0][5], 0.248969, places=5)
|
||||
|
||||
def test_scale_count_defaults_to_the_three_scale_table(self):
|
||||
"""With no scale_count given, a frame that only ever shows layer 0
|
||||
@ -271,6 +443,56 @@ class TestDeepxPpuDecode(unittest.TestCase):
|
||||
|
||||
self.assertTrue(np.all(detections == 0))
|
||||
|
||||
def test_corner_boxes_are_read_as_corners_until_a_record_rules_them_out(self):
|
||||
"""A decode-in-head PPU record holds either a centre and size or,
|
||||
for a corner-format head, the two corners in the same four fields.
|
||||
Corners always run x1 <= x2 and y1 <= y2, so a frame is read as
|
||||
corners unless one of its records has a width below its x or a
|
||||
height below its y, which only a centre box can."""
|
||||
detections = decode_ppu(
|
||||
[build_ppu_record((100.0, 50.0, 300.0, 250.0), label=2)],
|
||||
640,
|
||||
640,
|
||||
0.25,
|
||||
0.45,
|
||||
)
|
||||
self.assertEqual(detections[0][0], 2)
|
||||
self.assertAlmostEqual(detections[0][2], 50 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][3], 100 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][4], 250 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][5], 300 / 640, places=5)
|
||||
|
||||
# the same record beside one only a centre box could be
|
||||
detections = decode_ppu(
|
||||
[
|
||||
build_ppu_records(
|
||||
build_ppu_record((100.0, 50.0, 300.0, 250.0), label=2),
|
||||
build_ppu_record((320.0, 160.0, 64.0, 32.0), label=3, score=0.5),
|
||||
)
|
||||
],
|
||||
640,
|
||||
640,
|
||||
0.25,
|
||||
0.45,
|
||||
)
|
||||
self.assertEqual(detections[0][0], 2)
|
||||
# centre (100, 50), size 300 x 250, clipped at the frame edge
|
||||
self.assertAlmostEqual(detections[0][2], 0.0, places=5)
|
||||
self.assertAlmostEqual(detections[0][3], 0.0, places=5)
|
||||
self.assertAlmostEqual(detections[0][4], 175 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][5], 250 / 640, places=5)
|
||||
|
||||
# and told centres outright, no record is read as corners
|
||||
detections = decode_ppu(
|
||||
[build_ppu_record((100.0, 50.0, 300.0, 250.0), label=2)],
|
||||
640,
|
||||
640,
|
||||
0.25,
|
||||
0.45,
|
||||
centre_boxes=True,
|
||||
)
|
||||
self.assertAlmostEqual(detections[0][5], 250 / 640, places=5)
|
||||
|
||||
def test_needs_grid_decode_true_is_kept_for_a_single_layer_frame(self):
|
||||
"""A caller that already proved the head needs grid decoding must
|
||||
not have that overridden by a frame that, alone, looks ambiguous."""
|
||||
@ -426,6 +648,28 @@ class TestDeepxDamoyoloValidation(unittest.TestCase):
|
||||
validate_damoyolo_outputs([(1, 8400, 80), (1, 8000, 4)], 80)
|
||||
|
||||
|
||||
class TestDeepxYoloxValidation(unittest.TestCase):
|
||||
"""model_type yolox takes the raw YOLOX head: one (1, N, 5+C) tensor with
|
||||
N the cells of strides 8, 16 and 32 for the configured input."""
|
||||
|
||||
def test_the_raw_head_passes_in_either_orientation(self):
|
||||
self.assertEqual(validate_yolox_outputs([(1, 8400, 85)], 80, 640, 640), 85)
|
||||
self.assertEqual(validate_yolox_outputs([(1, 85, 8400)], 80, 640, 640), 85)
|
||||
# 52*52 + 26*26 + 13*13 cells at 416
|
||||
self.assertEqual(validate_yolox_outputs([(1, 3549, 85)], 80, 416, 416), 85)
|
||||
|
||||
def test_the_wrong_input_size_or_label_map_is_named(self):
|
||||
with self.assertRaisesRegex(ValueError, "width and height"):
|
||||
validate_yolox_outputs([(1, 8400, 85)], 80, 416, 416)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "labelmap_path"):
|
||||
validate_yolox_outputs([(1, 8400, 85)], 91, 640, 640)
|
||||
|
||||
def test_another_head_names_yolo_generic(self):
|
||||
with self.assertRaisesRegex(ValueError, "yolo-generic"):
|
||||
validate_yolox_outputs([(1, 8400, 80), (1, 8400, 4)], 80, 640, 640)
|
||||
|
||||
|
||||
class TestDeepxClassCount(unittest.TestCase):
|
||||
def test_the_highest_id_bounds_the_count(self):
|
||||
self.assertEqual(class_count({0: "person", 79: "toothbrush"}), 80)
|
||||
@ -743,7 +987,11 @@ class TestDeepxModelType(unittest.TestCase):
|
||||
DeepxDetectorConfig(type="deepx", model=model_with_type(model_type))
|
||||
|
||||
def test_supported_model_types_are_accepted(self):
|
||||
for model_type in (ModelTypeEnum.yologeneric, ModelTypeEnum.damoyolo):
|
||||
for model_type in (
|
||||
ModelTypeEnum.yologeneric,
|
||||
ModelTypeEnum.yolox,
|
||||
ModelTypeEnum.damoyolo,
|
||||
):
|
||||
with self.subTest(model_type=model_type):
|
||||
config = DeepxDetectorConfig(
|
||||
type="deepx", model=model_with_type(model_type)
|
||||
@ -865,7 +1113,14 @@ class TestDeepxDetectorLayout(unittest.TestCase):
|
||||
"""The detector asks the runtime about the loaded model and keeps the
|
||||
answer; the per-frame decode only dispatches on it."""
|
||||
|
||||
def _detector(self, model_type, outputs_info, ppu=False, dynamic=False):
|
||||
def _detector(
|
||||
self,
|
||||
model_type,
|
||||
outputs_info,
|
||||
ppu=False,
|
||||
dynamic=False,
|
||||
model_path="/nonexistent/model.dxnn",
|
||||
):
|
||||
dx_engine = MagicMock()
|
||||
dx_engine.Configuration.ITEM.SERVICE = object()
|
||||
session = dx_engine.InferenceEngine.return_value
|
||||
@ -876,10 +1131,14 @@ class TestDeepxDetectorLayout(unittest.TestCase):
|
||||
config = DeepxDetectorConfig(
|
||||
type="deepx",
|
||||
model=ModelConfig(
|
||||
model_type=model_type, labelmap_path=None, labelmap={79: "toothbrush"}
|
||||
model_type=model_type,
|
||||
labelmap_path=None,
|
||||
labelmap={79: "toothbrush"},
|
||||
width=640,
|
||||
height=640,
|
||||
),
|
||||
)
|
||||
config.model.path = "/nonexistent/model.dxnn"
|
||||
config.model.path = model_path
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"dx_engine": dx_engine}),
|
||||
@ -993,8 +1252,8 @@ class TestDeepxDetectorLayout(unittest.TestCase):
|
||||
]
|
||||
third = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
# the same layer-0 record as the first frame, now read against the
|
||||
# two-scale table proven by the second frame: stride 16, anchor 37x58
|
||||
self.assertAlmostEqual(third[0][3], 0.232094, places=5)
|
||||
# two-scale table proven by the second frame: stride 16, anchor 23x27
|
||||
self.assertAlmostEqual(third[0][3], 0.236031, places=5)
|
||||
|
||||
def test_ppu_scale_count_grows_when_a_grid_cell_proves_a_third_scale(self):
|
||||
"""A three-scale head whose early frames only ever hold layers 0
|
||||
@ -1025,6 +1284,107 @@ class TestDeepxDetectorLayout(unittest.TestCase):
|
||||
# the same record, now layer 1 of 3: stride 16, anchor 30x61
|
||||
self.assertAlmostEqual(third[0][3], 0.0975, places=5)
|
||||
|
||||
def _ppu_detector_for(self, ppu, layers):
|
||||
"""A PPU detector loaded from a model file that describes its head."""
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
path = write_dxnn(tmp.name, ppu, layers)
|
||||
|
||||
detector, _ = self._detector(
|
||||
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True, model_path=path
|
||||
)
|
||||
detector.width, detector.height = 640, 640
|
||||
return detector
|
||||
|
||||
def test_the_ppu_scale_count_is_read_from_the_model_at_load(self):
|
||||
"""A three-scale head reads against the three-scale table from its
|
||||
first frame. A layer-0 record near the origin followed by a layer-1
|
||||
one is exactly what record evidence alone would call a two-scale
|
||||
head; the model file says otherwise and the count never moves."""
|
||||
detector = self._ppu_detector_for(
|
||||
ANCHOR_BASED_PPU, [(80, 80, 3), (40, 40, 3), (20, 20, 3)]
|
||||
)
|
||||
self.assertEqual(detector.ppu_scale_count, 3)
|
||||
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((0.6, 0.4, 0.3, 0.7), grid=(7, 9, 1, 0))
|
||||
]
|
||||
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
# layer 0 of 3: stride 8, anchor 16x30
|
||||
self.assertAlmostEqual(first[0][3], 0.116750, places=5)
|
||||
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((0.5, 0.5, 0.4, 0.4), grid=(3, 4, 0, 1))
|
||||
]
|
||||
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
# layer 1 of 3: stride 16, anchor 30x61, not layer 1 of 2
|
||||
self.assertAlmostEqual(second[0][3], 0.0975, places=5)
|
||||
self.assertEqual(detector.ppu_scale_count, 3)
|
||||
|
||||
def test_an_anchor_free_model_settles_the_head_at_load(self):
|
||||
"""With the head kind in the file, a sub-pixel box in the corner,
|
||||
which the per-frame heuristic would take for an anchor-based
|
||||
record, is read as the pixels it is; and a YOLOX-style head is
|
||||
grid-decoded from a first frame holding one record at one layer."""
|
||||
detector = self._ppu_detector_for(ANCHOR_FREE_PPU, [(100, 84, 1)])
|
||||
self.assertTrue(detector.ppu_anchor_free)
|
||||
self.assertFalse(detector.ppu_needs_grid_decode)
|
||||
|
||||
detector.session.run.return_value = [build_ppu_record((0.6, 0.4, 0.3, 0.7))]
|
||||
detections = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
self.assertAlmostEqual(detections[0][3], 0.45 / 640, places=6)
|
||||
|
||||
detector = self._ppu_detector_for(
|
||||
ANCHOR_FREE_PPU, [(80, 80, 1), (40, 40, 1), (20, 20, 1)]
|
||||
)
|
||||
self.assertTrue(detector.ppu_needs_grid_decode)
|
||||
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((1.2, 0.5, 1.0, 0.5), grid=(9, 10, 0, 2), label=7)
|
||||
]
|
||||
detections = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
self.assertAlmostEqual(detections[0][3], 0.492043, places=5)
|
||||
|
||||
def test_an_unreadable_model_file_falls_back_to_record_evidence(self):
|
||||
with self.assertLogs(
|
||||
"frigate.detectors.plugins.deepx", level="WARNING"
|
||||
) as logs:
|
||||
detector, _ = self._detector(
|
||||
ModelTypeEnum.yologeneric, [{"shape": [8400]}], ppu=True
|
||||
)
|
||||
|
||||
self.assertTrue(any("PPU" in r.getMessage() for r in logs.records))
|
||||
self.assertIsNone(detector.ppu_layout)
|
||||
self.assertEqual(detector.ppu_scale_count, 0)
|
||||
|
||||
def test_centre_boxes_are_kept_once_a_record_proves_them(self):
|
||||
"""A centre-format head is settled by its first record with a width
|
||||
below its x, and a later frame whose boxes all happen to sit far
|
||||
enough left and down to pass for corners is still read as centres.
|
||||
A corner-format head never produces such a record and stays corners."""
|
||||
detector = self._ppu_detector_for(ANCHOR_FREE_PPU, [(100, 84, 1)])
|
||||
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((320.0, 160.0, 64.0, 32.0), label=3)
|
||||
]
|
||||
detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
self.assertTrue(detector.ppu_centre_boxes)
|
||||
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((100.0, 50.0, 300.0, 250.0), label=2)
|
||||
]
|
||||
second = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
# centre (100, 50), size 300 x 250: the right edge lands at 250
|
||||
self.assertAlmostEqual(second[0][5], 250 / 640, places=5)
|
||||
|
||||
detector = self._ppu_detector_for(ANCHOR_FREE_PPU, [(100, 84, 1)])
|
||||
detector.session.run.return_value = [
|
||||
build_ppu_record((100.0, 50.0, 300.0, 250.0), label=2)
|
||||
]
|
||||
first = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
self.assertFalse(detector.ppu_centre_boxes)
|
||||
self.assertAlmostEqual(first[0][5], 300 / 640, places=5)
|
||||
|
||||
def test_needs_grid_decode_is_kept_once_proven(self):
|
||||
"""A frame proving a YOLOX-style grid-relative head must not fall
|
||||
back to reading raw pixels just because a later frame holds only
|
||||
@ -1089,6 +1449,43 @@ class TestDeepxDetectorLayout(unittest.TestCase):
|
||||
self.assertTrue(np.all(detections == 0))
|
||||
self.assertFalse(detector.ppu_anchor_free)
|
||||
|
||||
def test_a_yolox_raw_head_is_decoded_through_the_grid(self):
|
||||
"""model_type yolox reads the raw head: a cell's x, y offsets are
|
||||
added to its grid position and scaled by the stride, its w, h are
|
||||
exp'd and scaled the same way, and confidence is objectness times
|
||||
the best class score."""
|
||||
detector, _ = self._detector(ModelTypeEnum.yolox, [{"shape": [1, 8400, 85]}])
|
||||
self.assertIs(detector.output.layout, YoloLayout.yolox)
|
||||
|
||||
# cell (x 10, y 5) of the stride-8 grid is row 5 * 80 + 10
|
||||
tensor = np.zeros((1, 8400, 85), np.float32)
|
||||
tensor[0, 410, :5] = [0.5, 0.5, np.log(4.0), np.log(2.0), 0.9]
|
||||
tensor[0, 410, 5 + 7] = 0.8
|
||||
detector.session.run.return_value = [tensor]
|
||||
detector.width, detector.height = 640, 640
|
||||
|
||||
detections = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
|
||||
# centre (84, 44), size 32 x 16
|
||||
self.assertEqual(detections[0][0], 7)
|
||||
self.assertAlmostEqual(detections[0][1], 0.72, places=5)
|
||||
self.assertAlmostEqual(detections[0][2], 36 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][3], 68 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][4], 52 / 640, places=5)
|
||||
self.assertAlmostEqual(detections[0][5], 100 / 640, places=5)
|
||||
|
||||
# a channel-major export is oriented first
|
||||
detector, _ = self._detector(ModelTypeEnum.yolox, [{"shape": [1, 85, 8400]}])
|
||||
detector.session.run.return_value = [np.swapaxes(tensor, 1, 2)]
|
||||
detector.width, detector.height = 640, 640
|
||||
detections = detector.detect_raw(np.zeros((1, 640, 640, 3), np.uint8))
|
||||
self.assertAlmostEqual(detections[0][5], 100 / 640, places=5)
|
||||
|
||||
def test_a_yolox_compiled_with_ppu_support_is_read_as_ppu(self):
|
||||
detector, _ = self._detector(ModelTypeEnum.yolox, [{"shape": [8400]}], ppu=True)
|
||||
|
||||
self.assertIs(detector.output.layout, YoloLayout.ppu)
|
||||
|
||||
def test_damoyolo_is_checked_against_the_model_at_load(self):
|
||||
"""A YOLO model or a wrong label map under damo-yolo fails at load
|
||||
with the reason, rather than returning nothing on every frame."""
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user