diff --git a/docs/data/object_detectors_models.yaml b/docs/data/object_detectors_models.yaml index d9592d85f0..2fcbe652b7 100644 --- a/docs/data/object_detectors_models.yaml +++ b/docs/data/object_detectors_models.yaml @@ -1205,3 +1205,33 @@ axengine: input_dtype: int input_pixel_format: bgr labelmap_path: /labelmap/coco-80.txt +axelera: + title: Axelera Metis + models: + - key: yolox + label: YOLOX + recommended: true + download: |- + Axelera models must be compiled for the Metis NPU with the Voyager SDK before Frigate can run them (compilation is not supported inside the container). Compile a model with the SDK, place the compiled output directory under `/config`, and point the model's `path` at it. The default recommendation is the Apache-2.0 licensed YOLOX (Megvii YOLOX, e.g. the `yolox-s-coco-onnx` model-zoo network). + ui: |- + Navigate to **Settings > System > Detection models** and select **Axelera Metis** from the **Hardware** dropdown. Then, on the same model, open the **Custom Model** tab and configure: + + | Field | Value | + | ---------------------------------------- | ------------------------------------------------------------ | + | **Custom object detector model path** | `/config/model_cache/axelera/yolox-s-coco-onnx` | + | **Label map for custom object detector** | `/labelmap/coco-80.txt` | + | **Object detection model input width** | `704` (the frame size Frigate sends; boxes map back to it) | + | **Object detection model input height** | `576` | + | **Model Input Pixel Color Format** | `rgb` (Frigate's default value) | + | **Model Input Tensor Shape** | `nhwc` (Frigate's default value) | + | **Model Input D Type** | `int` (Frigate's default value) | + | **Object Detection Model Type** | `yolox` | + yaml: |- + models: + - devices: + - axelera:PCIe + path: /config/model_cache/axelera/yolox-s-coco-onnx + model_type: yolox + width: 704 + height: 576 + labelmap_path: /labelmap/coco-80.txt diff --git a/docs/docs/configuration/object_detectors.md b/docs/docs/configuration/object_detectors.md index 755d4e24ff..bcb3596a02 100644 --- a/docs/docs/configuration/object_detectors.md +++ b/docs/docs/configuration/object_detectors.md @@ -60,6 +60,10 @@ Frigate supports multiple different detectors that work on different types of ha - [AXEngine](#axera): axmodels can run on AXERA AI acceleration. +**Axelera** + +- [Axelera Metis](#axelera-metis): The Metis AI Accelerator is available in M.2 and PCIe formats, running compiled YOLO models through the vendor runtime. + **For Testing** - [CPU Detector (not recommended for actual use](#cpu-detector-not-recommended): Use a CPU to run tflite model, this is not recommended and in most cases OpenVINO can be used in CPU mode with better results. @@ -97,7 +101,7 @@ models: - openvino:GPU ``` -Coral EdgeTPU and MemryX accelerators can only be opened by one process, so those devices can not be repeated. +Coral EdgeTPU, MemryX, and Axelera Metis accelerators can only be opened by one process, so those devices can not be repeated. ### Running more than one model @@ -851,3 +855,45 @@ The AXEngine python package is not part of the Frigate image. It is downloaded a When configuring the AXEngine detector, you have to specify the model name. + +--- + +## Axelera Metis + + + +Hardware accelerated object detection is supported on the Axelera Metis AI Accelerator, available in M.2 and PCIe card formats. Inference runs through Axelera's low-level runtime (`axelera.runtime`), which executes compiled YOLO-family models with a flat per-frame latency. + +See the [installation docs](../frigate/installation.md#axelera-metis) for information on configuring the Metis hardware. + +:::info + +The Axelera runtime is not part of the Frigate image. It is downloaded and installed into `/config/.local` the first time an Axelera detector is configured, verified against pinned checksums, and updated automatically when a Frigate release pins a new version. If the container has no internet access, see [Detector runtimes](/frigate/network_requirements#detector-runtimes) for how to provide the files yourself. + +::: + +### Configuration {#configuration-axelera} + + + +#### Using a Custom Model + +Models must be compiled for the Metis NPU with the Axelera Voyager SDK before Frigate can run them. The runtime consumes the compiled output directory, which contains `1/model.json`, `1/manifest.json`, and the baked `1/postprocess_graph.onnx`. + +Compile the model on a machine with the Voyager SDK installed (compilation is not supported inside the Frigate container), for example with the SDK's `deploy.py` against a model-zoo yaml. Then either: + +- place the compiled directory under `/config` and point the model's `path` at it, or +- zip the compiled directory and point `path` at the zip (bind-mounted, or at a URL). + +```yaml +models: + - devices: + - axelera:PCIe + path: /config/model_cache/axelera/yolox-s-coco-onnx + width: 704 # the frame size Frigate sends to the model + height: 576 + model_type: yolox + labelmap_path: /labelmap/coco-80.txt +``` + +A bare preset name (no slash or dot, e.g. `yolox-s-coco-onnx`) is also accepted and resolved under `AXELERA_BUILD_ROOT` (default `/opt/voyager-sdk/build`) for setups that build models into the image. diff --git a/docs/docs/frigate/installation.md b/docs/docs/frigate/installation.md index 880292f232..349fa46166 100644 --- a/docs/docs/frigate/installation.md +++ b/docs/docs/frigate/installation.md @@ -503,6 +503,41 @@ If you are using `docker run`, add this option to your command `--device /dev/ax Finally, configure [hardware object detection](/configuration/object_detectors#axera) to complete the setup. +### Axelera Metis + +The Axelera Metis AI Accelerator is available in the M.2 and PCIe card form factors and is supported on x86 (Intel/AMD) PCs and Rock 5B class arm64 boards. + +The Axelera runtime used inside the container is not part of the Frigate image; Frigate downloads and installs it at first start once an Axelera detector is configured. Containers without internet access can provide the files themselves, see [Detector runtimes](/frigate/network_requirements#detector-runtimes). The host side kernel driver still has to be installed as described below. + +#### Installation + +Install the Metis PCIe kernel driver and firmware on the **host** following [Axelera's driver installation guide](https://developer.axelera.ai/axelera-documentation/Getting-Started/GettingStartedWithMetis/InstallingAxeleraMetisDriver). The driver creates the `/dev/metis*` device nodes. Reboot after installing it. + +#### Setup + +To set up Frigate, follow the default installation instructions, for example: `ghcr.io/blakeblackshear/frigate:stable` + +Next, grant Docker permissions to access your hardware by adding the following lines to your `docker-compose.yml` file: + +```yaml +devices: + - /dev/dma_heap/system + # the exact node is printed by `ls /dev/metis*`, e.g. /dev/metis-0-1-0 + - /dev/metis-0-1-0 +``` + +If you are using `docker run`, add `--device /dev/dma_heap/system --device /dev/metis-0-1-0` to your command. + +:::warning + +DMA-BUF must be available on the host kernel. On Debian/Ubuntu this is provided by the default kernel; on other systems it may need enabling (`CONFIG_DMABUF_HEAPS_SYSTEM`). If Frigate's log reports that no DMA heap can be allocated, check that `/dev/dma_heap/system` exists on the host and is passed into the container. + +::: + +#### Configuration + +Finally, configure [hardware object detection](/configuration/object_detectors#axelera-metis) to complete the setup. + ## Docker Running through Docker with Docker Compose is the recommended install method. diff --git a/docs/docs/frigate/network_requirements.md b/docs/docs/frigate/network_requirements.md index 692811041c..c8363d2e16 100644 --- a/docs/docs/frigate/network_requirements.md +++ b/docs/docs/frigate/network_requirements.md @@ -65,6 +65,7 @@ The SDKs for a few hardware detectors are not shipped in the Frigate image. They | [Hailo 8 / 8L](/configuration/object_detectors#hailo-8) | 4.21.0 | `hailort-debian12-amd64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_x86_64.whl` on x86, `hailort-debian12-arm64.tar.gz` and `hailort-4.21.0-cp311-cp311-linux_aarch64.whl` on arm64 | [GitHub release](https://github.com/frigate-nvr/hailort/releases/tag/v4.21.0) | | [MemryX MX3](/configuration/object_detectors#memryx-mx3) | 2.1.0 | `mx_accl_frigate-2.1.0.zip` (the release source archive, renamed) | [GitHub archive](https://github.com/memryx/mx_accl_frigate/archive/refs/tags/v2.1.0.zip) | | [AXERA AXEngine](/configuration/object_detectors#axera) | 0.1.3 | `axengine-0.1.3-py3-none-any.whl` | [GitHub release](https://github.com/AXERA-TECH/pyaxengine/releases/tag/0.1.3-frigate) | +| [Axelera Metis](/configuration/object_detectors#axelera-metis) | 1.8.0 | `axelera_runtime-1.8.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl` and `axelera_runtime2-0.2.0-cp311-cp311-manylinux_2_28_x86_64.whl` on x86, `axelera_runtime-1.8.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl` and `axelera_runtime2-0.2.0-cp311-cp311-manylinux_2_28_aarch64.whl` on arm64, plus `axelera_firmware-1.8.0-py3-none-any.whl` on both | [Axelera Artifactory PyPI](https://software.axelera.ai/artifactory/#web/ui/redirect/pypi/axelera-pypi/simple) | If the container cannot reach GitHub, provide the files yourself: