Compare commits

..

No commits in common. "0febc4d4564ee0ddb6aa3334391b8d72d3e330ed" and "d3af748366ff8f880771d57a3ee0743f79d9a467" have entirely different histories.

21 changed files with 43 additions and 276 deletions

View File

@ -79,6 +79,4 @@ tflite_runtime @ https://github.com/feranick/TFlite-builds/releases/download/v2.
sherpa-onnx==1.12.*
faster-whisper==1.1.*
librosa==0.11.*
soundfile==0.13.*
# DeGirum detector
degirum == 0.16.*
soundfile==0.13.*

View File

@ -14,7 +14,6 @@ Frigate supports multiple different detectors that work on different types of ha
- [Coral EdgeTPU](#edge-tpu-detector): The Google Coral EdgeTPU is available in USB and m.2 format allowing for a wide range of compatibility with devices.
- [Hailo](#hailo-8): The Hailo8 and Hailo8L AI Acceleration module is available in m.2 format with a HAT for RPi devices, offering a wide range of compatibility with devices.
- [MemryX](#memryx-mx3): The MX3 Acceleration module is available in m.2 format, offering broad compatibility across various platforms.
- [DeGirum](#degirum): Service for using hardware devices in the cloud or locally. Hardware and models provided on the cloud on [their website](https://hub.degirum.com).
**AMD**
@ -246,6 +245,8 @@ Hailo8 supports all models in the Hailo Model Zoo that include HailoRT post-proc
---
## OpenVINO Detector
The OpenVINO detector type runs an OpenVINO IR model on AMD and Intel CPUs, Intel GPUs and Intel VPU hardware. To configure an OpenVINO detector, set the `"type"` attribute to `"openvino"`.
@ -1199,101 +1200,6 @@ Explanation of the paramters:
- **example**: Specifying `output_name = "frigate-{quant}-{input_basename}-{soc}-v{tk_version}"` could result in a model called `frigate-i8-my_model-rk3588-v2.3.0.rknn`.
- `config`: Configuration passed to `rknn-toolkit2` for model conversion. For an explanation of all available parameters have a look at section "2.2. Model configuration" of [this manual](https://github.com/MarcA711/rknn-toolkit2/releases/download/v2.3.2/03_Rockchip_RKNPU_API_Reference_RKNN_Toolkit2_V2.3.2_EN.pdf).
## DeGirum
DeGirum is a detector that can use any type of hardware listed on [their website](https://hub.degirum.com). DeGirum can be used with local hardware through a DeGirum AI Server, or through the use of `@local`. You can also connect directly to DeGirum's AI Hub to run inferences. **Please Note:** This detector *cannot* be used for commercial purposes.
### Configuration
#### AI Server Inference
Before starting with the config file for this section, you must first launch an AI server. DeGirum has an AI server ready to use as a docker container. Add this to your `docker-compose.yml` to get started:
```yaml
degirum_detector:
container_name: degirum
image: degirum/aiserver:latest
privileged: true
ports:
- "8778:8778"
```
All supported hardware will automatically be found on your AI server host as long as relevant runtimes and drivers are properly installed on your machine. Refer to [DeGirum's docs site](https://docs.degirum.com/pysdk/runtimes-and-drivers) if you have any trouble.
Once completed, changing the `config.yml` file is simple.
```yaml
degirum_detector:
type: degirum
location: degirum # Set to service name (degirum_detector), container_name (degirum), or a host:port (192.168.29.4:8778)
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start. If you aren't pulling a model from the AI Hub, leave this and 'token' blank.
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the [AI Hub](https://hub.degirum.com). This can be left blank if you're pulling a model from the public zoo and running inferences on your local hardware using @local or a local DeGirum AI Server
```
Setting up a model in the `config.yml` is similar to setting up an AI server.
You can set it to:
- A model listed on the [AI Hub](https://hub.degirum.com), given that the correct zoo name is listed in your detector
- If this is what you choose to do, the correct model will be downloaded onto your machine before running.
- A local directory acting as a zoo. See DeGirum's docs site [for more information](https://docs.degirum.com/pysdk/user-guide-pysdk/organizing-models#model-zoo-directory-structure).
- A path to some model.json.
```yaml
model:
path: ./mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1 # directory to model .json and file
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
#### Local Inference
It is also possible to eliminate the need for an AI server and run the hardware directly. The benefit of this approach is that you eliminate any bottlenecks that occur when transferring prediction results from the AI server docker container to the frigate one. However, the method of implementing local inference is different for every device and hardware combination, so it's usually more trouble than it's worth. A general guideline to achieve this would be:
1. Ensuring that the frigate docker container has the runtime you want to use. So for instance, running `@local` for Hailo means making sure the container you're using has the Hailo runtime installed.
2. To double check the runtime is detected by the DeGirum detector, make sure the `degirum sys-info` command properly shows whatever runtimes you mean to install.
3. Create a DeGirum detector in your `config.yml` file.
```yaml
degirum_detector:
type: degirum
location: "@local" # For accessing AI Hub devices and models
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start.
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the [AI Hub](https://hub.degirum.com). This can be left blank if you're pulling a model from the public zoo and running inferences on your local hardware using @local or a local DeGirum AI Server
```
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
```yaml
model:
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
#### AI Hub Cloud Inference
If you do not possess whatever hardware you want to run, there's also the option to run cloud inferences. Do note that your detection fps might need to be lowered as network latency does significantly slow down this method of detection. For use with Frigate, we highly recommend using a local AI server as described above. To set up cloud inferences,
1. Sign up at [DeGirum's AI Hub](https://hub.degirum.com).
2. Get an access token.
3. Create a DeGirum detector in your `config.yml` file.
```yaml
degirum_detector:
type: degirum
location: "@cloud" # For accessing AI Hub devices and models
zoo: degirum/public # DeGirum's public model zoo. Zoo name should be in format "workspace/zoo_name". degirum/public is available to everyone, so feel free to use it if you don't know where to start.
token: dg_example_token # For authentication with the AI Hub. Get this token through the "tokens" section on the main page of the (AI Hub)[https://hub.degirum.com).
```
Once `degirum_detector` is setup, you can choose a model through 'model' section in the `config.yml` file.
```yaml
model:
path: mobilenet_v2_ssd_coco--300x300_quant_n2x_orca1_1
width: 300 # width is in the model name as the first number in the "int"x"int" section
height: 300 # height is in the model name as the second number in the "int"x"int" section
input_pixel_format: rgb/bgr # look at the model.json to figure out which to put here
```
# Models
Some model types are not included in Frigate by default.

View File

@ -899,7 +899,7 @@ cameras:
# Optional: Configuration for triggers to automate actions based on semantic search results.
triggers:
# Required: Unique identifier for the trigger (generated automatically from friendly_name if not specified).
# Required: Unique identifier for the trigger (generated automatically from nickname if not specified).
trigger_name:
# Required: Enable or disable the trigger. (default: shown below)
enabled: true

View File

@ -335,7 +335,7 @@ class WebPushClient(Communicator):
camera: str = payload["after"]["camera"]
camera_name: str = getattr(
self.config.cameras[camera], "friendly_name", None
self.config.cameras[camera], "nickname", None
) or titlecase(camera.replace("_", " "))
current_time = datetime.datetime.now().timestamp()
@ -410,7 +410,7 @@ class WebPushClient(Communicator):
camera: str = payload["camera"]
camera_name: str = getattr(
self.config.cameras[camera], "friendly_name", None
self.config.cameras[camera], "nickname", None
) or titlecase(camera.replace("_", " "))
current_time = datetime.datetime.now().timestamp()

View File

@ -52,14 +52,12 @@ class CameraTypeEnum(str, Enum):
class CameraConfig(FrigateBaseModel):
name: Optional[str] = Field(None, title="Camera name.", pattern=REGEX_CAMERA_NAME)
friendly_name: Optional[str] = Field(
None, title="Camera friendly name used in the Frigate UI."
)
nickname: Optional[str] = Field(None, title="Camera nickname. Only for display.")
@model_validator(mode="before")
@classmethod
def handle_friendly_name(cls, values):
if isinstance(values, dict) and "friendly_name" in values:
def handle_nickname(cls, values):
if isinstance(values, dict) and "nickname" in values:
pass
return values

View File

@ -1,135 +0,0 @@
import logging
import queue
import degirum as dg
import numpy as np
from pydantic import Field
from typing_extensions import Literal
from frigate.detectors.detection_api import DetectionApi
from frigate.detectors.detector_config import BaseDetectorConfig
logger = logging.getLogger(__name__)
DETECTOR_KEY = "degirum"
### DETECTOR CONFIG ###
class DGDetectorConfig(BaseDetectorConfig):
type: Literal[DETECTOR_KEY]
location: str = Field(default=None, title="Inference Location")
zoo: str = Field(default=None, title="Model Zoo")
token: str = Field(default=None, title="DeGirum Cloud Token")
### ACTUAL DETECTOR ###
class DGDetector(DetectionApi):
type_key = DETECTOR_KEY
def __init__(self, detector_config: DGDetectorConfig):
self._queue = queue.Queue()
self._zoo = dg.connect(
detector_config.location, detector_config.zoo, detector_config.token
)
logger.debug(f"Models in zoo: {self._zoo.list_models()}")
self.dg_model = self._zoo.load_model(
detector_config.model.path,
)
# Setting input image format to raw reduces preprocessing time
self.dg_model.input_image_format = "RAW"
# Prioritize the most powerful hardware available
self.select_best_device_type()
# Frigate handles pre processing as long as these are all set
input_shape = self.dg_model.input_shape[0]
self.model_height = input_shape[1]
self.model_width = input_shape[2]
# Passing in dummy frame so initial connection latency happens in
# init function and not during actual prediction
frame = np.zeros(
(detector_config.model.width, detector_config.model.height, 3),
dtype=np.uint8,
)
# Pass in frame to overcome first frame latency
self.dg_model(frame)
self.prediction = self.prediction_generator()
def select_best_device_type(self):
"""
Helper function that selects fastest hardware available per model runtime
"""
types = self.dg_model.supported_device_types
device_map = {
"OPENVINO": ["GPU", "NPU", "CPU"],
"HAILORT": ["HAILO8L", "HAILO8"],
"N2X": ["ORCA1", "CPU"],
"ONNX": ["VITIS_NPU", "CPU"],
"RKNN": ["RK3566", "RK3568", "RK3588"],
"TENSORRT": ["DLA", "GPU", "DLA_ONLY"],
"TFLITE": ["ARMNN", "EDGETPU", "CPU"],
}
runtime = types[0].split("/")[0]
# Just create an array of format {runtime}/{hardware} for every hardware
# in the value for appropriate key in device_map
self.dg_model.device_type = [
f"{runtime}/{hardware}" for hardware in device_map[runtime]
]
def prediction_generator(self):
"""
Generator for all incoming frames. By using this generator, we don't have to keep
reconnecting our websocket on every "predict" call.
"""
logger.debug("Prediction generator was called")
with self.dg_model as model:
while 1:
logger.info(f"q size before calling get: {self._queue.qsize()}")
data = self._queue.get(block=True)
logger.info(f"q size after calling get: {self._queue.qsize()}")
logger.debug(
f"Data we're passing into model predict: {data}, shape of data: {data.shape}"
)
result = model.predict(data)
logger.debug(f"Prediction result: {result}")
yield result
def detect_raw(self, tensor_input):
# Reshaping tensor to work with pysdk
truncated_input = tensor_input.reshape(tensor_input.shape[1:])
logger.debug(f"Detect raw was called for tensor input: {tensor_input}")
# add tensor_input to input queue
self._queue.put(truncated_input)
logger.debug(f"Queue size after adding truncated input: {self._queue.qsize()}")
# define empty detection result
detections = np.zeros((20, 6), np.float32)
# grab prediction
res = next(self.prediction)
# If we have an empty prediction, return immediately
if len(res.results) == 0 or len(res.results[0]) == 0:
return detections
i = 0
for result in res.results:
if i >= 20:
break
detections[i] = [
result["category_id"],
float(result["score"]),
result["bbox"][1] / self.model_height,
result["bbox"][0] / self.model_width,
result["bbox"][3] / self.model_height,
result["bbox"][2] / self.model_width,
]
i += 1
logger.debug(f"Detections output: {detections}")
return detections

View File

@ -78,7 +78,7 @@ class StorageMaintainer(threading.Thread):
)
camera_key = (
getattr(self.config.cameras[camera], "friendly_name", None) or camera
getattr(self.config.cameras[camera], "nickname", None) or camera
)
usages[camera_key] = {
"usage": camera_storage,

View File

@ -1,6 +1,6 @@
import * as React from "react";
import * as LabelPrimitive from "@radix-ui/react-label";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
import { CameraConfig } from "@/types/frigateConfig";
interface CameraNameLabelProps
@ -12,7 +12,7 @@ const CameraNameLabel = React.forwardRef<
React.ElementRef<typeof LabelPrimitive.Root>,
CameraNameLabelProps
>(({ className, camera, ...props }, ref) => {
const displayName = useCameraFriendlyName(camera);
const displayName = useCameraNickname(camera);
return (
<LabelPrimitive.Root ref={ref} className={className} {...props}>
{displayName}

View File

@ -16,7 +16,7 @@ import axios from "axios";
import { toast } from "sonner";
import { Toaster } from "../ui/sonner";
import { Trans, useTranslation } from "react-i18next";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type CameraInfoDialogProps = {
camera: CameraConfig;
@ -75,7 +75,7 @@ export default function CameraInfoDialog({
return b === 0 ? a : gcd(b, a % b);
}
const cameraName = useCameraFriendlyName(camera);
const cameraName = useCameraNickname(camera);
return (
<>

View File

@ -37,7 +37,7 @@ import ImagePicker from "@/components/overlay/ImagePicker";
import { Trigger, TriggerAction, TriggerType } from "@/types/trigger";
import { Switch } from "@/components/ui/switch";
import { Textarea } from "../ui/textarea";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type CreateTriggerDialogProps = {
show: boolean;
@ -162,7 +162,7 @@ export default function CreateTriggerDialog({
onCancel();
};
const cameraName = useCameraFriendlyName(selectedCamera);
const cameraName = useCameraNickname(selectedCamera);
return (
<Dialog open={show} onOpenChange={onCancel}>

View File

@ -24,7 +24,7 @@ import { baseUrl } from "@/api/baseUrl";
import { PlayerStats } from "./PlayerStats";
import { LuVideoOff } from "react-icons/lu";
import { Trans, useTranslation } from "react-i18next";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type LivePlayerProps = {
cameraRef?: (ref: HTMLDivElement | null) => void;
@ -77,7 +77,7 @@ export default function LivePlayer({
const internalContainerRef = useRef<HTMLDivElement | null>(null);
const cameraName = useCameraFriendlyName(cameraConfig);
const cameraName = useCameraNickname(cameraConfig);
// stats
const [stats, setStats] = useState<PlayerStatsType>({

View File

@ -21,7 +21,7 @@ import {
usePreviewForTimeRange,
} from "@/hooks/use-camera-previews";
import { useTranslation } from "react-i18next";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type PreviewPlayerProps = {
previewRef?: (ref: HTMLDivElement | null) => void;
@ -149,7 +149,7 @@ function PreviewVideoPlayer({
const { t } = useTranslation(["components/player"]);
const { data: config } = useSWR<FrigateConfig>("config");
const cameraName = useCameraFriendlyName(camera);
const cameraName = useCameraNickname(camera);
// controlling playback
const previewRef = useRef<HTMLVideoElement | null>(null);
@ -466,7 +466,7 @@ function PreviewFramesPlayer({
}: PreviewFramesPlayerProps) {
const { t } = useTranslation(["components/player"]);
const cameraName = useCameraFriendlyName(camera);
const cameraName = useCameraNickname(camera);
// frames data
const { data: previewFrames } = useSWR<string[]>(

View File

@ -107,7 +107,7 @@ export default function CameraEditForm({
const cameraInfo = useMemo(() => {
if (!cameraName || !config?.cameras[cameraName]) {
return {
friendly_name: undefined,
nickname: undefined,
name: cameraName || "",
roles: new Set<Role>(),
};
@ -121,14 +121,14 @@ export default function CameraEditForm({
});
return {
friendly_name: camera?.friendly_name || cameraName,
nickname: camera?.nickname || cameraName,
name: cameraName,
roles,
};
}, [cameraName, config]);
const defaultValues: FormValues = {
cameraName: cameraInfo?.friendly_name || cameraName || "",
cameraName: cameraInfo?.nickname || cameraName || "",
enabled: true,
ffmpeg: {
inputs: [
@ -169,18 +169,18 @@ export default function CameraEditForm({
const saveCameraConfig = (values: FormValues) => {
setIsLoading(true);
let finalCameraName = values.cameraName;
let friendly_name: string | undefined = undefined;
let nickname: string | undefined = undefined;
const isValidName = /^[a-zA-Z0-9_-]+$/.test(values.cameraName);
if (!isValidName) {
finalCameraName = generateFixedHash(finalCameraName);
friendly_name = values.cameraName;
nickname = values.cameraName;
}
const configData: ConfigSetBody["config_data"] = {
cameras: {
[finalCameraName]: {
enabled: values.enabled,
...(friendly_name && { friendly_name }),
...(nickname && { nickname }),
ffmpeg: {
inputs: values.ffmpeg.inputs.map((input) => ({
path: input.path,
@ -235,7 +235,7 @@ export default function CameraEditForm({
if (
cameraName &&
values.cameraName !== cameraName &&
values.cameraName !== cameraInfo?.friendly_name
values.cameraName !== cameraInfo?.nickname
) {
// If camera name changed, delete old camera config
const deleteRequestBody: ConfigSetBody = {

View File

@ -33,7 +33,7 @@ import { Link } from "react-router-dom";
import { LiveStreamMetadata } from "@/types/live";
import { Trans, useTranslation } from "react-i18next";
import { useDocDomain } from "@/hooks/use-doc-domain";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type CameraStreamingDialogProps = {
camera: string;
@ -57,7 +57,7 @@ export function CameraStreamingDialog({
const { getLocaleDocUrl } = useDocDomain();
const { data: config } = useSWR<FrigateConfig>("config");
const cameraName = useCameraFriendlyName(camera);
const cameraName = useCameraNickname(camera);
const [isLoading, setIsLoading] = useState(false);

View File

@ -8,14 +8,14 @@ export function resolveCameraName(
) {
if (typeof cameraId === "object" && cameraId !== null) {
const camera = cameraId as CameraConfig;
return camera?.friendly_name || camera?.name.replaceAll("_", " ");
return camera?.nickname || camera?.name.replaceAll("_", " ");
} else {
const camera = config?.cameras?.[String(cameraId)];
return camera?.friendly_name || String(cameraId).replaceAll("_", " ");
return camera?.nickname || String(cameraId).replaceAll("_", " ");
}
}
export function useCameraFriendlyName(
export function useCameraNickname(
cameraId: string | CameraConfig | undefined,
): string {
const { data: config } = useSWR<FrigateConfig>("config");

View File

@ -61,7 +61,7 @@ export default function useStats(stats: FrigateStats | undefined) {
return;
}
const cameraName = config.cameras?.[name]?.friendly_name ?? name;
const cameraName = config.cameras?.[name]?.nickname ?? name;
if (config.cameras[name].enabled && cam["camera_fps"] == 0) {
problems.push({
text: t("stats.cameraIsOffline", {
@ -82,7 +82,7 @@ export default function useStats(stats: FrigateStats | undefined) {
memoizedStats["cpu_usages"][cam["pid"]]?.cpu_average,
);
const cameraName = config?.cameras?.[name]?.friendly_name ?? name;
const cameraName = config?.cameras?.[name]?.nickname ?? name;
if (!isNaN(ffmpegAvg) && ffmpegAvg >= CameraFfmpegThreshold.error) {
problems.push({
text: t("stats.ffmpegHighCpuUsage", {

View File

@ -33,7 +33,7 @@ export type SearchModel = "jinav1" | "jinav2";
export type SearchModelSize = "small" | "large";
export interface CameraConfig {
friendly_name: string;
nickname: string;
audio: {
enabled: boolean;
enabled_in_config: boolean;

View File

@ -49,7 +49,7 @@ import {
} from "@/components/ui/select";
import { IoMdArrowRoundBack } from "react-icons/io";
import { isDesktop } from "react-device-detect";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
import { CameraNameLabel } from "@/components/camera/CameraNameLabel";
type CameraSettingsViewProps = {
@ -98,7 +98,7 @@ export default function CameraSettingsView({
return [];
}, [config]);
const selectCameraName = useCameraFriendlyName(selectedCamera);
const selectCameraName = useCameraNickname(selectedCamera);
// zones and labels

View File

@ -30,7 +30,7 @@ import { isDesktop } from "react-device-detect";
import { Trans, useTranslation } from "react-i18next";
import { useDocDomain } from "@/hooks/use-doc-domain";
import { getTranslatedLabel } from "@/utils/i18n";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
import { AudioLevelGraph } from "@/components/audio/AudioLevelGraph";
import { useWs } from "@/api/ws";
@ -129,7 +129,7 @@ export default function ObjectSettingsView({
}
}, [config, selectedCamera]);
const cameraName = useCameraFriendlyName(cameraConfig);
const cameraName = useCameraNickname(cameraConfig);
const { objects, audio_detections } = useCameraActivity(
cameraConfig ?? ({} as CameraConfig),

View File

@ -23,7 +23,7 @@ import { cn } from "@/lib/utils";
import { formatUnixTimestampToDateTime } from "@/utils/dateUtil";
import { Link } from "react-router-dom";
import { useTriggers } from "@/api/ws";
import { useCameraFriendlyName } from "@/hooks/use-camera-friendly-name";
import { useCameraNickname } from "@/hooks/use-camera-nickname";
type ConfigSetBody = {
requires_restart: number;
@ -79,7 +79,7 @@ export default function TriggerView({
const [triggeredTrigger, setTriggeredTrigger] = useState<string>();
const [isLoading, setIsLoading] = useState(false);
const cameraName = useCameraFriendlyName(selectedCamera);
const cameraName = useCameraNickname(selectedCamera);
const triggers = useMemo(() => {
if (
!config ||

View File

@ -14,7 +14,7 @@ import {
import useSWR from "swr";
import { useTranslation } from "react-i18next";
import { CameraNameLabel } from "@/components/camera/CameraNameLabel";
import { resolveCameraName } from "@/hooks/use-camera-friendly-name";
import { resolveCameraName } from "@/hooks/use-camera-nickname";
type CameraMetricsProps = {
lastUpdated: number;