Ensure landmark detector has a defualt value (#17420)

* reindex with api endpoint and zmq

* threading

* frontend

* require admin role

docker-compose.yml

@@ -36,6 +36,7 @@ services:
      # - /dev/bus/usb:/dev/bus/usb # Uncomment for Google Coral USB
   mqtt:
     container_name: mqtt
-    image: eclipse-mosquitto:1.6
+    image: eclipse-mosquitto:2.0
+    command: mosquitto -c /mosquitto-no-auth.conf # enable no-auth mode
     ports:
       - "1883:1883"

docker/main/build_nginx.sh

@@ -2,7 +2,7 @@
 
 set -euxo pipefail
 
-NGINX_VERSION="1.25.3"
+NGINX_VERSION="1.27.4"
 VOD_MODULE_VERSION="1.31"
 SECURE_TOKEN_MODULE_VERSION="1.5"
 SET_MISC_MODULE_VERSION="v0.33"

docs/docs/configuration/face_recognition.md

@@ -31,6 +31,7 @@ In both cases a lightweight face landmark detection model is also used to align
 ## Minimum System Requirements
 
 The `small` model is optimized for efficiency and runs on the CPU, most CPUs should run the model efficiently.
+
 The `large` model is optimized for accuracy, an integrated or discrete GPU is highly recommended.
 
 ## Configuration
@@ -65,7 +66,7 @@ Fine-tune face recognition with these optional parameters:
 - `blur_confidence_filter`: Enables a filter that calculates how blurry the face is and adjusts the confidence based on this.
   - Default: `True`.
 
-## Dataset
+## Creating a Robust Training Set
 
 The number of images needed for a sufficient training set for face recognition varies depending on several factors:
 
@@ -74,11 +75,9 @@ The number of images needed for a sufficient training set for face recognition v
 
 However, here are some general guidelines:
 
-- Minimum: For basic face recognition tasks, a minimum of 10-20 images per person is often recommended.
+- Minimum: For basic face recognition tasks, a minimum of 5-10 images per person is often recommended.
-- Recommended: For more robust and accurate systems, 30-50 images per person is a good starting point.
+- Recommended: For more robust and accurate systems, 20-30 images per person is a good starting point.
-- Ideal: For optimal performance, especially in challenging conditions, 100 or more images per person can be beneficial.
+- Ideal: For optimal performance, especially in challenging conditions, 50-100 images per person can be beneficial.
-
-## Creating a Robust Training Set
 
 The accuracy of face recognition is heavily dependent on the quality of data given to it for training. It is recommended to build the face training library in phases.
 
@@ -89,7 +88,8 @@ When choosing images to include in the face training set it is recommended to al
 - If it is difficult to make out details in a persons face it will not be helpful in training.
 - Avoid images with extreme under/over-exposure.
 - Avoid blurry / pixelated images.
-- Be careful when uploading images of people when they are wearing clothing that covers a lot of their face as this may confuse the model.
+- Avoid training on infrared (grayscale). The models are trained on color images and will be able to extract features from grayscale images.
+- Using images of people wearing hats / sunglasses may confuse the model.
 - Do not upload too many similar images at the same time, it is recommended to train no more than 4-6 similar images for each person to avoid overfitting.
 
 :::
@@ -124,4 +124,4 @@ This can happen for a few different reasons, but this is usually an indicator th
 
 ### I see scores above the threshold in the train tab, but a sub label wasn't assigned?
 
-The Frigate face recognizer collects face recognition scores from all of the frames across the person objects lifecycle. The scores are continually weighted based on the area of the face, and a sub label will only be assigned to person if there is a prominent person recognized. This avoids cases where a single high confidence recognition result would throw off the results.
+The Frigate considers the recognition scores across all recogntion attempts for each person object. The scores are continually weighted based on the area of the face, and a sub label will only be assigned to person if a person is confidently recognized consistently. This avoids cases where a single high confidence recognition would throw off the results.

frigate/api/classification.py

@@ -298,3 +298,49 @@ def reprocess_license_plate(request: Request, event_id: str):
         content=response,
         status_code=200,
     )
+
+
+@router.put("/reindex", dependencies=[Depends(require_role(["admin"]))])
+def reindex_embeddings(request: Request):
+    if not request.app.frigate_config.semantic_search.enabled:
+        message = (
+            "Cannot reindex tracked object embeddings, Semantic Search is not enabled."
+        )
+        logger.error(message)
+        return JSONResponse(
+            content=(
+                {
+                    "success": False,
+                    "message": message,
+                }
+            ),
+            status_code=400,
+        )
+
+    context: EmbeddingsContext = request.app.embeddings
+    response = context.reindex_embeddings()
+
+    if response == "started":
+        return JSONResponse(
+            content={
+                "success": True,
+                "message": "Embeddings reindexing has started.",
+            },
+            status_code=202,  # 202 Accepted
+        )
+    elif response == "in_progress":
+        return JSONResponse(
+            content={
+                "success": False,
+                "message": "Embeddings reindexing is already in progress.",
+            },
+            status_code=409,  # 409 Conflict
+        )
+    else:
+        return JSONResponse(
+            content={
+                "success": False,
+                "message": "Failed to start reindexing.",
+            },
+            status_code=500,
+        )

frigate/comms/embeddings_updater.py

@@ -17,6 +17,7 @@ class EmbeddingsRequestEnum(Enum):
     register_face = "register_face"
     reprocess_face = "reprocess_face"
     reprocess_plate = "reprocess_plate"
+    reindex = "reindex"
 
 
 class EmbeddingsResponder:

frigate/data_processing/common/face/model.py

@@ -20,6 +20,7 @@ class FaceRecognizer(ABC):
 
     def __init__(self, config: FrigateConfig) -> None:
         self.config = config
+        self.landmark_detector: cv2.face.FacemarkLBF = None
         self.init_landmark_detector()
 
     @abstractmethod

frigate/embeddings/__init__.py

@@ -250,3 +250,6 @@ class EmbeddingsContext:
         return self.requestor.send_data(
             EmbeddingsRequestEnum.reprocess_plate.value, {"event": event}
         )
+
+    def reindex_embeddings(self) -> dict[str, any]:
+        return self.requestor.send_data(EmbeddingsRequestEnum.reindex.value, {})

frigate/embeddings/embeddings.py

@@ -3,6 +3,7 @@
 import datetime
 import logging
 import os
+import threading
 import time
 
 from numpy import ndarray
@@ -74,6 +75,10 @@ class Embeddings:
         self.metrics = metrics
         self.requestor = InterProcessRequestor()
 
+        self.reindex_lock = threading.Lock()
+        self.reindex_thread = None
+        self.reindex_running = False
+
         # Create tables if they don't exist
         self.db.create_embeddings_tables()
 
@@ -368,3 +373,27 @@ class Embeddings:
         totals["status"] = "completed"
 
         self.requestor.send_data(UPDATE_EMBEDDINGS_REINDEX_PROGRESS, totals)
+
+    def start_reindex(self) -> bool:
+        """Start reindexing in a separate thread if not already running."""
+        with self.reindex_lock:
+            if self.reindex_running:
+                logger.warning("Reindex embeddings is already running.")
+                return False
+
+            # Mark as running and start the thread
+            self.reindex_running = True
+            self.reindex_thread = threading.Thread(
+                target=self._reindex_wrapper, daemon=True
+            )
+            self.reindex_thread.start()
+            return True
+
+    def _reindex_wrapper(self) -> None:
+        """Wrapper to run reindex and reset running flag when done."""
+        try:
+            self.reindex()
+        finally:
+            with self.reindex_lock:
+                self.reindex_running = False
+                self.reindex_thread = None

frigate/embeddings/maintainer.py

@@ -206,6 +206,9 @@ class EmbeddingMaintainer(threading.Thread):
                             self.embeddings.embed_description("", data, upsert=False),
                             pack=False,
                         )
+                    elif topic == EmbeddingsRequestEnum.reindex.value:
+                        response = self.embeddings.start_reindex()
+                        return "started" if response else "in_progress"
 
                 processors = [self.realtime_processors, self.post_processors]
                 for processor_list in processors:

notebooks/YOLO_NAS_Pretrained_Export.ipynb

@@ -8,14 +8,14 @@
       },
       "outputs": [],
       "source": [
-        "! pip install -q super_gradients==3.7.1"
+        "! pip install -q git+https://github.com/Deci-AI/super-gradients.git"
       ]
     },
     {
       "cell_type": "code",
       "source": [
-        "! sed -i 's/sghub.deci.ai/sg-hub-nv.s3.amazonaws.com/' /usr/local/lib/python3.10/dist-packages/super_gradients/training/pretrained_models.py\n",
+        "! sed -i 's/sghub.deci.ai/sg-hub-nv.s3.amazonaws.com/' /usr/local/lib/python3.11/dist-packages/super_gradients/training/pretrained_models.py\n",
-        "! sed -i 's/sghub.deci.ai/sg-hub-nv.s3.amazonaws.com/' /usr/local/lib/python3.10/dist-packages/super_gradients/training/utils/checkpoint_utils.py"
+        "! sed -i 's/sghub.deci.ai/sg-hub-nv.s3.amazonaws.com/' /usr/local/lib/python3.11/dist-packages/super_gradients/training/utils/checkpoint_utils.py"
       ],
       "metadata": {
         "id": "NiRCt917KKcL"

web/public/locales/en/views/settings.json

@@ -87,9 +87,15 @@
       "title": "Semantic Search",
       "desc": "Semantic Search in Frigate allows you to find tracked objects within your review items using either the image itself, a user-defined text description, or an automatically generated one.",
       "readTheDocumentation": "Read the Documentation",
-      "reindexOnStartup": {
+      "reindexNow": {
-        "label": "Re-Index On Startup",
+        "label": "Reindex Now",
-        "desc": "Re-indexing will reprocess all thumbnails and descriptions (if enabled) and apply the embeddings on each startup. <em>Don't forget to disable the option after restarting!</em>"
+        "desc": "Reindexing will regenerate embeddings for all tracked object. This process runs in the background and may max out your CPU and take a fair amount of time depending on the number of tracked objects you have.",
+        "confirmTitle": "Confirm Reindexing",
+        "confirmDesc": "Are you sure you want to reindex all tracked object embeddings? This process will run in the background but it may max out your CPU and take a fair amount of time. You can watch the progress on the Explore page.",
+        "confirmButton": "Reindex",
+        "success": "Reindexing started successfully.",
+        "alreadyInProgress": "Reindexing is already in progress.",
+        "error": "Failed to start reindexing: {{errorMessage}}"
       },
       "modelSize": {
         "label": "Model Size",

web/src/views/settings/ClassificationSettingsView.tsx

@@ -21,11 +21,21 @@ import {
   SelectTrigger,
 } from "@/components/ui/select";
 import { Trans, useTranslation } from "react-i18next";
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+} from "@/components/ui/alert-dialog";
+import { buttonVariants } from "@/components/ui/button";
 
 type ClassificationSettings = {
   search: {
     enabled?: boolean;
-    reindex?: boolean;
     model_size?: SearchModelSize;
   };
   face: {
@@ -48,39 +58,22 @@ export default function ClassificationSettingsView({
     useSWR<FrigateConfig>("config");
   const [changedValue, setChangedValue] = useState(false);
   const [isLoading, setIsLoading] = useState(false);
+  const [isReindexDialogOpen, setIsReindexDialogOpen] = useState(false);
 
   const { addMessage, removeMessage } = useContext(StatusBarMessagesContext)!;
 
   const [classificationSettings, setClassificationSettings] =
     useState<ClassificationSettings>({
-      search: {
+      search: { enabled: undefined, model_size: undefined },
-        enabled: undefined,
+      face: { enabled: undefined, model_size: undefined },
-        reindex: undefined,
+      lpr: { enabled: undefined },
-        model_size: undefined,
-      },
-      face: {
-        enabled: undefined,
-        model_size: undefined,
-      },
-      lpr: {
-        enabled: undefined,
-      },
     });
 
   const [origSearchSettings, setOrigSearchSettings] =
     useState<ClassificationSettings>({
-      search: {
+      search: { enabled: undefined, model_size: undefined },
-        enabled: undefined,
+      face: { enabled: undefined, model_size: undefined },
-        reindex: undefined,
+      lpr: { enabled: undefined },
-        model_size: undefined,
-      },
-      face: {
-        enabled: undefined,
-        model_size: undefined,
-      },
-      lpr: {
-        enabled: undefined,
-      },
     });
 
   useEffect(() => {
@@ -89,32 +82,26 @@ export default function ClassificationSettingsView({
         setClassificationSettings({
           search: {
             enabled: config.semantic_search.enabled,
-            reindex: config.semantic_search.reindex,
             model_size: config.semantic_search.model_size,
           },
           face: {
             enabled: config.face_recognition.enabled,
             model_size: config.face_recognition.model_size,
           },
-          lpr: {
+          lpr: { enabled: config.lpr.enabled },
-            enabled: config.lpr.enabled,
-          },
         });
       }
 
       setOrigSearchSettings({
         search: {
           enabled: config.semantic_search.enabled,
-          reindex: config.semantic_search.reindex,
           model_size: config.semantic_search.model_size,
         },
         face: {
           enabled: config.face_recognition.enabled,
           model_size: config.face_recognition.model_size,
         },
-        lpr: {
+        lpr: { enabled: config.lpr.enabled },
-          enabled: config.lpr.enabled,
-        },
       });
     }
     // we know that these deps are correct
@@ -125,10 +112,7 @@ export default function ClassificationSettingsView({
     newConfig: Partial<ClassificationSettings>,
   ) => {
     setClassificationSettings((prevConfig) => ({
-      search: {
+      search: { ...prevConfig.search, ...newConfig.search },
-        ...prevConfig.search,
-        ...newConfig.search,
-      },
       face: { ...prevConfig.face, ...newConfig.face },
       lpr: { ...prevConfig.lpr, ...newConfig.lpr },
     }));
@@ -141,10 +125,8 @@ export default function ClassificationSettingsView({
 
     axios
       .put(
-        `config/set?semantic_search.enabled=${classificationSettings.search.enabled ? "True" : "False"}&semantic_search.reindex=${classificationSettings.search.reindex ? "True" : "False"}&semantic_search.model_size=${classificationSettings.search.model_size}&face_recognition.enabled=${classificationSettings.face.enabled ? "True" : "False"}&face_recognition.model_size=${classificationSettings.face.model_size}&lpr.enabled=${classificationSettings.lpr.enabled ? "True" : "False"}`,
+        `config/set?semantic_search.enabled=${classificationSettings.search.enabled ? "True" : "False"}&semantic_search.model_size=${classificationSettings.search.model_size}&face_recognition.enabled=${classificationSettings.face.enabled ? "True" : "False"}&face_recognition.model_size=${classificationSettings.face.model_size}&lpr.enabled=${classificationSettings.lpr.enabled ? "True" : "False"}`,
-        {
+        { requires_restart: 0 },
-          requires_restart: 0,
-        },
       )
       .then((res) => {
         if (res.status === 200) {
@@ -156,9 +138,7 @@ export default function ClassificationSettingsView({
         } else {
           toast.error(
             t("classification.toast.error", { errorMessage: res.statusText }),
-            {
+            { position: "top-center" },
-              position: "top-center",
-            },
           );
         }
       })
@@ -169,9 +149,7 @@ export default function ClassificationSettingsView({
           "Unknown error";
         toast.error(
           t("toast.save.error.title", { errorMessage, ns: "common" }),
-          {
+          { position: "top-center" },
-            position: "top-center",
-          },
         );
       })
       .finally(() => {
@@ -191,6 +169,43 @@ export default function ClassificationSettingsView({
     removeMessage("search_settings", "search_settings");
   }, [origSearchSettings, removeMessage]);
 
+  const onReindex = useCallback(() => {
+    setIsLoading(true);
+
+    axios
+      .put("/reindex")
+      .then((res) => {
+        if (res.status === 202) {
+          toast.success(t("classification.semanticSearch.reindexNow.success"), {
+            position: "top-center",
+          });
+        } else {
+          toast.error(
+            t("classification.semanticSearch.reindexNow.error", {
+              errorMessage: res.statusText,
+            }),
+            { position: "top-center" },
+          );
+        }
+      })
+      .catch((error) => {
+        const errorMessage =
+          error.response?.data?.message ||
+          error.response?.data?.detail ||
+          "Unknown error";
+        toast.error(
+          t("classification.semanticSearch.reindexNow.error", {
+            errorMessage,
+          }),
+          { position: "top-center" },
+        );
+      })
+      .finally(() => {
+        setIsLoading(false);
+        setIsReindexDialogOpen(false);
+      });
+  }, [t]);
+
   useEffect(() => {
     if (changedValue) {
       addMessage(
@@ -262,28 +277,18 @@ export default function ClassificationSettingsView({
               </Label>
             </div>
           </div>
-          <div className="flex flex-col">
+          <div className="space-y-3">
-            <div className="flex flex-row items-center">
+            <Button
-              <Switch
+              variant="default"
-                id="reindex"
+              disabled={isLoading || !classificationSettings.search.enabled}
-                className="mr-3"
+              onClick={() => setIsReindexDialogOpen(true)}
-                disabled={classificationSettings.search.reindex === undefined}
+              aria-label={t("classification.semanticSearch.reindexNow.label")}
-                checked={classificationSettings.search.reindex === true}
+            >
-                onCheckedChange={(isChecked) => {
+              {t("classification.semanticSearch.reindexNow.label")}
-                  handleClassificationConfigChange({
+            </Button>
-                    search: { reindex: isChecked },
-                  });
-                }}
-              />
-              <div className="space-y-0.5">
-                <Label htmlFor="reindex">
-                  {t("classification.semanticSearch.reindexOnStartup.label")}
-                </Label>
-              </div>
-            </div>
             <div className="mt-3 text-sm text-muted-foreground">
               <Trans ns="views/settings">
-                classification.semanticSearch.reindexOnStartup.desc
+                classification.semanticSearch.reindexNow.desc
               </Trans>
             </div>
           </div>
@@ -316,9 +321,7 @@ export default function ClassificationSettingsView({
               value={classificationSettings.search.model_size}
               onValueChange={(value) =>
                 handleClassificationConfigChange({
-                  search: {
+                  search: { model_size: value as SearchModelSize },
-                    model_size: value as SearchModelSize,
-                  },
                 })
               }
             >
@@ -346,6 +349,35 @@ export default function ClassificationSettingsView({
           </div>
         </div>
 
+        <AlertDialog
+          open={isReindexDialogOpen}
+          onOpenChange={setIsReindexDialogOpen}
+        >
+          <AlertDialogContent>
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {t("classification.semanticSearch.reindexNow.confirmTitle")}
+              </AlertDialogTitle>
+              <AlertDialogDescription>
+                <Trans ns="views/settings">
+                  classification.semanticSearch.reindexNow.confirmDesc
+                </Trans>
+              </AlertDialogDescription>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel onClick={() => setIsReindexDialogOpen(false)}>
+                {t("button.cancel", { ns: "common" })}
+              </AlertDialogCancel>
+              <AlertDialogAction
+                onClick={onReindex}
+                className={buttonVariants({ variant: "select" })}
+              >
+                {t("classification.semanticSearch.reindexNow.confirmButton")}
+              </AlertDialogAction>
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialog>
+
         <div className="my-2 space-y-6">
           <Separator className="my-2 flex bg-secondary" />