CodeProject.AI Discussions

CodeProject.AI is our standalone, self-hosted, fast, free and Open Source Artificial Intelligence microserver for any platform, any language.

Re: Custom models question [Solved]

actran7020-Nov-23 7:03

actran70

20-Nov-23 7:03

Example when you want to use 1 model only and turn off license plate/alpr:

In your camera trigger tab -> AI -> Custom models:

ipcam-combined,alpr:0

Re: Custom models question [Solved]

Mike Lud20-Nov-23 8:10

Mike Lud

20-Nov-23 8:10

I see you are using the Object Detection (YOLOv5 6.2) module. Give the Object Detection (YOLOv5 .NET) module a try. I have the same GPU as you, see the below screenshot of my detection times.

Re: Custom models question [Solved]

Fun-Fisherman-58220-Nov-23 8:42

Fun-Fisherman-582

20-Nov-23 8:42

Will do. How many additional images and time for each are you using?

Re: Custom models question [Solved]

Mike Lud20-Nov-23 9:24

Mike Lud

20-Nov-23 9:24

Below are my typical AI settings

Re: Custom models question [Solved]

Fun-Fisherman-58220-Nov-23 12:38

Fun-Fisherman-582

20-Nov-23 12:38

Hum. It is faster, but I am not seeing the same things in the logs as you are. It is identifying stuff, but not at the speed of your machine (this is at my business and it is a i9 with ZOTAC Gaming GeForce RTX 3060 Twin Edge OC 12GB GDDR6 192-bit 15 Gbps PCIE 4.0 Graphics Card). BI instructions note that if there is a alp Blush | :O

that this makes the custom model list an exclusion so it will run everything except this item. So does this mean all the models on the machine (in the CPAI folder) or all the custom models listen in the settings. I feel like I should be getting pretty low times and yet I am not near what you have.

Re: Custom models question [Solved]

Mike Lud20-Nov-23 14:05

Mike Lud

20-Nov-23 14:05

In your cameras AI setting do you have Use main stream if available checked. If so uncheck it, having it check does not improve the AI accuracy it only slows it down because the image gets resized to a 640 resolution before doing the detection.
If Custom models is blank it will use all the models and if you have a custom model in the box it will only use that model.
Also if you have Default object detection, or Facial recognition, or ALPR for plates check and you do not want to use it for a specific camera you need to add objects:0 or faces:0 or alpr:0 to Custom models

Re: Custom models question [Solved]

Fun-Fisherman-58220-Nov-23 16:04

Fun-Fisherman-582

20-Nov-23 16:04

Thanks so much for the tip on the use main tream if available. That greatly changed the times!

Also it seems that the YOLOv5 6.2 with CUDA isn't as fast as the YOLOv5 .net without the CUDA. I assumed that the CUDA would be faster, but I guess I am wrong.

BTW, would you share your motion sensor settings in BI (I know this is CPAI) but this might also help me get this set up correctly. Thank you again. Super helpful

Does the high def box change the level of time here that CPAI needs (I suspect not as this is just for motion trigger)

modified 20-Nov-23 22:09pm.

Re: Custom models question [Solved]

Mike Lud20-Nov-23 18:06

Mike Lud

20-Nov-23 18:06

The high def box is just for motion trigger

Re: Custom models question [Solved]

Mike Lud20-Nov-23 18:15

Mike Lud

20-Nov-23 18:15

Also if you want to get some better speed out of your GPU see the below post. The speed shown in the below post is for one of my PC that has a RTX 4090

CodeProject.AI Server: AI the easy way.[^]

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 4:33

Fun-Fisherman-582

21-Nov-23 4:33

Wow...just wow. I couldn't understand why others were getting such fast detections and I wasn't even thought I have what I would consider a higher performance machine but with those simple tweaks, this baby is hardly breaking a sweat and processing a ton of triggers. I changes the pstate for the GPU and that was amazing. My home machine is under powering the graphic card and so when I change it, the CPAI seemed to freeze so I turned it back.

This is on a 30 camera system. I cannot thank you enough for the feed back and help. This little tutorial should be in the BI set up area as this really changes the entire game for me.

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 4:53

Fun-Fisherman-582

21-Nov-23 4:53

So my business machine is running smoothly, but my home one isn't now. It seems like the CPAI server isn't getting anything from BI now. So I changes the gpu pstate to max power, it seems to just hang so I figured it wasn't getting enough power (the power supply is only 260w) and so changed it back to optimum power. I have rebooted between changes. What do you think about this?

When I clear the logs and wait, nothing populates into the logs making me think that the CPAI isn't getting any requests for detection

Al I changed was the pstate to max power then back to optimized and here is where I am at. Thoughts?

Re: Custom models question [Solved]

Mike Lud21-Nov-23 7:54

Mike Lud

21-Nov-23 7:54

Try rebooting the PC

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 8:10

Fun-Fisherman-582

21-Nov-23 8:10

I did between changing the power option for the GPU. Maybe I need to pull the cord from the machine and let it sit for a bit.

Just did it again and no go. Weird. I'll pull the cord when I am home...

modified 21-Nov-23 13:27pm.

Re: Custom models question [Solved]

Mike Lud21-Nov-23 8:31

Mike Lud

21-Nov-23 8:31

Post a screenshot of your main AI settings like the below

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 8:35

Fun-Fisherman-582

21-Nov-23 8:35

Ugh. I'm dumb. The wrong port is in there. How did that change??? Anything else look off?

Re: Custom models question [Solved]

Mike Lud21-Nov-23 8:39

Mike Lud

21-Nov-23 8:39

Change the port to 32168 then reboot

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 11:22

Fun-Fisherman-582

21-Nov-23 11:22

Got it going, but when I try set it to max power, it won't work. Again, maybe a power supply to the card, but as of now, it is working. Thanks.

Re: Custom models question [Solved]

Mike Lud21-Nov-23 11:54

Mike Lud

21-Nov-23 11:54

You can check what Pstate your GPU is in by opening up a command prompt and run nvidia-smi

Re: Custom models question [Solved]

Fun-Fisherman-58221-Nov-23 22:39

Fun-Fisherman-582

21-Nov-23 22:39

OK now I am confused. I though that if I set this it would put it into Pstate 0. So on my robust machine that is working I see this

pstate of 2 and this machine is flying and working well.

I am still having trouble with my less robust machine as it seems like it won't use the GPU now.

Here you can see that it isn't reporting and power usage??? and a pstate of 0!

I have no idea what is going on now....

Re: Custom models question [Solved]

Fun-Fisherman-58222-Nov-23 16:09

Fun-Fisherman-582

22-Nov-23 16:09

I suspect there is something with the card. Here a found another graphic card and put it in. My power supply does not have another 6 pin plug so I had to borrow power from another machine to test this out and check out the speed now!

Once I get a new power supply, I'll be able to button this up and have a neat little system.

Re: Custom models question [Solved]

Mike Lud22-Nov-23 16:14

Mike Lud

22-Nov-23 16:14

What GPU did you put in?

Re: Custom models question [Solved]

Fun-Fisherman-58222-Nov-23 16:33

Fun-Fisherman-582

22-Nov-23 16:33

I'm not sure. I had a card from a friend and put it in.

Re: Custom models question [Solved]

Fun-Fisherman-58222-Nov-23 21:52

Fun-Fisherman-582

22-Nov-23 21:52

You can see it is old. The Bios date is 2016!!!

Coral: The interpreter is in use. Please try again later

SlothCroissant19-Nov-23 5:44

SlothCroissant

19-Nov-23 5:44

Hey all - been toying around with cp.ai and I'm struggling a bit with Coral latency (I have a couple of dual-TPU M.2s). Only the Coral module installed, nothing else.

I'm seeing issues when making calls to cp.ai (from Frigate or Blue Iris, same result) where cp.ai reports that the interpreter is in use.

I've run through Google's coral getting started, and everything seems to work fine (2-3ms inference on their image classification demo).

Anything I'm missing here? I'd like to centralize my Coral TPUs and enable services to call into it, rather than use the Coral directly in Frigate, etc.

Run details:

sudo docker run --name cpai --privileged --device /dev/apex_0 -d -p 32168:32168 codeproject/ai-server

Relevant trace logs:

23:03:17:Module ObjectDetectionCoral installed successfully.
23:03:17:Module ObjectDetectionCoral not configured to AutoStart.
23:03:17:Installer exited with code 0
23:03:38:Command: /app/modules/ObjectDetectionCoral/bin/linux/python39/venv/bin/python3
23:03:38:
23:03:38:Module 'ObjectDetection (Coral)' 1.6.3 (ID: ObjectDetectionCoral)
23:03:38:Module Path:   /app/modules/ObjectDetectionCoral
23:03:38:AutoStart:     True
23:03:38:Starting /app...onCoral/bin/linux/python39/venv/bin/python3 "/app...ionCoral/objectdetection_coral_adapter.py"
23:03:38:
23:03:38:Attempting to start ObjectDetectionCoral with /app/modules/ObjectDetectionCoral/bin/linux/python39/venv/bin/python3 "/app/modules/ObjectDetectionCoral/objectdetection_coral_adapter.py"
23:03:38:Queue:         objectdetection_queue
23:03:38:Platforms:     windows,linux,linux-arm64,macos,macos-arm64
23:03:38:GPU Libraries: installed if available
23:03:38:GPU Enabled:   enabled
23:03:38:Parallelism:   1
23:03:38:Accelerator:
23:03:38:Half Precis.:  enable
23:03:38:Runtime:       python3.9
23:03:38:Runtime Loc:   Local
23:03:38:FilePath:      objectdetection_coral_adapter.py
23:03:38:Pre installed: False
23:03:38:Start pause:   1 sec
23:03:38:LogVerbosity:
23:03:38:Valid:         True
23:03:38:Environment Variables
23:03:38:MODELS_DIR = %CURRENT_MODULE_PATH%/assets
23:03:38:MODEL_SIZE = tiny
23:03:38:
23:03:38:Started ObjectDetection (Coral) module
23:03:40:objectdetection_coral_adapter.py: CPAI_MODULE_REQUIRED_MB not found. Setting to default 0
23:03:40:objectdetection_coral_adapter.py: NUM_THREADS not found. Setting to default 1
23:03:40:objectdetection_coral_adapter.py: MIN_CONFIDENCE not found. Setting to default 0.5
23:03:40:objectdetection_coral_adapter.py: MODULE_PATH:    /app/modules/ObjectDetectionCoral
23:03:40:objectdetection_coral_adapter.py: MODELS_DIR:     /app/modules/ObjectDetectionCoral/assets
23:03:40:objectdetection_coral_adapter.py: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
23:03:40:objectdetection_coral_adapter.py: MODEL_SIZE:     small
23:03:40:objectdetection_coral_adapter.py: CPU_MODEL_NAME: tf2_ssd_mobilenet_v2_coco17_ptq.tflite
23:03:40:objectdetection_coral_adapter.py: TPU_MODEL_NAME: tf2_ssd_mobilenet_v2_coco17_ptq_edgetpu.tflite
23:03:40:objectdetection_coral_adapter.py: Edge TPU detected
23:03:40:objectdetection_coral_adapter.py: Input details: {'name': 'serving_default_input:0', 'index': 0, 'shape': array([  1, 300, 300,   3], dtype=int32), 'shape_signature': array([  1, 300, 300,   3], dtype=int32), 'dtype': , 'quantization': (0.007843137718737125, 127), 'quantization_parameters': {'scales': array([0.00784314], dtype=float32), 'zero_points': array([127], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}
23:03:40:objectdetection_coral_adapter.py: Output details: {'name': 'StatefulPartitionedCall:3;StatefulPartitionedCall:2;StatefulPartitionedCall:1;StatefulPartitionedCall:02', 'index': 8, 'shape': array([ 1, 20], dtype=int32), 'shape_signature': array([ 1, 20], dtype=int32), 'dtype': , 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}
23:03:40:Request 'detect' dequeued from 'objectdetection_queue' (...45fa9b)
23:03:40:Request 'detect' dequeued from 'objectdetection_queue' (...c536c3)
23:03:40:Running init for ObjectDetection (Coral)
23:03:40:ObjectDetection (Coral): ObjectDetection (Coral) started.
23:04:37:Client request 'detect' in queue 'objectdetection_queue' (...add625)
23:04:37:Request 'detect' dequeued from 'objectdetection_queue' (...add625)
23:04:37:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:37:ObjectDetection (Coral):  [RuntimeError] : Traceback (most recent call last):
  File "/app/modules/ObjectDetectionCoral/objectdetection_coral_adapter.py", line 104, in do_detection
    result = do_detect(opts, img, score_threshold)
  File "/app/modules/ObjectDetectionCoral/objectdetection_coral.py", line 191, in do_detect
    interpreter.invoke()
  File "/app/modules/ObjectDetectionCoral/bin/linux/python39/venv/lib/python3.9/site-packages/tflite_runtime/interpreter.py", line 941, in invoke
    self._interpreter.Invoke()
RuntimeError: Encountered an unresolved custom op. Did you miss a custom op or delegate?Node number 4 (EdgeTpuDelegateForCustomOp) failed to invoke.
23:04:37:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...add625) took 21ms
23:04:37:Response received (...add625)
23:04:37:Client request 'detect' in queue 'objectdetection_queue' (...c798be)
23:04:37:Request 'detect' dequeued from 'objectdetection_queue' (...c798be)
23:04:37:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:38:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...c798be) took 1003ms
23:04:38:Response received (...c798be): The interpreter is in use. Please try again later
23:04:38:Client request 'detect' in queue 'objectdetection_queue' (...5dc3cf)
23:04:38:Request 'detect' dequeued from 'objectdetection_queue' (...5dc3cf)
23:04:38:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:39:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...5dc3cf) took 1003ms
23:04:39:Response received (...5dc3cf): The interpreter is in use. Please try again later
23:04:39:Client request 'detect' in queue 'objectdetection_queue' (...93b11a)
23:04:39:Request 'detect' dequeued from 'objectdetection_queue' (...93b11a)
23:04:39:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:40:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...93b11a) took 1002ms
23:04:40:Response received (...93b11a): The interpreter is in use. Please try again later
23:04:40:Client request 'detect' in queue 'objectdetection_queue' (...5c8043)
23:04:40:Request 'detect' dequeued from 'objectdetection_queue' (...5c8043)
23:04:40:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:41:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...5c8043) took 1003ms
23:04:41:Response received (...5c8043): The interpreter is in use. Please try again later
23:04:41:Client request 'detect' in queue 'objectdetection_queue' (...718079)
23:04:41:Request 'detect' dequeued from 'objectdetection_queue' (...718079)
23:04:41:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:42:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...718079) took 1003ms
23:04:42:Response received (...718079): The interpreter is in use. Please try again later
23:04:42:Client request 'detect' in queue 'objectdetection_queue' (...0c35b8)
23:04:42:Request 'detect' dequeued from 'objectdetection_queue' (...0c35b8)
23:04:42:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:43:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...0c35b8) took 1003ms
23:04:43:Response received (...0c35b8): The interpreter is in use. Please try again later
23:04:43:Client request 'detect' in queue 'objectdetection_queue' (...6bb356)
23:04:43:Request 'detect' dequeued from 'objectdetection_queue' (...6bb356)
23:04:43:ObjectDetection (Coral): Retrieved objectdetection_queue command
23:04:44:ObjectDetection (Coral): Rec'd request for ObjectDetection (Coral) command 'detect' (...6bb356) took 1003ms
23:04:44:Response received (...6bb356): The interpreter is in use. Please try again later
23:04:44:Client request 'detect' in queue 'objectdetection_queue' (...588c5e)
23:04:44:Request 'detect' dequeued from 'objectdetection_queue' (...588c5e)

Server info (running in Proxmox):

Server version:   2.3.2-Alpha
System:           Docker
Operating System: Linux (Linux 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29))
CPUs:             Common KVM processor
                  1 CPU x 4 cores. 4 logical processors (x64)
System RAM:       8 GiB
Target:           Linux
BuildConfig:      Release
Execution Env:    Docker
Runtime Env:      Production
.NET framework:   .NET 7.0.13
Video adapter info:
System GPU info:
  GPU 3D Usage       0%
  GPU RAM Usage      0
Global Environment variables:
  CPAI_APPROOTPATH = <root>
  CPAI_PORT        = 32168

Re: Coral: The interpreter is in use. Please try again later

Chris Maunder20-Nov-23 3:58

Chris Maunder

20-Nov-23 3:58

This has been bugging us, and many others, for ages. I came across a writeup yesterday regarding the memory handling issues in Coral which make it fundamentally broken, and read then an amusing hack this morning to try and get the Coral back up and running from a hung state.

We restart the module every hour to try and manage this, but it seems we need to be more proactive.

I love the Coral, but if I was choosing again I'd simply move to an Orange Pi or a Raspberry Pi 5. Just way too finicky and, it seems, another abandoned Google project Frown | :(

cheers
Chris Maunder

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.