[_the_inflator]

Exactly, and all on an embedded system with quite restrictive settings and no overclocked Intel lastest generation combined with NVIDIA's 10k graphic cards.

[charcircuit]

Embedded systems can make network calls to powerful, GPU equipped servers.

[ceejayoz]

Sure. Claude does that. "Cogitated for 1m 50s" doesn't work for real-time applications.

[charcircuit]

You can submit many queries in parallel to increase throughout. Smaller models and faster hardware can reduce the time per query too.

[ceejayoz]

None of that gets you the 100ms response time the parent poster talked about, for something like "who is at my doorbell?" real-time uses.

[sebmellen]

Ok. Claude will not work for this use case because none of the sample data (weirdly blurry ID images) is in the training data.

[Chu4eeno]

They really shouldn't, though.

[charcircuit]

It can offer a ton of user value. There is a whole industry built upon this idea, Internet of Things.

[ceejayoz]

IoT wasn't not built on "send all the data off to a hosted GenAI". It predated them by quite a few years.

[charcircuit]

The GPUs were doing video transcoding instead of GenAI.

[ceejayoz]

You can run OpenCV on a GPU-less Raspberry Pi or other IoT device just fine.

And most IoT devices aren't doing video transcoding at all. You're making some very odd assertions in this thread.