[canada_dry]

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation.

Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject:

User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json

Llama: { "subject": "light", "action": "turned off", "time": "15 minutes from now", "location": "living room" }

Several of the latest models are on par to the results from Gpt4 in my tests.

[polishdude20]

What about like, if I said "switch off the lamp at 3:45"

How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

[canada_dry]

User: in the sentence "switch off the lamp at 3:45" output the subject, action, time, and location as json

Llama: { "subject": "lamp", "action": "switch off", "time": "3:45", "location": "" }

Where there is an empty parameter the code will try to look back to the last recent commands for context (e.g. I may have just said "turn on the living room light"). If there's an issue it just asks for the missing info.

Translating the parameters from the json is done with good old fashion brute force (i.e. mostly regex).

It's still not 100% perfect but its faster and more accurate than the cloud assistants and private.

[polishdude20]

So you'd need to somehow know that a lamp is also a light eh

[coder543]

With a proper grammar, you can require the "subject" field to be one of several valid entity names. In the prompt, you would tell the LLM what the valid entity names are, which room each entity is in, and a brief description of each entity. Then it would be able to infer which entity you meant if there is one that reasonably matches your request.

If you're speaking through the kitchen microphone (which should be provided as context in the LLM prompt as well) and there are no controllable lights in that room, you could leave room in the grammar for the LLM to respond with a clarifying question or an error, so it isn't forced to choose an entity at random.

[fragmede]

In all seriousness, I have names for my lights for this very reason.

[BrandoElFollito]

Same here (and for sub-areas). Now: this is sometimes stressful when I say "OK Google, switch off .... errrr ... pffff ..." and Google responds with a "come on, make up your mind" (or similar :))

[sprobertson]

I do something similar but I just pre-define the names of lights I have in Home Assistant (e.g. "lights.living_room_lamp_small" and "lights.kitchen_overhead") and a smart enough LLM handles it.

If you just say "the lamp" it asks to clarify. Though I hope to tie that in to something location based so I can use the current room for context.

[jorvi]

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input.

A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”, assistant just assumes you ‘deserve’ two even if your verbal command is to order one.

A much scarier example is asking the assistant to “preheat the oven when I move downstairs” a few times. Then finally one day you go on vacation and tell the assistant “I’m moving downstairs” to let it know it can turn everything off upstairs. You pick up your luggage in the hallway none the wiser, leave and.. yeah. Bye oven or bye home.

Edit: enjoy your unlocked doors, burned down homes, emptied powerwalls, rained in rooms! :)

[coder543]

No. LLMs do not have memory like that (yet).

Your 'scary' examples are very hypothetical and would require intentional design to achieve today; they would not happen by accident.

[jorvi]

I love how burning your house down is something that deserves air quotes according to you.

All I can tell you is this: LLM’s frequently misinterpret, hallucinate and “lie”.

Good luck.

[amluto]

Preventing burning your house down belongs on the output handling side, not the instruction processing side. If there is any output from an LLM at all that will burn your house down, you already messed up.

[flemhans]

I'd go as far as saying it should be handled on the "physics" level. Any electric apparatus in your home should be able to be left on for weeks without causing fatal consequences.

[lacrimacida]

Im not taken aback by the current AI hype but having LLMs as an interface to voice commands is really revolutionary and a good fit to this problem. It’s just an interface to your API that provides the function as you see fit. And you can program it in natural language.

[jodrellblank]

Chapter 4: In Which Phileas Fogg Astounds Passepartout, His Servant

Just as the train was whirling through Sydenham, Passepartout suddenly uttered a cry of despair.

"What's the matter?" asked Mr. Fogg.

"Alas! In my hurry—I—I forgot—"

"What?"

"To turn off the gas in my room!"

"Very well, young man," returned Mr. Fogg, coolly; "it will burn—at your expense."

- Around The World in 80 Days by Jules Verne, who knew that leaving the heat on while you went on vacation wouldn't burn down your house, 1872.

[jpsouth]

When you think about the damage that could be done with this kind of technology it’s incredible.

Imagine asking your MixAIr to sort out some fresh dough in a bole and then leaving your house for a while. It might begin to spin uncontrollably fast and create an awful lot of hyperbole-y activity.

[jorvi]

I suggest looking up how electric motors work lest you continue looking stupid :)

[jpsouth]

I’ll just not worry myself over seemingly insane hypotheticals, lest I continue looking stupid, thank you.

[jorvi]

I mean there is multiple people all over the main post pointing out how LLMs aren’t reliable but you do you.

[05]

All of those outcomes are already accessible by fat fingering the existing UI. Oven won’t burn your house down, most modern ones will turn off after some preset time, but otherwise you’re just going to overpay for electricity or need to replace the heating element. Unless you have a 5 ton industrial robot connected to your smart home, or have an iron sitting on a pile of clothes plugged in to a smart socket, you’re probably safe.

[kybernetikos]

If it wasn't dangerous enough by default, he specifically instructs it to act as much like a homocidal AI from fiction as possible, and then hooks it up to control his house.

I think there's definitely room for this sort of thing to go badly wrong.

[jasonjmcghee]

Out of curiosity what are you using the vision aspect for?

Fwiw bakllava is a much more recent model, using mistral instead of llama. Same size and capabilities

[canada_dry]

> vision aspect

It checks a webcam feed to tell me the current weather outside (e.g. sunny, snowing) though the language parsing is a more important feature.

> more recent model

Yes... models are coming out quicker every week - it's hard to keep up! But I put this one in place a few months ago and its been working fine for my purposes (basic voice controller home automation).

[ilaksh]

Does anyone know if there is something like bakllava but with commercial use permitted?

[shortrounddev2]

But why use an llm for that? This kind of intent recognition has existed for a while now and we already have it in the form of smart speakers. It seems like an overkill tool for the job

[dr_dshiv]

> Several of the latest models are on par to the results from Gpt4 in my tests.

Wow! So almost as good as alexa?

[AdrienBrault]

Probably much better than alexa. Gpt 3.5 is miles ahead alexa

[dr_dshiv]

Sorry that was a bad joke