The smartest home assistant I could build

"Just noticed I'm recording this video in the dark in the office. Can you do something about that?"

Siri would shrug. My assistant turned the office light on and said so.

Version one: plain English as the program

My house runs on HomeKit. Dozens of lights, thermostats, underfloor heating, ventilation and cameras. In January 2023 I replaced Siri's brain with GPT-3, and I did it without writing a line of conventional code.

The whole thing is a prompt. It tells the model it's the brain of the house, lists the rooms and devices, gives it the time and location, and asks for every reply as structured JSON in one of four categories:

• command: do something ({"location": "office", "target": "light", "value": "on"})
• query: read something, like the kitchen temperature
• answer: a general question, answered in context
• clarify: ask the user to rephrase

An Apple Shortcut sends the spoken request, parses the JSON and fires the right HomeKit action.

What surprised me was the judgement. "My son's reading for another 20 minutes, switch his light off when it's time to sleep" came back with the bedroom light, off, and a timestamp twenty minutes ahead. "Set the bedroom to a temperature that'll help me sleep" came back as 19°C. Nobody wrote a rule for either.

The article hit around 3,000 claps on Medium, and the reaction was the same everywhere. Why can't the big assistants do this?

Version two: real conversation, real access

By late 2024 OpenAI's Realtime API made natural, interruptible voice possible. So I rebuilt it properly. The assistant now talks to a small Node.js server, and every capability is its own API endpoint:

• Switch lights and report their state
• Read and set the heating per room, with temperature and humidity
• Look at the CCTV cameras and describe what's there
• Read the latest photo on the phone
• Get weather and news, and discuss them
• Manage shopping and to-do lists, including adding items in bulk
• Read and update the calendar
• Draft and send WhatsApp messages
• Print things

The server translates each call into an Apple Shortcut, runs it, and hands the result back so the conversation carries on with the new information. Adding a skill means one endpoint, one Shortcut, one line in the schema.

What businesses can take from this

Strip away the house and you've got a pattern for any voice or chat interface that needs to do things: a natural-language front end, a model that turns intent into structured actions, and a thin, secure API layer onto real systems. Booking platforms, field-service tools, internal ops dashboards and hotel rooms all fit.

I design and build these end to end, including the dull-but-critical parts: authentication, permissions, cost control and keeping the model's actions predictable. Tell me what you'd want yours to do.

More on this project

• The voice demo on YouTube
• ChatGPT in an iOS Shortcut: the original HomeKit assistant (Medium)
• Building the real-time voice assistant, step by step (Medium)
• A five-minute Siri and ChatGPT shortcut (Medium)
• Try the original Shortcut