Touch-free screens with Kinect gesture tracking
Full-body skeleton tracking streamed into the browser, turning any big screen into something people control with their hands.
Mate MarschalkoFounder & ConsultantCompleted
- Expertise
- Spatial Experiences
- Type
- Prototype
- Status
- Completed
- Year
- 2015
- Focus
- Kinect, Gesture Control, WebSockets, Kiosks, Installations
The project
Walk up to a screen. Raise a hand. The cursor follows. Swipe and the content moves. Nobody touched anything, nobody downloaded an app, and nobody had to be shown how.
How it works
A Kinect 2 sensor tracks up to six people at once, 25 joints each, thirty times a second. I read that stream in Node.js, packaged every frame as JSON and pushed it over WebSockets. Any browser on the network could subscribe, including a Mac across the room that had no idea a Kinect was involved.
From there it's just web development. The browser gets hand positions, head position and body pose, and I turn those into gestures: point, grab, swipe, wave, step closer, walk away.
That last part matters. Because the interface is HTML, CSS and JavaScript, it can be anything a website can be. Video, 3D, live data, animation. And it runs full screen on whatever display you've got.
Where this belongs
Shop windows. A passer-by stops, and the display reacts. Wave to browse the new collection after hours, while the shop is closed.
In-store kiosks. Browse products, compare options, or check stock on a big screen without touching a grubby surface. Hygiene stopped being a niche concern a while ago.
Museums and visitor centres. Step into a zone and the exhibit wakes up. Lean in to zoom. Groups can explore together, because the sensor tracks several people at once.
Events and launches. A product reveal that responds to the crowd. A photo moment that tracks poses. Something people film and share.
Receptions and waiting areas. Wayfinding, check-in or a playful idle screen that notices when someone approaches.
Today's version
The Kinect itself has been retired, but the idea is stronger than ever. Depth cameras, webcams with on-device pose estimation, and AI vision models now do the same job with smaller, cheaper hardware. I'd also add things that weren't practical in 2015: recognising what people are holding, estimating crowd size and dwell time, or adapting content to who's actually standing there.
I build the full stack for this: sensor integration, real-time data plumbing, and the interface people actually see. If you've got a space and a screen and want them to do more, let's talk.
Related projects
All Spatial Experiences
React IoT: building electronics like we build websites
Two ideas for bringing web-style structure to hardware: a markup language for physical devices, and React components that switch real lights and read real sensors.
Billboards that read the room
A vision model looks at who's walking past a digital billboard, then picks the ads most likely to land. Online-style targeting, out on the street.