Hand Tracking Website: A Glass Flower That Blooms When You Wave

How my hand tracking website works: MediaPipe reads 21 points on your hand, one number opens a glass flower, a rude gesture wilts it. Code-level breakdown.

Written by
Ayoub Kada
Published on
November 19, 2026
Read time
8 min
Category

A hand tracking website reads your hand through the webcam and turns its shape into input, the way a mouse turns movement into a cursor. In my Verdure experiment, an environmental landing page, you open your hand and a glass flower blooms, close it and the flower closes, and make a rude gesture at it and it wilts and drops its petals. It runs entirely in the browser: Google's MediaPipe Hand Landmarker finds 21 points on your hand in each video frame, a few lines of math turn those points into one "openness" number between 0 and 1, and that number drives the flower. The camera only switches on after you ask for it, and the page still works with a mouse. Below is how each layer works in the shipped build, the design decisions behind it, and when hand tracking is the wrong choice.

What the Verdure hand tracking website actually does

Verdure is a fictional sustainability brand with a classic hero: "A Cleaner Planet Blooms Further", a Get Started button, a few partner and impact numbers. Under the headline sits a still pond with a glass flower on it. A card in the corner says "Use your hand". Tap it, allow the camera, and the page starts watching.

Open your palm toward the screen and the flower opens with you, petal by petal. Half open your hand and the flower stops halfway. Show it your middle finger and it reacts the way a living thing might: the petals lose their colour, the stem droops, and the petals fall. Keep being nice and it comes back. Without a camera, moving your pointer over the water stirs it, so nobody gets a dead page.

The item on my Verdure experiment page lists the tools: Figma, Claude, the Figma MCP, Higgsfield, Kling video, MediaPipe hand tracking and WebGL water. The live demo opens from that page; it plays best in a desktop browser with a webcam.

How the hand tracking works, layer by layer

Layer 1: the camera, only after a click

The page does not ask for the camera on load. A "Use your hand" card waits until you click it, then calls getUserMedia for the front camera and plays the stream muted and inline in a hidden video element. If the browser has no camera API, the card switches to a no-camera state and the pointer fallback stays in charge. This is the most important design call in the whole build. A permission prompt that fires before people know why is the fastest way to lose them, the same lesson I wrote about in onboarding UX that took activation from 13% to 65%: show value first, ask for trust second.

Layer 2: MediaPipe finds 21 landmarks per frame

The hand model is the MediaPipe Hand Landmarker, loaded from the @mediapipe/tasks-vision package (version 0.10.14) with its WebAssembly runtime. The build asks for the GPU delegate and falls back to the CPU if that fails. Instead of polling on a timer, it uses requestVideoFrameCallback, so detection runs once per real camera frame and never on a frame it already saw. Each result is a list of 21 landmarks: the wrist, then four joints for each finger, numbered 0 to 20. A small overlay canvas can draw those points so you can see what the model sees.

Layer 3: turning 21 points into one number

Raw landmarks are useless to a flower. It needs one value: how open is this hand? The function in the build does it like this:

  • Measure the palm: the distance from the wrist (point 0) to the base of the middle finger (point 9). Every other distance is divided by it, so the result does not change when your hand moves closer to the camera.
  • For each of the four fingers, take the distance from the wrist to the fingertip minus the distance from the wrist to that finger's knuckle. A curled finger has its tip close to the knuckle, an extended one is far past it. Average the four.
  • Measure the thumb spread: thumb tip (point 4) to the base of the little finger (point 17).
  • Blend them: 85% finger extension, 15% thumb spread, each remapped to a working range, then clamped between 0 and 1.

That number is the whole interface. It goes out as a custom verdure:hand browser event, and the rest of the page listens for it.

Layer 4: detecting the rude gesture

The middle finger check uses the same normalised extension, finger by finger. The gesture counts when the middle finger is clearly extended (above 0.42 of the palm length), the index and ring fingers are each less than 45% as extended as the middle one and both stay under 0.22, and the little finger stays under 0.3. Five plain comparisons, no machine learning beyond the landmarks. I like this because it is readable: when it misfires, you can see exactly which threshold to move.

Layer 5: the flower and the water are WebGL

The bloom is AI video, made with Kling through Higgsfield, and the openness value scrubs through it, the same scrub-a-film idea behind my scroll animation examples with a perfume film. The wilt is not a second video. A WebGL2 shader takes the flower apart using a petal ID texture, a colour texture and a separate stem image, then drives uniforms named wilt, droop, sway and drop. Wilting mixes the colour toward a dusty grey using luminance weights, the stem rotates down around a pivot, and petals detach and fall one by one. The pond is a small fluid simulation in WebGL2: velocity is advected, divergence and pressure are solved, curl adds swirl, and each pointer move "splats" force into it. The result warps the still photo of the water, so the ripples follow you.

Hand tracking vs mouse vs touch: when each input fits

InputWhat it is good atWhat it costsUse it when
Mouse or trackpadPrecision, hover, every desktop user has oneNo sense of body or emotionAnything people must complete: forms, checkout, navigation
TouchDirect manipulation, gestures people already knowFingers hide the content, no hoverMobile, physical feeling interactions like the drag and throw in my physics pages
Hand tracking (webcam)Whole-hand expression, delight, a moment people film and shareCamera permission, light conditions, CPU or GPU load, privacy worries, not usable by everyoneAn optional layer on a brand or campaign page, never the only way in

The design decisions that matter more than the model

One number, not a gesture vocabulary. It is tempting to map ten gestures to ten actions. Nobody remembers them. Openness is something every person already does without instruction, and the flower answers instantly, so the page teaches itself in one second.

A metaphor that matches the hand. An open hand and an opening flower are the same shape. That is why it feels obvious. If the hand had controlled a slider or a carousel, it would feel like a gimmick.

Give it a personality. The wilt is the part people remember. Interfaces should be fun and alive, and a flower that sulks when you are rude to it turns a tech demo into a story. I used the same idea with the moody bird on my interactive landing page examples.

Never require it. The headline, the buttons and the content all work without the camera. Hand tracking is the reward for curious visitors, not a gate.

Where AI fits in a build like this

The layout started in Figma and went to code through the Figma MCP with Claude, the workflow I describe in my AI design workflow from idea to App Store. The bloom footage came from an AI video model. The shaders, the fluid sim and the landmark math are the kind of code I write with Claude and then tune by hand, constant by constant, until the motion feels right. What AI did not decide: that the camera waits for a click, that openness is the only control, that rudeness gets a reaction. Those are product calls. It is the same split I lived as a lead product designer and design engineer at ViewStats, where I designed features and also shipped the React components.

When you should not build a hand tracking website

Do not use it for anything a person must finish. Do not use it on pages where most traffic is mobile, where the front camera faces the wrong way for this kind of play and battery matters. Do not use it where trust is fragile, like banking or health, because a camera prompt reads as a threat there. And test it in bad light, because the model will. If any of those apply, a pointer or scroll interaction will give you most of the delight at none of the risk.

FAQ: hand tracking websites

How does hand tracking work in a browser?

A model such as MediaPipe Hand Landmarker runs in the page through WebAssembly and WebGL. For each webcam frame it returns 21 points on each hand. Your code turns those points into values, like how open the hand is, and uses them as input.

Does a hand tracking website send my camera video to a server?

It does not have to. In Verdure the model runs in the browser and the frames are only read locally to compute landmarks. As a designer, say this clearly next to the camera button, because people will wonder.

Is MediaPipe hand tracking free to use?

MediaPipe is an open source Google project and the web package is published on npm. Check the license and model card on the official MediaPipe page for your use case before shipping.

Can hand tracking work on mobile websites?

Technically yes, but performance and the front-camera angle make it a weak fit for most phone visits. I treat it as a desktop enhancement with a touch or pointer fallback.

What is the easiest hand gesture to design for?

Open versus closed. It needs no explanation, it is robust to camera distance once you normalise by palm size, and it maps to many natural metaphors: blooming, grabbing, revealing.

Try it, then steal the checklist

Open the demo, open your hand, then be a little rude to it. If you build your own, keep this list: ask for the camera only after a click, reduce the hand to one or two values, normalise by palm size, pick a metaphor that matches the hand's shape, give the object a reaction worth sharing, and keep a full experience for people without a camera. More playable builds live on my AI design experiments page, and if you want a landing page that people play with instead of scroll past, hire a product designer who ships the code.

If you have a web or mobile app idea in mind or you need some advice about product design, feel free to get in touch ✉️