Gaze Interaction on Rainy Days

Eye tracking is finally becoming mainstream. It’s been around for accessibility for years, but now mixed reality headsets come with built-in eye trackers. This isn’t exactly new either -— head-mounted eye trackers have existed for a while -— but now that big tech companies are investing in it, more people are paying attention.

The most exciting part? Exploring gaze-based interaction in 3D. Apple Vision Pro, for example, lets users interact with objects using a combination of gaze and hand movements. If you’re interested in this area, I highly recommend following Ken Pfeuffer ’s work — he’s done some incredible research on the topic. My colleagues at UIUC recently published a paper showing a fascinating technique: selecting objects by changing gaze depth.

That paper sparked an idea in my mind. I won’t be working on it anytime soon, so I’m writing it down here.

Why is Gaze Interaction Hard?

Interacting with computers using our eyes might sound futuristic and fun. Or maybe you find it creepy, invasive, and pointless -— I respect that.

But here’s the thing: we move our eyes unconsciously most of the time. They’re great at reacting to stimuli but surprisingly hard to control on demand.

Let’s try something. Below, you’ll see two circles: one static, one moving. Try “drawing” each circle with your eyes. If you want, you can enable your webcam to record yourself (don’t worry, these videos stay on your computer).

If you don’t want to record, just pay attention to how your eyes move. I recorded myself doing this -— yes, it’s weird to watch someone else’s eyes, but trust me, it’s worth it. Can you guess which circle I was looking at in each video?

Try it yourself:

1. Click/tap on the circle you want to draw.
2. Move your mouse/finger to the “Start” button (this will also be the Stop button).
3. Position your face within the circle.
4. Click/tap “Start” and trace the circle with your eyes.
5. Click/tap “Stop.”
6. You can delete the video and retry by clicking the trash icon.

What Just Happened?

In the first video, I was looking at the moving circle. In the second, I was looking at the static circle.

How can you tell?

• In the second video, my eyes move in quick, jumpy steps. These are saccades —- the default way we look at things.
• In the first video, my eyes move smoothly. These are pursuit movements, and they only happen when following a moving object.

So, What’s the Big Idea?

This experiment shows that visual cues are crucial for gaze interaction.

The question I’ve been thinking about is: How can we indicate that a layer of interactive objects is available -— without disrupting the user’s field of view?

And that’s when I realized: our brains are great at ignoring things that are right in front of our eyes.

Example 1: The Aquarium

Imagine you’re at an aquarium, watching fish swim. You take a photo, and later, you notice something unexpected:

What happened? The camera captured reflections on the glass -— but you didn’t notice them while looking at the fish. Your brain filtered them out.

Example 2: The Nose You Forgot You Had

Have you noticed your nose lately? You’re always seeing it, but your brain ignores it. Until now. Sorry.

Example 3: Raindrops on a Window

If it’s lightly raining, you can still see through a window. You’re aware of the raindrops, but you can also choose to focus on them or ignore them.

The Idea: Interactive Raindrops

What if we used raindrops on a virtual window as a subtle visual cue for interactable layers?

• Users could ignore the raindrops and focus on the content behind them.
• Or they could shift their focus to the raindrops to interact with objects.

Would this be uncomfortable? Useful? I have no idea. But I think it’s worth exploring.

Maybe I’ll prototype it someday. We’ll see.