3D Interaction Design

Interaction in three dimensions is fundamentally different from interaction in two. Every designer trained on windows and cursors must unlearn assumptions about planar layout, pixel-precise pointing, and implicit desktop context. This skill catalogs the major input modalities, selection and manipulation techniques, navigation metaphors, and feedback strategies, with heuristics for when each applies.

Agent affinity: bret-victor (direct manipulation and feedback), sutherland (input device foundations), krueger (gestural and responsive interaction)

Concept IDs: spatial-reasoning-3d, spatial-coordinate-navigation, spatial-iterative-build-process

Input Modalities

Modality	Degrees of freedom	Precision	Fatigue	Best for
6DoF controller	6 per hand	High	Low	VR manipulation, precise placement
Gaze	2 (direction)	Medium	Low	Targeting, dwell-select
Gesture (free-hand)	Variable	Medium-low	High	Expressive, social, short sessions
Voice	Symbolic	High (recognition)	None	Labeling, commands, dictation
Eye tracking	2	High	None	Foveated rendering, implicit targeting
Brain-computer	Symbolic	Very low	Variable	Research, accessibility
Props / tangibles	Varies	High	Low	Authoring, haptic grounding

The designer's first decision is which modality or combination the interaction will use. Voice plus gaze is a common accessibility-friendly pair. 6DoF plus audio is a common VR game pair. Gesture plus projected video is Krueger's VIDEOPLACE legacy.

Selection Techniques

Selecting an object in 3D is the precursor to manipulating it. The difficulty depends on distance, occlusion, density of candidates, and input modality.

Ray-cast (pointing)

Cast a ray from the controller (or from the head) into the scene. The first intersected object is the candidate. Simple, familiar, works at arbitrary distance. Fails when candidates are densely packed or partially occluded.

Virtual hand

The user's hand (represented by a 3D model) reaches into the scene and touches the object. Direct, tangible, satisfying. Limited by arm length.

Go-go

Extends the virtual hand nonlinearly as the user reaches further. Close objects use 1:1 mapping; distant objects get amplified reach. Popularized by Poupyrev et al. 1996.

World-in-miniature (WIM)

Present a small replica of the scene in the user's hand. Selecting in the miniature selects the corresponding object in the full scene. Excellent for selecting occluded or distant objects.

Image-plane selection

Treat the user's hand as if it were projecting onto a 2D plane at arm's length. Selection becomes 2D pointing in a virtual window. Easier precision, loses depth cues.

Manipulation Techniques

Once selected, the object must be moved, rotated, scaled, or otherwise transformed.

6DoF translate-rotate

Attach the object to the controller transform. Intuitive, fast, but constrained by the user's physical reach. Standard in VR building tools.

Widget-based

Display manipulation widgets (arrows for translation, rings for rotation) around the selected object. Precision improves, exploration slows.

Constrained manipulation

Lock the object to a grid, a plane, or an axis. Essential for building tasks where alignment matters. Trades freedom for precision.

Two-handed manipulation

Use both hands to specify transformations: distance between hands for scale, vector between hands for orientation. Natural for symmetric tasks.

Navigation Techniques

Moving through a 3D environment is its own interaction problem, distinct from object manipulation.

Teleport

Point to a destination; the user's position jumps there. Comfortable (no vection), discontinuous (breaks immersion). Standard for comfort-first VR.

Continuous (thumbstick)

Smooth locomotion via thumbstick. Immersive, cinematic, but causes simulator sickness in a meaningful fraction of users. Mitigation: blinders, low field-of-view during motion.

Redirected walking

Subtly rotate the virtual environment so the user walks in physical circles while believing they are walking in a straight line in the virtual environment. Allows large virtual spaces in small rooms. Requires careful calibration.

World-in-miniature navigation

Grab and drag the miniature to reposition. Used in architectural walkthroughs.

Portal

Step through a portal to move to a distant location. Narratively expressive, spatially discontinuous.

Fitts' Law in 3D

Fitts' law (1954) relates selection time to target distance and size: MT = a + b log2(D/W + 1). In 3D, W must be generalized to the smallest angular dimension of the target from the user's viewpoint, and D is path length through 3D space. The implications for designers:

• Large targets are easier. Minimum touchable target in VR is about 2-3 cm at arm's length.
• Distance matters more than in 2D because the user's hand must actually move through physical space.
• Depth occlusion adds cost. A target behind another requires rotation or navigation before selection.

Feedback Design

Feedback closes the loop between user action and system response. Without feedback, the user does not know whether the interaction succeeded.

Visual

Highlight the target on hover. Show the trajectory of a thrown object. Animate the manipulation in real time. Use motion blur or trails to indicate velocity.

Audio

Play a click on selection. Play a whoosh on fast movement. Play a clunk when an object lands on a surface. Spatial audio cues direction.

Haptic

Vibrate the controller on contact. Increase vibration intensity with resistance. Use haptic patterns to distinguish events (short pulse for selection, long rumble for impact).

Proprioceptive

Align virtual hand with physical hand position. When the two diverge (because the virtual object was released or because collision overrode the transform), the user gets a subtle mismatch cue that something changed.

Comfort and Simulator Sickness

Visual-vestibular mismatch causes simulator sickness. The common triggers:

• Continuous locomotion without head movement
• Sudden acceleration or deceleration
• Low frame rate (under 90 Hz)
• High field of view during motion
• Camera shake or roll

Mitigations: teleport, comfort vignettes, stable horizon lines, reduced FoV during movement, 90 Hz minimum, seated modes for longer sessions.

When to Use This Skill

• Designing VR/AR applications where the user's body is the primary input
• Reviewing 3D interfaces for discoverability, comfort, and efficiency
• Choosing selection/manipulation/navigation techniques for a specific task shape
• Diagnosing user complaints about difficulty, fatigue, or nausea

When NOT to Use This Skill

• Pure 2D UI design
• Screen-based 3D (Unity Editor, CAD tools on a desktop) where the input is mouse/keyboard
• Non-interactive 3D rendering

Common Mistakes

Mistake	Why it fails	Fix
Porting 2D widgets into 3D unchanged	Widgets overlap, occlude, fail to afford 3D use	Redesign for 3D placement and scale
Small distant targets	Fitts penalty compounds with hand movement	Enlarge or use WIM
No feedback on selection	User cannot tell if tap registered	Always provide multi-modal feedback
Continuous locomotion without comfort options	Fraction of users get sick	Offer teleport as primary, continuous as opt-in
Gesture without discoverability	Users do not know what gestures exist	Provide a tutorial and on-demand cheat sheet
Ignoring fatigue in free-hand interfaces	Gorilla arm after 5 minutes	Allow arm rest, add input alternatives

Cross-References

• sutherland agent: History of input devices, from Sketchpad's light pen onward
• bret-victor agent: Direct-manipulation principles and feedback immediacy
• krueger agent: Responsive environments and gestural input lineage
• furness agent: Military and medical VR, high-stakes interaction
• azuma agent: AR registration and tracking that underlies interaction precision
• immersive-environment-design skill: Environment design that sets the stage for interaction

References

• Sutherland, I. E. (1968). "A head-mounted three-dimensional display." AFIPS Proceedings, 33, 757-764.
• Fitts, P. M. (1954). "The information capacity of the human motor system." Journal of Experimental Psychology, 47(6), 381-391.
• Poupyrev, I., Billinghurst, M., Weghorst, S., & Ichikawa, T. (1996). "The go-go interaction technique." UIST '96.
• Bowman, D. A., Kruijff, E., LaViola, J. J., & Poupyrev, I. (2005). 3D User Interfaces: Theory and Practice. Addison-Wesley.
• Razzaque, S., Kohn, Z., & Whitton, M. (2001). "Redirected walking." Eurographics 2001.