Site-wide links

Gestural Interface & Design Guidelines

Contents:

Design & Specification of Existing Interfaces

     Playstation Eye, Playstation Move, Nintendo Wii

Things to Avoid

     General rules, specific things to avoid

Things to Consider

     Context sensitivity, control actions, chaining/combo, speed of actions

Haptic Feedback

     Overview, applications, future usage

Augmented Reality

     Usage terms, things to be aware of

Kinect-Related Terms

References

 


Design & Specification of Existing Interfaces

PlayStation Eye

  • Richard Marks (original Eye Toy designer) spearheaded development efforts for Sony’s next gen device for PS3
  • Design criteria:

                o   New innovation & improvements over previous design of EyeToy

                o  Supports 640 x 480 at 60fps / 320 x 240 at 120 fps

                o  Upgraded microphone to use modern array technology

                o  Fixed focus camera with manual zoom for variety fields of view

  • Largely a technology upgrade to EyeToy that takes advantage of modern chip technologies and processing power

PlayStation Move

  • Designed to work in conjunction with PlayStation Eye
  • Controller:

                o   Various buttons for user interaction

                o   LED lighted orb at top used for distance calculation from camera

                             o  Calculation simple & fast because of spherical, known shape/size

                o   Three-axis linear accelerometer

                o   Three-axis angular rate sensor (similar to PS3 SIXAXIS technology)

                o  Highly accurate tracking of controller motion

                o   Changeable orb color for higher background contrast

                o   Works with secondary controller for higher range of input

  • Secondary Controller:

                o   Called the navigation controller

                o   Similar in functionality to the Nunchuck controller on the Nintendo Wii

                o   Takes the place of the left analog stick of a normal PS3 controller

  • PS3 capable of tracking 4 orbs at once

Nintendo Wii

  • Smaller, cheaper console with different, intuitive interface as an alternative to the “arms race” of gaming console power
  • Input controlled primarily by WiiMote

               o   Infrared sensors triangulate position of WiiMote in real space using Infrared LEDs placed in television-mounted sensor bar to map position of cursor on screen

               o   Accelerometer within Wiimote also used, but as an auxiliary to infrared sensing

               o   Buttons on main WiiMote and optional secondary controller provide further input control

  • Secondary Controller: Nunchuck

               o   Can be plugged into WiiMote for added analog stick functionality, along with additional buttons

  • Add-On: Wii Motion Plus

               o   Added greater accelerometer capabilities for more precise controls

For additional designs of older systems, see Historical Motion Tracking Systems

Things to Avoid

General Rules:

  • Keep in mind possible ergonomic issues
  • Avoid fatigue
  • Remember that interactions should be limited to short discrete actions that mimic normal human object interaction
  • Map repetitive long actions (like running) to more discrete start and stop gestures
  • The result of holding one’s arms horizontally for long periods of time, causing fatique and the feeling that one’s arms are large and heavy
  • Repeating the same interaction repeatedly for long periods of time can lead to both fatigue and other aches and pains

Specific Things to Avoid:

  • Gorilla Arm: The result of holding one's arms horizontally for long periods of time, causing fatigue and the feeling that one's arms are large and heavy

  • Repetitive Motion: Repeating the same interaction repeatedly for long periods of time can lead to both fatigue and other aches and pains

Things to Consider

Context Sensitivity

  • Purpose:

              o   Reduce commands necessary to learn

              o   Reduce clicks/keystrokes

              o   Reduce screen clutter

  • Disadvantage: requires large amount of meta data, specialized algorithms (case analysis or AI)
  • Common uses:

              o   Help buttons

              o   Suggestions for similar topics

              o   Devices only necessary in certain situations (special weapons or abilities)

              o   Action buttons

  • Increasingly being used as keyboard-less devices become more popular
  • Context Sensitivity gone Wrong: MS Word Paperclip

Control Actions (specific to context sensitivity)

  • Hold arm up to indicate actual command

              o   Move hand/arm to direct pointer movement

              o   “push” forward to indicate selection

  • Mimic motions used with physical items

              o   Vacuum cleaner

              o   Interactive storybooks

              o   Etc…

  • Intuitive, well-known gestures known to work well

              o   Pinch to change zoom

              o   Two-finger rotate

Chaining/Combos

  • Must be simplistic enough for user to easily understand
  • Combo depth (similar to folder depths in filesystems)
  • Dependent on sensitivity of input device
  • May need to account for Kinect lag time
  • Need well defined intervals of speed

Speed of Actions

  • Too many small intervals can trigger unwanted commands

Haptic Feedback

  • Technology that involves some form on physical interaction with the user
  • Arose from the need for users to be able to interact more thoroughly with their working environment
  • Early Examples: ranged grabber mechanisms, early airplanes (feedback eliminated, then later restored artificially to notify pilots if plane is stalling)
  • Goal: add another level of sensing to virtual objects, applications, or machines
  • Currently sight & sound are only senses commonly taken advantage of in everyday technology

             o   Sight: monitors & projectors

             o   Sound:  speakers & headphones

  • This deals with output, not input, meaning that force-measuring hardware would be the tactile equivalent of a camera or microphone
  • Current Applications:

             o   Video game controllers that send feedback to the user (shaking controller)

                           Simulates driving over rough terrain, turbulence, hit by enemy fire, etc..

             o   Medical simulators, flight simulators, graphic design

                          §  Physical/digital robotic arms move with user and stop in places where a virtual object is supposed to be solid

  • Future Uses:

             o   Interactions with 3D holograms

                          §  Acoustic radiation created throughout model

                          §  User putting hand inside hologram area result sin sound waves creating pressure on the user

                          §  Potential uses ranging from video games and simulations to medical procedures and robot teleoperation

Augmented Reality

  • Focal awareness:

             o   allocation of high levels of attention to and high levels of modeling of the states of physical, virtual, or imaginary people or objects

             o   Uses conscious attention resources

  • Peripheral awareness:

             o   A limited consciousness of people or objects

             o   Based largely on automatic attention processes indicating that people or objects are co-present

  • Within-Interactant Social Presence Symmetry:

             o   index of the degree to which an individual perceives their sense of social presence to be equal or symmetrical to the other interactants

  • Cross-Interactant social presence symmetry:

             o    index reports degree to which each of two or more individuals perceive the other’s level of social presence as matching the other’s own self-assessed social presence

  • Gesture Recognition:

             o   system must recognize user’s gesture in 3D space independently of the user’s location

             o   Gestures can be trained before recognition

  • Extending Reality:

             o   system must provide means for presenting auxiliary information in the field of view of a user’s VR headset

             o   If particular gestures are recognized, the system must change the state of the correspondent virtual object

  • Posture Analysis (two parameters):

             o   Location of the fingertip used for pointing

             o   Number of distinct fingers found during extraction selection

 

Kinect-related Terms (paraphrased from OpenNI documentation)

  • Image Map – an array of pixels that represents an image             
  • IR Image Map – and image map in which each pixel represents the brightness of the pixel in grayscale
  • Depth Map – image map with pixels represented by distance from sensor
  • Color Image Map – imag emap with pixels represented by an RGB value
  • Hand Point – the location of the hand palm
  • Gesture – instruction expression using bodily motion, usually referring to movements beside traditional controller motions (button pressing)

Avatar Movement Glossary (follow link)

References:

Eye Toy:
http://articles.latimes.com/2004/jan/18/business/fi-eyetoy18/3
http://en.wikipedia.org/wiki/EyeToy
“GestureTek Grants Patent License to Sony Computer Entertainment America for EyeToy and PlayStation2 Game Development. " Business Wire  17 February 2005 ABI/INFORM Dateline, ProQuest. Web.  13 Apr. 2011.
Business Editors/High-Tech Writers E3Expo 2003.  "Sony Computer Entertainment Europe -- SCEE -- Selects OmniVision's CameraPlay Technology for EyeToy USB Camera For PlayStation2. " Business Wire  30 May 2003 ABI/INFORM Dateline, ProQuest. Web.  13 Apr. 2011.
Minority Report:
http://en.wikipedia.org/wiki/Minority_Report_(film)
http://oblong.com/
http://www.youtube.com/watch?v=dyMVZqJk8s4
http://www.ted.com/talks/john_underkoffler_drive_3d_data_with_a_gesture.html
http://web.mit.edu/invent/iow/underkoffler.html
http://web.mit.edu/newsoffice/2002/underkoffler-0717.html
http://dir.salon.com/story/ent/movies/int/2002/07/10/underkoffler_belker/index.html
Eye Toy:
http://en.wikipedia.org/wiki/PlayStation_Eye
http://www.maxconsole.net/showthread.php?59874-Behind-The-Scenes-Of-Playstation-Eye&p=510013&langid=2
PlayStation Move:
http://en.wikipedia.org/wiki/PlayStation_Move
http://community.eu.playstation.com/t5/PlayStation-Move/Move-FAQ/td-p/11447251
http://us.playstation.com/ps3/playstation-move/faq/
http://gdc.gamespot.com/story/6253435/sony-reveals-what-makes-playstation-move-tick
VideoPlace
http://en.wikipedia.org/wiki/Videoplace
GestureTek
http://www.vjvincent.com/
Nintendo Wii:
http://en.wikipedia.org/wiki/Wii_Remote
http://arstechnica.com/hardware/news/2006/10/wii.ars
http://electronics.howstuffworks.com/wii.htm
Guidelines (for web development):
http://www.readwriteweb.com/archives/designing_for_iphone_wii.php
http://www.springerlink.com/content/h850v803qxp537w9/
Things to Avoid
http://en.wikipedia.org/wiki/Touchscreen#Ergonomics_and_usage 
http://en.wikipedia.org/wiki/Gestural_interface 
http://gesturedesignblog.com/?page_id=19 
http://www.billbuxton.com/multitouchOverview.html
Things to Consider
http://en.wikipedia.org/wiki/Context-sensitive_user_interface
http://kinecthacks.net/hands-free-vacuum-cleaner/
http://kinecthacks.net/interactive-book-with-kinect/
http://www.kinect-hacks.com/kinect-hacks/2011/03/30/finger-tracking-and-...
Haptic Feedback:
http://electronics.howstuffworks.com/gadgets/other-gadgets/haptic-technology.htm
http://en.wikipedia.org/wiki/Haptic_technology
http://krafttelerobotics.com/force_feedback.html
Augmented Reality:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.114.6155&rep=rep1&type=pdf
http://www.buccleuchpark.net/MUCOM/publi/acmMobility09.pdf
http://people.scs.carleton.ca/~roth/iit-publications-iti/docs/NRC-46491.pdf

 

 

 

 

 

 

·         Eye Toy:

  • http://articles.latimes.com/2004/jan/18/business/fi-eyetoy18/3 
  • http://en.wikipedia.org/wiki/EyeToy 
  • “GestureTek Grants Patent License to Sony Computer Entertainment America for EyeToy and PlayStation2 Game Development. " Business Wire  17 February 2005 ABI/INFORM Dateline, ProQuest. Web.  13 Apr. 2011.
  • Business Editors/High-Tech Writers E3Expo 2003.  "Sony Computer Entertainment Europe -- SCEE -- Selects OmniVision's CameraPlay Technology for EyeToy USB Camera For PlayStation2. " Business Wire  30 May 2003 ABI/INFORM Dateline, ProQuest. Web.  13 Apr. 2011.

·         Minority Report:

·         Eye Toy:

·         PlayStation Move:

·         Nintendo Wii:

·         Guidelines (for web development): 

·         Things to Avoid

·         Things to Consider

·         http://en.wikipedia.org/wiki/Context-sensitive_user_interface 

·         http://www.kinect-hacks.com/kinect-hacks/2011/03/30/finger-tracking-and-... 

·         Haptic Feedback:

·         Augmented Reality:

·         http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.114.6155&rep=rep1&type=pdf 

·         http://www.buccleuchpark.net/MUCOM/publi/acmMobility09.pdf 

·         http://people.scs.carleton.ca/~roth/iit-publications-iti/docs/NRC-46491.pdf