Apple Patents for Multi-Touch and Teaching Gestures
On July 3rd, the US Patent and Trademark Office published a patent application from Apple computer for "Gesture Learning."
Through this patent, Apple seeks to protect a method for teaching hundreds of gestures that are used or will be used to control multi-touch sensitive devices. This "Gesture Learning" patent follows last year's "Multi-Touch Gesture Dictionary" patent that I wrote about earlier this year in "Gesture Patents Point Way to Full-Body Interfaces."
Image from Gesture Learning Patent that shows interface for teaching Gestures
What I find so intriguing about these two patent filings is that Apple is creating a rich gesture-based movement language for controlling computers, mobile phones, music players and other devices. And Apple is also devising a computer-based system for teaching users how to master what will inevitably become a sophisticated gestural language with many options and possibilities.
My question for choreographers and dancers is whether you believe that Apple has entered into territory traditionally thought of as the domain of dancers and movement experts? And what will happen as new patent applications eventually go well beyond seeking to protect hand gestures and attempt to protect full-body movements generated by the arms, legs and torso? Are we just talking about ways to control computer interfaces or do these Apple filings have much greater importance--in particular to the dance community?
Background: Gesture Learning Patent
According to this patent application:
A multi-touch gesture can be as simple as using one or two fingers to trace out a particular trajectory or pattern, or as intricate as using all the fingers of both hands in a complex sequence of movements reminiscent of American Sign Language. Each motion of hands and fingers, whether complex or not, conveys a specific meaning or action that is acted upon by the computer or electronic device at the behest of the user. The number of multi-touch gestures can be quite large because of the wide range of possible motions by fingers and hands. It is conceivable that an entirely new gesture language might evolve that would allow users to convey complex meaning and commands to computers and electronic devices by moving their hands and fingers in particular patterns.
The fingers used and paths that a gesture may take
An individual multi-touch gesture, according to patent, includes two phases. The first phase, the "chord," consists of the hand parts used (e.g, a specified number of fingers and a thumb used in a particular configuration). The second phase consists of the motion of the specific hand parts. "This motion may take the form of lateral motions such as rotation, translation, scaling (expansion and contraction)...in some embodiments, this may comprise moving the hand parts around on the multi-touch surface. In such embodiments, the second phase of the gesture may also comprise vertical motions (relative to the multi-touch surface) such as tapping, double-tapping, etc."
With each hand able to execute twenty-five or more chords, and with each chord having thirteen or more motions associated therewith, there may be over three hundred possible gestures for each hand. Many more gestures are possible if both hands are used together.
New Interactive Application for Learning Gestures
In the summary section of the Gesture Learning patent application, you will find a detailed section describing the possible configuration of a graphical user interface for teaching this embodied language:
A flow chart of the gesture learning software application
The present invention can relate, for example, to a method for teaching gestures. The method can include presenting a display having two display areas, one of which can be a multi-touch monitor window. The multi-touch monitor window can be used to display interactive feedback to a user indicating what gesture the user is performing. The multi-touch monitor window can be overlaid on the first display area or can be a separate window, such as a side-by-side arrangement...
The interactive feedback can take a variety of forms. For example, the interactive feedback can comprise an animated hand together with one or more motion indicators. The animated hand can be a line drawings, a three-dimensional rendering, a translucent shadow of a hand, or other representation. The motion indicators can include motion trails or other representations, which can also be color-coded. The interactive feedback may also be superimposed with an animated display of a correctly performed gesture so that the user can see the difference, if any, between the gesture he performs and the idealized gesture.
Gesture, Movement and the Body Control New Interfaces
Today's post is an update to my video background page, "Movement Is at the Heart of Scientific and Technological Change." Below, you'll find eight videos that demonstrate how gesture, movement and the body are being coupled with new technologies and interfaces for manipulating, viewing and controlling art and objects:
Using Your Body to Manipulate an Art Installation
Light Rain from wowlab is an art installation that users manipulate with their hands and body - text in Japanese, but you'll find pictures and a video clip:
And watch this earlier clip from the creators of the above video to see how they set-up the Wii balance board to control these applications.
Manipulating Computer Animations with Your Feet
LM3Labs developed Catchyoo Sprites, which allows users to manipulate animations on large-screen display with their feet and hands:
Gestures Control Toshiba Notebook
Hand gestures, without physical contact with a computer, can be used to control
Toshiba Qosmio G55 notebook [via Laptop, Trend Hunter and Fresh Creation]:
FlicFlex
The Flicflex concept is based on a bendable, flexible computer interface created by Chris Woebken [via Fresh Creation]:
Take A Seat
Jelte van Geest, a student at the Design Academy Eindhoven in the Netherlands, created these movable, robotic chairs with embedded RFID chips that follow the owner around in preparation for the next seating -- one or more chairs can be controlled simultaneously [via momeld, technabob and Grinding] :
TelePresence in Action
Here's an example of telepresence: a business meeting with John Chambers, CEO of Cisco:
Dance.Draw - Interdisciplinary Project Fuses Interaction Design and Dance
The Dance.Draw Exquisite Interaction project is an interdisciplinary collaboration at the University of North Carolina at Charlotte that combines human-computer interaction (HCI) and dance. In the project's performance work, each dancer holds two wireless Logitech Air mice (one per hand). The movements of the mice serve as input for real-time visualizations that are projected on to a large screen behind the dancers.
Click the following screen shot to watch a video clip of the performance on the Dance.Draw website:
Participating in this project were Dr. Celine Latulipe in the Software and Information Systems Department and Professor Sybil Huskey from the Department of Dance along with student dancers and other contributors. In her research in interaction design, Dr. Latulipe explores two-handed interaction techniques or symmetric interactions that require both hands to actively contribute to a task.
This research focus is reflected in this collaborative dance project. You can learn a bit more about the software, visualization and input techniques behind this dance performance on the project's technology page.
How do choreographers create dances that speak to software developers? In other words, how can layers of meaning and "hidden features and functionality" be made known to people who specialize in creating user interfaces?
First, I encourage you to read Alex Iskold's post "The Rise of Contextual User Interfaces" in ReadWriteWeb. This article points out that software interfaces have for too long been difficult to decipher and encumbered with too many features. We are moving to contextual user interfaces that present users with just the minimal number of features that they need when they want them.
Vimeo is offered as a good illustration of the new generation of minimalist user interfaces. As you'll see in the following video, once you start to play the clip and then move your cursor away from the video player, the available features will disappear from the screen. Then, a small number of features reappear once you move your cursor back over the player. Here's a site-specific performance by Sahra Huby and Philip Bussmann:
To see the contextual functions of the Vimeo player in action, place your cursor over the video player, then click the "Embed" link. Finally, select "Customize size, color, and other options." In all instances, just the functionality you want at any given point in time is made available to you. You are not presented with too many options or irrelevant functions.
Revealing the Layers of Complexity in a Dance
Getting back to my initial question: Say, you were creating a dance piece for a geeky audience of software developers. How would you go about creating and structuring a dance so that developers could relate to it as if it were, in some sense, the user interface for a software program?
How is meaning, in all its forms, made known and revealed? How do the layers of a work that build one on top of another reveal themselves to viewers? How do hidden elements of a work all of a sudden make sense and, possibly, transform the dance? What contextual cues are offered to viewers so that they can get inside a work? How dense or minimalist is a work and what are the trade-offs of each approach? How, overall, does the audience make sense of the structure, movement vocabulary, connections (or lack of connections) among dancers, the emotional quality and changes over time, the narrative and aesthetic flow?
I would find it very interesting to speak with choreographers who have pondered some variation of my questions connecting dance to user interface design. Or who have created dances for non traditional dance audiences.
Siftables, Choreographing a Tangible Interface of Sensing Video Blocks
Created by David Merrill and Jeevan Kalanithi at MIT Media Lab, Siftables aim to make it as easy to manipulate virtual objects as it is for us to handle and organize everyday physical objects.
I've included the video first followed by a brief description. As you watch the video, consider how the small Siftable video blocks interact with each other based on their proximity, content, function and groupings:
Imagine overturning a container of nuts and bolts, then looking through the resulting pile for a particular item. Or spreading photographs out on a tabletop and then beginning to sort them into piles. During these activities we interact with large numbers of small objects at the same time, and they utilize all of our fingers and both hands together. We humans are skilled at using our hands in these ways, and can effortlessly sift and sort - focusing on our higher level goals rather than the items themselves.
And a bit about the technology behind Siftables:
As an interaction platform, Siftables applies technology and methodology from wireless sensor networks to tangible user interfaces. Siftables are independent, compact devices with sensing, graphical display, and wireless communication capabilities. They can be physically manipulated as a group to interact with digital information and media. Siftables can be used to implement any number of gestural interaction languages and HCI applications.
At the heart of this project is the idea of using multiple sensors as a tangible user interface. And as you'll see in the above video, the possibilities for the individual Siftable blocks to influence and manipulate each other seems limitless. Colors can be transfered from one block to another. Video images of faces turn to each other on adjoining blocks. Gestures can change multiple video block displays simultaneously. And physical organizations of blocks can be reflected on a nearby computer screen.
I never considered that gestures could be patented until reading about Apple's efforts last year to apply for patents that describe how users interact with its Macs, iPods and iPhones. Will body movements be patented next as more interfaces accept input from heads, arms and legs?
To ask a bit of an improbable question, will dancers be prohibited from certain movement sequences because they are protected by the US Patent and Trademark Office? This is not likely to happen. But what might happen is that the most natural of human gestures and movements may eventually become proprietary instruments of interface designers.
On another front, I would like to create a video contest where dancers were invited to create their own series of gestures and movements that were intended to control new PC and mobile interfaces. I think that dancers would come-up with some highly innovative approaches that had not previously been considered.
In news that I believe I'm the first to write about, Ohad Naharin is already working with Steve Jobs on a new interface that responds to full-body movements. Here's a clip from the research lab:
For the record, I'm just having fun with the idea that Ohad Naharin's Batsheva Dance Company is working with Apple. But I am serious about the premise that choreographers and dancers would have some very innovative and worthwhile approaches to controlling new interfaces. Choreographers are essentially rapid-prototoypers who can create the basics of new movement vocabularies in a few hours.
Apple Multi-Touch Patent Application
The following diagram shows a mockup of a Mac OS X gesture-control panel with multi-touch capabilities from a recent patent application [via MacRumors.com]:
"If Apple's patents are granted, the company could absolutely stop others from using similar technology," says Raj Abhyanker, a patent lawyer who used to write patent applications for Apple. "They'd also be in an especially good position to stop others from including certain features. Apple could stop [their use] not only on mobile devices but also desktops."
Samsung has recently patented a system of cell phone and mobile device control which responds to a users gestures. It doesn't do this on the display as how the iPhone currently does, but as recognized in the space around the handset courtesy of the handset's front-mounted camera. The pre-loaded software will recognize preset motions, translating them into on-screen control. Take for example, pointing at the display and moving the finger to control a mouse/cursor whilst rotating the wrist with the hand outstretched in order to flip an image or layer.
Will Muscle and Musical Memory Inform New Interface Designs?
Summary: Over many years of practice, dancers hone their muscle and rhythmic memories. They can watch extended dance sequences and often perform what they've just seen demonstrated after a single viewing...As new movement-based interfaces increasingly permeate the environments in which we live and work, will muscle and rhythmic memory become an important skill for successfully interacting with and responding to our digital surroundings? And what insights do dance-makers have about the challenges and possibilities of these human-computer interactions?
--------------------
I've always been impressed by the muscle and rhythmic memories of dancers. When I started taking jazz and modern dance classes, my body simply could not remember a series of moves that were demonstrated by a teacher. But after about two years, my body seemed to develop a magical capacity to watch and then perform movement phrases. I'm still very much a beginner dancer, but it does feel good to be able to replicate sequences that I've just seen performed for the first time moments before.
In this video interview with choreographer Mark Morris, you can watch company members rehearse for "Mozart Dances," and get a glimpse of the highly developed rhythmic and muscle memory abilities of professional dancers:
I also encourage you to watch this related Mark Morris video, "Taking Sides."
When TV show host Ellen DeGeneres visited the dancers of Alvin Ailey American Dance Theater at their studio on 55th Street and 9th avenue here in Manhattan (a great place to take dance classes), she has fun with the talented dancers knowing that they can instantly watch and then "playback" her dance moves:
Last week for the fourth episode of the new reality TV dance show "Step It Up & Dance, the contestants had to quickly learn the challenging rhythmic patterns taught by the cast members of Stomp Out Loud:
Also, watch this clip from last week's Step It Up & Dance rehearsal with the Stomp crew as the dancers learn to do a trashcan lid routine.
Digital Interfaces Based on Muscle and Rhythmic Memory
What's interesting about muscle memory (remembering and playing back physical movements over time) and rhythmic memory (remembering and playing back sound patterns over time) is that they are both time-based. In other words, no value is derived by freezing the action or sound at a single point in time.
If you take a look at my video guide to digital interfaces, you'll notice that the movement of your body does control these digital tools and software applications. But in most cases there is only a one-to-one relationship between a specific movement and a response to that movement.
In this demonstration of a gesture-controlled robot, one movement of the user controls either the specific direction or rotation of the robot:
The demonstration of the "Shadow Reaching" technology is a bit more interesting. If you jump to the 47-second mark in the below video, you'll see how Shadow Reaching can be used to move objects on a large-screen display. You'll see that it takes three actions of the user (grab, drag and release) to complete the process of moving an object across the screen. I don't know the specifics of the software, but I'd imagine that each action "grab," "drag" and "release" is viewed as a discrete event by the program. But the overall effort of changing the position of an object does have the feel of a movement routine.
Questions for interaction designers and dance-makers:
- As digital interfaces become more complex and pervasive, will they begin to accept input in the form of movement and rhythmic patterns? (There may be examples out there already that I'd like to learn about).
- What types of interfaces might rely on this type of time-based input?
- How specifically would we control interfaces that used movement and rhythmic patterns?
- What insights can be learned about controlling these interfaces from dancers and others whose artistic work is based on the mastery of muscle and rhythmic memory? And,
- What are examples of dance works that have relied upon movement patterns to control interactive environments that are incorporated into a performance?
