The Future of the Controller


"New interaction styles (...) increase the realism of interface objects and allow users to interact even more directly with them-using actions that correspond to daily practices within the non-digital world."

Since multi-touch screen technology began flooding the market over 5 years ago, buttons, knobs, and dials have been disappearing from electronic interfaces across the board. Smartphones, E-readers, tablets, and surface computers have all been undergoing a slow formal convergence towards a common "glass-slab" model - a sandwich of performance touchscreen and cardboard-thin computers. Devices themselves are losing their specificity and identity as products, dissolving into a generic "touchfrastructure," as Frog's Fabio Sergio recently termed the phenomenon, that can access a universe of software and content through a single standard physical interface.

As the growing hype around Microsoft's body-tracking Kinect gaming technology suggests, "hands-free" interfaces may in fact overtake the current finger-on-screen paradigm over the next decade. Already, hackers are proving that the Kinect is a low cost, highly versatile tool for interacting with a broad range of virtual tasks and environments. Users can "mime" a steering wheel, a paintbrush, or a guitar with equal ease, suggesting the potential of body-tracking interfaces to move out of the gaming sphere and into all sorts of real-world applications.

There are of course downsides to reducing the physical clutter of computer interfaces. Without tactile feedback, touchscreen and projected keyboards systems have both proven difficult to operate (without looking down at one's fingers). Several studies with consumer electronics have also shown that lack of tactile feedback may make it harder to form the muscle memory necessary to learn physical tasks. Certainly frets and strings make it easier to play a physical guitar than a completely virtual one. As these examples illustrate, in gaming and productive tasks both, we clearly need haptic and tactile feedback systems paired with the graphical virtualities that we are so adept at creating.

One of the key questions for designers and product planners is how and why should physicality be preserved, especially in relation to behaviors and media that have become entirely digital? Furthermore, given the benefits of compatibility and broad utility that tend to drive physical interfaces into formal convergence, what business and technology strategies can be used to add value to diverse, physically-specific controllers and interfaces?

The Unique Physical Object

David Small's Illuminated Manuscript (2002) offers an important lesson in how the interplay between the physical and digital can be orchestrated to transmit a cultural meaning in addition to an innovative interaction model. Projected typography is virtually printed into the blank pages of a large hand-bound book. Sensors embedded in the pages tell the computer as the pages are turned, and sonar sensors allow visitors to run their hands over and to disrupt, combine and manipulate the text on each page. This piece communicates a poetic message by contrasting the sanctity and reverence of a unique handmade object with the open-ended textual manipulations possible of the digital content.

How can digital technologies enhance rather than compete with the phenomenological qualities of physical objects?

Bridging New Technologies and Physical Traditions

Serato Scratch Live is a vinyl emulation software that allows the user to physically manipulate the playback of digital audio files on a computer using the turntables as an interface, thus preserving the hands-on 'feel' of deejaying with vinyl while allowing playback of audio recordings not available in phonograph form. This allows DJs to scratch, beatmatch, and perform other turntablism that would be impossible with a conventional keyboard-and-mouse computer interface. While similar functionality is available with a touchscreen (e.g. Algoriddim's Djay App), a majority of DJs continue to stick with emulators (or real vinyl) due to the sense of familiarity, better ergonomics, and tactile feedback.

How can bridge products-part digital, part physical-be designed to introduce new functionalities and performance standards while retaining familiar emotional, stylistic, and ergonomic qualities?

Skeuomorph or Natural UI?

The Apple iPhone's Notes, Compass, and iBooks applications were all designed to mirror real-world objects (Post-it notes, old-fashioned compasses, and wooden bookshelves). Their designers did this in order to aid in comprehension-you can pick up an iPhone and know pretty much right away what these icons mean. The wood-veneered virtual bookshelf may even make users feel subconsciously more comfortable, even if it takes up a lot of screen space.

On the other hand, as some critics have rightly questioned, shouldn't Apple, a company obsessed with the perfect marriage of form and function, be more aggressively weaning us off these vestigial elements of pre-digital culture?

When does a "natural design" feature become a skeuomorph -an empty symbol that has persisted without necessity or function?

Realistic Interfaces

Arcade games have long celebrated the use of physical props and accessories to make play seem more real, exciting, and responsive. These include race car games with real steering wheels and gearshifts, dance games with pressure-sensitive mats, and even a firefighter game with a nozzle that replicates the weight, grip and controls of a real fire hose. Arcades, however, have been in decline since the early 1980's, superseded by home gaming systems that favor generic ergonomics and simple inputs that can control thousands of game titles instead of specific interfaces tailored to just one game. This was the dominant paradigm until the breakout success of Guitar Hero and Guitar Band in 2005, which proved that consumers were ready to pay for game-specific controllers given the right value proposition: a quantum leap in dexterity compared to generic controllers, real-world social value (as performance prop, collector's item, etc), and a compelling modification that distinguishes the controller from merely being a cheap copy of the "original" physical object.

How can gaming and productivity interfaces balance physical specificity with breadth of application?