Update — 29. Mai 2015


The main concept of the Shelf Project: finds and accesses virtual objects (which would be digital files). Physical objects recquire visual recognition; this task may take too much complexity for the scope of presenting a prototype of a virtual shelf.

In order to expand the project’s scope of being functional (eBooks for example), there would be music, video, image album. Thus, more interactive and entertaining.

Interesting features about the output modality: Kinect would plays songs whenever demanded by user; or it would declare „start“, „finished”; or play sounds during the movements.

To make evident what file within all files is „onstage“ at the moment, the object would be outstanding from the rest of them. For visual output, a light would focus on this popped-out file. The user’s selection of the object would consist of either pointing on it, or using a voice command to enunciate letters and numbers to identify the desired object’s location.

Topics to be done:

  • Graphical representations (creating visually the virtual shelf): Starting with a static shelf. How to implement? Using Processing. How the objects would be portraied? The book would be represented by its cover or by its spine with a barcode? Since Kinect Is not ideal for recognizing book cover, we will not use recognition by colour; In this way, the files would be identified by its name and displayed with its cover.
  • Positioning files in columns and lines: we start with a two-dimensional location.system; Lines correspond to how up and down shelf and column correspond to how much left or how much right is the object found
  • Identifying gestures and voice commands
  • Firstly the shelf would be moved by arrow keys (can be used for initial features and then for final application): it works for people who have a laptop and a beamer and also for ones who own only a laptop;
  • Someone may have provided voice recognition but it just works online (api); doesn’t work well offline.


1) Access shelf by letter-number or by file’s name; commands like: go right, go left, go down, up, down, open file X

important: coordinates of hands state movements of gestures

2) Once your hands are down, the system becomes steady

The idea to use a mobile with touch screen to cooperate brings the disadvantage of needing a zusätsliches device