Weekly Progress — 30. Juni 2015

Weekly Progress

图片1Our Group work after the intermediate presentation:

What we have done so far:
– shelf will be presented to the user
– items will show dynamically depending on the files stored in the directory on the disc
– images will be shown directly as icon in the shelf
– audio files need an image file which is named like the audio file appended with a ‚-cover‘ (e.g. audio_file-cover.mp3)
– if a media file is accessed a pop up opens
– it will close after the file has finished (except images)
– shelf moves right after the hand movements of the user
– the shelf can be used with keyboard input already

What we need to do:
– Smooth the movements of the shelf (use average values between to recognized points)
– Make the items accessible via Kinect
– The image of a movie is not shown on the display yet. (But it already runs in background (sound is hearable))
– Protect for recognizing two people at the same time
– Introduce a cursor which is related to the other hand of the user (use this instead of speech commands)
– Add button overlays which can be used by hovering them
– Adding response texts which will be synthesized by the system

Things that may will drop at the end:
– Speech recognition (caused by running problems on Mac)
– Zoom the shelf (a factor is influencing a lot of values used for calculation und visualization)

The Result of Brainstorming — 23. Mai 2015

The Result of Brainstorming

Following you will see our ideas we had for this years Multimodal Interaction course


Smart Home Simulation (SHS)

The idea is to have a three-dimensional view of a house and the Kinect camera focused on a user. The user has the chance to rotate the house and look at it from each angle. The house itself contains of different rooms like the kitchen or the living room. Each of these rooms contains a user control which can be activated by hovering over them when standing in front of the camera. If the control has been activated the user have different actions available for this room (e.g. dimming the light, change light colors, close or open a curtain). I don’t know if some has experiences with controlling hardware. But if so I thinking about steering some lights for presentation. So if the light has been changed in the simulation the real lights will also be changed. But that is just to put the cherry on the cake. 😉

Another cherry: Measure the height of a person to give her kind of access rights for some functions (e.g. heating or electricity so they don’t be able to play PlayStation :D)

Opinion of tutors: TO HARD, DO SOME ELSE


Kind of ikea kitchen simulation:

You can place your furniture in a room and have a 3D look of how it looks.

Opinion of tutors: Also to hard to implement


Idea of tutors: Simulation of a single room and control the light

People stay in a room and he/she will be filmed by the Kinect. When he/she points on one of the lights in the room, and says for example “TURN ON”, then the light in the physical room will be turned on.


Virtual Artist:

Draw something in front of the Kinect. The camera will recognize it and process the result to be shown on a physical wall by a beamer. The used tool or color can be chosen by voice.



The idea is a alarm clock to wake you after a nap or at the morning by flashing some lights or turn on the radio. The fact that a person is sleeping will be recognized by the Kinect camera. To stop the alarm the user has to answer a question or to do a task. The alarm only stops if the user has answered.



Recognizing whether a baby is leaving a predefined area (e.g. the bed). The system will raise the alarm. Also if the baby is crying can be recognized.


Recognizing the presence of  a second person in another environment by movement or audio recognition.




Height-variable shelf:

  1. Recognize the movement of the person in front of the Kinect and move the shelf if the person is trying to reach the shelf
  2. Move the shelf up again if the user says: “Thank you” or some similar
  3. Two options for the course:
    • a physical shelf on the table
      • disadvantages:
        • have to build for real
        • needs wood or Lego blocks for the shelf
        • needs motors to move the shelf
        • needs a controller for the motors
      • advantages:
        • PRETTY COOL!!! 😀
    • a virtual shelf presented on the wall by a beamer
      • disadvantages:
        • a virtual environment have to be build
        • hard to visualize a shelf which looks real
      • advantages:
        • a lot easier to realize than the first option
  4. Modalities:
    • Input:
      1. Gesture (Kinect camera)
      2. Voice (Also Kinect)
    • Output:
      1. Visual (Movement of the shelf)
      2. Audio (Response for the actions of the shelf)