The Result of Brainstorming — 23. Mai 2015

The Result of Brainstorming

Following you will see our ideas we had for this years Multimodal Interaction course


Smart Home Simulation (SHS)

The idea is to have a three-dimensional view of a house and the Kinect camera focused on a user. The user has the chance to rotate the house and look at it from each angle. The house itself contains of different rooms like the kitchen or the living room. Each of these rooms contains a user control which can be activated by hovering over them when standing in front of the camera. If the control has been activated the user have different actions available for this room (e.g. dimming the light, change light colors, close or open a curtain). I don’t know if some has experiences with controlling hardware. But if so I thinking about steering some lights for presentation. So if the light has been changed in the simulation the real lights will also be changed. But that is just to put the cherry on the cake. 😉

Another cherry: Measure the height of a person to give her kind of access rights for some functions (e.g. heating or electricity so they don’t be able to play PlayStation :D)

Opinion of tutors: TO HARD, DO SOME ELSE


Kind of ikea kitchen simulation:

You can place your furniture in a room and have a 3D look of how it looks.

Opinion of tutors: Also to hard to implement


Idea of tutors: Simulation of a single room and control the light

People stay in a room and he/she will be filmed by the Kinect. When he/she points on one of the lights in the room, and says for example “TURN ON”, then the light in the physical room will be turned on.


Virtual Artist:

Draw something in front of the Kinect. The camera will recognize it and process the result to be shown on a physical wall by a beamer. The used tool or color can be chosen by voice.



The idea is a alarm clock to wake you after a nap or at the morning by flashing some lights or turn on the radio. The fact that a person is sleeping will be recognized by the Kinect camera. To stop the alarm the user has to answer a question or to do a task. The alarm only stops if the user has answered.



Recognizing whether a baby is leaving a predefined area (e.g. the bed). The system will raise the alarm. Also if the baby is crying can be recognized.


Recognizing the presence of  a second person in another environment by movement or audio recognition.




Height-variable shelf:

  1. Recognize the movement of the person in front of the Kinect and move the shelf if the person is trying to reach the shelf
  2. Move the shelf up again if the user says: “Thank you” or some similar
  3. Two options for the course:
    • a physical shelf on the table
      • disadvantages:
        • have to build for real
        • needs wood or Lego blocks for the shelf
        • needs motors to move the shelf
        • needs a controller for the motors
      • advantages:
        • PRETTY COOL!!! 😀
    • a virtual shelf presented on the wall by a beamer
      • disadvantages:
        • a virtual environment have to be build
        • hard to visualize a shelf which looks real
      • advantages:
        • a lot easier to realize than the first option
  4. Modalities:
    • Input:
      1. Gesture (Kinect camera)
      2. Voice (Also Kinect)
    • Output:
      1. Visual (Movement of the shelf)
      2. Audio (Response for the actions of the shelf)
Optical illusion: ‚Rotating Snakes‘ — 6. Mai 2015

Optical illusion: ‚Rotating Snakes‘

Optical illusion from

The optical illusion above is a so called ‚Rotating Snakes‘ illusion and belongs to the class of ‘Peripheral Drift’ illusions. These kind of illusions is characterized in the way that motion signals can only be recognized in the periphery of the focused visual field. Responsible for this motion signals in this case is the order of the used colors. The effect would also show up in a gray-scaled pattern. However, critical for the motional effect is the luminance and the contrast of the used colors in the pattern.

The whole image consists of concentric ordered blocks which build up circular rings. Each of these rings consists of repeated constructs of four colored elements which have the following order

Black -> Blue -> White -> Yellow

The observed rotational movements in this statically image can be explained by the mechanisms of the human eye. First of all the motional effect is essentially influenced by the so called saccades. These are fast parallel movements of the eye which are used to directing the eye towards an object or scanning the environment. The rate is two to three movement per second. The influence can be observed by an subject by fixating a specific point and afterwards look a bit around on the image. These tiny movements changing the image displayed on the eye’s retina. This on the other hand stimulates the neurons in the single layers of the retina in frequently different ways.

To get a better understanding of what is happening while this illusory effect we have to dive into the structure of the retina. As explained at the beginning the effect relies on the luminance and also the contrast of the blocks in the concentric circles.

Every time the eye is looking onto a specific point on an image or a landscape, a reflection of the scene is projected onto the retina. This projected image will be processed by the neurons of the retina layers (rods, cones, bipolar cells, etc.). If the scene changes, e.g. by a happened saccade, the informations on the retina will be changed and the affected neurons will send rapidly the new informations. After this is happened the signaling slows down until the next change is occurred. This decrease of signaling is called ‚adaption‘ and results in a more efficient way of processing because unnecessary informations won’t be send by the neurons highly frequent.

The interesting part for this explanation is the difference in how fast the various contrasts ‚adapt‘. Higher contrast also results in higher neural activity whereas moderate contrast just results in moderate activity. The rational changes of neural signaling will be detected by motion mechanisms which causing the illusion of a movement in the peripheral view.