Controlling NAOs arms with color markers

Denis Lalanne
Matthias Schwaller
Kevin Mettenberger
Project status: 


The general idea is to control NAO [1] through the movements of the user.  In this Project I’m only going to control the arms of NAO through the movements/positions of the users ‘hands.


There are basically two ways to recognize the movements or positions of the hands:

  • Use a device like Wiimote [2], which can recognize movements and then send it do NAO

  • Use a camera to capture  the positions/movements of the hands

I decided to implement the second option because I don’t want, that the user needs to hold something in his hands.

To simplify the image processing part the user sits in front of a computer with a webcam (for stable lighting conditions) and has colored markers on his fingers (Matthias Schwaller already implemented the tracking of colors). After the users’ movements are being recognized, commands are sent to NAO over Wi-Fi. NAO then moves the arms according to the movement of the users’ hands

Because I’m not using a three dimensional camera there are two ways to recognize movements in the third dimension:

- Use different colors to control the second and the third dimension. For Example: If the user shows the blue color NAO moves his arms only in the x and y axes. If he shows the green color NAO moves his arms in the z axes.

- Use the size of the recognized color. If the user holds his hands close to the camera the marker appears bigger.

The project is divided in the following tasks:


  • T1.  Technology for movement tracking:  Research and testing of possible technologies to recognize the users’ movements.

  • T2.  Movement recognition: Implement the movement recognition. Choose solution for “third dimension Problem”.

  • T3.  Set up the NAO SDK and simulation software and learn how to work with it.

  • T4.  Animation of NAO: I need to be able to control the arms over Wi-Fi. One of the challenges is going to be, that NAO moves his arms smoothly.

  • T5.  Join the animation of NAO und movement recognition to the final program.

  • T6.  Test the prototype

  • T7.  Improve to the final solution.

  • T8.  Evaluation of the solution.

  • T9.  Write the bachelor thesis report.