[Video] Alternative Kinect Control (using depth images)

[omwdunkley]

I found the problem!

I missed a step in installing the drivers for the kinect. You should blacklist the default kernel module if you are using Ubuntu 13.04 using:

Code: Select all

sudo modprobe -r gspca_kinect
echo 'blacklist gspca_kinect' | sudo tee -a /etc/modprobe.d/blacklist.conf

Now I can access the parameters from the Reconfigure gui. But, of course we are faced with a new error. Under TF->Status:Warn->cf0 we get "No transform from [cf0] to frame [world]" and theres still no dots nor picture

Patrik

Hm. Lets try to acheive the following in this order:

• view a rgb image from the kinect
• view a depth image from the kinect
• view a point cloud in rviz
• set up all the frames and detect the flie

That error you are seeing is fine for now. Its the last last step of the whole process and means the kinect did not identify the flie. If you hold the flie infront of the kinect while running the detector from my GUI, it still might not detect the flie because of the size (flie+arm > size of flie + tolerance).

Viewing images from the kinect
For now, start the kinect driver, check the depth_registered check box, and run

Code: Select all

rqt_image_view

That should give you a drop down of all the image topics, try clicking through some of them and see if you can visualise anything. If you cannot, it means the kinect is not working properly. Keep me posted

Here is a tutorial with screenshots, can you do all of this stuff or does it error out somewhere along the way?

[Patrik]

Hi Oliver,

We haven't made any progress. Using freenect drivers and rqt_image_view we don't get any image from any of the topics. Following the guide using the openni driver we don't get any PointCoud nor any rgb picture using "rosrun image_view image_view image:=/camera/rgb/image_color". Only thing positive is that we hade tested the kinect in windows and got it working there at least ^^

So whats next?

Patrik

[omwdunkley]

Hi,

So if you cannot complete the tutorial to get your kinect working, then the problem is with your hardware/system setup. Im not sure how much I can help in this case. I know the kinect can bitch with usb3, so try your other (hopefully some are) usb2 ports.

The tracking algorithm just needs depth images, it doesnt matter where they come from. So if you have different hardware (Xtion PRO?) or want to try the freenect drivers and can get something working, we can move on from there.

If you have no luck, I point you to Ros Answers. Maybe some one has been having similar problems. If not, thats the place to ask for help Lots of people are using the kinect so I'm sure whatever difficulties you are having are solvable.

Good luck

[Patrik]

Thanks for the help!

I'll have a look at the ROS forum and post here if I find a solution

Patrik

[Patrik]

We've solved the problem (after many agonizing hours). We were running on a virtual machine using virtualbox wich seemed to mess up something with the usb drivers for the cameras. Switching to VMware player solved the problem (somewhat). We did most of the steps in the openni tutorial and got pointclouds from both the kinect and xtion pro. But then it seems the old problems reoccurred and nothing works anymore. So we'll move to another computer that directly runs linux on monday. The next step was setting up the the frames and detecting the flie?

[omwdunkley]

Ah, VMs were in the play

Yup, so next up is makeing sure the transform tree is set up as expected, running the pid controller node, running the custom cfclient, detecting the background (uses depth registered depth images, dont use the pointcloud at all) and then detecting the flie

Lemme know how it goes. Good luck!

[fetrit]

Im looking for a way to implement an Kalmanfilter to your code omwdunkley. Is it possible to just add a method in your trackManager.py or do you have to create a node to do this?

The Kalman filter should have the x,y,z coordinates as input and send the processed x,y,z coorinates as output where then the rest of your code will be run.

I'm trying to understand your code in the trackManager.py, is it in the method "def projectRay(self, u,v,d):" where the incoming x,y,z coordinates is registered from the camera? Would it then be possible to send these coordinates to a method "def kalmanFilter(x,y,z):"? I also need to have some global variabels to use for the kalman filter.

something like this:

Code: Select all

 def projectRay(self, u,v,d):
        (x,y,z) = self.cameraModel.projectPixelTo3dRay((u,v))
        kalmanFilter(x,y,z)  #the kalman filter will return processed x,y,z coordinates
        return [x*d,y*d,z*d]

Or is this wrong? (I've never programmed in python before, only in java)

[Patrik]

At long last, we have gotten somewhere. Here's a picture of our ROS tree:

I think it looks good, but really I have no idea

This is the picture we get in Rviz, whats wrong with the picture/transform?


We watched your youtube video but couldn't figure out exactly how you got the program to track the flie. Also i couldn't figure out some of the settings you wanted us to set in the readme e.g. the set goal option in joysettings.

Patrik

[Veerachart]

Hi Oliver,

Thanks for this good work. I have been trying on modifying the kinect control by BitCraze for a while, but didn't make a lot of progress. This one is just amazing.

Now I am trying to follow your work. I think I have got everything installed. May I ask if it is possible to use the tracking without any joystick (from the video you just use it to lift the flie from the ground and after that it's the work of Kinect.) I tried following the instruction and things seemed to work, except the error that there is no joystick. Then I tried start the tracking, set the background, then place the flie in the frame, and then connect GUI to the flie. (This is to avoid the flie starting while I am placing it in the frame.) The tracking worked and the flie was detected (green dot with numbers in the Camera frame with frame /cf_xyz, and 3-axis frame on the black screen in rviz.) However, there was no /goal frame as seen in your video and the flie didn't fly, also the /cf_xyz didn't rotate when I rotate the flie. So I wonder if it needs to be started up first and then controlled by the Kinect.

Thank you,
Veerachart

[omwdunkley]

Hi guys, sorry for the slow reply.

Im looking for a way to implement an Kalmanfilter to your code omwdunkley. Is it possible to just add a method in your trackManager.py or do you have to create a node to do this?

Good question, I guess there are multiple points of entry. I am not sure which is best
A separate node might cause a bit of overhead at first, but then you also learn how to use ROS and Python a bit, which is a plus. Also, it makes a clear separation between input/output etc. Might be a little overkill and inflexible though.

I guess the easiest would be to add it directly to the GUI program (that way you could also add a tab with the kalman filter settings, etc). You could create a signal which the trackManager emits and you listen to, process, and then send out the /cf_xyz transform. This would keep your code separated out too. The idea is the track manager just blindly tracks what it is given.

I'm trying to understand your code in the trackManager.py, is it in the method "def projectRay(self, u,v,d):" where the incoming x,y,z coordinates is registered from the camera?

If I were you, I would hook in here.
This if statement checks if there is a detection (after the filtering). There might be multiple, which might be important for you kalman filter, as you might need to decide which detection you want to use. They are sorted depending on the settings used.
In that if block, I take the first sorted detection, extract the x,y,z position, and send it as a TF transform. You will want to process this X,Y,Z before sending it out.

Would it then be possible to send these coordinates to a method "def kalmanFilter(x,y,z):"? I also need to have some global variabels to use for the kalman filter.

If you want "global variables" I would suggest you make a kalman filter class that stores all the variables you need within it and add the function you mention to the class as well.

This is the picture we get in Rviz, whats wrong with the picture/transform?

So, Im not really sure what is going on there, but: RVIZ can super-impose / overlay the 3d scene onto the 2d camera view. So what you might be seeing is some TF frame that is in the optical center of the camera. This isnt a bug or anything. You can just disable the visualisation of that particular TF frame in RVIZ. Look at the camera tab in rviz, you can also set the overlay transparencies and which parts of the scene should be overlayed.

We watched your youtube video but couldn't figure out exactly how you got the program to track the flie.

So, what happens is the following:
There are two main components, the tracker and the controller.
The tracker is part of the GUI, listens to the depth images, and sends out a TF camera -> flie_position of the estiamted flie position in the camera frame.
The controller is integrated into the joystick controller and run in the python script (joy_driver_pid.py). It takes care of the user controlling the flie directly, or controlling the flie via the position estimate. To activate the latter, you need to hold and press (down all the way) the hover button. Then, if the controller can look up the transform world-> camera -> flie_position -> flie_rotation (given by imu) and world -> goal it controls the flie with a PID controller (error = flieEstimate (position and rotation) - goal).

Also i couldn't figure out some of the settings you wanted us to set in the readme e.g. the set goal option in joysettings.

So, the node joy_driver_pid.py advertises some parameters via the dynamic reconfigure interface. To access it, just run the node and access the settings from the dynamic reconfigure gui dropdown (in your screenshot, the very top right drop down). You should find the settings there. The rest of the settings are in the actual GUI. I still need to get around to merging the two programs...one day...

Hope that helps!

Thanks for this good work. I have been trying on modifying the kinect control by BitCraze for a while, but didn't make a lot of progress. This one is just amazing.

Thanks for your kind words!

Now I am trying to follow your work. I think I have got everything installed. May I ask if it is possible to use the tracking without any joystick (from the video you just use it to lift the flie from the ground and after that it's the work of Kinect.)

In theory yes, right now as it stands, no. So right now the joystick controller is responsible for two things:
- activating the hover mode. However you could probably remove this requirement, by always having the hover mode activated or adding a button to the GUI.
- invoking a signal at 100hz. So the pid controller listens to the joystick commands, which come in at 100hz. If the automatic control is activated, it then computes a command for the crazyflie using the PID controller, and sends this to the flie (ie at 100 hz, as a direct response to the joystick command). If automatic control is not activated, it directly forwards the commands to the flie.

I guess the easiest (even if rather silly) solution would be to quickly make a node that blindly sends "fake" joystick messages at 100hz on the right channel. While you do that, you could even set the "automatic mode" button to set (1.0 I think)). Alternatively you could modify the joy_driver_pid.py file to do that for you.

However, there was no /goal frame as seen in your video and the flie didn't fly, also the /cf_xyz didn't rotate when I rotate the flie. So I wonder if it needs to be started up first and then controlled by the Kinect.

No goal frame might be because the "automatic contorl" button was never pressed, so no goal was ever sent out.
"/cf_xyz" is only the flie position (not rotation) as estimated by the tracker. The tracker cannot estimate the rotation. The rotation estimate comes from the flie itself (so you must be connected to it and receiving roll, pitch and yaw). The GUI program sends out a /cf_xyz->/cf0 transform with just the rotation. The static transform publisher in the guide then links /cf0->/cf_gt (ground truth), which is compared against /goal in the PID controller.
I know the names are not the best, the whole thing is still in dev (even though I have not touched the code in a long time...other priorities for now).


I hope this clears things up! Thanks for taking the time to try it all out, I appreciate the feedback.
Good luck!