Physics-based picking

I've been trying to implement physics-based picking based on the Bullet demos, DemoApplication.cpp in particular.

First, I've added a pointer to the corresponding SIO2object for each btRigidBody:

Code:

...
      _SIO2physic->_btDiscreteDynamicsWorld->addRigidBody( _btRigidBody );
      
      _SIO2object->_btRigidBody = _btRigidBody;
      _btRigidBody->setUserPointer( _SIO2object ); // Enables getting a SIO2object* from a btRigidBody
   }
...
}

Then, I've implemented a SIO2 physics function which corresponds basically to what is implemented in Bullet's DemoApplication::mouseFunc and DemoApplication::getRayTo:

Code:

inline btVector3 getCameraPosition( SIO2camera *_SIO2camera )
{
   return btVector3( _SIO2camera->pos->x, _SIO2camera->pos->y, _SIO2camera->pos->z );
}

inline btVector3 getCameraTargetPosition( SIO2camera *_SIO2camera )
{
   return btVector3( _SIO2camera->tar->x, _SIO2camera->tar->y, _SIO2camera->tar->z );
}

inline btVector3 getCameraUpVector( SIO2camera *_SIO2camera ) {
   return btVector3( _SIO2camera->up->x, _SIO2camera->up->y, _SIO2camera->up->z );
}

btVector3 getRayTo( SIO2camera *_SIO2camera,
               float x,
               float y )
{
   // Several attributes which were hardcoded in the DemoApplication are available from SIO2,
   // e.g. the FoV is calculated there but we have it already available in the camera structure.

   float screenWidth = sio2->_SIO2window->size->x;
   float screenHeight = sio2->_SIO2window->size->y;
   float fov = _SIO2camera->fov * SIO2_DEG_TO_RAD;

   btVector3 rayFrom = getCameraPosition( _SIO2camera );
   btVector3 rayForward = ( getCameraTargetPosition( _SIO2camera ) - rayFrom );
   rayForward.normalize();
   float farPlane = _SIO2camera->cend;
   rayForward *= farPlane;
   
   btVector3 vertical = getCameraUpVector( _SIO2camera );
   
   btVector3 hor;
   hor = rayForward.cross( vertical );
   hor.normalize();
   
   float tanFov = tanf( 0.5f * fov );
   btScalar aspect = screenHeight / (btScalar) screenWidth;
   
   hor *= 2.f * farPlane * tanFov;
   vertical *= 2.f * farPlane * tanFov;
   
   if( aspect < 1 ) {
      hor /= aspect;
   } else {
      vertical *= aspect;
   }
   
   btVector3 rayToCenter = rayFrom + rayForward;
   btVector3 dHor = hor * 1.f / screenWidth;
   btVector3 dVert = vertical * 1.f / screenHeight;
   
   btVector3 rayTo = rayToCenter - 0.5f * hor + 0.5f * vertical;
   rayTo += x * dHor;
   rayTo -= y * dVert;
   
   return rayTo;
}

bool sio2PhysicPick( SIO2physic *_SIO2physic,
                  SIO2camera *_SIO2camera,
                  float x,
                  float y,
                  SIO2object** object,
                  float* px,
                  float* py,
                  float* pz )
{
   btVector3 rayTo = getRayTo( _SIO2camera, x, y );
   btVector3 camPos = getCameraPosition( _SIO2camera );
   btCollisionWorld::ClosestRayResultCallback rayCallback( camPos, rayTo );
   _SIO2physic->_btDiscreteDynamicsWorld->rayTest( camPos, rayTo, rayCallback );
   if( rayCallback.hasHit() )
   {
      btRigidBody* body = btRigidBody::upcast( rayCallback.m_collisionObject );
      if( body )
      {
         // For debugging purposes: check which object was hit.
         SIO2object *test = (SIO2object *) body->getUserPointer();
         if( !body->isStaticOrKinematicObject() ) {
            *object = (SIO2object*) body->getUserPointer();
            btVector3 pos = rayCallback.m_hitPointWorld;
            *px = pos.x();
            *py = pos.y();
            *pz = pos.z();
            return true;
         }
      }
   }
   return false;
}

Now I have a test scene with a huge cube as a container where the front and top faces are removed, and a little dynamic cube in the container which falls down once the application starts. Whenever I try to pick this little cube, I get the big one returned from this method.

Is someone willing to try this method out and help me out with it? I can't see any differences from the original Bullet code which could lead to problems.

Thanks,
Matt

Does this cube is an instance? (ex: Cube, Cube.001) If yes just get the handle of the parent instead...

SIO2object *tmp = _SIO2object->_SIO2object ? ( SIO2object * )_SIO2object->_SIO2object : _SIO2object;

By the way picking with bullet is relatively slow compare to the method used by SIO2 (like in tutorial06)... Depending on what kind of scene you have and the type of performance you are looking for maybe you should give a try to the embedded SIO2 method...

sio2interactive wrote:Does this cube is an instance?

No, it shouldn't be. I'll double-check this.

[quote?"sio2interactive"]By the way picking with bullet is relatively slow compare to the method used by SIO2 (like in tutorial06)... Depending on what kind of scene you have and the type of performance you are looking for maybe you should give a try to the embedded SIO2 method...[/quote]
It's okay if you just need to know which object was clicked. But I also need the location in 3D world space where a ray from the users mouse position through the camera planes hits an object.

Best,
Matt

sio2Unproject, will return you the XYZ position in worldspace from any coordinate on the screen (in other words 2D to 3D)

sio2interactive wrote:sio2Unproject, will return you the XYZ position in worldspace from any coordinate on the screen (in other words 2D to 3D)

Hm, mind to provide an example?

From my understanding, I would have to provide a z-buffer-value as the "winz" parameter. Normally, in OpenGL, you'd retrieve that value with glReadPixels and GL_DEPTH_COMPONENT as the type, but OpenGL|ES doesn't support that. So what's your solution? How do you use sio2Unproject?

Moreover, once this is solved, how can I quickly get the modelview/projection matrix without calling glGetDoublev... for all three? I guess SIO2 already has them available somewhere? Does the method already take into account that the OGL viewport has (0,0) at the bottom left or do I have to take care of this myself?

Best,
Matt

Same way you are using gluUnproject, however it is pretty slow due to the calculation, on a second thought maybe bullet method is faster cuz they already have a tree built...

I now mixed "best of both worlds" and use sio2UnProject for generating a test ray through far and near clipping plane, but still go for the Bullet stuff to find the hit object / point.

Code:

inline btVector3 getCameraPosition( SIO2camera *_SIO2camera )
{
   return btVector3( _SIO2camera->pos->x, _SIO2camera->pos->y, _SIO2camera->pos->z );
}

inline btVector3 getCameraTargetPosition( SIO2camera *_SIO2camera )
{
   return btVector3( _SIO2camera->tar->x, _SIO2camera->tar->y, _SIO2camera->tar->z );
}

inline btVector3 getCameraUpVector( SIO2camera *_SIO2camera ) {
   return btVector3( _SIO2camera->up->x, _SIO2camera->up->y, _SIO2camera->up->z );
}

btVector3 getRayTo( SIO2camera *_SIO2camera,
               float x,
               float y )
{
   GLfloat worldX1, worldY1, worldZ1;
   GLfloat worldX2, worldY2, worldZ2;
   GLfloat model[ 16 ];
   glGetFloatv( GL_MODELVIEW_MATRIX, model );
   GLfloat proj[ 16 ];
   glGetFloatv( GL_PROJECTION_MATRIX, proj );
   GLint viewport[ 4 ];
   glGetIntegerv( GL_VIEWPORT, viewport );
   
   y = viewport[ 3 ] - y;
   
   sio2UnProject( x, y, 0,
               model, proj, viewport,
               &worldX1, &worldY1, &worldZ1 );
   sio2UnProject( x, y, 1,
               model, proj, viewport,
               &worldX2, &worldY2, &worldZ2 );
   
   btVector3 v = btVector3( worldX2 - worldX1,
                        worldY2 - worldY1,
                        worldZ2 - worldZ1 );
   return getCameraPosition( _SIO2camera ) + v;
}

bool sio2PhysicPick( SIO2physic *_SIO2physic,
                  SIO2camera *_SIO2camera,
                  float x,
                  float y,
                  SIO2object** object,
                  float* px,
                  float* py,
                  float* pz )
{
   btVector3 rayTo = getRayTo( _SIO2camera, x, y );
   btVector3 camPos = getCameraPosition( _SIO2camera );
   btCollisionWorld::ClosestRayResultCallback rayCallback( camPos, rayTo );
   _SIO2physic->_btDiscreteDynamicsWorld->rayTest( camPos, rayTo, rayCallback );
   if( rayCallback.hasHit() )
   {
      btRigidBody* body = btRigidBody::upcast( rayCallback.m_collisionObject );
      if( body )
      {
         *object = (SIO2object*) body->getUserPointer();
         btVector3 pos = rayCallback.m_hitPointWorld;
         *px = pos.x();
         *py = pos.y();
         *pz = pos.z();
         return true;
      }
   }
   return false;
}

Works for me, maybe it's useful for others as well. Now I need to have a look how I could get more information about the triangle which was hit and its material...

Matt

Something must have changed from 1.1 to 1.2... I changed a test project to work with the new version. In 1.1, I could pick an object, move it around, and drop it. It was correctly rendered in all of these phases. In 1.2, after picking an object, it remains at the same place visually, but interacts with other objects at it's new position physically. Once I drop it, its rendered at correct position.

The position change takes place like in all the Bullet demos by changing a pivot point position on a btPoint2PointConstraint.

sio2interactive, do you have any ideas where I could look?

Best,
Matt

Hummmm try to comment _SIO2object->_btRigidBody->getActivationState( ) == ACTIVE_TAG is sio2_object.cc since when you pick your object is not necessarily activated... Lemme know if it work

sio2interactive wrote:Hummmm try to comment _SIO2object->_btRigidBody->getActivationState( ) == ACTIVE_TAG is sio2_object.cc

Yes, that's it. But I'm actually disabling deactivation when I pick the object...

Code:

selectedObject->_btRigidBody->setActivationState( DISABLE_DEACTIVATION );

Strange. Any ideas?

Matt

Heuuuuuu not strange, the state is DISABLE_DEACTIVATION and not == to ACTIVE_TAG

sio2interactive wrote:Heuuuuuu not strange, the state is DISABLE_DEACTIVATION and not == to ACTIVE_TAG

Yeah, but doesn't DISABLE_ACTIVATION imply ACTIVE_TAG? Shouldn't your if include DISABLE_ACTIVATION as a possible condition? I'm not a Bullet expert so I might be wrong here but from my understanding DISABLE_ACTIVATION means the rigid body is active and remains in that state, while ACTIVE_TAG means it's active but could get deactivated by Bullet.

Matt

Yeah, but doesn't DISABLE_ACTIVATION imply ACTIVE_TAG? Shouldn't your if include DISABLE_ACTIVATION as a possible condition?

Yeah I think that'll do the trick...

Hello,

Can anyone send me an example to pick/move an object? I can select an object but I do not have an idea how I can move the selected object. The problem is to convert between the touch coordinates to the 3d word coordinates.

Thanks a lot
Ludi

ludi wrote:Can anyone send me an example to pick/move an object? I can select an object but I do not have an idea how I can move the selected object. The problem is to convert between the touch coordinates to the 3d word coordinates.

All relevant parts are provided right here in this thread. See my earlier post where I showed how I created a ray through near/far-clipping plane and then use Bullet to pick an object. One important thing if you want to use Bullet for picking is that you should set a btRigidBody's userPointer to the SIOobject* so that you know which body belongs to which SIO2 object. I have changed SIO2 in sio2PhysicsAdd...

Matt

I am quite new in Object-C and iPhone programming. It was not possible to create a working application with your hints. I will try it again.
It would be great if you can provide me an example.

Thanks
Ludi

matt wrote:

ludi wrote:Can anyone send me an example to pick/move an object? I can select an object but I do not have an idea how I can move the selected object. The problem is to convert between the touch coordinates to the 3d word coordinates.

All relevant parts are provided right here in this thread. See my earlier post where I showed how I created a ray through near/far-clipping plane and then use Bullet to pick an object. One important thing if you want to use Bullet for picking is that you should set a btRigidBody's userPointer to the SIOobject* so that you know which body belongs to which SIO2 object. I have changed SIO2 in sio2PhysicsAdd...

Matt

Hi Matt,

I started with tutorial 6 and added your code and comments to the template.mm file.
Therefore I added your methods:
-getCameraPosition
-getCameraTargetPosition
-getCameraUpVector
-getRayTo
-sio2PhysicPick

Now I tried to add code to select the right object at the function templateScreenTap and move the object in templateScreenTouchMove.
I used the method getRayTo to calculate the touch coordinates to the coordinates in the 3D World and afterwards the sio2PhysicPick to move the object.
The program still does not work.

Can you give me an additional hint or a small example? That would be nice.

Thanks a lot
Ludi

Could you provide me with the code you already have so that I can see some more details on how you used the methods? Unfortunately, I cannot provide you with an isolated example myself because I've incorporated that into a larger project.

For example, did you remember to set the UserPointer in sio2PhysicAdd as shown in this posting? And do you also have the relevant properties set in Blender?

Best,
Matt

thanks rom and matt, it is a very very good post.

After updating to 1.4, this picking method works after i call sio2getModel/Proj during render. The other problem is the actual picking precision... ? I have a horizontal plane, and when i pick on it, the point picked seems skewed... as if the frustrum was inverted or something. Its precise when i click in the middle of screen, but seems to have an ever growing offset when i click further from the center of the screen... ?

Any help appreciated, i use same code as on this post.

Thanks !

Check tutorial062, 1.4 have 3D pick and drag based on physic...

Yup, forgot to reply, it was due to cend of camera. Wasn't far enough apparently and created that offset.

Thanks

Mostly this makes sense to me, the only question I have is where do you get the values for the near and far clipping plane? In order for it to be a constant value it would have to be relative to the camera since the camera moves and this picking method still works. Are 1 and 0 just arbitrary values or is there a reason you picked them?

Also, I still haven't seen an explanation that I can understand that really explains the z coordinate for SIO2UnProject. The way I am thinking about it is the x, y is the location you click on the screen and the z would be the distance from the camera to the point you're trying to get the coordinates for in worldspace. So if the camera were looking straight down the y axis you would do camera.y - target.y to get the depth. So if my objects were at y=2.0 and the camera was located at y=6.0 the z coordinate for unproject would be 4.0. Of course this doesn't work if you're not looking directly down an axis, but for simplicity am I thinking the right way?

*Edit* As I've had more time to research I figured out that the z value is a float ranging 0.0 to 1.0 with 0.0 representing the near clipping plane and 1.0 representing the far clipping plane. So, .5 would be a depth of halfway between the near and far clipping plane, .33 would be 1/3 (closest to the near clipping plane) .66 would be a depth 2/3 of the distance between the near and far clipping plane and so on... Understanding that fundamental was really key in making sense of both of my questions...