I am struggling with implementing collision detection for 2 shapes. The project where it is needed is constructive solid geometry. The catch is that I need a "precise" detection between 2 shapes. One is always convex and second is concave after modifying it.

I already tried to use collision detection based on bsp tree generated from a mesh (bsp tree is just a representation of a mesh in that form). In this approach I just checked whether at least one point behind a planes. If yes, then they are colliding

With second approach I tried to use polygon vs bsp tree. In this approach I checked if polygon behind or straddling. If at least one is doing so, then 2 meshes are colliding

The issue is that one of the shapes is modifiable(mesh A) and on collision I modify this mesh substruct with other mesh (mesh B). And after I update mesh with modifying mesh A with mesh B, the collison detection doesn't work anymore.

So, I am wondering, if there is other way to implement collision between mesh A and mesh B or I just introduced unnecessary complexity in my collision detection test and introduced a bug? Is there any algorithms that work on polygon soup A and polygon soup B? Or maybe there could be algorithm to calculate hull of meshes so that I could use gjk algorithm or sat? Note that mesh A and mesh B only consists of polygons and I don't think there is possibility of using bounding boxes for this. Sorry for some really bad English here. Thanks