hello, I am trying to implement a realistic simulation of a roulette wheel. it is not clear for me what is the proper way to simulate the initial status of the ball, when it spins against the edge of the wheel until it loss energy and start falling towards the centre.

I modelled the conic table as a height map, as I assume that would provide the smoother surface. but I see anyway there is rough squared corners everywhere, so really I don't have a smooth inner wall to slide against.

I wonder if I should ignore the wall and simulate the sliding by code. i.e: apply force (or impulse?) each frame to keep the ball at a fixed radius and somehow force it to follow a desired angular speed..  later, when I want to execute the falling behaviour, just stop applying that forces and let the simulator and the gravity do their work.. makes sense?

any advice is highly appreciated!

Personally I wouldn't recommend doing that. Even if you use a high-end physics system there will be tiny variations in the physics system or model or other properties that won't give a perfect distribution. 

Use a solid math library to compute the proper wheel roll to avoid biases. 

After you have a mathematically-sound number, do whatever you need to make that expected value show up.  For a wheel that could mean rotating the texture and replaying one of several known animations. For dice that can mean applying the right texture that gives the right result for the animation.