I haven't quite got the picking to work again. I adapted the Mesa gluUnProject into my own camera code, but there's something wrong here... not 100% sure what the modelView matrix should be here... I only have the view matrix from the camera as far as I can understand...
Anyway, scaling works and the projection is back to where it should be. Tomorrow I should start to see some shaders that do lighting and whatnot.