You ignore rotations.
If you rotate the upper arm, you want to rotate the lower arm, hands etc. with it.
But currently you only propagate translation to child bones: 

auto matrix = parentMatrix * Mat4::Translate(jointPos);

jointPos tells where the bone has its center of rotation, but you also need…

Sorry for the bump. Any clues as to what could be causing the problem above?