Personally, I would ask the doctor. He is, in fact, a doctor.
It depends on how long the cast is too.
From my own personal experience in a full arm cast, from above my elbow to my fingers, your mobility will be shit. I don't think you're going to get that off and then just go straight to full elbow bending, wrist turning and the like. Things get tight.
How long until the mobility returns? I don't know, I could fully extend my elbow after a week but my surgically repaired wrist took 4 months or so. That was probably caused by the surgery though.