REMOVAL
Grasp the arm and push it (firmly) upwards-and-forwards, towards the roof and windscreen. There's a spring-loaded lug at the top, and a fixed lug at the bottom. If it doesn't come easily, that spring-loaded lug has probably frozen. So spray plenty of penetrating oil (WD-40) at top centre, so it runs down onto the spring-loaded spigot and frees things up. Give it a few minutes for the WD-40 to do its stuff, then try again. You might also want to tape a couple of sheets of corrugated cardboard to the windscreen, in case you slip.
I've just removed mine because it was wobbling around. I think that the spring had frozen, which was stopping it pressing firmly into the bracket, hence the spring wasn't holding the assembly tight enough, and it started wobbling as I pulled on the arm when adjusting the mirror. (The free movement of the spring appears to be essential for clamping the bracket and keeping it from wobbling after refitting.)
REFITTING
As others have said, you need to thoroughly lubricate and 'work' the spring-loaded bit, until it moves easily, before attempting to put the mirror arm back onto the bracket.
When refitting, you press the (top) spring-loaded part into the fixed bracket, then, maintaining pressure on the spring-loaded part, you should be able to slip the fixed lug over the bottom lip, and releasing the upwards pressure, the thing will clamp itself into place (it's all coming back to me now). As I say, the free movement of that spring-loaded lug is what holds the mirror arm firmly in the bracket, so it doesn't wobble around.
Time invested in getting the spring to work freely will save a lot of pain. Last time I did this it went in quite easily. I'd recommend washing out any gunge with WD-40 to get things moving, then graphite dry lubricant or graphite/moly greace, as the WD-40 will dry out over time, and you don't want that spring-loaded top pin to seize up, else the darned thing won't be firmly gripped (by the spring's pressure) and it'll start wobbling around like mine did.
If it doesn't go back (relatively) easily, just keep lubricating and working that spring until the spring-loaded part moves easily when you press it against a solid object (e.g. a shoe last, a vice, or a house brick).
Thanks for the reminders guys. I'd have been struggling but for your good advice. A timely reminder.
Cheers,
Rick