Have you got an Overdrive?
It could be the nut on the back of the main shaft. (Has to be undone to fit an overdrive, so be have been reassembled with a knackered lock wash, and so my first question, but can happen with or without O/D)
Does the gearlever move back and forwards on it's own as you come on and off the power?