Whatever you do, before anything else, follow carefully the advice about connections from/to the battery. And yes this CAN be intermittent. Damp and cold can make all the difference.
First the terminal connections to the battery on a TD5 are both peculiar and SH!TE. The connectors are almost designed to come undone when you turn the tightening bolt. Bang them down over the posts with a socket and a hammer, first, before tightening.
Next check the connections between the battery earth and the engine, use a jump lead if necessary.
Next check the bolts securing the starter to the engine/gearbox are good and tight.
Next check the positive lead and the lead to the solenoid are both tight.
Once all this is done, test it.
If the problem still occurs, disconnect the lead from the solenoid that comes from the switch and use a jump lead to connect positive from the battery to the solenoid, if it all now works, you know the problem is a wiring or switch problem.
It you still get the noise then take the starter off and test it off the car. You can clean it up, get rid of dust, which doesn't have to be clutch dust, it can be ordinary road muck. But as it works a lot of the time, I really don't think the starter itself is the problem. Unless the pinion, (which is the little gear wheel that spins in contact with the much bigger gear, called the ring gear, cos it's a big ring,) is worn on the end and doesn't always mesh properly with the ring gear. Or, sadly, if the ring gear is worn particularly badly on the contact points, you could investigate this, which needs a torch and the ability to turn the gear, i.e. the crank.
Get back to us if this doesn't work.
Crossed fingers!!