As long as you haven't got any petroleum based chemicals in there then yes.
Not sure what rubber the diaphragm is made of but brake fuid is not normally associated with any rubber deterioration (though the nitrile/Buna types that are petrol resistant don't like brake fuid).
The brake system seals are nothing other than basic EDPM material, which is likely the diaphragm material too.
Therefore washing out the unit to protect it from paint loss and rust through would probably be more important.