As im sure has been said before... The only serious downside i have ever been told - never done it meself, is that the taper on the drum can force the recess on the back of the alloy, so stressing it and making it more liable to crack.
Oh yeah, make sure you have the correct wheel nuts too!
Alloys also are not as strong, so they are obviously not as good off road.
why dont you all just buy steel wolf rims dont get much better than that
G