Hand brake sticking on a bit; such that you are effectively only using the front brakes.
Thats all I can think of. Easy to test my theory....leave it in gear on a downhill (without handbrake on) and see if that makes a difference. Point front wheels into the kerb if you are worried.