My Hannibal rack stays on all the time because its too heavy to get on/off by myself, takes too long, and I've got nowhere else to store it.
As my defender is shaped like a brick I dont suppose the rack makes a lot of difference to mpg although I'll concede it will have some effect.
Also I find it useful to have the rack in place as otherwise you'll be somewhere and it will be at home when it could be most useful. Like the time I passed a building site and they were flogging 6x2 timber in 5.2 meter lengths for £1 each because the H&SE inspector had told them to reduce the flammable material stored on site. Spent £20 and loaded up my rack (overhangs were slightly illegal) to build my decking and pergola.
Went straight back for a second load and they had all been sold so if I'd had to go home and fit the rack I'd have missed out.