Just whip the belt off, no need to refit the fan pulley.
All the same, when checked, you could refit the fan before the belt to eliminate the viscous part of the fan unit
I know it sounds like teaching granny to suck eggs, but it can be a good idea to draw a small diagram of the belt routing, makes refitting easier
Also a possibility that the alternator bearing is noisy, there's a hydraulic pump on the front of it.
Hope you get sorted