So firstly i dont know the engine, But.
Can I clear up somethings first.
You dont have a tapered crank?
You have a sloppy fit. I.E the pully just goes on with no resistance?
Ok so my thoughts.
The straight shaft indicates to me that it has to be a clamping force that holds it together. the keyway is there so the timing marks are in the right place. (how am I doing so far).
For the pulley to spin on the shaft suggests that either the bolt wasnt tight or there was a lot of resistance on the belt, or a mix of both.
So taking material off the back of the pully does nothing except put the belt out of alignment.
You have a Mill

.
So, Put 4 blobs of weld on the dampner/pulley inside both ends, that wont heat it to much. Mill it out to the interference fit you want and use as suggested above a filler type of your choice.
You now have it centered with a bit of grip over its full internal.
Then do up the bolt kin tight.
If I have no idea of what you have then please ignore my post.
J