Basically your swivels need rebuilding.
I did the same to get through my MOT this year and fully expect to rebuild the swivels with new bearings in the next couple of years.
The shim removed was probably from the top pin, removing it increases the preload on a worn bearing. If it was the last shim then there is no way to adjust for wear other than to fit new bearing and bush.