Pots are basically rubbish unless you turn them to full output or near that by default. As a matter of fact this becomes the addiction mostly by users of pots as they have the tendency to deteriorate the signal most at the lowest levels. This is very easy to explain as the lower you set it the longer the carbon track is for the signal to pass through and this is then forming the series resistor in the signal path. Very poor quality conductor....
Also the lower you set it the greater the differences in resistance value become between the left and right channel.
The stepped attentuator is ALWAYS a quantum leap ahead of even the best potentiometer, though a poor quality switch can ruin the joy prematurely. Many offered rotary switches use silver plated very thin contacts which are fine for an input selector, but for volume controls they wear out far too quick. And the contacts are often very open and vulnerable and easily break in the process of soldering the resistors on them.
I highly recommend to choose a switch with a quality contact that is spring loaded and nicely thick. Elma switches are a good example, but there are more brands that endorse the same type of construction.
Then there are two kinds of stepped attenuators, series and ladder type. Series is simpler and you will need half of the contacts of a ladder type. It is already way ahead of any potentiometer, but it still has a multitude of series resistors in the signal path of which the bad properties are multiplied by the amount, again the down side is the lower it is set the poorer the signal transfer. And the tolerance influenced value differences also add up by the used amount, so again not perfect equal left and right attentuation. (though this can be fairly well matched by hand)
The ladder type is by far superior, always just one series resistor in the signal path.
Oh and before you buy a switch, make sure it is a make before break switch!! Some switches (including Elma) can be configured by opening them up and turning the 'wiper' internally 180 degrees. The 'wiper' contact has a wide and a narrow end, the wide end will already make contact with the next piont whereas the narrow end will break contact with the previous before making it with the next. Sounds complicated? Well all I can say is if the contact to ground is broken in between the steps it will go MAX volume periodically, which can be quite a shocking experience...
Pics can help a lot to diagnose something from any place on the globe...