If thats whats happening, it means that you've accidently installed a reverse taper audio pot. An "ordinary" audio taper should only cut about 10% of the signal, at approx. 50% of rotation.
A linear taper control will be at half resistance @ half rotation.
An audio taper control has very little effect until you turn it pretty far. The resistance element is "tapered", so that you only have about 10% of the resistance @ about halfway down.