Familiarity is a big factor. I use a range of familiar, high-quality recordings or recordings that have familiar specific attributes. (This is contrary to famous Norwitz/Qvortrup "Are You On The Road To... Audio Hell?" advice: see HERE. But that's 50% bunkum anyway)

I have often had the following sort of comparative experience. I compare A to B and hear little or no discernible difference; I compare B to C and likewise hear little difference. But comparing A to C I hear a distinct difference.