The best way to perform an AB comparison IMO is to listen to the same musical passages on each in fairly rapid succession. The there should be a wide variety of musical selections showing different aspects of the kind of music you like to listen to. There should be a wide variety of dynamics, instruments, diffferent combinations, and the recordings should be of high quality. The levels must be exactly matched. The selected passages should be long enough for you to get a flavor of each sample but not so long as to forget what the previous sample sounded like. All other variables should be ruled meaning that the tests should be repeated on a wide variety of equipment.

For a cd player, this means two identical discs synchronized between the players, a variable output with a volume control on at least one of the players, the louder of the two, A-B repeat, and several sound systems to try them out with. Synchronizing cds is probably easier than with any other program source except FM radio. If you want to make the test scientifically fair, someone else should do the switching and you should have several listeners with a large number of samples.

I have never done any such test myself. However, I have determined to my own satisfaction that there are subtle audible differences between some players. This was done by a much less formal process where I did the switching myself. I was curious to see how a very highly rated JVC 1 bit unit I bought for my parents stacked up against a 4 times as expensive 20 bit Denon uint I had bought for myself several years earlier. At that time, I concluded that the differences were slight having to do with the relative high frequency balance and what I call steeliness of violin strings, slight on the Denon, non existant on the JVC. The JVC was also slightly brighter. That was about 13 years ago. Ironically, this year, the Denon unit finally failed and I decided not to repair it but to replace it with the JVC unit. Slight adjustments to the high end response made the JVC sound about identical as to what I remembered from the Denon. Not scientific but adequate for my purposes.

I am extremely suspicious when someone says one cd player blew another one away. From what I can tell, the differences are always subtle. If you will not take any steps to change the frequency response of a sound system other than to experiment with expensive cables or reposition your loudspeakers, then the sonic signiture of one cd player due to its deviation from flat frequency response can add or detract from deviations inherent in the rest of a sound system to the benefit or detriment of overall sound differently from another unit. If that's what makes some people think brand x is better than brand y and therefore justifies 5x or 10x the cost, then let them pay the high price. Most of todays so called high fidelity loudspeakers suffer from being too bright in many rooms and a cd player with a slight high end rolloff may sound better. BTW, the current unit in my second system is a 5 year old 5 disc RS 1 bit carousel unit and it sounds just fine. My chief conmplaints, it has no volume control, no AB repeat, and won't display remaining time per track.