I would change the amp and try and get a decent tube amplifier. A used Sugden A48b would easily be had under your budget and would be nice upgrade. I'm not a fan of the 320Bee. You could upgrade it in stages but I think that is an area to be replaced so I agree with your assessment. After that I would look at the speakers and look to something like the Audio Note AX two or something from Omega loudspeakers, Both will run over your budget however. The B&W's are alright for now.

Since you don't play super loud I would look at a tube amplifier from Grant Fidelity who are relatively new to the scene and have prices that are very reasonable (and I mean very reasonable!). Antique Sound Labs and Jolida are others that on the used market would be easily attained. And before anyone says - yes they have enough power to drive B&W.

If you're truly impressed by the B&W's then you could look for a Matrix 805 or CDM 2SE as IMO these were the two best standmounts they made that might fall in your budget *The matrix might not but the 2SE will." I prefer the music reproduction of the 2SE to the 1SE and 1NT and 705.

You could also check out Tannoy - they've been making solid strides at the budget end of the spectrum and generally are tube and low power amp friendly.