I just got my first pair of grown-up speakers, and am curious about keeping 1 or both grilles on or not?

Is there a difference in sound? Is it just aesthetically pleasing?
When I see pictures of speakers there is always one w/o grille and one with?

What do most people do? Is this just a totally silly subject?