Mas OK ang may grilles.
Grilles are intended to protect the speakers from the elements and from accidents. The only drawback is that grilles will unavoidably cause some diffraction for high frequencies, even if the grille's material is highly transparent sonically. Thus, removing the grilles should produce slightly less muffled highs.
However, some speakers are designed to be used with the grilles on. When speakers are already factory-voiced with grilles on, you get the correct frequency response with the grilles on, but a brighter output with the grilles off. In this case, removing the grilles would not be a good idea if you're looking for accurate reproduction.
Isa pang problema, saan mo ilalagay ang grilles tuwing tatanggalin mo? Dadagdagan mo lang ng kalat ang listening room (clutter drives me nuts!), baka mabutas pa ang cloth, o mabali ang frame, etc. At pag laging tanggal-kabit ang grilles, luluwag ang clip ng frame, at baka ma-damage pa ang tela sa lugar na laging hinahawakan.