Usually it doesn't affect the sound from the speakers, but aesthetically it's a problem.
Some people use vacuum cleaner to suck out the cone again.
Me, I use masking tape, apply it on the cone, and pull it swifty (YMMV, be careful if the material is delicate such as paper)