If you are on a swim team you could wear a skirted suit as a drag suit, that way you shave even more time at meets... If you just mean in a local pool or in your back yard, of course. The only thing...
Though its primary purpose is to stop water from going into a swimmer's ears, (and keeps hair fairly dry), it can help a swimmer to move faster. The cap has a tiny bit less friction with the water...