It's complicated to calculate precisely, but cotton is widely considered to be almost as awful as polyester. Production takes up huge areas of land and a bunch of water, and the carbon footprint (depending on production type and how it's calculated) can end up being roughly the same as polyester. Here's a Guardian piece for more info.
There are some efforts to push for more sustainable cotton production, and it is making a change. So buying organic/sustainably sourced cotton could possibly half the climate footprint without any visible change to the material - it's not necessarily just a green washing marketing trick.
As with everything else, the best you can do is to make sure you don't buy more than you need, that you make sure to keep things as long as they can last, and repair them whenever possible. The lowest climate footprint you're going to get is obviously from the clothes already in your wardrobe; the follow-up is buying second hand. It's beyond obvious, but it bears repeating.