It depends. If you are acclimatized to the desert, you would say yes. But if you live in Maryland and you were visiting during the winter, you would say that the desert winters are warm.
That really depends on which desert and what you consider "really cold". As an example: Antarctica (the coldest region on the earth) is considered a desert and no matter what you consider "really...