The early church celebrated everything on Epiphany - birth, Magi, and baptism, all together. Celebrating one's birthday was considered a pagan custom in the first few centuries of Christianity. Some say Christ was most likely born in spring, because no one calls a census in winter time (Mary and Joseph travelled to Bethlehem for the census). There's also some discussion on whether or not shepherds would be pasturing flocks at night certain times of the year. Some Egyptian scholars in the 3rd century decided Christ was probably born around May 20th, and Mesopotamian scholars set it a month earlier. These hypotheses were based on the idea of Christ being born in the 9th month, which in the Jewish calendar roughly corresponds to September, but months at the time in their own calendars differed in length. Epiphany, on the other hand, seems to have been rather regularly celebrated in early January (roughly, again).
It's possible that selection of the date was influenced by the Mithras myth, which apparently was also influenced by the story of Christ (the source I found suggested that the Mithras myth borrowed the idea of shepherd adoration, just as Christianity borrowed or was inspired by the timing of the Mithras myth). The logic behind the Mithras myth fits very well with Christian theology. Christ is associated with the sun very much, and right around Christmastide is the time when, from the darkest time of the year, the sun is "reborn." It also works with the story of the Magi - in the dark sky appears a bright star which leads them to the Lord.
So, basically, sometime in the late 4th century into the early 5th century, the Church ended up adopting 25 December as the day of the feast of Christ's birth. It seems different bishops 'conformed' to this at different times.
For a much fuller, somewhat mindnumbing analysis, check out: http://www.newadvent.org/cathen/03724b.htm This is a Catholic encyclopedia, BUT it cites its sources in text, such that practically every line has some sort of citation, including church documents and published texts, modern and ancient.
In response to the earlier answer: At that point in time, there were no "denominations" - so one could say that the church at the time was the Catholic Church, or, just as accurately, the Orthodox Church, which only split around the 11th century, predating any Protestant churches by four centuries. At the time, of course, as there was no schism, the church was basically mere Christianity, to borrow a phrase. At that time, the Church was more concerned with eliminating heresies and settling beliefs - such as at the Council of Nicea in 325, where the quintessial statement of the Christian faith, the Nicene Creed, was written. The Church had no compunction about the use of the term "catholic" for the Church or in the creed because it means "universal" - a reflection both of early Christian unity and Christ's commission to 'make disciples of all nations.'