Except that is for Benjamin Disraeli and Tony Blair. Officially they lived at No 10 Downing Street but physically lived at No 11. The apartments of No 10 were bigger than No 11 and too big for Mr Disraeli so he swapped with his chancellor, Stafford Northcote. Remodelling over the years reversed the situation and because of the size of his family, Tony Blair moved into the now larger apartments of No 11 and Gordon Brown, single at the time, moved into No 10s apartment.
So to answer the question: Benjamin Disraeli was the first Prime Minister to live at No 11 Downing street.