When nobody wants to live someplace, it's cheap to buy a house there. That's why it's cheap to buy a house in Detroit. By contrast, you'll find that buying a house in Manhattan or San Francisco is extremely expensive. This seems to me to be the overwhelming reason to doubt Greg Mankiw's contention that high-tax states are bleeding high-income residents.
Mark Thoma has some empirical information on this, but I think the housing price data is really the most important thing to consider. The point is that without denying that if San Francisco somehow achieved a more optimal tax/service mix that would increase demand for San Francisco living, it takes a very outmoded view of the American landscape to say this would lead to more people living in San Francisco. What it would lead to is continuation of the trend whereby high income people displace low-income and middle-class residents. This then actually becomes a reason for voters in high-cost supply-constrained cities to deliberately select a non-optimal tax/service mix in an effort to prevent rich people from outbidding them for a limited stock of houses.
Now Mankiw's overall argument is that because of population migration we ought to favor decentralization, because decentralization will make it impossible for the government to raise the living standards of the least-fortune people. That strikes me as a morally perverse perspective, so I'm not really sure how a clearer understanding of land use issues would change his thinking. But I think it's fairly clear that population migration is driven by job availability and housing costs much more than by tax policy changes.