It's not obvious at first blush why a metropolitan area as a whole would find itself underproviding housing and other valuable structures. Development has both costs and benefits, but as some jurisdictions adopt anti-development rules to avoid the costs the benefits to other jurisdictions of letting building happen go up. What's more, developers would seem to have a strong interest in lobbying to be allowed to build things. And yet we can see in practice that this doesn't happen. Real estate all throughout certain metro areas is way more expensive than in lower-cost areas, indicating that structures are being systematically underprovided not just blocked out of a few places.
David Schleicher has a neat paper situating this in a broader account of urban governance:
This paper seeks explain these changes with a story about big city land use that places the legal regime governing land use decisions at its center. Using the tools of positive political theory, I argue that, in the absence of strong local political parties, land use law sets the voting order in local legislatures, determining policy from potentially cycling preferences. Specifically, these laws create a peculiar procedure, a form of seriatim decision-making in which the intense preferences of local residents opposed to re-zonings are privileged against more weakly-held citywide preferences for an increased housing supply. Without a party leadership to organize deals and whip votes, legislatures cannot easily make deals for generally-beneficial legislation stick. Legislators, who may have preferences for building everywhere to not building anywhere, but stronger preferences for stopping construction in their districts, “defect” as a matter of course and building is restricted everywhere. Further, the seriatim nature of local land use procedure results in a large number of "downzonings," or reductions in the ability of landowners to build "as of right", as big developers do not have an incentive to fight these changes. The cost of moving amendments through the land use process means that small developers cannot overcome the burdens imposed by downzonings, thus limiting incremental growth in the housing stock.
Finally, the paper argues that, as land use procedure is the problem, procedural reform may provide a solution. Land use and international trade have similarly situated interest groups. Trade policy was radically changed, from a highly protectionist regime to a largely free trade one, by the introduction of procedural reforms like the Reciprocal Trade Agreements Act, adjustment assistance, and "safeguards" measures. The paper proposes changes to land use procedures that mimic these reforms. These changes would structure voting order and deal-making in local legislatures in a way that would create support for increases in the urban housing supply.
In other words, if U.S. cities had regularized party systems each city would probably have something like a "growth and development party" that pushed systematically for greater density. Its members and elected officials would, of course, have idiosyncratic interests and concerns that would sometimes cut across the main ideology. But the party leaders would be able to exercise discipline, the party activists and donors would push for consistency and ideological rigor, and it'd be off to the races. Instead, most big cities feature what really amounts to no-party government in which each elected official stands on his or her own and overwhelmingly caters to idiosyncratic local concerns rather than any kind of over-arching agenda. But different institutional processes could change this, and create a dynamic where growth, development, and density are more viable.