Here's an interesting chart from Brad DeLong showing that with the exception of the Reagan years, the interest rate the federal government pays on debt is normally lower than the growth rate of the economy. That's to say that borrowed money (rather than taxes) has normally been a good way to pay for the marginal government service.
Since people have a lot of very strong, very emotional, very ideologically charged opinions about the appropriate shape and scope of government spending, they sometimes have difficulty thinking clearly about the question of how the government should be financed. The message here isn't necessarily that going into debt to spend more is usually a good idea. The message is that for whatever the appropriate level of spending is, the appropriate level of taxation is (usually) lower. Fully paying for government spending with taxes represents an unnecessary burden on the present-day private sector relative to the fiscal capacity of tomorrow's richer private sector.