First, man does not live by bread alone. Our happiness depends partly on our incomes, but also on the time we spend with our friends, our hobbies, and our favorite TV shows. So, it's a good exercise in perspective to remember that by and large, the big winners in the income derby have been the small winners in the leisure derby, and vice versa.
Second, a certain class of pundits and politicians are quick to see any increase in income inequality as a problem that needs fixing—usually through some form of redistributive taxation. Applying the same philosophy to leisure, you could conclude that something must be done to reverse the trends of the past 40 years—say, by rounding up all those folks with extra time on their hands and putting them to (unpaid) work in the kitchens of their "less fortunate" neighbors. If you think it's OK to redistribute income but repellent to redistribute leisure, you might want to ask yourself what—if anything—is the fundamental difference.