The above chart of real GDP per capita in the late 1970s is, I think, an important bit of context to keep in mind as the question "are you better off than you were four years ago" re-emerges in American politics. At the time Ronald Reagan asked it during the 1980 campaign the correct answer was that, yes, on average Americans were better off than they had been four years earlier during the waning days of the Ford administration.
Jimmy Carter's actual problem was that economic performance in 1979 and 1980 was really bad. So even though on net people were more prosperous at the end of his term than they had been at the beginning, voters felt that conditions were deteriorating and punished Carter for it. Obama is, if anything, in the opposite situation—dismal economy at the beginning of his administration that's been partially offset by better performance later.