In Australia, nonvoters need to pay a fine but it's a small one. Just 20 Australian dollars (which is about $20 these days) for the first offense with some upscaling for repeat nonvoters. That doesn't seem like a huge deal, but according to Dylan Matthews studies show it leads to a fairly large change in behavior and "compulsory voting reduced the vote share of the victorious right-wing Liberal Party/National Party coalition by five points."
If Barack Obama is re-elected today, we're likely to get a test of this kind of phenomenon not in terms of voting but in terms of health insurance. That's because the much-discussed Affordable Care Act "individual mandate" is far from tyannical. It's actually quite weak involving a fine of just under $700 and a somewhat flimsy enforcement mechanism. The point of the fine, recall, is that you want young and healthy people for whom comprehensive health insurance isn't a great deal to buy it anyway in order to create a large risk pool that works for everyone. For that logic to really hold, the penalty has to be pretty big. Nobody wants to lose $700, but $700 is a lot less than the cost of a health insurance plan. So the extent to which this works is going to come down in part to the psychology. Most people follow the law most of the time, even when they could get away with breaking it. If people understand nonparticipation in the insurance market as a morally wrong violation of the rules that happens to be backed up with a $700 fine, then the odds are you'll get a lot of compliance. But if it's understood as just a morally neutral option—either you buy this overprices insurance plan or else you pay $700—then a lot of people will do the math and realize it doesn't make sense.