These three economists say they've assembled "the largest price and quantity dataset ever employed in economics." Sadly this fully operational dataset does not allow them to destroy entire planets. It does, however, let them test government CPI methodologies for external validity. They find that they work pretty well when inflation is high (perhaps not so surprised because that was the case when the methods were developed) but not when inflation is low:
Official price indexes, such as the CPI, are imperfect indicators of inflation calculated using ad hoc price formulae different from the theoretically well-founded inflation indexes favored by economists. This paper provides the first estimate of how accurately the CPI informs us about “true” inflation. We use the largest price and quantity dataset ever employed in economics to build a Törnqvist inflation index for Japan between 1989 and 2010. Our comparison of this true inflation index with the CPI indicates that the CPI bias is not constant but depends on the level of inflation. We show the informativeness of the CPI rises with inflation. When measured inflation is low (less than 2.4% per year) the CPI is a poor predictor of true inflation even over 12-month periods. Outside this range, the CPI is a much better measure of inflation. We find that the U.S. PCE Deflator methodology is superior to the Japanese CPI methodology but still exhibits substantial measurement error and biases rendering it a problematic predictor of inflation in low inflation regimes as well.
In my view this is a key part of the fussy and not politically interesting case for a nominal GDP targeting policy from central banks. The rhetoric of inflation targeting has us accustomed to the idea that CPI or PCE adjusted numbers are "real," while nominal figures are perhaps ghostly or fake. But in fact nominal quantities can be measured much more rigorously and uncontroversially than inflation adjusted ones.