When measuring fraud prevention systems, honest analysis is critical to improvement
Two weeks ago, the Centers for Medicare & Medicaid Services (CMS) announced that its Fraud Prevention System (FPS) had "identified or prevented" $454 million in fraudulent claims in 2014. That figure made up more than half of $820 million identified or prevented claims over the system's three-year life span.
CMS touted one figure in particular: Last year's $454 million represented a 10 to 1 return on investment.
"We are proving that in a modern healthcare system you can both fight fraud and avoid creating hassles for the vast majority of physicians who simply want to get paid for services rendered. The key is data," CMS Acting Administrator Andy Slavitt said in the announcement. "Very few investments have a 10:1 return on taxpayer money."
That figure was featured prominently as news outlets reported the story across the country, and with good reason. When calculating fraud, it's often difficult to really conceptualize the accompanying dollar figure. The ROI value allowed us to see what real value these programs have to offer. Moreover, Slavitt was right. A 10 to 1 investment is hard to find anywhere, let alone a massive government program.
But, as with most statistical breakdowns, numbers and figures can be twisted and manipulated almost at will. A concurrent analysis of those numbers by the Office of the Inspector General (OIG) found that the ROI for the FPS was closer to 3 to 1.
The OIG analysis found that FPS realized $133 million in actual and projected savings in 2014, about one-third of CMS' estimate. Why the large discrepancy? The OIG only accounted for money that was likely to be recovered. In other words, the remaining balance of the $454 million was more or less imaginary. It would probably never be seen, and yet CMS willingly counted it up as if it were sitting right in front of it.
Imagine your financial manager saying that you made a tenfold return on your stock investment. Then when you go to cash out, he gives you 30 percent of that.
"Well, theoretically you made three times as much," he tells you, enthusiastically.
There are two problems with the way CMS approached its FPS statistical analysis. First, it offers an unrealistic measuring stick for both current and future fraud prevention efforts. Much like the golfer that takes a dozen mulligans before announcing that he shot a 76, it's difficult to really measure improvement over time. How do you get better if you don't actually know where you stand?
Second, it overshadows the fact that a 3 to 1 ROI is actually pretty reasonable. As the OIG points out, the adjusted actual and projected savings collected by the FPS last year more than doubled the $54.2 million it collected in 2013, during just the second year of the program.
Those numbers signify notable progress within the FPS system, which is important. With the government's increasing reliance on data and predictive analytics to fight fraud, it's encouraging to see that preventive systems are being refined to the point that we are seeing significant year-over-year gains in fraud recovery. It's just not the 10 to 1 ratio that CMS would have you believe.
There are important lessons here for payers of all types that are molding and tweaking their predictive analytics systems aimed at preventing fraud. Perhaps the most notable takeaway is that honesty is the best policy, not just for the sake of public perception, but because these new systems need a chance to mature. Unfortunately, overestimating their value actually stunts their growth.
In some ways, I don't blame CMS for touting its more impressive ratio. It's difficult not to get wrapped up in the wealth of numbers tied to these new systems. But as fighting fraud transitions to a system that relies on data, it's important that we evaluate the functionality of these systems with the same critical eye that is applied to the claims data itself. This way, the remaining gaps become evident, rather than boisterously waving around an inflated dollar amount. - Evan (@HealthPayer)