Fitness trackers have come under a bit of scrutiny lately, with Fitbit facing a class action lawsuit over the accuracy of its heart rate tracking last year and studies raising questions about the accuracy of such devices. In more grist for the mill, a new study from Stanford into the accuracy of seven popular devices indicates their energy expenditure estimates are way off. But it’s not all bad news, with heart rate data found to be relatively accurate.
The team from Stanford tested seven wrist-based fitness trackers: the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn and the Samsung Gear S2. Sixty participants were monitored wearing the devices while using stationary bicycles and walking or running on treadmills.
The study compared the fitness tracker measurements for heart rate to a medical-grade electrocardiograph, while energy expenditure data was compared against an instrument that monitors oxygen and carbon dioxide in breath.
The heart rate data offered unexpectedly accurate results with the study finding six out of the seven devices accurately recording heart rate with a median error rate of less than 5 percent. The worst performing device in this area was the Samsung Gear S2, but it was still accurate to within 5.1 percent.
While the heart rate measurements were unexpected in their degree of accuracy, especially after the last 12 months of fitness tracker controversies, it was the energy expenditure data that really shocked the researchers.
“The heart rate measurements performed far better than we expected but the energy expenditure measures were way off the mark,” says Professor Euan Ashley, senior author of the study. “The magnitude of just how bad they were surprised me.”
The most accurate device in regards to energy expenditure was the Fitbit Surge with a median error rate of 27 percent, while and the worst performing device was the PulseOn, with an extraordinary error rate of 92.6 percent. As each device has its own proprietary algorithm for calculating energy expenditure, the team could not offer a comprehensive explanation for the inaccuracies.
“My take on this is that it’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight, etc.” suggests the study’s lead author, graduate student Anna Shcherbina.
The study also found that darker skin tone, larger wrist circumference and higher body mass index correlated with higher error rates when measuring heart rate across multiple devices.
In the next phase of testing, the team will move out of the laboratory to evaluate the fitness trackers’ measurements within the context of participants going about their normal day.
“In phase two we actually want a fully portable study,” explains Shcherbina. “So volunteers’ ECG will be portable and their energy calculation will also be done with a portable machine.”
The good news here is that many of these wrist-worm fitness trackers offer reliable heart rate measurements. But if you’re using these devices to count your calories, and maybe provide an excuse for having an extra piece of cake at the end of a long day, then maybe you should think again.