Customer testimonials have become synonymous with all health products from pharmaceuticals to quackery. While a testimonial in of itself is not a particularly strong piece of evidence, the widespread consistency in anecdotal reports leads to the necessity of further research. The issue is when companies stop at testimonials as the end pursuit of evidence, and do not pursue proper research, or make existing research available for those trained and competent to analyze and critique.
This pyramid is as accurate as a simple diagram can be. I would argue that a very large and well controlled clinical trial can be superior in information to a systematic review, if the review is combining a large percentage of flawed data to try to ascertain an overall statistical significance. Many groups purporting to teach the population about skepticism rightfully include this pyramid and encourage their followers to question testimonials. Others go a step further and mock the hierarchy of evidence in pseudoscience with photos such as the on below:
The issues with this messaging
Issues with intent
Let’s get this straight, testimonials ARE a form of evidence, just a poor one. I’ve seen deceptive practices on both sides of the fence. I’ve spoken to marketers who’ve exclaimed “get those studies away from me”, and “if you insist on having a science based section, bury it down so far you can only share it when it is requested”. Most good marketers know testimonials work, and science and statistics aren’t just ineffective, they make the testimonials less effective. While I agree with this, and understand the psychology and reasoning, my issue is when marketers go a step further and rationalize that because all that is needed is testimonials and those are easy to come by, no scientific validation of any type is required. This, to me, is wildly unethical.
On the other hand, I’ve observed many examples of self proclaimed skeptics with reasonably large audiences mocking a company for leading with testimonials, “debunking” claims using sarcasm, memes and the diagram above, without bothering to do their due diligence in properly reviewing a subject. I’m sorry wannabe skeptics, but a quick page 1 google search, or even a PubMed search without knowing what to look for, does not constitute as a review of the evidence.
Even with Hydrogen Water, numerous “legitimate articles” in mainstream media have claimed a search “turns up no” or “turns up 1 or 2” human trials and a “few early trials in rodents”. This is in part due to the issue with how the media reports on science, the topic of an upcoming article of mine, part of the issue being the use of “go to” experts who are friends with the author but have little or no knowledge on the subject they are being asked about. Often multiple media outlets will pick up the same story with little or no independent fact checking.
Hydrogen therapy has over 1200 publications in 12 years, 65 (at the time of this writing) published human trials with dozens underway, and the average impact factor of the journals publishing this research is about 3. While hydrogen water, and hydrogen therapy as a whole, is still very much green science, there are dozens of randomized controlled studies and even multiple systematic reviews, including reviews on specific indications. Criticizing a company for employing testimonials in a subject with this volume of research is dishonest and contributes to a divide in messaging.
Issues with understanding
A big issue with both supposed health influencers, marketers and skeptics is a failure to fully comprehend just how much impact a testimony is in terms of evidence. Often, those disseminating the testimonials will be utterly convinced that it constitutes irrefutable evidence. Even the suggestion that it does not will cause offence, typically leading to the individual spreading what they believe as fact to bear down and justify the criticism as lack of understanding.
Conversely, many supposed skeptics will train themselves to call bullshit whenever they see marketing lead with testimonies, or even the insinuation of a vague endorsement in an ad. These “skeptics” neither understand that a testimonial is still a form of evidence, albeit a weak one, nor do they understand the necessity for leading with this in marketing. I’ve personally rebutted hundreds of these “skeptics” who just comment on posts “this is pseudoscience” or “lies” or countless other trite statements without bothering to review the claims in the post, or any empirical literature. What is even more concerning is when confronted these individuals typically will ignore the debate, deleting their messages, and go back to what they have come to believe is a necessary style of debunking quackery- without bothering to verify if it is quackery.
The Psychology behind testimonials
There is a passage I quite like but for the life of me cannot find, where I believe it is Dostoevsky’s Underground Man musing about how we are more concerned with our more irrelevant and inconsequential problems than the greatest tragedies befalling strangers across the world. We may momentarily be troubled by the plight of strangers in incomprehensible volumes, but quickly turn our attentions back to ourselves, even paying more concern to a broken nail on our little finger. (if you know the passage I am speaking about, even if it is not Dostoevsky, please write in an email)
While the above may not seem tied to testimonials at first thought, it is the core of why they are successful. We care about ourselves, and we care about others- so long as they are individuals which can trigger an emotional response. On a talk I listened to between Daniel Kahneman and Sam Harris, at about the 30 minute mark Harris brings up a point he references as from Paul Slovic. Harris goes on to cite that in gathering evidence on altruism with charitable donations, that if a single starving little girl is presented there is an overwhelming sense to help. He goes on to say that when a brother is added, so it is two young children, rather than increasing in response the response decreased. Finally, if on top of the emotional plea with the starving children statistics are presented of how many children like this exist, levels of altruism hit the floor.
To paraphrase Kahneman’s response, ““we know that it is better to have significant data than 1 case, but we feel better about the 1 case” and “if you want to convince people, you need to convince them by telling them stories about individuals because numbers just don’t catch the imagination of people”
Likewise, people resonate with a single testimonial better than a mountain of research, especially if that one person is someone they know and trust, or a celebrity or authority figure they look up to. I often quip one celebrity endorsement is worth more than all of the worlds scientists and doctors combined (I would say this is hyperbole, however recent trends have shown otherwise)
Efficacy without influence is irrelevant
One of the conundrums facing anyone intending to market a health product- supplement, device or drug alike, is that with what we know about human psychology and marketing, some of which is detailed above, by necessity messaging needs to hit emotional triggers while even downplaying statistical analysis. If a product is efficacious and contributes to a net good, marketing it in a neutral, cautious and academic manner will essentially negate the potential to do good. If a product works but cannot trend, it becomes irrelevant. If no one takes it, it isn’t doing any good.
This extends to the gathering of evidence, and how efficacy is established. With how fast scientific research is moving it is impossible for researchers to keep up. Teams will seek to explore a therapy based on popular trends, to evaluate whether the mechanisms are rooted in a specific mode of action, or simply placebo. We see certain sham therapies that have compiled thousands, even tens of thousands, of studies on them even when showing little or no efficacy. They continue to be studied, receiving grants to do so, due to their popularity within consumers.
Rebuttal to the Skeptical Rebuttal
A common skeptical rebuttal is that a product should have established efficacy, such as a medicine, before ever being marketed, and before testimonials are employed. This position is inherently flawed for anyone not supporting an oligarchy, and further able to differentiate between medicine and complementary and adjuvant therapies.
For something like our hydrogen tablets, or hydrogen water, the cost to launch as a drug could exceed 100 million USD and take a decade or more. To be frank hydrogen water may never be successful in the drug pipeline as it would need to be targeted for a single and specific indication. The mechanisms of action and outcomes of hydrogen water administration are significantly different than that of what we term to be drugs and their specific targets.
While hydrogen therapy, most prevalently hydrogen water, has shown to positively impact 170 different models across every organ of the mammalian body, in 1200 publications and 65 and counting human trials, the chance that it would be a lone treatment for a specific disease is modest at best. Moreover, with the wide range of benefits for overall health in many different functions, it would be a disservice to the population to not make it available to the masses. It doesn’t fit the drug model, and as such the bureaucracy and regulatory progression surrounding drugs is largely irrelevant.
In terms of funding, from a philosophical stand point the notion that the Pharma route is the only route is incredibly troubling. Pharmaceuticals must be funded by large corporations or the ultra rich just in terms of pure allocation of funds and time needed. This also dramatically drives up the cost of goods to end users. Something like what I have done, I funded entirely from my own personal savings and that of a few friends. I do not come from money, in fact growing up my family would have been considered low income. That said myself personally and my company combined have invested over $1 million into the regulatory framework and R&D to bring hydrogen tablets to market. While I have made extreme efforts to ensure safety and compliance, and to reach out to public researchers to further explore and hopefully validate the efficacy of hydrogen water and our hydrogen tablets under no publication agreement, often in the industry this isn’t the case. A large percentage of “bad actors” end up causing skeptical rebuttals decrying the entire industry.
Responsibilities of the Supplement Industry and Health Influencers
While it is a complete misconception that there is no regulatory framework within the supplement industry in the USA (it is actually more stringent than most jurisdictions, for those that “opt in” to compliance), it is true that enforcement is laughably inadequate. I cover this in an upcoming piece “Are supplements actually regulated?”. Since skeptics love to state that supplements are not regulated and that manufacturers simply need to “notify 75 days in advance”, which shows a complete and utter lack of understanding of the process, it befalls on industry to ensure we check the practices of bad actors. As a note for skeptics who parrot this laughably false statement, I will quote Sam Harris’s rebuttal to Deepak Chopra regarding physics.
“I would never be tempted to lecture a room full of 1000 people at Cal Tech on Physics. I’m not a Physicist. You’re not a Physicist.. and basically every sentence demonstrates that.”
In reality the New Dietary Ingredient Notification entails rigorous review, and complete compliance of everything relevant under Title 21 of the Code of Federal Regulations. Further, significant safety data and evidence of human use is required. While the FDA no longer rules supplements “safe” under the GRAS process, it befalls on companies to evaluate the safety themselves, and the FDA then verifies this and either objects or does not object.
“information that is the basis on which the manufacturer or distributor has concluded that a dietary supplement containing such new dietary ingredient will reasonably be expected to be safe. FDA reviews this information to determine whether it provides an adequate basis for such a conclusion. Under 21 U.S.C. § 350b(a)(2), there must be a history of use or other evidence of safety establishing that the new dietary ingredient, when used under the condition recommended or suggested in the labeling of the dietary supplement, will reasonably be expected to be safe. If this requirement is not met, the dietary supplement is considered to be adulterated under 21 U.S.C. § 342(f)(l)(B) because there is inadequate information to provide reasonable assurance that the new dietary ingredient does not present a significant or unreasonable risk of illness or injury.”
As I will delve into further in the article on various strengths and shortcomings of world bureaucratic organizations in the realm of the supplement space, the NDI process has a staggering rate of failure/rejection, with upwards of 85% of applications being objected some years(and rates of applications being only a few per month submitted). Even for the hydrogen tablets, despite dozens of human studies, a century of use of H2 gas, hydrogen water already having GRAS status in foods, and our own human studies the process took almost two years and between consulting fees, analytical testing, reports and staff hours added to around $200,000 in expenses to net a “no objection”.
In industry we need to speak out more about the bad actors, at least pertaining to policing what we supply to them. To quote Christopher Hitchens (in a different, but relatable context) “If you will claim it as one you must accept it for the more. If you don’t it’s flat out dishonest.” While many supplements maintain a high propensity for good, some have little evidence or can actively do harm based on dosing suggestions. Some distributors actively create misinformation and lies based on some pathological need to create a fear-based story. Because we are profiting off of the power of testimonials, we need to be to be cognizant of the harm they can do in the wrong hands.
Everyone in industry needs to hold themselves accountable for ensuring what we distribute maintains evidence of safety and efficacy, and the messaging distribution employs maintains at the very least a semblance of accuracy, without actively doing harm in another regard. I say this regarding the products we supply or market, as it is unreasonable to presume the ability to police what others are doing on products unrelated to what we supply, or distribute. At that point, there would be no industry and no one to spread influence. More on this and the blurred line, in my article next week “100 voices” I discuss the need to ally with others, even when there are philosophical differences, and when I believe it is ethically right or wrong to maintain a relationship, and the obligations of maintaining that relationship.
We also need to speak up regarding issues that may not pertain to us, but are inherently wrong and cause black eyes to the goal of providing efficacious products to positively impact health. In every other field whether it be industry, politics or theology, quiet complacency of the most extreme views by moderates is met with condemnation from the critics, even viewed as tacit endorsement. It is not and should not be different for our industry. Trends often begin from the power of personal story, testimonials, devolving into a movement of “me too”, creating an almost insurmountable surge of false information. Do not join a trend for the sake of a trend, verify its accuracy.
Closing Message to Skeptics
The obligation of skeptics is to do their due diligence in researching a topic before dismissing it. The onus of evidence is on the person making the claim, which goes both ways. Health influencers, corporations or individuals alike, need to be able to back up their positions with evidence and provide said evidence when challenged. Skeptics have their own platform to present their positions, and as such their refutations need to be backed up and well thought. Often those positions, companies or individuals they are dismissing have no channel to engage, and if they do, the audience may never see the rebuttal. We all have an obligation for truth whether we accept it or not. Extreme views and misinformation should not be employed to battle extreme views and misinformation. It is a disservice to the truth, and further drives the divide.
Comments will be approved before showing up.