Not enough scientists have the gallantry to pull a Barry Marshall and jump into the arena themselves. Try doom-looping on TikTok for 5 hours a day for a month straight and see if your own well-being is affected. Or perhaps watch just a single episode of Skibidi Toilet. While there certainly is a need to robustly demonstrate effects across large sample sizes, sometimes statistics aren't absolute over all other arguments.
“But again and again there comes a time in history when the man who dares to say that two and two make four is punished with death. The schoolteacher is well aware of this. And the question is not one of knowing what punishment or reward attends the making of this calculation. The question is one of knowing whether two and two do make four.”
― Albert Camus, The Plague
“We must keep in mind the story of the statistician who drowned while trying to wade across a river with an average depth of four feet.”
Thank you Jon and Zach for taking the time to deliver and present a thorough break-down of Ferguson's meta-analysis.
Too often meta-analyses are taken at face value without much scrutiny being given to the underlying studies or the black box that some authors use to determine effect size. It's unfortunate that articles with flaws like these pass review from major scientific institutions. Especially for the Ferguson meta-analysis where one need not delve through every underlying study (an onerous ordeal as shown here) to find fatal flaws in the report.
I think this is a classic case of researcher bias. Also, as one of the commenters below put it, "how much data do we need to prove the sky is blue"? At the end of the day, researchers can play with numbers until they are blue in the face, but truth remains. We all see what's happening. We all feel it. Still, settling the debate on a scientific basis is needful, if a bit futile in the long run. You can't fix abtuseness, and there are a lot of abtuse researchers trying to be edgy and get culturally fashionable outcomes. This is similar to a study I read about where reserachers "proved" a link between swearing and honesty. Subsequent researchers debunked the paper because it was based on faulty metrics. Research needs morality and honesty.
Great post. Your point about Lepp and Barkley (2022) is spot on. If one were to compare people who were stabbed with a knife to people who were punched, and notice the latter group suffers feom fewer negative health consequences, one should not conclude that getting punched is beneficial.
Like the subjects in the Wilson study, I would rather shock myseslf than have to do the work entailed in disentangling this metanalysis. This is very disheartening as aside from disliking shocks I am a mental health professional who depends on understandable research to further my perspectives of the people I care for. Unlike other professionals (I am a psychiatrist) we do not use the same end points we see in research. Blood pressure, tumor response, blood sugars, etc are all part of practice and measurable endpoints in medical research. In mental health research we have interviews used only by researchers, which are then subjected to statistics, also used only by researchers. This is opaque to most of us so we depend on the investigators to be transparent and frank about what they have found. Since around 1990 this has meant that psychiatrists have been sold a bill of goods regarding what we really know about how the brain produces mental disorders and their treatments. The story changes every decade or so and is always delivered with utmost certainty. Until it disappears. A simple place to start is to report effect sizes and not p values. This could be followed by more openness about where the patients came from and what they are like. In this vital issue of what to do with the adolescents who are coming into everyone's offices, we sure could use some straight forward reporting of the nature of these experiments. Thanks to Jon, Zach, et al for doing this dissection for us.
What would be a suitable control in Lepp and Barkley (2022)? Obviously an activity that is worse than getting electric shocks won't do. What is a good placebo? Watching tv (taking out some of the interactivity), or a magazine, or a book?
I'm finding this series to be pretty interesting, if only to see the amount of research that's been put in so far to identify a relationship between social media and poor mental health. It's inspired me to write an article asking what would happen if we built our regulation argument with research on harmful design patterns instead of social media as a whole. https://substack.com/home/post/p-153030889
Thanks to social media, psuedoscientists and hacks were bound to come out of the woodwork. Attention is the winning metric nowadays. Dr. Haidt has been preparing for this for a long time. Cheers to him, as always.
In my opinion, it is very unsuprising that a study was produced that show no connection between smartphone use and depression. It would definitely be in the interest of tech corporations to hide this phenomenon. Unfortunately, public libraries are often complicit in this problem, as they promote internet access and computers far more than books. For a practical alternative, read this:
Having worked at the interface of business and science, in nutritional/health research and food marketing, I know how corrupt such research can be.
Based on my read here; there appears to be (potentially) motivated “errors” to reach (potentially) predetermined conclusions.
Its easy to hide and hard to find when an industry sponsor doesn’t want to be found. Its not like there are laws against sponsoring scientific research.
I am glad people like Dr Haidt are working on this but GD. How much data do we need to prove that the sky is blue before we'll believe it?
Not enough scientists have the gallantry to pull a Barry Marshall and jump into the arena themselves. Try doom-looping on TikTok for 5 hours a day for a month straight and see if your own well-being is affected. Or perhaps watch just a single episode of Skibidi Toilet. While there certainly is a need to robustly demonstrate effects across large sample sizes, sometimes statistics aren't absolute over all other arguments.
Exactly.
“But again and again there comes a time in history when the man who dares to say that two and two make four is punished with death. The schoolteacher is well aware of this. And the question is not one of knowing what punishment or reward attends the making of this calculation. The question is one of knowing whether two and two do make four.”
― Albert Camus, The Plague
“We must keep in mind the story of the statistician who drowned while trying to wade across a river with an average depth of four feet.”
― Neil Postman, Technopoly
It doesn't take a hypothesis when kids never look up from their screens to converse.
Thank you Jon and Zach for taking the time to deliver and present a thorough break-down of Ferguson's meta-analysis.
Too often meta-analyses are taken at face value without much scrutiny being given to the underlying studies or the black box that some authors use to determine effect size. It's unfortunate that articles with flaws like these pass review from major scientific institutions. Especially for the Ferguson meta-analysis where one need not delve through every underlying study (an onerous ordeal as shown here) to find fatal flaws in the report.
I think this is a classic case of researcher bias. Also, as one of the commenters below put it, "how much data do we need to prove the sky is blue"? At the end of the day, researchers can play with numbers until they are blue in the face, but truth remains. We all see what's happening. We all feel it. Still, settling the debate on a scientific basis is needful, if a bit futile in the long run. You can't fix abtuseness, and there are a lot of abtuse researchers trying to be edgy and get culturally fashionable outcomes. This is similar to a study I read about where reserachers "proved" a link between swearing and honesty. Subsequent researchers debunked the paper because it was based on faulty metrics. Research needs morality and honesty.
Great post. Your point about Lepp and Barkley (2022) is spot on. If one were to compare people who were stabbed with a knife to people who were punched, and notice the latter group suffers feom fewer negative health consequences, one should not conclude that getting punched is beneficial.
Like the subjects in the Wilson study, I would rather shock myseslf than have to do the work entailed in disentangling this metanalysis. This is very disheartening as aside from disliking shocks I am a mental health professional who depends on understandable research to further my perspectives of the people I care for. Unlike other professionals (I am a psychiatrist) we do not use the same end points we see in research. Blood pressure, tumor response, blood sugars, etc are all part of practice and measurable endpoints in medical research. In mental health research we have interviews used only by researchers, which are then subjected to statistics, also used only by researchers. This is opaque to most of us so we depend on the investigators to be transparent and frank about what they have found. Since around 1990 this has meant that psychiatrists have been sold a bill of goods regarding what we really know about how the brain produces mental disorders and their treatments. The story changes every decade or so and is always delivered with utmost certainty. Until it disappears. A simple place to start is to report effect sizes and not p values. This could be followed by more openness about where the patients came from and what they are like. In this vital issue of what to do with the adolescents who are coming into everyone's offices, we sure could use some straight forward reporting of the nature of these experiments. Thanks to Jon, Zach, et al for doing this dissection for us.
I am not a scientist, but am I right in thinking that the lack of transparency alone should be enough to dismiss the meta-analysis as worthless?
What would be a suitable control in Lepp and Barkley (2022)? Obviously an activity that is worse than getting electric shocks won't do. What is a good placebo? Watching tv (taking out some of the interactivity), or a magazine, or a book?
On a related note, I posted this piece - https://morfmorford.substack.com/p/multi-tasking-r-us
I'm finding this series to be pretty interesting, if only to see the amount of research that's been put in so far to identify a relationship between social media and poor mental health. It's inspired me to write an article asking what would happen if we built our regulation argument with research on harmful design patterns instead of social media as a whole. https://substack.com/home/post/p-153030889
Thanks to social media, psuedoscientists and hacks were bound to come out of the woodwork. Attention is the winning metric nowadays. Dr. Haidt has been preparing for this for a long time. Cheers to him, as always.
In my opinion, it is very unsuprising that a study was produced that show no connection between smartphone use and depression. It would definitely be in the interest of tech corporations to hide this phenomenon. Unfortunately, public libraries are often complicit in this problem, as they promote internet access and computers far more than books. For a practical alternative, read this:
https://swiftenterprises.substack.com/p/great-american-libraries
Having worked at the interface of business and science, in nutritional/health research and food marketing, I know how corrupt such research can be.
Based on my read here; there appears to be (potentially) motivated “errors” to reach (potentially) predetermined conclusions.
Its easy to hide and hard to find when an industry sponsor doesn’t want to be found. Its not like there are laws against sponsoring scientific research.