Luke Muehlhauser recently published 'EA needs consultancies'. This is an example of an EA consulting project. If you're interested in consulting services from me, feel free to get in touch at: H@EA.do

Our process

This report was written in roughly 20 days of desk research in early 2020. The evaluation was partly based on a prototype beta version of Mindease and pilot data on the effectiveness of it dating back even further. Other parts of our analysis assume a more idealized and further optimized version of Mind Ease. Mindease's team is constantly working to improve the app and has for instance added an additional intervention since this review was conducted and made lots of other improvements. Since the review, the Mind Ease team reports to have 'additional evidence that was not evaluated in this review, in particular: (1) lots of app usage data from our actual users with information about how much they improve, and (2) a randomized controlled trial that we conducted early on in the life of the project that we used to narrow down our set of planned interventions to use in the app.'

Acknowledgements

This work was funded by the Total Portfolio Project. Thanks to Derek Foster from Rethink Priorities and the Total Portfolio Project for their help in improving this manuscript. All opinions are mine.

Executive Summary

This report evaluates the social impact of Mind Ease, an app to reduce anxiety. The main goal of this report is to estimate the number of disability adjusted life years (DALYs) averted per user. This is to inform a separate analysis on whether to make an impact investment.

Our key question here was: how large of a counterfactual social impact does Mind Ease have?

To answer this, we tackled the following four related questions:

As a cause area, is anxiety a large and neglected problem?
Are Mind Ease exercises based on evidence-based psychotherapeutic interventions that are effective in an offline context?
Is there evidence from the scientific literature that anxiety can be effectively reduced if these offline psychotherapeutic interventions are ported over to mental health apps?
Is there evidence that Mind Ease in particular is effective in reducing anxiety?

In chapter 1, we briefly consider anxiety as a cause area. In brief, we find that the scale of untreated anxiety is relatively big and neglected.

We find that more than 1% of all ill-health and death—27 million disability adjusted life years—were caused by anxiety. This is similar to the global burden of violence. Globally, 284 million people—3.8% of all people—have anxiety disorders. Other estimates suggest that this might be even higher: according to the CDC, 11% of U.S. adults report regular feelings of worry, nervousness, or anxiety and ~19% had any anxiety disorder in the past year according to the NIH and Anxiety and Depression Association of America.

Mental health in general has been described as a 'truly neglected area of global health policy'. The net present cost of scaling up anxiety treatment to an adequate level from 2016–30 has been estimated to be $56 billion. This could perhaps be seen as the overall value of Mind Ease's market size that could be 'disrupted'. The expected returns of scaled-up treatment would be 6 million extra years of healthy life valued at $52 billion. Given that this is the case, the average cost-effectiveness of anxiety treatment might be quite high already (roughly $10k per QALY gained=$56 billion / 6 million years of healthy life), it is plausible that there are even more highly cost-effective interventions to treat anxiety. Scaled-up treatment might also increase labour productivity by $169 billion, leading to benefit cost ratios 3 (or 4, when the value of health returns is also included).

In chapter 2, we review the scientific literature on the psychotherapeutic interventions Mind Ease tries to reduce anxiety with (e.g. Cognitive Therapy, Progressive Muscle Relaxation).

On the whole, Mind Ease's current ten interventions seem carefully selected to be evidence-based. Though the scientific literature in clinical psychology often has poor methodology, many of Mind Ease's interventions are backed up by relatively substantial evidence from systematic reviews and meta-analyses, and, where those are unavailable, plausible other evidence exists that suggest that the interventions are beneficial. The interventions usually have small to moderate effects in an offline therapeutic context, but these effect sizes might be reduced when the interventions are implemented in an app.

In chapter 3, we review this literature on mobile health (mHealth), a burgeoning field in health where apps are used to treat illness effectively at scale. In particular, we focus on mental mHealth to treat anxiety. We find that there is some emerging evidence that the effects of the interventions reviewed in the previous chapter transfer to an online context and that apps can reduce anxiety, but likely with a smaller effect size. We find that there is some evidence from systematic reviews and meta-analyses that apps, especially evidence-based apps like Mind Ease, can indeed reduce anxiety, but the effect sizes will likely be smaller than in an offline context.

In chapter 4, we evaluate direct evidence for Mind Ease's efficacy such as user reports, internal self-evaluation, and their internal study. We find that results from internal studies are promising and perhaps hint at Mind Ease being effective to reduce anxiety. However, there are several methodological concerns and only one study outcome is useful for comparing it to other health interventions. We suggest an ideal trial to test Mind Ease's effectiveness going forward. Taken together with user reports and the evidence reviewed in the previous chapters, it is plausible that the usage of an optimized future version of Mind Ease might significantly reduce anxiety for some people—also with non-trivial effect sizes. We also review the evidence of competitor apps and find they might also be quite effective at reducing anxiety, which could reduce Mind Ease's counterfactual impact.

Chapter 1-4 served as priors for chapter 5, where we analyse the cost-effectiveness of Mind Ease in terms of quality adjusted life years gained (or disability adjusted life year averted) per additional user.

In a final literature review we find that, generally, studies often show that mental health apps have very high cost-effectiveness. This is due to their zero marginal cost per user, compared to the high cost of conventional psychotherapy, which is highly-skilled labor intensive. We also review what the size of the quality of life gains / reductions in disability weights of reducing anxiety in other studies are. We use these to feed them into our cost-effectiveness analysis.

Ultimately, we find that the long-term benefits of anxiety reduction of consistent long-term usage of an optimized future version of Mind Ease targeted at moderately anxious populations could conceivably have a benefit of 1 DALY averted per user. However, due to some inherent uncertainty in the data, estimated conservatively it might be as low as 0.002 DALY averted per user or as high as 6.11 DALY averted per user in an optimistic scenario. To counterfactually adjust for the fact that users might find other treatment, through apps or other forms of therapy, our best guess is that Mind Ease counterfactually averts 0.25 DALY per user. However, again due to the inherent uncertainty in the data, the counterfactual might be as low as 0.0001 DALY per user under conservative assumptions, and 4.07 DALY per user averted under optimistic assumptions.

For a crude comparison, the Against Malaria Foundation has a cost-effectiveness of roughly $50 per DALY averted (this figure might be somewhat out of date, but should be roughly correct). This is because a death of an under 5-year-old is equivalent to ~34 Years of Life lost (YLL) per AMF death. One of the most effective global health charities—the Deworm the World Initiative—roughly averts a DALY equivalent for $14, while GiveDirectly, the philanthropic benchmark averts a DALY equivalent for roughly $860. This means that if Mind Ease can reach ~4 users for less than $50 (or $12.5 per user), and avert 0.25 DALYs in each, then it could be as cost-effective as AMF. The promise of mobileHealth (mHealth) is that at scale apps often have 'zero marginal cost' per user (much less than $12.50) and so plausibly are very cost-effective. One can calculate the cost-effectiveness of an impact investment differently, and this conservative 'societal perspective' includes the cost to the user and assumes a philanthropic subsidy. In contrast, if investing in Mindease returns profits above market rate in expectation, if we only look at the cost to the government and the philanthropist, the cost-effectiveness of MindEase might even be 'negative', i.e. they might save users, the government, and the philanthropist money.^[1]

We close with some qualitative arguments in the form of crucial considerations. For instance, funding Mind Ease might have benefits beyond its direct impact, through the value of improved global mental health and the value of information of researching what works in mental mHealth. We provide some toy models showing that Mind Ease's research might plausibly have even higher cost-effectiveness than its direct impact if it were to improve the field of anxiety treatment as a whole.

Academic Experts that could be consulted

We came across many scientists that have an academic focus on how to use apps to improve mental health. Most notably:

1. Cause area: Anxiety

What is the problem?

Anxiety disorders are

a group of mental disorders characterized by significant feelings of anxiety and fear. Anxiety is a worry about future events, while fear is a reaction to current events.[2] These feelings may cause physical symptoms, such as increased heart rate and shakiness.^[2]

There are several anxiety disorders such as phobias but here we focus on generalized anxiety disorder, as well as trait anxiety below the clinical threshold.

Generalized anxiety disorder (GAD) is an anxiety disorder characterized by excessive, uncontrollable and often irrational worry about events or activities. This excessive worry often interferes with daily functioning, and sufferers are overly concerned about everyday matters such as health issues, money, death, family problems, friendship problems, interpersonal relationship problems, or work difficulties. Symptoms may include excessive worry, restlessness, trouble sleeping, feeling tired, irritability, sweating, and trembling.^[3]

Taken from^[4]

One review describes anxiety symptoms as follows:

All of these conditions are typically characterized by states of hyper-arousal, cognitive beliefs that focus on risk and danger, and excessive fear and worry, all of which are symptoms that allow anxiety to be distinguished from other psychopathologies (Olthuis et al., 2016). Anxiety symptoms have a debilitating impact on wellbeing, quality of life and general functioning, and involve considerable costs to individuals and to society at large (Simpson et al., 2010).^[5]

Psychiatrist Scott Alexander writes:

[...] it's hard to describe how disabling anxiety can be. A lot of people with nominally much worse conditions – depression, bipolar, even psychosis – will insist that they want their anxiety treated before anything else, because they can live with the rest. On the other hand, while a lot of people with psychosis have enough other problems that treating the psychosis barely puts a dent in their issues, a lot of people with anxiety would be happy and productive if they could just do something about it.'^[6]

How important and neglected is it?

Importance- what's the scale of the problem? How large is the group it affects and how badly does it affect them?

Anxiety features prominently in any discussion of mental health. An excellent recent report by Founders Pledge reviews Mental Health more generally as a cause area –- the interested reader is referred to this. Here we will focus solely on anxiety.

In 2017, the global disability burden of disease study found that more than 1% of all ill-health and death, 27 million disability adjusted life years, were caused by anxiety – similar to the global burden of violence and drugs.^[7]

Globally, 3.8% [2.5-7%] of people have anxiety disorders- this amounts to 284 million people.^[8]

A 2012 systematic review and meta-analysis finds that this is even higher and that the global current prevalence of anxiety disorders was 7.3% (4.8–10.9%).^[9]

Anxiety is a highly prevalent condition, with lifetime rates for its derived mental disorders between 14.5% and 33.7% in Western countries (Alonso and Lepine, 2007; Kessler et al., 2012), and global estimates across countries between 3.8% to 25.0% (Remes et al., 2016).^[10]

Many more might have trait social anxiety which is not quite clinical yet still causes suffering. Indeed, trait social anxiety may have evolved to protect our ancestors from social threat.^[11] Similarly, generalized anxiety might have evolved to protect us from other threats. Thus, anxiety might be natural and very widespread.

Mind Ease writes that

'We very roughly estimate that about 34% of people [in the U.S.] suffer from substantial anxiety. According to the NIMH, 'An estimated 19.1% of U.S. adults had any anxiety disorder in the past year.' However, it is clear that more people suffer from substantial anxiety (causing considerable suffering) than would meet the clinical cutoff criteria.'

It is unclear how ME arrived at the 34% guesstimate based on the study. The source they cite already includes 'people with any anxiety disorder that experienced mild impairment' as opposed to only counting moderate or severe impairment. Thus, there are perhaps fewer hidden cases of 'substantial anxiety causing considerable suffering'. The study is also an estimate of total prevalence.

However, the study is based on data from 2007 and depression (and probably anxiety) has increased substantially since then:

One study concludes that 'Rates of major depressive episode in the last year increased 52% 2005–2017 (from 8.7% to 13.2%) among adolescents aged 12 to 17 and 63% 2009–2017 (from 8.1% to 13.2%) among young adults 18–25. Serious psychological distress in the last month and suicide-related outcomes (suicidal ideation, plans, attempts, and deaths by suicide) in the last year also increased among young adults 18–25 from 2008–2017 (with a 71% increase in serious psychological distress), with less consistent and weaker increases among adults ages 26 and over.'^[12]
A 2019 analysis of large-scale, nationwide surveys in US college student found that from 2007 to 2018 rates of depression, anxiety, nonsuicidal self-injury, suicidal ideation, and suicide attempts markedly increased, with rates doubling in many cases.^[13]
A recent critical literature review also concludes that there is a large and consistent increase in adolescent mood disorders especially amongst girls since 2010, likely due to smartphone use.^[14]
Psychiatrist Scott Alexander writes that anxiety disorders are the most common class of psychiatric disorders, yet being also among the least recognized and least treated. He suggests that recently depression has become more socially accepted, but anxiety has not ('it isn't just being sad, and you can't just turn your frown upside down'), but the most common response to anxiety disorders is still 'Anxiety? So what, everyone gets that sometimes.').^[15]
What are the macro drivers of anxiety? If we run into economic depression will there be more anxiety? Unemployment? Is it a hedge against society becoming depressed during a depression?

Economic crises excerbate anxiety- one systematic review concluded 'unemployment, increased workload, staff reduction, and wages reduction were linked to an increased rate of mood disorders, anxiety, depression, dysthymia, and suicide.'^[16]
Similarly, another systematic review of 101 papers showed that ' recessions and mediators such as unemployment, income decline, and unmanageable debts are significantly associated with poor mental wellbeing, increased rates of common mental disorders, substance-related disorders, and suicidal behaviours.'^[17]
One paper on research priorities for the COVID-19 pandemic found that 'The general population survey, done by Ipsos MORI, revealed widespread concerns about the effect of social isolation or social distancing on wellbeing; increased anxiety, depression, stress, and other negative feelings; and concern about the practical implications of the pandemic response, including financial difficulties. '^[18]
One paper^[19] even highlights the importance of digital mental health during the COVID-19 pandemic and argues that increased investments in digital health today will yield unprecedented access to high-quality mental health care. They further argue that apps can soon play a larger role.

Figure 2 from^[20]. Bending the curve further on access and quality of care will require increased efforts around safety, evidence, engagement, outcomes, and implementation. However, these increased efforts will yield greater returns at each step. The COVID-19 crisis has (at least temporarily) removed implementation barriers to synchronous telehealth through regulatory changes, and the evidence, safety, and engagement were already in place before. The next steps to use apps toward asynchronous telehealth will require continued effort but yield even greater increases in access to high-quality care.

However, some argue that anxiety is often justified and as such, anxiety is overdiagnosed. For instance one study suggests that high anxiety seen in poor mothers is caused by poverty itself, not mental illness. Because anxiety is a natural response to poverty, 'assessment and interventions should be targeted at the environmental level and diagnostic labels should be used judiciously.'^[21]

Neglectedness: who else is working on this? What sorts of activities do they fund?

The Happier Lives Institute argues that mental health generally is a neglected problem-especially globally:

One third of Lower and Middle Income Countries do not have a designated mental health budget,[11] and for those that do the average expenditure is 0.5% of their total health budget.[12] In such countries, the treatment gap for mental health (i.e. the number who don't get treatment as a percentage of those who need it) is 76-85%.[13] A Centre for Global Development report describes mental illness as a 'truly neglected area of global health policy'.[13] What this implies is that there may be many high-impact ways to improve mental illness that are not (yet) being pursued.^[22]

Anxiety in particular is also very neglected: The World Mental Health (WMH) survey collected data in 21 countries and 51,547 respondents and found that almost 10% had an anxiety disorder, of which not even a third received any treatment, and of which only 10% received possibly adequate treatment. Further only 41% of people with anxiety disorder even perceived a need for care.^[23]

The table on the next page is adapted from a recent paper in the Lancet on 'Scaling-up treatment of depression and anxiety: a global return on investment analysis'. They found large treatment gaps and modelled the global return on investment to estimate treatment costs and health outcomes in 36 countries between 2016 and 2030.

The net present cost of scaling up anxiety treatment from 2016–30 was estimated to be $56 billion. This could perhaps be seen as the overall value of Mind Ease's market size that could be 'disrupted'.

The expected returns of scaled-up treatment were 6 million extra years of healthy life valued at $52 billion economic value.

Scaled-up treatment would also increase labour productivity by $169 billion with benefit cost ratios 3 or 4, when the value of health returns is also included.

Table 1 Current and target levels of scaled-up treatment coverage for anxiety disorders (all interventions combined), by country income level * Source: Lancet 'Scaling-up treatment of depression and anxiety: a global return on investment analysis'
	Current coverage	Target coverage	Current gap	Reduced gap	% gap reduction
Anxiety disorders
Low-income countries	5%	20%	95%	80%	16%
Lower middle-income countries	10%	30%	90%	70%	22%
Upper middle-income countries	15%	35%	85%	65%	24%
High-income countries	20%	40%	80%	60%	25%
* Treatment coverage was modelled to increase from current to target rates linearly.

Table 2 Costs and benefits of scaled up treatment of depression and anxiety disorders, 2016–30
	Low-income countries (N=6)	Lower middle-income countries (N=10)	Upper middle-income countries (N=10)	High-income countries (N=10)	All countries (N=36)
Total population of countries analysed (millions, 2013)	443	2,215	2,101	992	5,751
Total investment (net present value, US$ millions)	$304	$3,797	$8,966	$42,668	$55,735
Average annual investment (net present value, US$ per person)	$0.1	$0.2	$0.5	$2.4	$0.9
Health returns (averted prevalent case)	$3,395,363	$1,659,719	$12,980,180	$12,077,053	$45,052,316
Health returns (healthy life-years gained)	416,232	2,220,716	1,711,767	1,604,069	5,952,783
Economic returns (US$ millions)	$824	$11,578	$26,691	$129,705	$168,797
Value of health returns (US$ millions) *	$181	$2,966	$8,453	$40,409	$52,009
Benefit cost ratio (economic returns)	3	3	3	3	3
Benefit cost ratio (economic and value of health returns)	3	4	4	4	4
* Healthy life-years gained multiplied by GDP per person multiplied by 0.5.

Neglectedness

Mental health has historically been neglected as a cause area by governments and philanthropists, but in recent years there have been encouraging signs of improvements.

For instance:

In January 2020, the Wellcome Trust pledged $258.8 million over five years to support 'efforts to advance understanding of basic mechanisms of mental health and improve treatments for depression and anxiety, including what works and why, and how best to tailor treatments to individuals who need them.'^[24]
Ted Stanley gave over $800 million to the Broad Institute for research in this area. The Fath and Lindner families made a joint $75 million commitment—to be given over their lifetimes—to the Lindner Center of HOPE for mental health care.^[25]
The Arnold foundation has recently focused on mental health as a cause area.^[26]

2. Literature review: Evidence-base for Mind Ease interventions

Summary

In this section, we review the literature of the efficacy of Mind Ease's interventions to reduce anxiety, depression and improve well-being.

The key question is the extent to which Mind Ease selects evidence-based interventions. Much of the evidence comes from the offline, non-app context, because the evidence base for mobile health (mHealth) is smaller. In the next section we will then review the more sparse evidence on how far the evidence for these interventions translates to mHealth.

We also highlight effect sizes to inform how large of an effect Mind Ease's interventions might be for the cost-effectiveness analysis in a later section.

Mind Ease's interventions can be broadly categorized in two forms:

Cognitive (behavioural) therapy techniques (e.g. Cognitive Therapy, Gratitude Listing).
Breathing and meditation activities (e.g. Progressive Muscle Relaxation, Mindfulness Meditation)

We will review the evidence base and effectiveness of each of these interventions in turn.

We will also review whether these interventions

reduce depression and suicide
are culturally specific to western countries, or could conceivably be scaled globally

Briefly, there is relatively good evidence that Mind Ease's interventions work, that they can work in the context of an app, and the average effect size is 0.3.

2.1 General effectiveness of clinical psychology and psychotherapy

Generally, clinical psychology (i.e. treating mental health conditions without medicine) is effective at treating conditions such as anxiety.

This is despite the fact that much biomedical science, of which clinical psychology is a subdiscipline, often has poor methodology. For instance, just 22% of taxpayer-funded trials comply with mandatory government registry of clinical studies^[27] and half of all clinical trials remain unpublished,^[28] leading to publication bias. Clinical psychology, psychotherapy and psychiatry research generally seem to have even more methodological problems than medicine. ^[29], ^[30], ^[31]

Examples of methodological problems within clinical psychology research are: ^[32]

Small sample size and power
diagnostic unreliability
publication bias
questionable research practices
variable adherence to registration guidelines
gaps between diagnostic assessment in the field and in research settings
and generalizability of randomized control trials to field and community settings

Psychotherapy also has some inherent difficulties that make it harder to study than medicine. For instance, it is difficult to conduct randomized double-blind placebo controlled trials in psychotherapy, ^[33] because therapists cannot be blinded and often not randomized to the treatment they administer. Studies in this area also mostly rely on self-report, which has many drawbacks, but can ultimately be considered a valid and useful measure.

Thus, the evidence in psychological therapy or drug treatment for mental health disorders is generally lower than in medicine. For instance, a widely-cited 2008 meta-analysis suggests that antidepressants, a very common treatment, mostly do not work compared to placebo.^[34] Other papers cite studies that question whether psychotherapy is very effective at all.^[35]

We're not going to evaluate this particular claim in detail – rather it is illustrative that clinical psychology is a softer science than biomedicine. In other words, it would surprise us much more to see a meta-analysis suggesting that chemotherapy does not actually cure cancer than the above meta-analysis suggesting psychotherapy does not actually cure depression.

A 2015 review on replication and contradiction of highly cited research papers in psychiatry looked at 83 articles recommending effective psychiatric interventions and found that '40 had not been subject to any attempt at replication, 16 were contradicted, 11 were found to have substantially smaller effects and only 16 were replicated. The standardised mean differences of the initial studies were overestimated by 132%.'^[36]

To address the methodological concerns of clinical psychology here as much as possible, we try to avoid looking at individual studies and focus mostly on meta-analyses and systematic reviews of randomized controlled trials. These are the 'gold standard' in terms of evidence – though of course they are far from perfect. As one discussion of CBT noted: 'large biases and high rates of false positives will often be found by conventional meta-analysis methods. Nonetheless, the routine application of meta-regression analysis and considerations of practical significance largely restore research credibility.'^[37]

However, especially recently, many methodologically rigorous meta-analyses have been published. And in fact, effect sizes for psychotherapy treatments are similar to pharmacological treatment.^[38]

Moreover, CBT is generally considered the gold standard in psychotherapy despite its limitations: it is the most researched form of psychotherapy and if there are systematic differences between psychotherapies, they typically favor CBT.^[39]

Effect sizes

A review of meta-analyses of pharmacotherapy and psychotherapy for adult psychiatric disorders finds effect sizes are usually roughly 0.5.^[40] What does this mean?

The average person in the treatment group has outcomes 0.5 standard deviations above the average person in the control group. Or: they have a higher score than 49% of patients in the control group. To visualize an effect size in psychotherapy: imagine assigning two groups of 20 people each randomly to have their happiness measured in the morning and afternoon. If people are happier in the afternoon and the effect size is 0.8, then the average person in the 'afternoon' group (i.e. ranked 10th/20 in the group) will score higher than 80% in the 'morning' group-- 4th (16/20=0.8).^[41]

Cohen's d and Hedges' g are both measures of effect size that are interpreted in a similar way. Cohen suggested using the following rule of thumb for interpreting results:

Small effect (cannot be discerned by the naked eye) = 0.2
Medium Effect = 0.5
Large Effect (can be seen by the naked eye) = 0.8

Note though that a 'small' reduction in suicide rates is invaluable, whereas 'small' weight loss may be meaningless.^[42] This is relevant for our purposes here: though it is not inconceivable that such a simple intervention as an app can have a large effect size, we think it is quite unlikely, if the 'market for ideas' is only ever so efficient. In other words, if there were simple exercises that robustly reduced anxiety by a lot, then people with anxiety would tell others about it and it would become 'common sense' (this is the case with much CBT advice, and might be why CBT's effectiveness has declined).^[43]

Further, if one third of users are 'responders' and the app helps them with an effect size of 1.0 (very large and impressive), and the other two-thirds of users are non-responders and the app does not work for them at all, then the effect size will be only 0.33 (one-third of patients get effect size of 1, two-thirds get effect size of 0).^[44] This is often seen in studies on antidepressants.^[45]

A study of clinical interventions in medicine and psychiatry across a range of domains indicates a median observed effect size of 0.37 for general medicine and 0.41 for psychiatry. A review of more than 7000 patients treated with antipsychotics in randomized clinical trials showed that the effect size of the newer atypical antipsychotics, compared with placebo, was 0.48. Finally, studies of schizophrenia clinicians show that they consider an improvement from baseline (which includes placebo response plus drug response) equivalent to an approximate effect size of d = 1.0 to be clinically significant.^[46]

Even skeptical critics acknowledge that psychotherapy usually has at least a small and clinically relevant effect size, especially cognitive behavioural therapy and when compared to patients on the waitlist.^[47] This is perhaps a relevant comparison for our purposes here given that many people who would benefit from therapy cannot get any at all (but see ^[48],^[49] for a discussion). Similarly, in a recent meta-analysis, most trials on CBT for social anxiety disorder, generalised anxiety disorder and panic disorder used a waiting list control group.^[50]

Yet, given that an app often has zero marginal cost per additional user at scale, even small effect sizes might be very cost-effective.

Recent papers suggest that publication bias and selective reporting of positive results leads to exaggerated effect sizes in psychology meta-analyses by a factor of three. ^[51], ^[52] We will try to account for that in our cost-effectiveness model.

In sum, while the methodology in clinical psychology is often poor and the effect sizes small, the evidence seems to suggest that psychotherapy can sometimes lead to clinically meaningful effects.

2.2. Cognitive behavioural therapy techniques

Cognitive Therapy

Mind Ease uses cognitive therapy as a direct 'Calm Me' exercise in its activities section ('Know your triggers', 'Know your early warning signs', 'Reassuring messages to self' are all CBT exercises).

Many different meta-analyses show that cognitive therapy (identical to certain aspects of cognitive behavioural therapy) is effective to reduce anxiety.

A 2016 meta-analysis shows that CBT in Generalized Anxiety Disorder (GAD) has an effect size of 0.59. However, because few trials were high‐quality these effects are still uncertain.^[53]
A 2018 meta-analysis showed the effect size of placebo-controlled CBT on other anxiety symptoms was 0.38 and 0.30 on quality of life. Interventions primarily using exposure strategies had larger effect sizes than those using CT or CBT, though this difference did not reach significance.^[54] Because whole books have been written on the behavioural aspect of CBT for anxiety,^[55] and Mind Ease only features cognitive therapy, this might reduce Mind Ease's effectiveness compared to CBT, because as far as we understand it cannot easily implement behavioural therapy aspects into an app.
Of 69 randomized clinical trials (4118 outpatients) that were mainly of low quality, cognitive behavioral therapy compared with control conditions was associated with improved outcomes after treatment completion, at 1 to 6 months, and at 6 to 12 months of follow-up for a generalized anxiety disorder (g=0.07-0.40). After 12-month follow-up, these associations were still significant (g=0.22). Relapse rates after 3 to 12 months were 0% to 14% but were reported in only 6 trials.^[56]
A 2016 meta-analysis showed that behavioural interventions can increase psychological well-being (d=0.44) and after two to ten months, the effect size is small but still significant (d=0.22).
A 2014 meta-analysis looking at the effect of CBT for anxiety disorders on quality of life showed relatively strong effects (g=0.55), though face-to-face therapy had significantly larger effect sizes than internet-delivered treatments.^[57]

In sum, cognitive therapy seems effective with small to medium effect sizes.

Positive Reappraisal

Positive Reappraisal (or cognitive reappraisal) is a CBT technique. A 2018 systematic review found that social anxiety is characterized by ineffective utilization of cognitive reappraisal.^[58]People who are depressed do not use cognitive reappraisal enough, which may be particularly detrimental in stressful situations. Cognitive reappraisal is important for CBT to reduce anxiety. For depression and anxiety, treatment intervention appears to address deficits in reappraisal: one RCT showed that twelve hours of positive reappraisal training reduces depressive symptoms more than psychoeducation-based interventions (d=0.46–0.50).^[59]

Overall, though there is not very much published literature on positive reappraisal, and though we cannot conclusively say whether it is effective, it seems somewhat evidence-backed and a promising technique.

Dare Response

There is no published academic literature on the dare response. However, it is a CBT-inspired technique and the book that first introduced the concept has sold 200,000 copies,^[60] has 4.26 out 5 stars based on 1,628 ratings and 162 reviews on Goodreads^[61] and 4.7 out of 5 stars based on 1,363 ratings on Amazon.

In a nutshell, this anecdotal, qualitative evidence suggests that the intervention is somewhat unlikely to be harmful, but there is insufficient evidence to say it is effective.

Visualisation (positive imagery)

Boosting positive mental imagery may provide a useful adjunct to CBT – which tends to focus on negative information processing – by encouraging vivid, field-perspective, positive mental imagery and by promoting imagery-based processing of positive information.^[62]

There is some evidence for this:

A 2017 systematic review and meta-analysis looking at autobiographical episodic memory-based training – of which positive visualization is one type – for the treatment of mood, anxiety and stress-related disorders found that it reduces depression (d=0.32; with variation in effect sizes from-0.18 to 1.91 across different protocols). There is also some evidence that it is beneficial for anxiety disorders.^[63]
Computerised positive imagery cognitive bias modification (CBM) can be delivered remotely and/or in combination with other Internet-delivered treatments such as Internet-delivered CBT. However, research in imagery CBM as an intervention is at an early stage, and findings are mixed. In an RCT with depressed participants, CBM resulted in no greater reduction in symptoms of depression than a control sham CBM intervention.^[64], ^[65]

In conclusion, visualization (positive imagery) seems effective, but with a relatively small effect size, especially when it is internet based.

Gratitude Listing

Gratitude listing is one the main interventions in the field of positive psychology.

Generally, meta-analyses suggest that positive psychology interventions (PPIs) are effective:

A highly cited 2009 meta-analysis suggests that positive psychology increases well-being (d = 0.61) and depression (d = 0.65).^[66] A different meta-analysis from 2013 shows smaller effect sizes for subjective well-being (d = 0.34), psychological well-being (d = 0.20), and depression (d = 0.23).^[67]
These analyses have been heavily criticised.^[68] A more recent 2019 meta-analysis that takes this criticism into account shows that PPIs on well-being were small but significant (approximately d = 0.20), whereas the effect of PPIs on depression are generally not statistically significant.^[69]
A 2020 meta-analysis of randomised controlled PPIs on subjective and psychological well-being, involving 16,085 non-clinical participants, showed that PPIs increase well-being (d=0.23, but it was 0.08 for psychological well-being, 0.22 for subjective well-being, and 0.43 when the studies targeted both types of well-being). Longer interventions showed stronger immediate effects than shorter ones, and interventions based on traditional methods were more effective than those that used technology-assisted methods (d=0.17). There was also evidence of long-term effects of the interventions.^[70]

We believe the more recent, more comprehensive, and more critical meta-analysis, and thus think that the evidence suggests that there is a small but significant effect on subjective well-being of roughly 0.2.

In terms of gratitude specifically, a recent qualitative review casts doubt on the efficacy of gratitude interventions, suggesting the need to carefully attend to the quality of comparison groups.^[71]

Looking specifically at the evidence for gratitude we find the following:

A 2019 meta-analysis of 38 articles concluded that gratitude can lead to small boosts in positive affect, well-being, happiness, and life satisfaction, while decreasing depressive symptoms. However, the effects are small (rather than life changing), and limited. Gratitude is no more effective at promoting well-being than other positive interventions. Gratitude interventions had small to medium effects (ds = 0.13 to 1.23), helping people to increase short-term well-being, happiness, life satisfaction, grateful mood, grateful disposition, positive affect, and depressive symptoms, with some long-term effects, but not health, sleep, exercise, prosocial behavior, or self-esteem.^[72] The authors argue that because gratitude is easily accessible and can be practiced for free at home, they are excellent candidates for far-reaching impact. A different recent meta-analysis also finds no evidence of gratitude interventions on health.^[73]

Overall, gratitude listing seems effective with small effect sizes.

Anxiety defusion and acceptance (acceptance and commitment therapy)

Mind Ease's anxiety defusion exercise is based on acceptance and commitment therapy (ACT), which is backed by the following evidence:

Traditional ACT with a therapist:

A 2017 review of RCTs of ACT to treat anxiety and depression shows that ACT improves depression relative to no treatment up to 6-months follow-up. (ds = 0.32 to 1.18). Two studies compared ACT with minimally active comparison conditions (expressive writing and minimal support group) and found ACT outperformed comparison conditions on depression at post, but were equivalent at follow-up.^[74]
A 2020 meta-analysis of 18 studies with 1,088 participants showed that ACT significantly reduced depression as compared with the control group (d= 0.59, 95% CI [0.38, 0.81]).^[75]

Self-help: Traditionally face-to-face, ACT is also delivered in self-help formats. A meta-analysis shows that ACT self-help showed significant small effect sizes favoring intervention for depression (g=0.34; 95% CIs [0.07, 0.61]; Z=2.49, p=0.01) and anxiety (g=0.35; 95% CIs [0.09, 0.60]; Z=2.66, p=0.008). Higher levels of clinician guidance improved outcomes but intervention format (e.g. book/computer) was unlikely to moderate results.^[76]
Internet-based ACT (iACT): A systematic review of internet-delivered ACT (iACT) for anxiety^[77] showed that 18 out 20 studies reported significant anxiety reduction after treatment. This was observed in studies that delivered iACT with (n=13) or without (n=5) therapist guidance. The average attrition rate during treatment was 19%. In 13 studies participants on average rated their iACT experience with above average to high treatment satisfaction.
App-based ACT: A recent RCT of ACT in an app form showed that help-seeking individuals vs. waitlist increased well-being with moderate effect sizes.^[78]

In aggregate, anxiety defusion and acceptance (acceptance and commitment therapy) seems effective with small to medium effect sizes.

Reflective writing

The evidence for reflective writing is as follows:

A 2018 meta-analysis with 39 RCTs found that brief, self-directed expressive writing did not decrease depressive symptoms. However, effects were larger with more sessions and more specific writing topics.^[79]
A 2019 meta-analysis of 29 studies found small effects for posttraumatic stress and negligible to small effects for posttraumatic growth. In some studies expressive writing even decreased quality of life (though this was a non-significant negligible to small effect size).^[80]
A 2019 systematic review of expressive writing as a therapeutic intervention for people with advanced disease found that studies had methodological shortcomings and evidence was generally of low quality. The authors found no evidence of expressive writing affecting sleep, anxiety or depression compared to an active control.^[81]

In sum, there is currently not very good evidence for the efficacy of expressive writing, and effect sizes seem small at best.

2.3 Breathing and meditation activities

Mindfulness and breathing meditation

The evidence for the effectiveness of mindfulness meditation is as follows:

A 2014 systematic review and meta-analysis of meditation programs for psychological stress and well-being (which used a definition of meditation that included mindfulness and other techniques based on transcendental and mantra meditation) found that:

'Mindfulness meditation programs had moderate evidence of improved anxiety (effect size, 0.38 [95% CI, 0.12-0.64] at 8 weeks and 0.22 [0.02-0.43] at 3-6 months), depression (0.30 [0.00-0.59] at 8 weeks and 0.23 [0.05-0.42] at 3-6 months), and pain (0.33 [0.03- 0.62]) and low evidence of improved stress/distress and mental health-related quality of life. [...] We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).'
The study included not only populations presenting high anxiety scores but also comorbidities such as depression, chronic pain, stress, insomnia, diabetes and hypertension, among others.^[82]

A 2010 meta-analysis of 39 studies found an effect size of d=0.6 in the general population and d=1 in people with anxiety disorders.^[83]
A 2007 review found no effect on depression and anxiety when active control groups were used.^[84]
A 2019 meta-analysis of 14 RCTs relaxation therapies for people with anxiety found that meditation is a bit more effective than relaxation therapies (effect size g = −0.23) and might also remain more effective at 12-month follow-up.^[85] However, few studies and interventions were of high quality, and allegiance might be moderating results.
Generally, mindfulness meditation has been described as hyped and often suffers from poor methodology.^[86] For instance, one study found considerable reporting bias in the mindfulness literature.^[87]
Moreover, a 2019 literature review of potential harmful outcomes of mindfulness meditation finds that a few studies have shown worsening symptoms due to mindfulness, though meta-analyses consistently report overall benefits for many outcomes. The authors argue that this is an under-researched topic and cite two studies in which a few participants reported increased anxiety during meditation (but no significant worsening of symptoms from pre- to post treatment).^[88] Others have reported challenging and adverse meditation experiences, but mostly when people meditate for a long time.^[89] Our sense that the risk of mindfulness causing harm, especially during a very brief meditation session as implemented in the Mind Ease app, is relatively low.

In sum, mindfulness meditation seems to be effective but the reported effect sizes are likely inflated and so we believe them to be small.

Breathing visualization / deep breathing

The evidence for the effectiveness of breathing visualization and deep breathing is as follows:

Breathing techniques have been proposed as firstline and supplemental treatments for stress, anxiety, depression, and some emotional disorders as they decrease the body's fight-or-flight stress response, and help with mood stabilization, anger control, and anxiety management.^[90]
One study^[91] reviews the 'Breathe2Relax mobile app', a stress-management tool that employs hands-on diaphragmatic breathing exercises. Breathe2Relax is possibly the most popular psychological health app currently in clinical and public use: +1.26 million downloads, with 53,371 active users and at least 130,000 uses each month. It underwent rigorous and iterative user experience/human factors testing with active duty military service members throughout design and development. The popularity of this app provides some evidence of the usefulness of breathing exercises, though of course this is not based on randomized studies.
One RCT^[92] tested the efficacy of an mHealth game, 'Flowy', that digitally delivered breathing retraining exercises for anxiety, panic, and hyperventilation symptom management. It found that the intervention did not reduce anxiety, panic, and self-report hyperventilation scores, but did increase quality of life.

However, the authors find that participants found 'Flowy' acceptable as an anxiety management intervention, engaged them sufficiently to endorse proactive gameplay, perceived 'Flowy' as a fun and useful intervention, proactively used 'Flowy' as part of their care, and would recommend 'Flowy' to family and friends.

PE Coach, an mHealth app developed by the US Dept. of Veterans Affairs, also has a breathing retraining tool.^[93]

Hence, while systematic evidence is lacking, one thesis on anxiety management argues that 'yogic breath control and attention to breathing have been employed in emotional somatic control for 5000 years.'^[94] As such, deep breathing exercises seem likely to have some limited efficacy in reducing anxiety.

Progressive muscle relaxation

The evidence for the effectiveness of progressive muscle relaxation is as follows:

A 2018 meta-analysis finds no evidence that relaxation therapies are less effective than CBT for anxiety in the short-term.^[95]
A 2019 meta-analysis^[96] suggests that meditation is only slightly more effective than relaxation for anxiety (g = 0.23 [95% CI: −0.40 to −0.07]), and might also remain more effective at 12-month follow-up, though few studies and interventions were of high quality, and allegiance might be moderating results.
A 2008 meta-analysis concluded that relaxation techniques such as progressive relaxation may have moderate effects (d=0.5).^[97]
A 2010 review supports the efficacy of relaxation training as a valid standalone or combined treatment for anxiety disorders or problems, and suggests a wider use of these techniques in clinical practice.^[98]

In sum, there is relatively good evidence that progressive muscle relaxation reduces anxiety.

Conclusion

On the whole, Mind Ease's ten interventions seem like they are carefully selected to be evidence-based. Though the scientific literature in clinical psychology often has poor methodology, many of Mind Ease's interventions are backed up by relatively substantial systematic reviews and meta-analyses, and, where those are unavailable, plausible other evidence exists that suggest that the interventions are beneficial. The interventions usually have small to moderate effects, and our sense from the literature is that effect sizes might be even further reduced when implemented in an app. Next, we review the mHealth literature more broadly.

Are these interventions culturally specific?

One reservation here is that most studies are conducted in 'Western Educated Industrialized Rich Democratic (WEIRD)' populations.^[99] If Mind Ease could scale to other demographics and poorer countries, perhaps using price discrimination or a 'freemium' model, then this might significantly increase Mind Ease's social impact.

For this to be true, we now look at whether the interventions are culturally specific, that is, only work in WEIRD or simply just 'western context'.

The evidence relating to this is as follows:

A 2020 meta-meta analysis^[100] finds a relatively large amount of evidence supporting psychosocial interventions for various mental health outcomes in low- and middle-income countries, yet the authors caution that the 'strength of associations and credibility of evidence were quite variable, depending on the target mental health condition, type of population and setting, and outcome of interest'.
A 2018 systematic review and meta-analysis found that positive psychological interventions are even more effective in non-western countries than in western countries in terms of subjective (g = 0.48) and psychological (g = 0.40) well-being, depression (g = 0.62) and anxiety (g = 0.95).^[101] In contrast, a 2019 meta-analysis suggests there is a lack of studies on gratitude practices within multi-cultural contexts.^[102]
A 2015 systematic review on online interventions for mental health conditions in low and middle income countries found only three studies on the subject and concluded that no firm conclusion can be drawn regarding its effectiveness.^[103]
A 2016 systematic review and meta-analysis showed that minimally guided interventions (including electronic) can be adapted to different cultures to reduce depression and anxiety (effect size:-0.81 [95% CI-0.10 to-0.62], but that higher cultural adaptation scores were significantly associated with greater effect sizes (P=.04).
A 2016 study suggests that social anxiety symptoms can be effectively reduced by internet-based CBT in Chinese people.^[104]
The WHO is developing technology supported mental health interventions for LMICs. Their 'Step-by-Step' intervention is a guided, technology supported intervention for depression. Similar to Mind Ease, 'it provides psychoeducation and training in behavioural activation through an illustrated narrative with additional therapeutic techniques such as stress management (slow breathing), identifying strengths, positive self-talk, increasing social support and relapse prevention. Step-by-Step has been designed so that it can be adapted for use in settings with different cultural contexts and resource availability and to be meaningful in communities affected by adversity.'^[105] One study showed that in Lebanon, Step-by-Step may be effective in reducing depression and anxiety symptoms and increasing well-being.^[106] Another study demonstrated a culturally appropriate adapted version of the Step-by-Step program for overseas Filipino workers.^[107]
There is also interest in mental health apps for Arabic speakers^[108] and for Indians^[109]- as evidenced by research papers on this topic.

In sum, even if Mind Ease's current interventions do not completely generalize to other non-WEIRD cultural contexts, the experience gained from the western context might still be very valuable when adapting the app to other cultural contexts, which some studies suggest is possible.

Suicide

As discussed above, the effect sizes of interventions Mind Ease's exercises are based on are likely to be small or medium. However, there is some variance within the groups that are studied and so there might be substantial differences between people in terms of how well the app works (which then averages out). Thus, though for some people the app might not work at all, for some people the app might work very well, and at scale, it is not inconceivable that it might prevent suicide. This would substantially contribute to Mind Ease's social impact. We now review the evidence whether this is plausible.

A 2016 meta-analysis found anxiety and its disorders (especially PTSD) is a statistically significant, yet weak, predictor of suicide ideation, attempts, but not deaths.^[110]
A 2018 meta-analysis found small to moderate associations between anxiety sensitivity and suicidal ideation (r = 0.24, 95% confidence interval (CI): [0.21, 0.26], p < 0.001) and suicide risk (r = 0.35, 95% CI [0.31, 0.38], p < 0.001).^[111]
A 2020 review^[112] finds

Anxiety disorders to be the most common class of disorders among people with suicide-related behaviors. Lifetime suicide attempt estimates among people who met criteria for an anxiety disorder ranged from approximately 42.0% for specifc phobia and social anxiety disorder.
Some evidence that CBT for anxiety is associated with reduced suicidal ideation for patients with post-traumatic stress disorder (PTSD) and seasonal affective disorder (SAD).
That patients with co-occurring suicidal ideation and anxiety symptoms found a single-session computerized intervention called the cognitive anxiety sensitivity treatment (CAST) to demonstrate reductions in suicidal ideation at a 4-month follow-up.

Globally there were 817,000 suicide in 2016^[113] and around 7.4% (3%–12.7%) might be attributable to anxiety disorders (equivalent to ~60k deaths globally).^[114]

In sum, there is some limited evidence that anxiety causes suicide and that CBT and even computerized exercises such as those by Mind Ease might reduce suicide. However, even though it is not inconceivable that Mind Ease will prevent a very limited number of suicides at scale, given that there are relatively few deaths due to anxiety globally, the effect on the social impact of Mind Ease will be relatively small and we decided not to model this in our cost-effectiveness analysis.

3. mHealth

In the last section, we showed that there is relatively good evidence that Mind Ease's interventions are evidenced-based and effective, but with a focus on an offline context.

Now, we turn to relevant considerations related to Mind Ease emerging from the scientific literature on mobile health (mHealth) with a focus on mobile mental health, and its effectiveness (effect size). We also highlight relevant findings from adjacent fields, such as internet-based mental health interventions, which likely generalize to apps such as Mind Ease, and findings on depression, which is very often comorbid with anxiety.

General considerations

A 2015 WHO survey of 15,000 mHealth apps revealed that 29% focus on mental health diagnosis, treatment, or support, and the UK's NHS and the U.S. National Institute of Mental Health see apps as cost-effective and scalable solutions to addressing the mental health treatment gap.^[115]
A 2019 US survey found that 44.5% of participants preferred in-person psychotherapy, 25.6% preferred self-guided digital treatment (like Mind Ease), 19.7% preferred expert-guided digital treatment, and 8.5% peer-supported digital treatment.^[116]
A 2020 paper argues that telehealth will increase during the COVID-19 global pandemic, that apps also have an important role given their availability and scalability, and calls for increased investments.^[117]
Of the current 10,000 mental health apps, just four (Headspace, Youper, Wysa, and Calm) account for ~90% downloads – 63% of all depression apps have zero monthly users.^[118]
A 2020 report by Deloitte estimates that poor mental health costs UK employers up to £45 billion a year and that offering mental health support for employees like CBT (including through digital platforms) has high ROI.^[119]
A 2020 critical review^[120] of background issues, current status and future concerns with mental health apps suggests:

In the future, data from apps and wearables, recommended by psychiatrists and selected by patients, might enter electronic medical records to enable immediate feedback to assist patients.
A review of digital self-help apps for depression and anxiety with 8 to 40,000 downloads per month reported 21–88% of users using it at least once, and 1–29% continuing after 6 weeks. Only 19% of a US Hispanic/Latino population enrolled in a depression clinical trial downloaded the treatment app.

Future trends in mental health apps are mobile virtual reality, wearable smartwatches^[121] and gamifying mental health through 'serious games'.^[122],^[123]
A 2019 review of publicly available mental health apps (MH apps) found that certain evidence-based treatment elements were rarely included in MH apps.^[124]
A 2017 systematic review of anxiety apps found that two thirds did not involve health care professionals in their development, and only 3.8% had been rigorously tested.^[125]
A 2019 review of evidence-based apps for anxiety and depression^[126] found 3% of apps had research to justify their claims of effectiveness, with the majority of that research undertaken by those involved in the development of the app. 30% of shortlisted apps claimed to have expert development input; 20% had an affiliation with a government body, academic institution, or medical facility; and 74% were free to download.
A 2020 systematic review^[127] compared empirically-supported internet-based CBT (iCBT) for depression with apps, and found that iCBT usually contained more psychoeducation, cognitive restructuring, behavioral activation, problem-solving, interpersonal communication, and anxiety content, but less mindfulness than apps. Yet, though iCBTs contain evidence-based content but few are available to the public. This suggests that Mind Ease might be one of the more evidence-based apps out there.
A 2019 review of 27 popular apps for depression and anxiety^[128] found that 23 included at least one common evidence-based element, with a median of three elements. Psychoeducation (in 52% of apps), relaxation (44%), meditation (41%), mindfulness (37%), and assessment (37%) were the most frequent elements, whereas several elements (e.g., problem solving) were not found in any apps. This analysis also identified gaps between app content and empirically supported treatments. Cognitive restructuring was more common in depression protocols than in depression apps (75% of protocols vs. 31% of apps), as was problem solving (34% vs. 0%). For anxiety, exposure (85%, 12%), cognitive restructuring (60%, 12%), and problem solving (25%, 0%) were more common in protocols than apps. The authors conclude that empirically supported treatment elements that are poorly represented in current MH apps.^[129] Yet, as shown in the table below many of the top competitor apps, such as Pacifica, have up to 11 (evidenced-based) treatment elements.

A recent narrative review^[130] of various schemes toward mental health app evaluations, including commercial app store metrics, government initiatives, patient-centric approaches, point-based scoring, academic platforms, and expert review systems highlights the American Psychiatric Association app rating framework as the first operational app evaluation framework to be endorsed by a US-based national medical association. In their assessment they give most weight to privacy and safety, followed by evidence, then usability and interoperability. It might be interesting for Mind Ease to complete this assessment (there is a similar evaluation framework by the UK's National Health Service^[131] to get on their app library^[132]).

A 2019 review of popular smartphone apps for depression and anxiety assessing the inclusion of evidence-based content^[133] warns that the vast majority of the limited research to date has been completed by those involved in an app's development. Further independent research and replication are required to demonstrate legitimacy and increase the acceptance of mental health apps as valid sources of therapy.^[134] The authors also propose that mental health apps undergo a new 'certification' process with the participation of app store marketplaces.
Some gaps for specialisation within the app market might exist. For instance, a 2020 systematic review on apps for the management of comorbid overweight/obesity and depression/anxiety found no apps for the simultaneous management of overweight/obesity and depression/anxiety, even though there is a significant association between them and obesity increases the odds of suffering from both anxiety and depression.^[135]

Evidence for effectiveness of mental health apps

A 2019 meta-analysis and systematic review examined 6 studies on apps to reduce depression and 4 on anxiety.^[136] It found apps to have a small effect on reducing depression (g = 0.33) and smoking (g = 0.39), but no significant effects were found for anxiety (g = 0.30, 95%CI −0.1 to 0.7, P = 0.145), alcohol use, and self-injurious thoughts and behaviors. Only when pooling the effects on depression (g = 0.34) and anxiety (g = 0.43), regardless of whether this was the primary aim of the intervention, were effects significant, but heterogeneity between studies was moderate to high. The paper concludes that 'smartphone apps as standalone psychological interventions cannot be recommended based on the current level of evidence'.
However, an unpublished 2020 synthesis of meta-analyses^[137] on standalone apps for anxiety and depression concludes that, though very few studies have been conducted, the extant evidence suggests that apps can significantly reduce anxiety and depression with a medium effect size when compared to no treatment at all, with similar or marginally superior performance to internet-based interventions. They criticized the aforementioned meta-analysis, arguing that

The pooled effect, which has a higher effect size, is more meaningful
Especially for anxiety the confidence intervals of the effect sizes ranged from-0.1 to 0.7, suggesting that there could be very effective interventions
The inclusion criteria were overly restrictive. For instance, a different meta-analysis from 2017 on the same topic included more participants (3,414), because it covered all trials that were reviewed in the newer meta-analysis, plus an additional RCT. The analysis found that both apps and computer interventions reduce depression.^[138] Another 2017 meta-analysis by the same authors also showed that apps reduced anxiety compared to waitlist (g=0.45, 95% CI=0.30–0.61), and even active control (g=0.19, 95% C.I.=0.07–0.31).^[139]
Mental health apps should be compared to no treatment and not to traditional in-person treatment, because most people with depression or anxiety do not get treatment and even if they seek in-person treatment, it is often delayed for weeks or months.^[140]
Some evidence suggests that adding an app to internet based treatment may be beneficial.^[141]
One study found no large differences between an unguided app-based internet intervention compared to a therapist guided app-based internet intervention.^[142]
How the app is designed has a large impact on the efficacy, with more traditional CBT interventions demonstrating promising early efficacy in apps. This is particularly relevant because Mind Ease is heavily based on CBT.

Another 2019 meta-analysis^[143] of 66 RCTs (n=66) on the efficacy of mental health apps concluded that:

Apps outperformed control conditions in improving depressive (g=0.28, n=54) and generalized anxiety (g=0.30, n=39) symptoms, stress levels (g=0.35, n=27), quality of life (g=0.35, n=43), general psychiatric distress (g=0.40, n=12), social anxiety symptoms (g=0.58, n=6), and positive affect (g=0.44, n=6), with larger effects for CBT‐based apps (depression: 0.34 (0.23-0.46), anxiety: 0.42 (0.26-0.57)) and those that offered professional guidance and reminders to engage and mostly even after adjusting for various possible biasing factors (type of control condition, risk of bias rating). But apps were not significantly better than control on panic symptoms (g=–0.05, n=3), post‐traumatic stress symptoms (g=0.18, n=4), and negative affect (g=–0.08, n=5). Also, apps were not significantly different from face‐to‐face or computerized treatment, though there were only a few studies on this.

A 2019 meta-analysis and review on Internet‐and mobile‐based interventions for anxiety disorders^[144] found

Adding apps to treatment for anxiety disorders was effective
Guided apps were superior to completely unguided interventions and improved adherence
The evidence base of Internet treatment and apps for anxiety disorders is improving. This may open numerous possibilities for mental health care and enable the scaling up of services for common mental disorders worldwide.

A 2020 systematic review and meta-analysis on user engagement in mental health apps concluded that all 40 studies that were included in the meta-analysis claimed positive user engagement indicators, but based on very different criteria, which makes it difficult to interpret.^[145]
Effect sizes might be reduced, because people with depression and anxiety especially might not have enough motivation to use a mental health app.^[146] A 2020 systematic review and meta-analysis of dropout rates in RCTs looking at apps for depression^[147] found that

Dropout rates from studies of apps for depressive symptoms is nearly 50% when accounting for bias.
Dropout did not vary between placebo apps and depressive symptoms apps.
Dropout was less for apps offering human feedback and mood monitoring.
High dropout rates present a threat to the validity of RCTs of mental health apps.
It is critical to consider bias when interpreting results of apps for depressive symptoms, especially given the strong indication of publication bias, and the higher attrition in larger studies.

Similarly, a 2019 meta-analysis of individual patient data found that a quarter of all patients do not respond to internet-based CBT, especially those with greater symptoms, anxiety disorders, and men.^[148]
A 2020 meta-analysis found that a tailored, integrative internet intervention can reduce depression, which is highly comorbid with anxiety. It can facilitate clinically relevant reduction of depressive symptoms (g = 0.51, 95% CI: 0.40–0.62) over 8–12 weeks across a broad range of initial symptom severity, and the intervention can be combined with other forms of depression treatment.^[149]
A 2020 meta-analysis concluded that study attrition and low adherence are common, problematic, and may undermine the validity of RCTs of mental health apps.^[150]
A 2018 Cochrane review (generally considered the highest quality systematic reviews) found that internet-based CBT reduced clinically important reduction in PTSD compared to waitlist (d= –0.60, 95% CI –0.97 to –0.24); however, this was based on very low quality evidence (i.e. the studies reviewed had methodological shortcomings) and there was no evidence of a difference in PTSD symptoms when follow‐up was less than six months.^[151]
A 2018 Cochrane review of e‐health interventions for anxiety and depression in children and adolescents with long‐term physical conditions found only very low quality evidence and concludes that e‐health interventions are uncertain at this time, especially in children aged under 10 years.^[152]
A 2018 systematic review and meta-analysis on technology delivered interventions for depression and anxiety in children and adolescents found^[153]

Technology delivered interventions had a small effect compared to a waiting list control group (g = 0.45 [95% CI 0.29, 0.60] p < 0.001), with CBT interventions having a higher medium effect size (n = 17, g = 0.66 [95% CI 0.42–0.90] p < 0.001) than attention bias modification interventions, which had a small effect size (g = 0.41 [95%CI 0.08–0.73] p < 0.01). Cognitive bias modification programs and other interventions failed to demonstrate a significant benefit over control groups.

A 2018 review finds^[154] that efficient mental health apps have high patient engagement, simple user interface (UI) and experience, transdiagnostic capabilities (e.g. watching for comorbidities like depression), and self-monitoring features. Mind Ease seems to do fairly well on most of these predictors, perhaps with the exception of transdiagnostic capabilities, i.e. they seem to focus almost exclusively on anxiety and not depression which is often comorbid.
A 2019 meta-analysis shows that apps that included acceptance and/or mindfulness components increased acceptance/mindfulness (k = 33; g = 0.29; 95% CI = 0.17, 0.41) lowered distress (k = 22; g = −0.32; 95% CI = −0.48, −0.16).^[155]
Finally, a 2019 review of recent meta-analyses of internet interventions for anxiety concludes that a growing number of meta-analyses now suggest that ICBT works and can be as effective as face-to-face therapy.^[156]

In sum, there is some evidence from several systematic reviews and meta-analyses of apps to reduce anxiety.

This effect size should be adjusted both upwards and downwards for Mind Ease's case. For instance, there is evidence that the meta-analyses reviewed here are based on studies with methodological shortcomings such as publication bias and high dropout rate.

The evidence reviewed here relied almost exclusively on self-report and not objective measures, like stress hormone levels. mHealth self-report measures like a subjective slider on a scale from one to five have been criticized as unreliable.^[157] Also, much of the evidence here comes from patient populations and presumably people with relatively severe symptoms.

On the other hand, Mind Ease is evidence- and CBT-based, and there is evidence that those apps are more effective. Also, the studies reviewed here usually only measured relatively short-term outcomes, and it could be that using an app over longer time scales might increase the effect size of Mind Ease's intervention.

We will not attempt a formal quantitative estimation of the true effect size based on the evidence reviewed here. Instead, our qualitative guesstimate from eyeballing these studies is that the effect size is around 0.3 (95% CI =-0.05 to 0.5), which we will use for further analysis. As such we will use the 0.3 effect size as a prior and combine it with Mind Ease's self-evaluation in the next section, which in turn will guide our cost-effectiveness analysis in the final section. In other words, we do not rule out that the true effect size of causing the Mind Ease app to be given to people in the real world is slightly negative, because it displaces other treatment or because the app is unhelpful and leads to distress (see Risks and Reservations section). However, we think this hypothetical negative effect and worst case scenario is very unlikely to be large. We also do not rule out that the effect size is medium (0.5) – there are very many people who report that interventions like CBT or meditation have made a big difference in their life and we could imagine that the effectiveness of apps will turn out to be roughly similar to those of other traditional clinical psychology interventions. However, again we do think this is unlikely that the effect is that large or even larger- given that it seems too good to be true that something as simple as an app could make a massive improvement to quality of life in the large share of all users- without spreading very quickly.

4. Review of Mind Ease

Request access to the specific Review of Mind Ease.

5. Benefits and Cost-effectiveness

Health economic cost-effectiveness analysis

To effectively allocate limited resources among different interventions cost–effectiveness analysis (CEA) is an important and commonly used approach.^[158] In CEA studies generic measures such as quality-adjusted life years (QALYs) and disability adjusted life-years (DALYs) are commonly used as the measure of effectiveness.^[159] The incremental cost-effectiveness ratio (ICER) provides cost per QALY gained or cost per DALY averted compared to the next best alternative.^[160]

The disability-adjusted life year (DALY) is a measure of the disease and disability burden in populations. DALYs are calculated by combining measures of life expectancy with adjusted health during a burdensome disease or disability for a population. Loosely, DALYs are the inverse of the quality-adjusted life year (QALY) measure: in health economics, one tries to avert DALYs and gain QALYs.^[161]

QALYs gained analysis

Studies often find that mental health apps have very high cost-effectiveness. This is due to their zero marginal cost per user, compared to the high cost of conventional psychotherapy, which is highly-skilled labor intensive. For comparison, the UK's National Health Service (NHS) usually aims to fund any intervention that is below $25-$37k per QALY^[162] – though one study suggests the threshold is closer to $21k.^[163]

A 2017 systematic review^[164] of the cost-effectiveness of anxiety interventions found that iCBT was cost-effective compared to control. They review studies that find:

One study in older adults with anxiety finds similarly high cost-effectiveness of iCBT versus waiting list: $6,175/QALY gained. This is in line with the previous RCT.
In patients with OCD

One study showed that iCBT versus internet-based supportive therapy had cost-effectiveness of $947 per relapse avoided; $7.307/QALY gained. This is interesting because it suggests that even if there is a very active control (supportive control) iCBT can be very effective.
Another study showed that additional booster treatment versus no booster treatment after receiving iCBT had a cost-effectiveness of $1,489/relapse avoided.
In a group of patients, iCBT has superior cost-effectiveness ratios in comparison with group CBT.

Pharmacological interventions are usually less cost-effective: for instance, Pregabalin cost $22,590 per QALY gained and $38,670/QALY in another study. However, sometimes antidepressants such as sertraline (Zoloft) can cause the high QALY gain at low cost.

One systematic review of 6 economic evaluations of internet- and mobile-based interventions (IMI) against depression found costs per clinically significant change in depressive symptom severity (or per depression-free years in the preventive IMI) ranged from €233 to €4,030 for the IMIs classified as cost-effective.^[165]

A 2018 systematic review evaluating the economic evidence of web- and mobile-based interventions for severe mental illness treatment similarly found low cost per QALY gained – sometimes even finding negative costs due to cost saving for the healthcare system. A 2019 meta-analysis and review on the effectiveness and cost-effectiveness of e-health interventions for depression and anxiety found:

One study showed that both guided and self-guided e-health interventions for depression were more cost-effective than care as usual, with incremental cost-effectiveness ratios (ICERs) of-€9.91 and-€98.37 and savings of €496.72 and €5140.40, respectively, per QALY gained.
Guided e-health interventions for anxiety were probably cost effective, with an ICER of-$1824 (-$616/0.34) after treatment in favor of guided ehealth over care as usual. For each incremental improvement in the outcome measure, iCBT by guided e-health generated a societal earning of $1824 relative to the control condition; each additional QALY therefore generated $7523 compared with care as usual.

A 2018 cost-effectiveness analysis of mobile and traditional CBT for anxiety disorders^[166] concluded that for a hypothetical population of 100,000 people in the US with anxiety, mobile CBT would be cost-saving and lead to gains of 34,108 QALYs and 81,492 QALYs and a cost reduction of $2.23 billion and $4.54 billion when compared to traditional CBT and no CBT, respectively.

In conclusion, all this suggests that mental health apps for anxiety can have high cost-effectiveness or lead to significant cost-savings.

GAD-7 to QALY

To transform Mind Ease's GAD-7 scores to QALY improvements, we now review the relevant literature of co-improvements in anxiety and GAD-7 scores:

One study showed that iCBT reduced GAD-7 by 3 points after 1 year at a cost of ~$656 and increased QALYs by 0.0379 (roughly $14,337 to 19,327 per QALY).^[167]
One 2015 RCT looked at the efficacy and cost-effectiveness of an 8 week iCBT course (that was therapist supported) for older adults with anxiety.^[168] The study found that iCBT reduced GAD-7 scores from 11.7 to 4 (also at 12 month follow-up) relative to the waitlist control group. iCBT cost $230, which was $92 (95% CI: $38.70 to $149.20) more than the control group and increased QALYs by 0.01 (95% CI: 0.003 to 0.018) relative to the control (through GAD-7 score reduction of 7.7). This translates to $8,806 per QALY gained (95% CI: $2,849 to $39,522).
Haemodialysis patients receiving unguided online CBT gained 0.144 QALYs when their GAD-7 scores reduced by 0.94.^[169]
Panic disorder patients receiving CBT gained 0.102-0.178 QALYs when their GAD-7 scores reduced by 4.2.^[170]
Epilepsy patients receiving ACT gained 0.08 QALYs when their GAD-7 scores reduced by 6.^[171]

Combining the results from the five studies above suggests that the average reduction in a one point on the GAD-7 coincides with a 0.04 increase in QALY score:

QALY gain	GAD-7 score reduction	QALYs/GAD-7 Score reduction
0.0379	3	0.013
0.0105	7.7	0.001
0.144	0.94	0.153
0.14	4.2	0.033
0.08	6	0.013

Average QALYs gained per 1 point GAD-7 score reduction		0.0428

Thus, if Mind Ease can reduce GAD-7 by 4 points for one year as suggested by the pilot data, then this would translate to 4*0.04 = 0.16 QALYs gained per user.

Mind Ease would have to serve roughly 6.25 users (=1/0.16) per QALY gained (disregarding retention effects for now – our cost-effectiveness analysis will include this).

This mapping is an inexact correlational approach. To be sure that reductions in GAD-7 are what really causes improvements in well-being is very difficult and the subject of mediation analysis in statistics:^[172]

In statistics, a mediation model seeks to identify and explain the mechanism or process that underlies an observed relationship between an independent variable and a dependent variable via the inclusion of a third hypothetical variable, known as a mediator variable (also a mediating variable, intermediary variable, or intervening variable).[1] Rather than a direct causal relationship between the independent variable and the dependent variable, a mediation model proposes that the independent variable influences the (non-observable) mediator variable, which in turn influences the dependent variable. Thus, the mediator variable serves to clarify the nature of the relationship between the independent and dependent variables.[2]

A simple mediation model is this:

Here we would assume that the independent variable is the anti-anxiety intervention, the dependent variable is QALY improvements, but it goes through GAD-7 score, which is the mediator variable.

Mediation statistics are said to infer causality if certain conditions are true, and without getting too much into philosophy of science, briefly and crudely, all variables should be able to predict each other, there should be strong a priori hypothesis for the causal model, there should not be any confounds, etc. and experimental evidence that shows temporal precedence increases the likelihood that the effects are causal.

We believe the literature provides evidence for most of these claims and it is intuitively plausible people's quality of life improves if one causally reduces their anxiety.

These factors also all predict each other:

One study showed that correlations of GAD-7 scores and SF-6D scores (which are used to compute QALY measures) ranged from 0.36 to 0.51. Further, despite evidence of poor predictive performance, predicted scores are able to discriminate across depression severity groups, which in turn can predict quality of life.^[173]

Given these high correlations and plausibility, we believe that it is very likely that reducing GAD-7 scores improves QALY scores.

DALYs averted analysis

In terms of DALYs, things look a little different. Consider the following diagram:^[174]

There, a 5 point reduction would result from going from moderate anxiety (12 point score on the GAD-7) to mild anxiety (7 point score).

When we consider the disability weights:

Mild anxiety disorders

Anxiety disorders, mild

Feels mildly anxious and worried, which makes it slightly difficult to concentrate, remember things, and sleep. The person tires easily but is able to perform daily activities.

0.030

(0.018-0.046)

Moderate anxiety disorders

Anxiety disorders, moderate

feels anxious and worried, which makes it difficult to concentrate, remember things, and sleep. The person tires easily and finds it difficult to perform daily activities.

0.133

(0.091-0.186)

Severe anxiety disorders

Anxiety disorders, severe

constantly feels very anxious and worried, which makes it difficult to concentrate, remember things and sleep. The person has lost pleasure in life and thinks about suicide.

0.523

(0.362-0.677)

This would only reduce disability weight by 0.103 (=0.133 to 0.03) or 0.02 DALYs averted per 1 point GAD-7 score decrease. However, reducing the score from 18 to 12, i.e. from severe anxiety to moderate anxiety, would reduce disability by 0.4 (=0.523–0.133) and 0.07 DALYs averted per 1 point GAD-7 score decrease.

So what weight should we plug into the cost-effectiveness analysis?

Luckily, Mind Ease gives us a more granular breakdown of the average reduction in GAD-7 scores by starting score in their internal study:

# of participants	Median change in 'pre' gad7	'Pre' gad7 cohort
36	3	10- 11
46	3.5	12- 13
47	5	14- 15
33	4	16- 17
32	7	18- 19
23	6	20- 21

217	4	Grand Total

With this, we can calculate that the (participant) weighted average reduction in DALY weight is 0.22- corresponding to a weighted average 0.04 DALYs averted per 1 point GAD-7 score decrease (calculation here).

Reassuringly, the average 0.0443 DALYs averted per 1 point GAD-7 score decrease, which are based on Mind Ease's internal study, and the average 0.0428 QALYs gained, based on the studies of anxiety reduction and the corresponding increase in QALYs, are roughly equal. This suggests that this is a realistic value to use for further cost-effectiveness analysis.

However, note the tables below with general population averages of GAD-7 scores. Recall that Mind Ease writes 'Our data so far suggests our core users will likely be: Technologically savvy, Active Online, Vocal / outspoken, More likely female'. This suggests high socioeconomic status.

Eyeballing the first table below that such demographics will only have an average GAD-7 score of around 2-3 with a standard deviation of 3-4.

In the second table below, we see that less than 5% of people have a GAD-7 score higher than 10. Of those might have high GAD-7 scores due to invariant forces, like low income (see correlation between low income and anxiety in table 4). This naturally limits the effective market size in which Mind Ease might have very high social impact.

From ^[175]

From ^[176]

Recall from above the effectiveness of mHealth:

A 2017 meta-analysis by the same authors also showed that apps reduced anxiety compared to waitlist (g=0.45, 95% CI=0.30–0.61), and even active control (g=0.19, 95% C.I.=0.07–0.31).^[177]
Another 2019 meta-analysis^[178] of 66 RCTs (n=66) on the efficacy of mental health apps concluded that apps outperformed control conditions in improving depressive (g=0.28, n=54) and generalized anxiety (g=0.30, n=39) symptoms, stress levels (g=0.35, n=27), quality of life (g=0.35, n=43).

Recall that the effect size roughly corresponds to a standard deviation measure.

This suggests that mental health apps effectiveness in the general population, where the average GAD-7 score of ~3 with a standard deviation of ~4, might reduce anxiety by 1.2 (4*0.3) and thus from 3 to 1.8 –- both of which fall into the 'no anxiety category' and would not lead to a high QALY improvement.

Yet, as one can see in Table 5 above around 5 percent of people do have a score of 10 and up, which just about classifies as moderate anxiety. This is perhaps the best 'customer group' for Mind Ease to target as well, because it hits the 'sweet spot' between relatively large DALY improvement, but also a 'large vertical', in the sense that it is a sizable chunk of the population.

Focusing on people with what would be classified as severe anxiety in the 15+ GAD-7 score range would lead to a bigger improvement per user, yet there are fewer than 1 percent of all people who have anxiety this severe. Nevertheless, the corresponding DALY reduction would be large, and of 1 billion people in developed countries, there seem to be around 10 million people (~1%) that could perhaps be targeted to become users. There seems to be a crucial strategic decision of whether to brand Mind Ease as a medical product or as a widely used app (note that again according to the table ~25% of the general population are classified as what one would begin to classify as 'mild anxiety').

Cost-effectiveness analysis

Spreadsheet can be found here.

Scenario	Conservative	Best guess	Optimistic	Sources
Average reduction in GAD-7 score per participant	1	4	6	Mind Ease internal evaluation
Disability weight of 1 point on GAD-7 scale	0.001	0.04	0.15	Studies showing concomitant GAD-7 score reductions and QALY improvements
DALYs averted per year per participant directly post-treatment	0.00	0.17	0.92	Calculation
Retention rate of benefits	50%	84%	86%	Long-term Outcomes of Cognitive Behavioral Therapy for Anxiety-Related Disorders
Total DALYs averted per user	0.002	1	6.11	Calculation
ME alternative GAD-7 reduction	1	3	2	Independent eval. of Pacifica competitor app shows 3 point reduction
Counterfactual GAD-7 reduction	0.05	1	4	Calculation (Row 2- Row 7)
Counterfactual adjustment: how much extra effect does ME have?	5%	25%	67%	Calculation
Counterfactual DALYs averted per user	0.0001	0.25	4.07
Cost per beneficiary	$54	$1.2	$1.2	Mind Ease cost-per-user
Cost per DALY averted	$528,000	$4.8	$0.3	Calculation
Cost per equivalent of a year of severe anxiety disorder prevented	$276,144	$3	$0.2	Calculation
Cost per DALY averted equivalent of Givewell charities	$14	$835	$2,770	Deworming, Cash-transfers, Action for Happiness
Health: Mind Ease is x times more effective than global development interventions	0.003%	172	9,396	Calculation

In our cost-effectiveness analysis above we multiply the DALY/QALY values by different assumptions for GAD-7 score improvement going from a 1 score improvement because of Mind Ease in a conservative case, to a 4 point improvement as our best guess, to a 6 point improvement, optimistically assuming that sustained app usage of an optimized version of Mind Ease would cause even greater improvement.

These resulting values then correspond to the DALYs averted per year per participant directly post-treatment and need to be multiplied by retention rates to get at the long-term reduction in DALYs (see methodology employed in a Founders Pledge report^[179]).

A 2019 systematic review and meta-analysis on the long-term outcomes of CBT for anxiety suggests that the effect size of CBT for GAD immediately upon treatment completion is 0.39 (0.12 to 0.66) reduces to 0.22 (0.02 to 0.42) at 12-month follow up.^[180] This suggests roughly a 1-year half-life of treatment effects- this will be our conservative assumption. The study also cites cross-sectional findings suggesting ~31%-55% of patients with remitted anxiety meet diagnostic criteria of the same or another disorder within 4 years, suggesting a 4 year half-life, which corresponds to 84% retention benefits and will be our best guess assumption. Finally, the study cites that relapse rates after 12 months were only 14%- and so 86% will be our optimistic assumption in the model below.

Next, we calculate the cost per beneficiaries: our conservative assumption here is $54- this includes the cost incurred for the user and the philanthropic subsidies. Our best guess assumption uses a 'zero marginal cost scenario' at $1.20- corresponding to a scenario where an additional user at scale only costs 10 cents per month and perhaps a 'freemium' model allows some users to access the app for free, resulting in low costs for both philanthropist and user (Mind Ease writes 'We're raising £1,100,000, to help 1,100,000 paid users by 2022').

According to our own calculations based on data by Givewell, the Against Malaria Foundation has a cost-effectiveness of roughly $50 per DALY averted. This is because a death of an under 5-year-old is equivalent to ~34 Years of Life lost (YLL) per AMF death. The most effective charity according to Givewell—the Deworm the World Initiative — roughly averts a DALY equivalent for $14, while GiveDirectly, the philanthropic benchmark averts a DALY equivalent for roughly $860.

The resulting cost effectiveness is given in terms of $ per DALY averted (or QALY gained), but depending on different assumptions this might be higher or lower than giving to Givewell charities.

Supplementary economic analysis

Would it be better than giving a grant in unconditional cash-transfers to poorest people in poorest countries or improve health in rich countries? We will have to adjust for disparities in income first.

Our supplementary analysis could be used to quantify this effect.

The income adjustment takes into account that 'your dollar does (>)100x or more good if you give to the poorest rather than people in high-income countries'). More on income weighting in Appendix 2. The optimistic case has an η value of 2, the realistic of 1.5, and the pessimistic of 1 (Source). This corresponds to 1 dollar being worth 120-13,610 more when it goes to the poorest people on the planet (e.g. via unconditional cash-transfers) than someone on Median US income (120 might be an underestimate according to some of my calculations and it might be 250). For higher values going towards 2, this can dominate the analysis.^[181]

Thus, for instance, a $100k grant to the poorest people on the planet could be worth ~$1.36 billion dollars if it's worth 13,600x as much or only $12 million if it's just 120x times as much.

Recall, that the National Health Service in the UK usually funds below $25k.^[182] (However, a 2019 study showed that cost per DALY averted in rich countries is usually around $69,499.^[183]).

This willingness to pay is a proxy for the benefit accrued to a rich country. If an intervention averts a DALY for $100, but a country is usually willing to spend up $25k, then a $100k grant could avert 1000 DALYs and thus create $25 million (1000x $25k) in value in a rich country. This would be more than twice as much as giving to Givedirectly.

However, note a grant $100k to GiveDirectly has roughly 17% overhead before it reaches a recipient, and so a $100k grant is only worth $8.3 million.

On the other hand, under most pessimistic assumptions, a $100k grant to Deworm the World Initiative which is 61x as effective would be worth the equivalent of ~$69 billion dollars given to the average person in rich countries, if the multiplier is 13,600 (=100,000 * 0.83 * 13,600 * 61).

General considerations and limitations

In this section, we list some general—qualitative—considerations why we believe that funding Mind Ease might be effective or not. Some of these are so-called crucial considerations that might completely dominate other considerations with regards to an interventions effectiveness.^[184]

There are several limitations and reservations to our model.

That our final bottom-line figure of the cost-effectiveness of Mind Ease in comparison with Givewell charities spans very many orders of magnitude reflects the uncertainty due to many of the key parameters in the analysis. Unfortunately, this inherent uncertainty cannot be reduced by much through more sophisticated error propagation techniques such as Monte Carlo simulation.
Mental health disorders might be more severe than their DALY weight. For instance, the Happier Lives Institute reviewed the literature on disability weights of mental health disorders and found that

'To highlight a particularly outstanding discrepancy, Dolan and Metcalfe (2012, from whom the above figure 4 is derived) report subjects agreed to hypothetically give up as many years of their remaining life, about 15%, to be cured of 'some difficulty walking' as they would to be cured of 'moderate anxiety or depression.' However, from SWB measures 'moderate anxiety or depression' is associated with 10 times a greater loss to life satisfaction, and 18 times a greater loss to daily affect, than 'some difficulty walking' is (note the time trade-off lines for 'mobility 2' and 'anxiety 2' in figure 4 are the same length but the two SWB lines are very different).'
'In fact, DALY weightings probably underestimate the severity of [mental] disorders. This is because DALY weightings are elicited through a preference-based method, by asking people to predict how bad different health states would be if they were to have them.9 Preference-based methods are flawed because when we ask people how they would feel in different health states, they are unable to predict how they will adapt to health conditions. In particular, people tend greatly to underestimate how bad mental illness would be relative to other kinds of illness. If instead people are asked to report how they are feeling when they have the condition (the subjective wellbeing method), mental health disorders tend to be evaluated as more severe. In particular, depression and anxiety are weighted as significantly worse than most physical health problems.'^[185]
However, the above points do not invalidate our calculations here for several reasons: our model is not very sensitive to whether anxiety disorder to variance in the disability weight of anxiety. Rather, it is sensitive to how well Mind Ease's interventions work, what the cost per beneficiary is, whether it scales, and how much further a dollar goes overseas. Moreover, there are many developing world mental health charities and charities that improve micronutrient status and prevent mental disorders that way, that have been shown to have roughly similar cost-effectiveness as top-recommended Givewell charities.

Different instruments can be used to calculate QALYs from GAD-7 scores (e.g. EQ-5D, SF-6D, AQoL-8D). Ideally, one would use the same instrument throughout. However, here we mixed different instruments in our calculation (e.g. EQ-5D-5L and SF-12), because that was the only data that was available. On the plus side, taking the average QALY gain due to reductions in GAD-7 score based on different instruments, makes our approach more robust to potential drawbacks of individual instruments.
Relatedly, we do not know whether reduction in anxiety as measured by GAD-7 scores causally improved quality of life. Some studies that we base this analysis on also report improvements in depression or other health indicators. However, given that we base our analysis on studies about therapeutic interventions improving anxiety, sometimes in patient populations, we think it is very plausible that the improvement of quality of life was indeed caused by the reduction in anxiety.
In the cost-effectiveness analysis in the previous section we found that competitor apps like Pacifica might also be effective in reducing anxiety. In our model above we did not assume that Mind Ease attracts entirely new users to its app, who would have not counterfactually used other apps like Pacifica and thus will accrue the full benefit of Mind Ease compared to no treatment at all. However, if Mind Ease users would not use a different app counterfactually, this might increase the counterfactual benefit they get from the app. Business case analysis could shine light on this and adjust the effectiveness by reducing this parameter– given that Pacifica has been shown to reduce GAD-7 scores by 3 points vs. the 4 point reduction that we assume for an idealized future version of Mind Ease. Our shallow review of competitor apps such as Pacifica suggests that they are more geared towards rapid user engagement than lasting improvements in flourishing. Spencer Greenberg puts this well: 'The key is, don't put too much candy on the broccoli, otherwise you defeat the purpose, I think there's a temptation for startups to just try to get user engagement, because they can go to investors with it. But that's not a long-term solution. If you're not actually providing value, engagement doesn't buy you much.' As such we think that Mind Ease might be better at reducing anxiety long-term but there is a market inefficiency here where for-profit companies optimize for the wrong metrics and the market is stuck in a local minimum, that philanthropic investment might help with.
Without a better and independent high-quality RCT of Mind Ease, our best guess scenario of its effectiveness of a 4 point reduction in GAD-7 scores remains an educated guess. This scenario assumes perhaps a more idealized and further optimized version of Mind Ease. However, given that Mind Ease seems very committed to continue to use evidence based interventions (Chapter 2), that have been shown to have similar effectiveness in an app setting (Chapter 3), and suggestive evidence of such effects based on internal studies and other studies in the published literature (Chapter 4), we believe this effect size is very much plausible.
Relatedly, at scale, small effect sizes can have outsized impact. For instance, Amazon.com found every 100ms of website loading time cost them 1% in sales.^[186] Conclusively establishing that loading time is important by getting feedback from users would not have been possible. Instead, finding such a small effect size is likely only possible because Amazon has a lot of data from its customers and thus a lot of statistical power to pick up relatively small effects. Similarly, mental health apps might have relatively small effect sizes that are difficult to pick up in RCTs. Sometimes, absence of evidence is not evidence of absence- just because a study failed to find an effect, does not mean it is not there (especially if it is small). Thus, perhaps even if the effect size is relatively small, the overall effect across users might be large, and the intervention thus very cost-effective. In theory, the zero marginal cost aspect of apps is what might drive a lot of the effectiveness. In other words, given that the app usage can be increased from 1 million to 2 million users (or 20 for that matter) at zero marginal (philanthropic) cost, arguments of how scalable the approach Mind Ease takes might dominate the cost-effectiveness calculation.
Related to the previous point above and also the uncertainty in our analysis: it is inherently hard to compare interventions that slightly increase the quality of life of a large number of, in global terms, very rich people with radically transforming the lives of a small number of the poorest people on the planet through cash-transfers. To simplify the issue and to illustrate the point very boldly: Some moral frameworks might question whether it is ever justified to solve 'first world problems' such as mild to moderate anxiety of the wealthy (Mind Ease writes 'Our data so far suggests our core users will likely be: Technologically savvy, Active Online, Vocal / outspoken, More likely female', this suggests high socioeconomic status) compared to helping people who are malnourished and at risk of death due to disease– no matter how many people are helped. This raises moral questions that are beyond the scope of this report to answer. To be sure, in our analysis we do account for this by conservatively assuming that a dollar does 1,260 times as much good when going to the poorest vs. the people in a rich country—which is usually considered reasonably conservative amongst economists.
Our sense from the literature is that the global mental health burden is large, growing and severely underfunded.
Mental health problems sometimes create externalities. For instance, anxiety might reduce productivity. Mental health disorder costs are projected to rise to $6 trillion in 2030 and alone account for 26% of productive time lost due to disability, more than any other category of disease.^[187] Our model does take this into account by assuming that rich countries spend adequately on this problem, but this is of course questionable, and might reduce Mind Ease's effectiveness.

Results seem to violate neoclassical consumer choice theory: 'Economists since the days of Adam Smith and Jeremy Bentham have traditionally viewed consumers as driven by relentless and consistent pursuit of self-interest, with their choices in the marketplace providing all the measurements needed to reveal their preferences and assess their well-being. This theory of consumer choice is empirically successful, and provides the foundation for most economic policy.'^[188] If neoclassical consumer choice theory is true, then it is puzzling why people with anxiety would not just spend the comparable trivial amount of money to buy the app given the large benefits that we see in our analysis. Is there unwillingness to buy mental health apps not a revealed preference that they do not actually benefit much from the app (i.e. lower than the price)? Only 31% of participants in Mind Ease's said they were willing to pay for the app, and the average of those chose an amount of payment as $9.99 once or $2.99 monthly. The traditional neoclassical consumer choice model has recently been challenged by evidence from psychology showing that consumers are often irrational.^[189] Thus, we do believe it is plausible that, if consumers are not fully informed, fully rational actors, might make suboptimal choices for themselves and can be nudged in a direction to improve their own preferences. This might be especially the case with people with mental health problems. However, some people object to this philosophy

'Consumer theory distinguishes between two different reasons why someone might not buy a Ferrari – budget constraints (they can't afford one) and preferences (they don't want one, or they want other things more). Physical diseases seem much like budget constraints – the reason a paralyzed person can't run a marathon is because it's beyond her abilities, simply impossible. Psychiatric diseases seem more like preferences. There's nothing obvious stopping an alcoholic from quitting booze and there's nothing obvious preventing someone with ADHD from sitting still and paying attention. Therefore they are best modeled as people with unusual preferences – the one with a preference for booze over normal activities like holding down a job, the other with a high dispreference for sitting still and attending classes. But lots of people have weird preferences. Therefore, psychiatric diseases should be thought of as within the broad spectrum of normal variation, rather than as analogous to physical diseases.'^[190] Yet, this view is generally considered very controversial and we are inclined to agree.

Cost-effectiveness estimates can be misleading in many different ways.^[191] For instance, Failing to take into account the counterfactual impact of altruistic employees (opportunity cost) might substantially increase the final cost-effectiveness. We feel for all intents and purposes our analysis here is sound and more sophisticated analysis outside of the scope of this analysis. However, others might object that for instance Mind Ease's impact is actually negative, because, as an example, its altruistically motivated employees could work on more impactful projects.
It seems that the grant investment is 'shovel-ready'. Mind Ease has the program set up, people are working on the topic, they are hiring, but they would like to intensify their work and seem funding constrained. If the program was to receive more funding, we imagine that they could scale up their work relatively quickly and hire additional staff.
Peter Brietbart, the Project Lead, and Spencer Greenberg, the Project Founder, seem very altruistically motivated. Their team members also seem to be members of the effective altruism community. For instance, Mihai Badic founded and launched EA forum. All else being equal, this is a good sign. For instance, giving the app to people in developing countries for free might substantially increase the cost-effectiveness of a grant. This seems plausible given the research interest outside of the western world mentioned in a previous chapter as well as the team seeming keen on launching the above mentioned freemium model (''We want to charge in the developed world, where people can afford to pay. In the developing world, we want to give the app away for free,' Spencer Greenberg).
Spencer Greenberg has also launched an anti-depression app: UpLift: The Depression App. This could have complementarities and synergies with Mind Ease.

Value of information

Mind Ease model might create valuable information by being a model for other mental health apps. This might spillover into their effectiveness. Mind Ease running studies might generate knowledge that is a public good and so their operations might lead to a better view of how apps can treat mental health conditions, which is another spillover effect.

Consider the value of information that the Against Malaria Foundation (AMF) creates. Givewell recently wrote about AMF being instrumental in running a study on the effectiveness of new improved malaria nets, with early evidence suggesting that they are more effective at reducing malaria cases than standard nets.^[192]

Givewell writes:

AMF reported that PBO nets were more effective at reducing malaria cases than standard nets six months after distribution (26% more), 12 months after distribution (27% more), and 18 months after distribution (16% more).

From 2000-2015 anti-malarial interventions have prevented about 663 million malarial fevers. Long-lasting insecticide treated bed nets stand out as being particularly effective — being responsible for around 68% of the malaria reduction. This means that bednets have prevented around 450 million cases of malaria. And globally, 6.2 million fewer people died of malaria over the last 15 years because of malaria interventions.^[193] Very roughly and crudely, 4.2 (68%*6.2) million deaths were averted due to bednets, or about 0.28 (4.2/15) million a year. If AMF speeds up the introduction of new nets that are ~25% more effective, this might save 0.07 million lives (0.28*0.25) or 70,000 lives. If the trial cost $1 million, then this would lead to a cost-effectiveness of $14 per life saved.

This is just a back of the envelope calculation and some of the numbers are very inexact. This is just to demonstrate that the value of information can be very high if a popular and effective intervention can be improved with a small capital investment.

Similarly, if Mind Ease might make the whole mental health industry more effective, this might have a very high value of information. Recall the recent paper in the Lancet on 'Scaling-up treatment of depression and anxiety: a global return on investment analysis'. They found large treatment gaps and modelled the global return on investment to estimate treatment costs and health outcomes in 36 countries between 2016 and 2030.

The net present cost of scaling up anxiety treatment from 2016–30 was estimated to be $56 billion. The expected returns of scaled-up treatment were 6 million extra years of healthy life. Thus the average cost-effectiveness of anxiety treatment is roughly $9,333 per QALY ($56 billion / 6 million).

If Mind Ease, which has higher cost-effectiveness, were to increase the average cost-effectiveness by 10%, this would lead to an additional 666 thousand years of healthy life ((56 billion / (9,333-(9,333*.1)))-6 million).

If this can be done through a $1 million dollar grant, this leads to a cost-effectiveness of the value information of $1.5 per QALY gained (1 million / ((56 billion / (9,333-(9,333*.1)))-6 million)).

Again these numbers are highly uncertain, and further research could try to make this calculation more precise, but it demonstrates the value of generating information through new research.

Also see:

Risks, Reservations, and Drawbacks

A recent systematic review demonstrated that interactive symptom checkers are the most frequently investigated category of diagnostic app in mHealth, yet Mind Ease does not seem to offer such diagnostics yet.^[194]
Some of the symptoms that are checked on the Mind Ease app are similar to that of stroke, perhaps there is a risk of misdiagnosis with a sufficiently large user base. The app should probably be checked by someone with a medical background.
Relatedly, Mind Ease might discourage users from seeking cheap and affordable pharmacological treatment for anxiety: one Cochrane review from 2017 on medication for social anxiety disorder found evidence of treatment efficacy for the SSRIs (though this was based on very low- to moderate-quality evidence).^[195] However, for patients who only partially respond to medication for anxiety, some trials have shown that the addition of CBT may lead to further reduction in symptoms.^[196]
Relatedly, currently the Mind Ease app does not seem to tackle potential biological causes of anxiety such as alcohol and excessive caffeine, which has also been linked to aggravating and maintaining anxiety is due to overactivation of the sympathetic nervous system.^[197] ('The one very important connection – if you drink too much coffee, or any other source of caffeine, that will make you anxious. I once had a patient come to me with severe recurrent anxiety. I asked her how much coffee she drank, and she said about twenty cups per day. Suffice it to say this was not a Dr. House-caliber medical mystery.').^[198] Further 'Pretty much every study – epidemiological or experimental, short-term or long-term, has shown that exercise decreases anxiety. The effect seems limited to aerobic exercise like walking, running or swimming, preferably for longer than twenty minutes. Various mechanisms have been postulated including norepinephrine, endogenous opioids, and decreased inflammation.'^[199] Our suggestion here would be to carefully consider whether Mind Ease should give evidence-based advice and guidance on these issues (i.e. reduce caffeine, alcohol, exercise more, etc.). If not, Mind Ease might risk having a negative impact, if users would get a larger effect from non-psychological interventions, but use Mind Ease and 'just get by'.
Though it is conceivable that Mind Ease might actually decrease smartphone usage, say because of its mindfulness interventions, it is equally plausible that Mind Ease might increase smartphone usage by virtue of being app and one having to open their phone to use it. Increased smartphone usage might have negative unintended consequences. A recent review concludes that 'Many studies, using a variety of methods, have found associations between heavy social media use and bad mental health outcomes, particularly for girls.'^[200] One 2020 paper on the mechanisms of rise in mental health problems amongst you people concludes: 'There is a growing consensus that these trends may be connected to the rise in technology use. Increased digital media and smartphone use may influence mental health via several mechanisms, including displacement of time spent in in-person social interactions, individually and across the generation, as adolescent cultural norms evolve; disruption of in-person social interactions; interference with sleep time and quality; cyberbullying and toxic online environments; and online contagion and information about self-harm.'^[201]. Another 2020 meta analysis concludes that meta-analytic evidence is not in support of dramatic claims relating social media use to mischief.^[202] Given the ubiquity of smartphones, we think the effect of Mind Ease increasing smartphone and social media usage and then in turn reducing wellbeing are likely to be small and offset by the positive effects of app usage.

A qualitative 2020 interview study of 14 people living in England with depression or an anxiety disorder using apps as part of self-care was very critical of apps to replace traditional healthcare and sees negative drawbacks. It has quotes from individuals that highlight certain themes that we list in Appendix 3.
Another study of client emails found that 61.5% of internet based CBT clients reported experiencing at least one negative effect during treatment.^[203] (see table below for example quotes).

Appendix 1: Other economic evaluations of mental-health apps

Taken from^[204]

Table 3: Characteristics of Included Economic Evaluations (continued)

N.R., not reported; QALY, quality-adjusted life years; ICER, incremental cost-effectiveness ratio; ICUR, incremental cost-utility ratio; WTP, willingness-to-pay; CBT, cognitive behavioral therapy; iCBT, internet-based Cognitive Behavioral Therapy; cCBT, computerized Cognitive Behavioral Therapy; UC, usual care; CES-D, Center for Epidemiological Studies Depression scale; HAI, Health Anxiety Inventory

Study ID	ICER or ICUR	WTP threshold, probability of being cost-effective	Economic evidence summary
1	$1,489 per relapse averted	$5,500 per relapse averted, 90%	CEA: Booster program dominates iCBT using a societal perspective. Each relapse averted saves $1,489.
2	N. R.	N. R.	CEA: iCBT dominates group CBT using an inferred health insurance perspective
3	£3,597 per QALY gained	£20,000 per QALY, 55%	CUA: Telephone-facilitated cCBT dominates minimally supported cCBT (i.e. lower mean costs and higher QALYs). At a WTP threshold of 20,000 per QALY, the probability that the telephone-facilitated cCBT is the cost-effective intervention is 0.55.
4	Societal: €314 per one point reduction in CES-D score Employer: €224 per one point reduction in CES-D score	Societal: €44,000 per one point reduction in CES-D score, 95% Employer: €3,500 per one point reduction in CES-D score, 95%	CEA: Using a societal perspective, a one-point reduction in depressive symptoms was associated with a cost-savings of €314. And every QALY lost was associated with cost-savings to the employer of €542,959. Using an employer perspective, a one point reduction in depressive symptoms was associated with a cost-savings of €224. And every QALY lost was associated with cost-savings to the employer of €382,354.
5	N.R.	CEA: N. R. CUA: €80,000 per QALY, 40%	CEA: Using a societal perspective, cCBT + UC dominates UC only at WTP threshold values above €7,000 per participant at 80% probability CUA: There are no significant group differences in quality of life or effectiveness.
6	ICER: £1,244 per unit reduction in HAI score ICUR: £6,533 per QALY gained	CEA: £5,000 per HAI unit reduction, 77% CUA: £5,000 per QALY, 96%	CEA: Using a societal perspective, iCBT dominates CBT. At each incremental improvement (no longer meeting diagnostic criteria for severe health anxiety) in iCBT relative to the control condition generated a societal earning of £1,244. CUA: At each additional QALY gained a societal earning of £6,533 when comparing iCBT to the control condition.
7	£17,173 per QALY gained	£30,000 per QALY, 71%	CUA: Guided iCBT is likely to be cost-effective (71% probability) compared with usual care if society is willing to pay at least £20 000 per QALY.
8	N. R.	N. R.	CUA: Although with uncertainty, there was no statistically significant difference in QALYs gained (0.082 for iCBT and 0.083 for control group). QALY estimates is suggestive of no difference in cost-effectiveness between the groups.
9	£4,392 per QALY gained	$50,000 per QALY, 95%	CUA: The results show that treatment is very likely to be cost-effective at a willingness-to pay threshold of $50,000 per QALY gained (probability of being cost-effective is 95%).
10	ICER: $-1,824 per QALY gained ICUR: $-7,563 per QALY gained	CEA: $1,00 per one additional patient case, 99% CUA: $3,000 per QALY, 95%	CEA: The results indicate that iCBT has a 99% probability of being cost-effective if society was willing to pay $1000 for one case of improvement. CUA: The results indicate if society were willing to pay $3000 per one additional QALY, the probability of iCBT being cost-effective would be 95%.
11	ICER: €3,222 per unit improvement in CES-D score ICUR: €157,900 per QALY gained	CEA: €15,000 per one unit improvement in CES-D score, 57% CUA: €30,000 per QALY, 30%	CEA: One point of improvement in the CES-D score (depressive symptoms) was associated with €3,222 higher costs as compared to the usual care group. CUA: One additional QALY in the intervention group was associated with an extra cost of €157,000 in comparison with the usual care group. The difference in improvement in depressive symptoms and QALYs between the two groups was not statistically significant either. The results show the intervention was not cost-effective in comparison with the usual care from a societal approach.
12	$8,806 per QALY gained	$50,000 per QALY, 95%	CUA: On average, the intervention will produce one additional QALY for an additional cost of $8,806. The results show that treatment is very likely to be more costly but more effective.
13	€-34,727 per additional treatment response	€30,000 per additional treatment response, 98.5%	CEA: Based on the results, it can be concluded that the intervention is acceptable from a cost-effectiveness point of view and that this conclusion is not sensitive to the WTP ceiling used.
14	£6,933 per QALY gained	£20,000 per QALY,76%	CUA: Intervention dominated usual care (lower mean costs and higher QALYS) and was likely to be cost-effective at a £20,000 per QALY threshold (0.76)
15	N. R.	£15,000 per QALY, 99%	CUA: Using a societal perspective, if there is a £5,000 value on each additional QALY gained, there is an 85% chance that cCBT is more cost-effective than the control. With an additional QALY valued at £15,000, the probability of the cCBT intervention becomes 99%.
16	Societal: £21,778 per QALY gained NHS: £-556 per QALY gained	Societal: £30,000 per QALY, 60% NHS: £30,000 per QALY, 80%	CUA: Using a societal perspective, society must pay £21,778 per participant in the intervention group for each additional QALY gained compared with usual care. Per the WTP threshold, society is willing to pay up to £30,000 per QALY. The probability cCBT is cost effective is 60%. Using a NHS perspective, each additional QALY gained will save £556 per QALY gained compared with usual care. The probability cCBT is cost effective is 80%.
17	£107 per unit improvement in outcome	N. R.	CEA: Further research is required to determine whether the cost-effectiveness ratio is acceptable.
18	€233 per QALY gained	€1,000 per QALY, 64%	CUA: Using a societal perspective, society must pay €10,708 for each additional QALY. Results show iCBT is likely to be more costly and more effective.
19	ICER: €-170 per per unit improvement in BDI-II score ICUR: €-11,390 per QALY gained	N. R.	CEA: Each point of improvement in BDI-II score using the iCBT intervention instead of usual care saves society €170. Results indicate the iCBT intervention is likely cost-effective using a societal perspective. CUA: Each additional QALY using the iCBT intervention instead of usual care saves society €11.390. The iCBT intervention is cost-effective for depressed patients in the Spanish primary care system using a societal perspective.
20	ICER: €1,248 per reliably improved participant ICUR: €11,523 per QALY gained	CEA: €30,000 per reliably improved participant, 95% CUA: €30,000 per QALY, 61%	CEA: Offering the iPST intervention instead of placing participants on a waiting list incur an extra cost of €1,248 for a health gain of one additional reliably improved participant, using a societal perspective. However, with a WTP threshold of €30,000, the iPST intervention shows to be more cost-effective (probability is 0.95) than waiting group control. CUA: The ICER for iPST compared to the waiting list placement resulted in extra costs of €11,523 per QALY gained. However, with a WTP threshold of €30,000, the iPST intervention shows to be more cost-effective (probability is 0.61) than waiting group control.

Appendix 2: Income-weighting

All else being equal, money going to poorer countries or people is better than money going to richer countries or people. Weyl suggests that assuming logarithmic utility giving 1 dollar to an extremely poor person is like giving 66 dollars to an American^[205]. ('That is, if marginal utility is declining in levels of income, say utility is the natural log of consumption, then the marginal utility is 1/consumption. This implies a dollar's worth of consumption in utility terms of a person at the global poverty line is worth 64 times as much as a dollar to a person in the highest decile of consumption in the USA (63.6=(so transferring income from a rich person in the USA to a globally poor person produces, in and of itself, massively higher total global utility (even if not Pareto improving).'^[206]).

Weyl^[207] suggests that logarithmic utility is canonical in economics and supported by a wide range of data, 'including recent happiness studies (Stevenson and Wolfers, 2008) and labour supply decisions (Chetty, 2006)'. This is also in line with work that finds a correlation between log income and happiness ^[208]:

The law of logarithmic utility can be found in other areas such as research funding as well ^[209].

The general form of modelling utility consumption relationships using isoelastic utility function is: ^[210]:

Ord ^[211] explains this function as follows:

'This equation has one free parameter, known as η ('eta', which sounds 'e' for 'elasticity'), which represents how steeply returns to consumption diminish. η must be between 0 and ∞, and can be estimated empirically.

The equation, for utility (u) at a given consumption level (c), and elasticity (η) is:

From this it follows that for η = 0 utility is linear in consumption, for η = ½ utility is the square root of consumption, and for η = 1 utility is logarithmic in consumption. Values of η above 1 correspond to utility having a finite upper bound, which is approached hyperbolically as consumption increases.

However, the main use of the equation is to just compare the slope of the curve at one consumption level to the slope at another consumption level. For example the ratio of the slope at $1,000 per annum to the slope at $10,000 per annum shows us the relative value of giving an extra dollar to someone with annual consumption $1,000 versus to someone with $10,000. When performing this calculation, the equation is very simple:

Giving a dollar to someone with k times as much consumption is worth only:

times as much.

There have been many attempts to measure η, and it is typically found to be between about 1 and 2. If η equals 1, then we have logarithmic utility of consumption and we have the very simple rule that a dollar is worth 1/k times as much if you are k times richer (and that doubling someone's income is worth the same amount no matter where they start). If η equals 2, then we have to raise this to the power of 2, so being 10 times richer would mean a dollar is worth just 1/100th as much (and doubling your income is worth much less the higher your starting income). The truth is probably in between these limits.'

Appendix 3: Interviews with mental health app users

A qualitative 2020 interview study of 14 people living in England with depression or an anxiety disorder using apps as part of self-care was very critical of apps to replace traditional healthcare and sees negative drawbacks. The authors conclude that: 'Apps as part of mental health care should consider means of design and delivery that can reduce apps' responsibilising features and isolating effects while maintaining their flexible, empowering benefits.'

It has quotes from individuals that highlight themes:

Theme 1: Apps within context

Desperate situations, lack of options:

'[my decision to use the app was influenced by] this feeling of desperation and feeling annoyed that no one was helping me and I had to take it into my own hands, quite literally (P11)'
'People didn't know whose responsibility I was and I ended up being re-referred three, four times... I just remember being at the mercy of the services. (P12)'

A priori positivity, trust towards apps and techno-optimism:

'[An app is] not going to just appear magically overnight, someone will have had to have planned it. So I believe someone has planned it. So someone has done some research, I believe it's been tested. (P08)'
'Before I'd taken the time to use it, even when I was having a bit of an anxious evening, I thought 'ah that's brilliant, there's something available on your smartphone'. (P02)'
'I really don't believe they could make – well I hope they couldn't make – [an app] that would make people feel worse about themselves. So personally, I'm not concerned about that. (P07)'

Imprecise, casual approach:

'At that point I didn't really know anything about what it did, or what have you, but... it was a freebie at that point... so [I] just took it from there really. (P04)'
'You download one of the apps, you do it for 1 or 2 weeks, you decide if it's worth it, and... if it seems like hogwash you delete the app. (P05)'

Apps as isolating and anti-social:

'[being able to use the app has] made me more aware of my mental health. And more aware that I need to take action, not expect other people to take action. (P09)'
'Having the apps has made me think, maybe I need to rely on [the NHS] less myself. Like if I can do more independently, then I'm [one] less [person] on the waiting list, stuff like that. (P12)'

Theme 2: Apps' affective capabilities: providing essential relief via uncomplicated engagement

Convenient, familiar, accustomed devices

'It's convenient and it's ready when I am, it's not like when you're relying on someone else like a therapist or like a medical appointment and you have to kind of wait for it. There's an immediacy. (P01)'
'If there's too many components to it, like measuring things, filling out things, then I get a bit bored. I like to keep it simple because I know I'm in a hurry, so [I like to] know I can do something quickly. (P12).'

Apps as tools

'They are a tool. They're a tool to aid my progression. I put a lot of energy and time into them because it's about building a more positive me. (P03)'
'[using the app] gets me to do something, to actually do something myself that I know is good for me that otherwise I'd struggle to get around to doing. (P01)'
'[I]f somebody said 'it doesn't work', I'd think: maybe it doesn't work for you, but it works for me. (P09)'

Immediate relief in reaction to raised distress

'Participants engaged frequently and actively with apps-as-tools, particularly during periods of acute emotional distress – be they brief panic attacks (P02, P07, P10), or longer 'spells' of emotional distress. Conversely, most participants engaged less with apps when feeling better.'
'In a bad spell I could probably spend 2–3 hours a day doing it. When I'm well, when I'm better I could go days without looking at it. (P06)'
Participants thus employed apps to gain instant, short-term alleviation from emotional distress. The main affective implication of apps used in this way was to allow people to manage or deflect negative thoughts, or provide distraction from present situations.
'I can zone into my phone. If I didn't have [apps on] my phone it would force me to engage with what's going on around me, and make me really paranoid. (P11)'

Imaginarily active apps

'I just log it when it asks me to. (P11)'
'I liked that I could set the app to remind me to fill it in [a mood-tracking questionnaire], so it alerts me every evening, otherwise I think I'd probably forget. (P01)'
'I've just come back from holiday and I'm pleased to say that I had quite a nice time and I didn't need to use the app or the [audio] tracks [included on the app], but it was nice knowing they were there. (P02)'

Stop-gap benefit

'[during a period of severe depression] I couldn't face the apps at all. Nothing, nothing at all. Just looking at my phone, opening it and seeing stuff that was pinging, it would just raise my anxiety levels off the scale. (P04)'
'I wanted to give it a chance, you know, I didn't dismiss it immediately. I wanted to sit down and give it a chance to see what's going on. But then I got to the stage where you know I felt I can't really go any further on here you know, I'd reached the limit (P10)'
'I use this app as something that's interim, a bit of a crutch I suppose, while I'm waiting to get myself sorted out. (P06)'
'I guess what I'm trying to say is with the apps, they're good at helping you, like track your mood or helping you challenge these thoughts, but then it's a bit like, well then what? They can't help you further than that. (P14)'
'I'd kind of outgrown what they could do for me. With a lot of these apps, because they scratch the surface and my problems are more deep-rooted, they're never going to be as helpful as seeing someone face to face would be. (P11)'

Theme 3: Responsibilising, apps, dutiful engagement

Personal responsibility:

'[the best aspect is] about empowering yourself to take responsibility without just relying on going to see a doctor. (P04)'
'I think before [using mobile apps] I thought it was somebody else's job, and they weren't doing their job. But now I think – no, actually it's my job (P09)'
'I know with the way the NHS is going... it's about empowering yourself and looking after yourself in other ways, rather than just waiting for your next appointment. (P12)'

Proactive and dutiful self-improvement

'I think having the app there is like something to snap you out of that rumination thing and take some action, positive action, yeah. (P12)'
'I guess I feel like I should use it because it's all about trying to help yourself. So by not engaging it, am I making myself worse by not helping myself? (P14)'
'I'm the one who suggested to my friend that she should probably use Headspace when she wasn't feeling great. Which is interesting isn't it; coz I'd recommend it even though it wasn't really helping me that much. (P13)'
'I feel (pause) a bit disappointed in myself for not using it as much as I should or as much as I think I should... it's always there, so there's no excuse to not use it. (P14)'

Apps afford reified knowledge of distress

'If I think back over the last month I find it hard to come up with an accurate sense of how I've been doing, whereas if I've been measuring it day-by-day on the app and then I look back then it's, obviously to a degree it's still subjective but there's an element of objectivity. (P01)'
'[T]hat outside opinion is quite useful. So it's given me that insight, that I don't always have the right insight into my wellbeing, and having an external measure like that really does help me, and helps me to self-manage better. (P09)'

Notes

[1] "Four quadrants of the cost-effectiveness plane: some considerations " https://www.tandfonline.com/doi/pdf/10.1586/14737167.4.6.599

[2] "Anxiety disorder - Wikipedia." https://en.wikipedia.org/wiki/Anxiety_disorder

[3] "Generalized anxiety disorder - Wikipedia." https://en.wikipedia.org/wiki/Generalized_anxiety_disorder

[4] "What is an anxiety disorder? - Craske - Wiley Online Library." https://onlinelibrary.wiley.com/doi/abs/10.1002/da.20633

[5] "Meditation techniques v. relaxation therapies when treating " https://www.ncbi.nlm.nih.gov/pubmed/31322102

[6] "Things That Sometimes Work If You Have Anxiety | Slate Star " '15 https://slatestarcodex.com/2015/07/13/things-that-sometimes-work-if-you-have-anxiety/

[7] https://vizhub.healthdata.org/gbd-compare/

[8] "Mental Health - Our World in Data." https://ourworldindata.org/mental-health

[9] "Global prevalence of anxiety disorders: a systematic review " https://www.cambridge.org/core/journals/psychological-medicine/article/global-prevalence-of-anxiety-disorders-a-systematic-review-and-metaregression/484845CE01E709EE4FB6554AA78E612F

[10] "Meditation techniques v. relaxation therapies when treating " '19 https://www.cambridge.org/core/journals/psychological-medicine/article/meditation-techniques-v-relaxation-therapies-when-treating-anxiety-a-metaanalytic-review/6F167C7F5B2A00CB2039C05E89F6E5C2

[11] "Trait social anxiety as a conditional adaptation - ScienceDirect " https://www.sciencedirect.com/science/article/pii/S027322971930125X

[12] "Age, period, and cohort trends in mood disorder indicators " https://psycnet.apa.org/record/2019-12578-001

[13] "Original article Trends in Mood and Anxiety Symptoms and " https://www.sciencedirect.com/science/article/abs/pii/S1054139X1930254X

[14] Mental Health

[15] "Things That Sometimes Work If You Have Anxiety | Slate Star " '15 https://slatestarcodex.com/2015/07/13/things-that-sometimes-work-if-you-have-anxiety/

[16] "The correlation between stress and economic crisis: a ... - NCBI." '16 https://www.ncbi.nlm.nih.gov/pubmed/27143898

[17] "Mental health outcomes in times of economic recession - NCBI." '16 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4741013/

[18] "Multidisciplinary research priorities for the COVID-19 pandemic." '20 https://www.sciencedirect.com/science/article/pii/S2215036620301681

[19] "Digital Mental Health and COVID-19 - JMIR Mental Health." '20 https://mental.jmir.org/2020/3/e18848/

[20] "Digital Mental Health and COVID-19 - JMIR Mental Health." '20 https://mental.jmir.org/2020/3/e18848/

[21] "Is it Generalized Anxiety Disorder or Poverty? An Examination " '12 https://link.springer.com/article/10.1007/s10560-012-0263-3

[22] "A large and neglected problem - Happier Lives Institute." https://www.happierlivesinstitute.org/large-and-neglected-problem.html

[23] "Depression and Anxiety - Wiley Online Library." '18 https://onlinelibrary.wiley.com/doi/abs/10.1002/da.22711

[24] "Mental health | Wellcome." https://wellcome.ac.uk/what-we-do/our-work/mental-health-transforming-research-and-treatments

[25] "Mental Health — Inside Philanthropy." https://www.insidephilanthropy.com/mental-health-grants

[26] "Why This Top Funder Has Put Mental Health on its Radar " '16 https://www.insidephilanthropy.com/mental-health/2016/9/26/why-this-top-funder-has-put-mental-health-on-its-radar.html

[27] "Compliance with mandatory reporting of clinical trial results on " '12 https://www.bmj.com/content/344/bmj.d7373

[28] "Timing and completeness of trial results posted at Clinical Trials ... - PloS." '13 http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001566

[29] "Psychology's Replication Crisis and Clinical Psychological " https://www.annualreviews.org/doi/abs/10.1146/annurev-clinpsy-050718-095710

[30] "Replication and contradiction of highly cited research ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pubmed/26159600

[31] "Biases in research: risk factors for non-replicability in " https://www.cambridge.org/core/journals/psychological-medicine/article/biases-in-research-risk-factors-for-nonreplicability-in-psychotherapy-and-pharmacotherapy-research/5D18546D585D42C44AB53E8F633F0B60

[32] "Psychology's Replication Crisis and Clinical Psychological " https://www.annualreviews.org/doi/abs/10.1146/annurev-clinpsy-050718-095710

[33] "Double blinding requirement for validity claims in ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4863672/

[34] "Initial Severity and Antidepressant Benefits: A Meta ... - PLOS." '08 https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0050045

[35] "Efficacy of pharmacotherapy and psychotherapy for adult " https://www.ncbi.nlm.nih.gov/pubmed/24789675

[36] "Replication and contradiction of highly cited research ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pubmed/26159600

[37] "Practical Significance, Meta-Analysis and the ... - SSRN." '19 https://www.ssrn.com/abstract=3427595

[38] "Efficacy of pharmacotherapy and psychotherapy for ... - NCBI." https://www.ncbi.nlm.nih.gov/pubmed/24789675

[39] "Why Cognitive Behavioral Therapy Is the Current ... - NCBI - NIH." '18 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5797481/

[40] "Efficacy of pharmacotherapy and psychotherapy for ... - NCBI." https://www.ncbi.nlm.nih.gov/pubmed/24789675

[41] "It's the effect size, stupid - Semantic Scholar." '12 https://pdfs.semanticscholar.org/c5ac/87df5d6e0e6b6de2f745284835c2a368b0f7.pdf

[42] "Hedges' g: Definition, Formula - Statistics How To." '16 https://www.statisticshowto.datasciencecentral.com/hedges-g/

[43] "The effects of cognitive behavioral therapy as an anti ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pubmed/25961373

[44] "SSRIs: An Update | Slate Star Codex." '18 https://slatestarcodex.com/2018/11/07/ssris-an-update/

[45] "Trajectories of Depression Severity in Clinical Trials of " https://jamanetwork.com/journals/jamapsychiatry/fullarticle/1107437

[46] "Small Sample Sizes and a False Economy for Psychiatric " https://research-information.bris.ac.uk/en/publications/small-sample-sizes-and-a-false-economy-for-psychiatric-clinical-trials(7571421b-30a6-4581-a119-20784badb3a1).html

[47] "Let's Distinguish Relative and Absolute Efficacy to Move " https://www.ncbi.nlm.nih.gov/pubmed/31154927

[48] "Is psychotherapy effective? A re-analysis of treatments for " '18 https://www.cambridge.org/core/journals/epidemiology-and-psychiatric-sciences/article/is-psychotherapy-effective-a-reanalysis-of-treatments-for-depression/5D8EC85B6FA35B5CEE124381F18E51B9

[49] "The waiting list is an inadequate benchmark for estimating the " https://www.cambridge.org/core/journals/epidemiology-and-psychiatric-sciences/article/waiting-list-is-an-inadequate-benchmark-for-estimating-the-effectiveness-of-psychotherapy-for-depression/2EC58914B2B4FB9C1F3A54198BE352E5

[50] "How effective are cognitive behavior therapies for ... - NCBI." '16 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5032489/

[51] "Comparing meta-analyses and preregistered multiple ... - Nature." '19 https://www.nature.com/articles/s41562-019-0787-z

[52] "The Meaningfulness of Effect Sizes in Psychological ... - NCBI." '19 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6470248/

[53] "How effective are cognitive behavior therapies for ... - NCBI." '16 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5032489/

[54] "Cognitive behavioral therapy for anxiety and related disorders " '18 https://www.ncbi.nlm.nih.gov/pubmed/29451967

[55] https://books.google.co.uk/books?hl=en&lr=&id=YZ-BDwAAQBAJ&oi=fnd&pg=PP1&dq=anxiety+meta-analysis&ots=BD3PGDLmIx&sig=aiUBVFRzvRzF83H9r-BJ1KT0bXI&redir_esc=y#v=onepage&q=anxiety%20meta-analysis&f =false

[56] "Long-term Outcomes of Cognitive Behavioral Therapy for " '19 https://jamanetwork.com/journals/jamapsychiatry/fullarticle/2756136

[57] https://psycnet.apa.org/buy/2014-01442-001

[58] "Emotion regulation in social anxiety and depression: a ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pubmed/30064053

[59] "Benefit-finding and effect on caregiver depression: A ... - NCBI." '17 https://www.ncbi.nlm.nih.gov/pubmed/28287803

[60] "Team – Dare Response." https://dareresponse.com/team/

[61] "Dare: The New Way to End Anxiety and Stop ... - Goodreads." https://www.goodreads.com/book/show/26452130-dare

[62] "Mental Imagery in Depression - Annual Reviews." https://www.annualreviews.org/doi/full/10.1146/annurev-clinpsy-021815-092925

[63] "Autobiographical episodic memory-based training for the " https://www.sciencedirect.com/science/article/pii/S0272735816301969

[64] "Positive imagery cognitive bias modification (CBM) and ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pubmed/25805405

[65] "Mental Imagery in Depression - Annual Reviews." https://www.annualreviews.org/doi/abs/10.1146/annurev-clinpsy-021815-092925

[66] "Enhancing well-being and alleviating depressive symptoms " https://www.ncbi.nlm.nih.gov/pubmed/19301241

[67] "Positive psychology interventions: a meta-analysis of " '13 https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-13-119

[68] "Synthesizing positive psychological " https://www.researchgate.net/publication/273024791_Synthesizing_positive_psychological_interventions_Suggestions_for_conducting_and_interpreting_meta-analyses

[69] "Meta-analyses of positive psychology interventions - PLOS." '19 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0216588

[70] "A Meta-Analysis of the Effectiveness of Randomized " '20 https://link.springer.com/article/10.1007/s11482-019-09788-z

[71] "Gratitude and well-being: a review and theoretical integration.." https://www.ncbi.nlm.nih.gov/pubmed/20451313

[72] "Gratitude and health: An updated review: The Journal of " https://www.tandfonline.com/doi/abs/10.1080/17439760.2019.1651888

[73] "Gratitude and health: An updated review: The Journal of " https://www.tandfonline.com/doi/abs/10.1080/17439760.2019.1651888

[74] "(ACT) to reduce depression: A systematic review and meta " https://www.sciencedirect.com/science/article/pii/S0165032719313023

[75] "(ACT) to reduce depression: A systematic review and meta " https://www.sciencedirect.com/science/article/pii/S0165032719313023

[76] "What is the evidence for the efficacy of self-help acceptance " https://www.sciencedirect.com/science/article/pii/S2212144717300753

[77] "Internet-Delivered Acceptance and Commitment ... - JMIR." '19 https://www.jmir.org/2019/1/e12530/

[78] "A Randomized Controlled Trial of Multiple Versions of ... - NCBI." https://www.ncbi.nlm.nih.gov/pubmed/29262693

[79] "Effects of expressive writing on depressive symptoms—A meta " '18 https://onlinelibrary.wiley.com/doi/full/10.1111/cpsp.12224

[80] "A Meta-Analysis of Expressive Writing on Posttraumatic Stress " '19 https://journals.sagepub.com/doi/abs/10.1177/1089268019831645

[81] "Expressive writing as a therapeutic intervention for ... - NCBI." '19 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6676535/

[82] "Meditation techniques v. relaxation therapies when treating " '19 https://www.cambridge.org/core/journals/psychological-medicine/article/meditation-techniques-v-relaxation-therapies-when-treating-anxiety-a-metaanalytic-review/6F167C7F5B2A00CB2039C05E89F6E5C2

[83] "The effect of mindfulness-based therapy on anxiety and " https://psycnet.apa.org/record/2010-05835-004

[84] "Does mindfulness meditation improve anxiety and mood " https://psycnet.apa.org/record/2007-10109-008

[85] "Meditation techniques v. relaxation therapies when treating " '19 https://www.cambridge.org/core/journals/psychological-medicine/article/meditation-techniques-v-relaxation-therapies-when-treating-anxiety-a-metaanalytic-review/6F167C7F5B2A00CB2039C05E89F6E5C2

[86] "Mind The Hype: A Critical Evaluation and ... - NCBI - NIH." '17 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5758421/

[87] "Reporting of Positive Results in Randomized Controlled Trials " '16 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0153220

[88] "Doing no harm in mindfulness-based programs: Conceptual " https://www.sciencedirect.com/science/article/pii/S0272735818301272

[89] "Challenging and Adverse Meditation Experiences: Toward a " https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780198808640.001.0001/oxfordhb-9780198808640-e-51

[90] "Self-Regulation of Breathing as a Primary Treatment for Anxiety." '15 https://link.springer.com/article/10.1007%2Fs10484-015-9279-8

[91] "Smartphone apps for psychological health: A brief state ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pubmed/30407057

[92] "New breathing therapy reduces panic and anxiety by " '10 https://www.sciencedaily.com/releases/2010/12/101220200010.htm

[93] "Comparison of prolonged exposure (PE) coach to treatment " https://psycnet.apa.org/record/2017-70145-001

[94] "Six interventions for Anxiety: Somatic training, affect regulation " '20 http://repository.cityu.edu/handle/20.500.11803/869

[95] "Is cognitive-behavioural therapy more effective than relaxation " '17 https://www.ncbi.nlm.nih.gov/pubmed/29037266

[96] "Meditation techniques v. relaxation therapies when treating " '19 https://www.cambridge.org/core/journals/psychological-medicine/article/meditation-techniques-v-relaxation-therapies-when-treating-anxiety-a-metaanalytic-review/6F167C7F5B2A00CB2039C05E89F6E5C2

[97] "Relaxation training for anxiety - BMC Psychiatry - BioMed " '08 https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-8-41

[98] "The efficacy of relaxation training in treating anxiety." https://neuroptimal.com/wp-content/uploads/2019/05/2010-18458-003.pdf

[99] "How WEIRD are positive psychology interventions? - Taylor " '18 https://www.tandfonline.com/doi/full/10.1080/17439760.2018.1484941

[100] "Articles Efficacy of psychosocial interventions for mental " https://www.sciencedirect.com/science/article/abs/pii/S2215036619305115

[101] "The efficacy of positive psychological interventions from non " https://internationaljournalofwellbeing.org/index.php/ijow/article/view/711

[102] "Gratitude Interventions: Meta-analytic Support for Numerous " '19 https://link.springer.com/chapter/10.1007/978-3-030-20020-6_6

[103] "A systematic review of online interventions for mental ... - NCBI." '15 https://www.ncbi.nlm.nih.gov/pubmed/28596860

[104] "Internet-Based Cognitive Behavioral Therapy for Social " '17 https://boris.unibe.ch/94889/

[105] "Step-by-Step: a new WHO digital mental health ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6131163/

[106] "Step-by-Step, an E-Mental Health Intervention for ... - NCBI - NIH." '20 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7034323/

[107] "Step-by-Step, an E-Mental Health Intervention for ... - NCBI - NIH." '20 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7034323/

[108] "The state of mental digi-therapeutics: A systematic " https://www.sciencedirect.com/science/article/pii/S1386505619301054

[109] "Is India ready for mental health apps (MHApps)? A ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pubmed/30231056

[110] "Anxiety and its disorders as risk factors for suicidal thoughts " https://www.sciencedirect.com/science/article/abs/pii/S0272735815300714

[111] "Anxiety sensitivity and suicidal ideation/suicide risk: A meta " https://psycnet.apa.org/record/2018-51572-006

[112] "Anxiety Disorders." https://link.springer.com/content/pdf/10.1007%2F978-3-030-30687-8.pdf

[113] "Global, regional, and national burden of " https://www.bmj.com/content/364/bmj.l94

[114] "The Burden Attributable to Mental and Substance Use ... - PLOS." '14 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0091936

[115] "Do mental health mobile apps work: evidence ... - NCBI - NIH." '18 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5897664/

[116] "Preference for in-person psychotherapy versus digital ... - Nature." '19 https://www.nature.com/articles/s41746-019-0077-1

[117] "Digital Mental Health and COVID-19 - JMIR Mental Health." '20 https://mental.jmir.org/2020/3/e18848/

[118] "The efficacy of smartphone-based mental health interventions " https://www.ncbi.nlm.nih.gov/pubmed/28941113

[119] "Mental health and employers: Refreshing the case ... - Deloitte." https://www2.deloitte.com/uk/en/pages/consulting/articles/mental-health-and-employers-refreshing-the-case-for-investment.html

[120] "Smartphones in mental health - International Journal of " '20 https://journalbipolardisorders.springeropen.com/articles/10.1186/s40345-019-0164-x

[121] https://luiseduve.github.io/files/2019_MasterThesis_LuisQuintero.pdf

[122] "Serious Games and Gamification for Mental Health: Current " '17 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5222787/

[123] "Reducing concussion symptoms among teenage youth " https://www.tandfonline.com/doi/abs/10.1080/02699052.2017.1332388?journalCode=ibij20

[124] "A review of popular smartphone apps for depression and anxiety." https://www.sciencedirect.com/science/article/pii/S0005796719301846

[125] "Anxiety: There is an app for that. A systematic review of ... - NCBI." https://www.ncbi.nlm.nih.gov/pubmed/28504859

[126] "The Digital Psychiatrist: In Search of Evidence-Based ... - NCBI." '19 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6872533/

[127] "The efficacy of smartphone-based mental health interventions " https://www.ncbi.nlm.nih.gov/pubmed/28941113

[128] "A review of popular smartphone apps for depression and anxiety." https://www.sciencedirect.com/science/article/abs/pii/S0005796719301846

[129] "A review of popular smartphone apps for depression and anxiety." https://www.ncbi.nlm.nih.gov/pubmed/31707224

[130] "The Emerging Imperative for a Consensus Approach Toward " https://journals.lww.com/jonmd/FullText/2018/08000/The_Emerging_Imperative_for_a_Consensus_Approach.13.aspx

[131] "How we assess health apps and digital tools - NHS Digital." '19 https://digital.nhs.uk/services/nhs-apps-library/guidance-for-health-app-developers-commissioners-and-assessors/how-we-assess-health-apps-and-digital-tools

[132] "NHS Apps Library - NHS." https://www.nhs.uk/apps-library/

[133] "A Review of Popular Smartphone Apps for Depression and " '19 https://psyarxiv.com/zp362/

[134] "A review of popular smartphone apps for depression and anxiety." https://www.sciencedirect.com/science/article/pii/S0005796719301846

[135] "Mobile Apps for the Management of Comorbid ... - Hindawi." https://www.hindawi.com/journals/jhe/2020/9317179/

[136] "Standalone smartphone apps for mental health—a systematic " '19 https://www.nature.com/articles/s41746-019-0188-8

[137] "Standalone Apps for Anxiety and Depression Show Promising " '20 https://psyarxiv.com/v48w9/

[138] "The efficacy of smartphone-based mental health interventions " https://www.ncbi.nlm.nih.gov/pubmed/28941113

[139] "A meta-analysis of randomized controlled trials." '17 https://www.sciencedirect.com/science/article/pii/S0165032717300150

[140] "Waiting times for primary care psychological therapy and " https://www.tandfonline.com/doi/pdf/10.1080/14733140600581358

[141] "Adding a smartphone app to internet-based self-help for social " https://psycnet.apa.org/record/2018-36910-011

[142] "Guided and unguided Acceptance and Commitment Therapy " '16 https://www.ncbi.nlm.nih.gov/pubmed/27721123

[143] "The efficacy of app-supported smartphone interventions for " https://www.ncbi.nlm.nih.gov/pubmed/31496095

[144] "Internet- and mobile-based interventions for anxiety ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pubmed/30450811

[145] "User Engagement in Mental Health Apps: A Review of " https://ps.psychiatryonline.org/doi/10.1176/appi.ps.201800519

[146] "Unguided Mental Health Self-help Apps: Reflections ... - NCBI." https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5688908/

[147] "Dropout rates in clinical trials of smartphone apps for " '20 https://www.sciencedirect.com/science/article/abs/pii/S0165032719326060

[148] "In the Absence of Effects: An Individual Patient Data Meta " '19 https://www.ncbi.nlm.nih.gov/pubmed/30984061

[149] "Effectiveness of a tailored, integrative Internet ... - PLOS." '20 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0228100

[150] "Attrition and adherence in smartphone-delivered ... - NCBI." '19 https://www.ncbi.nlm.nih.gov/pubmed/31697093

[151] "Internet-based cognitive and behavioural therapies for post " '18 https://www.cochrane.org/CD011710/DEPRESSN_internet-based-cognitive-and-behavioural-therapies-post-traumatic-stress-disorder-ptsd

[152] "E-health interventions for anxiety and depression in children " '18 https://www.cochrane.org/CD012489/DEPRESSN_e-health-interventions-anxiety-and-depression-children-and-adolescents-long-term-physical-conditions

[153] "Technology Delivered Interventions for Depression ... - NCBI." https://www.ncbi.nlm.nih.gov/pubmed/30229343

[154] "Do mental health mobile apps work: evidence ... - NCBI - NIH." '18 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5897664/

[155] "Can Acceptance, Mindfulness, and Self-Compassion Be " '19 https://www.sciencedirect.com/science/article/abs/pii/S0005789419301236

[156] "Internet Interventions for Adults with Anxiety and Mood ... - NCBI." '19 https://www.ncbi.nlm.nih.gov/pubmed/31096757

[157] "Anxiety apps: Can you lessen anxiety by playing a game ... - Vox." '20 https://www.vox.com/the-highlight/2019/9/17/20863016/anxiety-app-phone-gamification

[158] "PubMed - NCBI." '17 https://www.ncbi.nlm.nih.gov/pubmed/28922737

[159] "PubMed - NCBI." '17 https://www.ncbi.nlm.nih.gov/pubmed/28922737

[160] "PubMed - NCBI." '17 https://www.ncbi.nlm.nih.gov/pubmed/28922737

[161] "Disability-adjusted life year - Wikipedia." https://en.wikipedia.org/wiki/Disability-adjusted_life_year

[162] "QALYs and their role in the NICE decisionâ - Wiley Online " https://onlinelibrary.wiley.com/doi/pdf/10.1002/psb.1562

[163] "Carrying NICE over the threshold | Blog | News | NICE." '15 https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold

[164] "Cost-effectiveness of interventions for treating anxiety disorders." https://www.ncbi.nlm.nih.gov/pubmed/27988373

[165] "Economic evaluations of internet- and mobile-based ... - NCBI." '17 https://www.ncbi.nlm.nih.gov/pubmed/28922737

[166] "Mobile and traditional cognitive behavioral therapy programs " '18 Mobile and traditional cognitive behavioral therapy programs for generalized anxiety disorder: A cost-effectiveness analysis.

[167] "Long-Term Effectiveness and Cost-Effectiveness of ... - NCBI." '20 Long-Term Effectiveness and Cost-Effectiveness of Videoconference-Delivered Cognitive Behavioral Therapy for Obsessive-Compulsive Disorder, Panic Disorder, and Social Anxiety Disorder in Japan: One-Year Follow-Up of a Single-Arm Trial.

[168] "Clinical and cost-effectiveness of therapist-guided internet " '14 Clinical and cost-effectiveness of therapist-guided internet-delivered cognitive behavior therapy for older adults with symptoms of anxiety: a randomized controlled trial.

[169] "Tailored online cognitive behavioural therapy with or without " '17 Tailored online cognitive behavioural therapy with or without therapist support calls to target psychological distress in adults receiving haemodialysis: A feasibility randomised controlled trial.

[170] "A feasibility study of the clinical effectiveness and cost " '16 A feasibility study of the clinical effectiveness and cost-effectiveness of individual cognitive behavioral therapy for panic disorder in a Japanese clinical setting: an uncontrolled pilot study

[171] "A prospective service evaluation of acceptance and ... - NCBI." '20 A prospective service evaluation of acceptance and commitment therapy for patients with refractory epilepsy.

[172] "Mediation (statistics) - Wikipedia." https://en.wikipedia.org/wiki/Mediation_(statistics)

[173] "Mapping mental health condition-specific measures to ... - NCBI." https://www.ncbi.nlm.nih.gov/books/NBK262023/

[174] "Mobile and traditional cognitive behavioral therapy ... - NCBI." '18 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5754075/

[175] "Validation and Standardization of the Generalized Anxiety Di " https://journals.lww.com/lww-medicalcare/Fulltext/2008/03000/Validation_and_Standardization_of_the_Generalized.6.aspx

[176] "Validation and Standardization of the Generalized Anxiety Di " https://journals.lww.com/lww-medicalcare/Fulltext/2008/03000/Validation_and_Standardization_of_the_Generalized.6.aspx

[177] "A meta-analysis of randomized controlled trials." '17 https://www.sciencedirect.com/science/article/pii/S0165032717300150

[178] "The efficacy of app-supported smartphone interventions for " https://www.ncbi.nlm.nih.gov/pubmed/31496095

[179] Founders Pledge cause area report on Women's Empowerment: cost-effectiveness analyses

[180] "Long-term Outcomes of Cognitive Behavioral Therapy for " '19 https://jamanetwork.com/journals/jamapsychiatry/fullarticle/2756136

[181] "The value of money going to different groups - Centre for " '17 https://www.centreforeffectivealtruism.org/blog/the-value-of-money-going-to-different-groups/

[182] "Carrying NICE over the threshold | Blog | News | NICE." '15 https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold

[183] "Cost Per Daly Averted in Low, Middle and High ... - SSRN." '19 https://www.ssrn.com/abstract=3420403

[184] "Crucial Considerations and Wise Philanthropy - Effective " '14 https://www.effectivealtruism.org/articles/crucial-considerations-and-wise-philanthropy-nick-bostrom/

[185] "Mental Health Cause Area Report - Founders Pledge." https://founderspledge.com/research/fp-mental-health

[186] "Amazon Found Every 100ms of Latency Cost them 1% in Sales." '19 https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/

[187] "Mental Health Cause Area Report - Founders Pledge." https://founderspledge.com/research/fp-mental-health

[188] "The New Science of Pleasure." http://www.nber.org/papers/w18687.pdf

[189] "The Irrational Consumer: Why Economics Is Dead Wrong " '13 https://www.theatlantic.com/business/archive/2013/01/the-irrational-consumer-why-economics-is-dead-wrong-about-how-we-make-choices/267255/

[190] Contra Contra Contra Caplan On Psych

[191] "List of ways in which cost-effectiveness estimates can be " https://forum.effectivealtruism.org/posts/zdAst6ezi45cChRi6/list-of-ways-in-which-cost-effectiveness-estimates-can-be

[192] "GiveWell donors supported more than direct delivery: AMF " '20 https://blog.givewell.org/2020/04/23/givewell-donors-supported-more-than-direct-delivery-amf-and-new-net-research/

[193] "Bednets Have Prevented 450 Million Cases of Malaria - the " '15 https://www.givingwhatwecan.org/post/2015/12/bednets-have-prevented-450-million-cases-of-malaria/

[194] "What is the clinical value of mHealth for patients? | npj Digital " '20 https://www.nature.com/articles/s41746-019-0206-x

[195] "Medication for social anxiety disorder (SAnD): a review of the " '17 https://www.cochrane.org/CD001206/DEPRESSN_medication-social-anxiety-disorder-sand-review-evidence

[196] "e-Therapy Interventions for the Treatment of ... - NCBI - NIH." '18 https://www.ncbi.nlm.nih.gov/books/NBK532212/

[197] https://en.wikipedia.org/wiki/Generalized_anxiety_disorder

[198] Things That Sometimes Work If You Have Anxiety

[199] Things That Sometimes Work If You Have Anxiety

[200] Social Media Use and Mental Health: A Review

[201] "Increases in Depression, Self-Harm, and Suicide Among U.S. " https://prcp.psychiatryonline.org/doi/full/10.1176/appi.prcp.20190015

[202] "Are Social Media Ruining Our Lives? A Review of Meta " '19 https://journals.sagepub.com/doi/10.1177/1089268019880891

[203] "Negative effects associated with internet " https://www.sciencedirect.com/science/article/pii/S2214782919300831

[204] "EVALUATING THE ECONOMIC EVIDENCE OF WEB AND " https://cdr.lib.unc.edu/concern/masters_papers/ht24wm28t

[205] "The Openness‐equality Trade‐off in Global ... - Wiley Online Library." https://onlinelibrary.wiley.com/doi/abs/10.1111/ecoj.12469

[206] "Alleviating Global Poverty: Labor Mobility, Direct Assistance, and " '18 https://www.cgdev.org/publication/alleviating-global-poverty-labor-mobility-direct-assistance-and-economic-growth

[207] "The Openness‐equality Trade‐off in Global ... - Wiley Online Library." https://onlinelibrary.wiley.com/doi/abs/10.1111/ecoj.12469

[208] "Subjective Well-Being, Income, Economic Development and Growth." http://www.nber.org/papers/w16441

[209] "The law of logarithmic returns - Future of Humanity Institute." '14 https://www.fhi.ox.ac.uk/law-of-logarithmic-returns/

[210] "Isoelastic utility - Wikipedia." https://en.wikipedia.org/wiki/Isoelastic_utility

[211] "The value of money going to different groups - Centre for Effective " '17 https://www.centreforeffectivealtruism.org/blog/the-value-of-money-going-to-different-groups/