Today, Nature published its third critique of Sala et al. 2021, the most covered ocean science paper of the past decade. The ‘official’ publication of Ovando et al. 2023 puts a final nail in the coffin of this case of poor science.
Ovando et al. 2023 criticizes the food provisioning model first proposed in Cabral et al. 2020—a since-retracted paper by the same group of authors that preceded Sala et al. 2021.
Sala et al. 2021 used 3 models—all 3 haven’t held up to close examination
Sala et al. 2021 recommended more marine protected areas (MPAs) using three different computer models to say that increased MPAs would improve ocean biodiversity, decrease carbon emissions, and increase seafood production. However, each of the models used bizarre assumptions to create their conclusions. For example, the biodiversity model assumed that creating new MPAs would eliminate fishing pressure rather than displace it. However, when fishermen are told they can’t fish in an area, they do not vanish, they fish elsewhere. An official critique of this point was published in April 2022.
The carbon emissions model was the most attention-getting and perhaps the most inaccurate. Nearly every media organization in the U.S. and Europe led their coverage of Sala et al. 2021 with the claim that bottom trawling emitted as much carbon as all of airline travel, but the estimate was off by at least two or three orders of magnitude, or 100 – 1000x! Their carbon model assumed that stored carbon in sediments was the same as carbon on the top layer of the ocean disturbed by trawling. Our explainer of the carbon model shows how illogical an assumption that was and a recent empirical study of carbon disturbance at trawled sites ended any notion of the model’s legitimacy. The official critique to correct the record was published in May 2023.
Food model also relied on poor assumptions
Now, Ovando et al. 2023 is illuminating the shortcomings of the food model used in Sala et al. 2021.
Sala et al. 2021 used a fisheries model that claimed a 5% increase in MPAs leads to 20% more fishing yield. (For added context, this model was adapted from the retracted Cabral et al. 2020). In their critique of the food production model, Ovando et al. 2023 point out two unreasonable assumptions in the model that led to implausible results:
- That density dependence is a function of total pooled population size, independent of how fish are distributed in space.
- Unassessed fish stocks of a given species are a single global interconnected population.
Ovando et al. 2023 say that “These two assumptions generate results that are neither consistent with their source material nor ecologically reasonable”
The global density dependence assumptions of Sala et al. 2021 essentially mean that birthrates of Atlantic cod off the coast of Norway are influenced by Atlantic cod off the coast of Massachusetts… those populations are thousands of miles away from each other and never interact!
Ovando et al. 2023 demonstrate that using regional rather than global assumptions (i.e. birthrates of Atlantic cod are influenced by the population they are a part of, not global Atlantic cod abundance) results in a 62% decrease in maximum yield from what Sala et al. 2021 report. Modeling regionally is more valid than modeling globally, and the authors of Sala et al. 2021 probably knew that as Ovando et al. 2023 point out: “The assumption that density dependence occurs at the local scales used in our regional results is common in the MPA modelling literature, including in studies authored by authors of the study by Sala et al.”
Another global vs regional assumption gone wrong
The other misleading assumption was that unassessed fish stocks (i.e. fish populations not reported in the RAM Legacy database) act as a single global, interconnected population. The choice to use that assumption led to absurd results: “MPAs around Australia [could] increase catches along the shores of North America, or a single fish population can be affected both by MPAs in the Caribbean and in the waters off of China.”
Unreasonable assumptions lead to unreasonable results
The strange outcomes could be seen in the maps published in Sala et al. 2021 showing where MPAs would be most effective for increasing food production. The maps implied heavy MPA coverage around the U.S. and relatively little around Southeast Asia. This should have been a dead giveaway to the peer reviewers that something in the model was off. The U.S. manages its fisheries extremely well while Southeast Asia has much more overfishing due to lower capacity for management and enforcement. From Ovando et al. 2023:
The global results place much of the west coast of North America in the top 30% of areas for protection, but omit much of the coastal Indian Ocean and the Coral Triangle. These results are flipped under our regional assumptions. The global assumptions of Sala et al. suggest that 46% of the exclusive economic zone of the USA could be placed in MPAs while increasing or maintaining food production, whereas under our regional assumptions that number drops to 13%.
Ovando et al. 2023 note that rerunning Sala et al. 2021’s model with regional density dependence and limiting the distribution of unassessed populations as opposed to globally interconnected significantly changed the outcome. They note:
We are not suggesting that the regional results are the ‘right’ findings. Instead, we are demonstrating that the central results of Sala et al. are not robust to changes to their core assumptions.
In a response to Ovando et al.’s criticism, Sala et al. note their goals for the paper were to create a framework to “prioritize MPAs in places that would result in multiple benefits today and in the future.” But, as with any place-based conservation measure, MPAs are highly context dependent. Removing local and regional context to create a “global” analysis does a disservice to conservation and science by increasing underlying uncertainties. Publishing in Nature and garnering significant media attention masks that uncertainty and creates misinformation.
The conclusion from our coverage of the Cabral et al. 2020 retraction remains:
The science in both Cabral et al. and Sala et al. is critically flawed, but being used to advocate for public policy. Both follow a recent trend of publishing predictions that use a limited set of assumptions (in a very uncertain world) to produce global maps that get published in high-profile journals and garner considerable media and political attention.
Computer models are essential tools for science and management, but the accuracy of their predictions depends on both the quality of the data and the assumptions they are based on. Often, a problem is so complex that several assumptions may be equally plausible; readers need to be made aware when different assumptions lead to vastly different outcomes.
The Cabral et al. and Sala et al. papers disregard uncertainty in favor of set values for their model parameters. They don’t account for the enormous uncertainty in these parameters and don’t provide strong evidence that their choice of values was correct. The assumptions and parameters produce big headlines, but are fundamentally unhelpful for the future of ocean governance and sustainability. We expect policy-makers and resource managers to make decisions based on the best available science. Inconsistent and unrealistic assumptions are not that.
Max Mossler
Max is the managing editor at Sustainable Fisheries UW.
