by Judith Curry

Tim Palmer’s new book has just been published: “The Primacy of Doubt: From Quantum Physics to Climate Change, How the Science of Uncertainty Can Help Us Understand Our Chaotic World”

This book is a physics-intellectual feast.  Must read.

The book can be purchased at amazon.com

Blurbs from the publisher:

“A bold, visionary, and mind-bending exploration of how the geometry of chaos can explain our uncertain world – from weather and pandemics to quantum physics and free will. Covering a breathtaking range of topics – from climate change to the foundations of quantum physics, from economic modelling to conflict prediction, from free will to consciousness and spirituality – The Primacy of Doubt takes us on a unique journey through the science of uncertainty. A key theme that unifies these seemingly unconnected topics is the geometry of chaos: the beautiful and profound fractal structures that lie at the heart of much of modern mathematics. Royal Society Research Professor Tim Palmer shows us how the geometry of chaos not only provides the means to predict the world around us, it suggests new insights into some of the most astonishing aspects of our universe and ourselves. This important and timely book helps the reader makes sense of uncertainty in a rapidly changing world.”

“In The Primacy of Doubt, Palmer argues that embracing the mathematics of uncertainty is vital to understanding ourselves and the universe around us. Whether we want to predict climate change or market crashes, understand how the brain is able to outpace supercomputers, or find a theory that links quantum and cosmological physics, Palmer shows how his vision of mathematical uncertainty provides new insights into some of the deepest problems in science. The result is a revolution—one that shows that power begins by embracing what we don’t know.”

“The Primacy of Doubt explains how the geometry of chaos allows us to understand why systems can be predictable most of the time, but spectacularly unpredictable on occasion”

Comments from leading physicists:

“The Primacy of Doubt provides a remarkably broad-ranging account of uncertainty in physics, in all its various aspects. I strongly recommend this highly thought-provoking book.” – Roger Penrose, OM, FRS, winner of the 2020 Nobel Prize in Physics

“Tim Palmer is a scientific polymath. It’s hard to think of anyone else who could have written so authoritatively-and so accessibly-on themes extending from quantum gravity to climate modelling. This fascinating and important book offers some profoundly original speculations on conceptual linkages across different sciences. – Lord Martin Rees, Astronomer Royal of the United Kingdom

“The Primacy of Doubt is an important book by one of the pioneers of dynamical weather prediction, indispensable for daily life.” – Suki Manabe, winner of the 2021 Nobel Prize in Physics

“In a whirlwind of a book that’s partly scientific autobiography and partly the manifest of a visionary, Tim Palmer masterfully weaves together climate change and quantum mechanics into one coherent whole. Using uncertainty as a unifying principle, Palmer puts forward new perspectives on old problems. A revolutionary thinker way ahead of his time.” – Sabine Hossenfelder, author of Lost in Math

From the PopScience review: “This is quite possibly the best popular science book I’ve ever read (and I’ve read hundreds).”  “What is astounding is the the way that Palmer rattles through a series of topics that are quite difficult to get our head around and, in several diverse cases, give the most approachable explanations I’ve ever seen.”

Palmer’s biosketch:  Tim Palmer, FRS, CBE is a Royal Society Research Professor in the department of physics at the University of Oxford. He pioneered the development of operational ensemble weather and climate forecasting. Palmer is a Commander of the British Empire, a fellow of the Royal Society and the U.S. National Academy of Sciences, and a recipient of the Institute of Physics’ Dirac Gold Medal.

The Author’s note is a fascinating read on Palmer’s scientific background and intellectual journey.  In the 1970’s at Oxford, Palmer studied general relativity and black holes under Dennis Sciama.  In considering what to do as a postgraduate, Palmer turned down a postdoc with Stephen Hawking(!).  Palmer became interested in meteorology in a chance meeting with Raymond Hide, who had interests in both meteorology and astronomy.  On a whim, he applied for a position at the UK Met Office, and was offered a job.  His wrenching decision that resulted in turning down the opportunity to work with Hawking is fascinating reading.  His growing interest in chaos theory was the source of many of his seminal contributions to meteorology, but also rekindled his interest in quantum physics.  There is no question that this is the journey of a unique and profound intellect, that the fields of meteorology and climate have been extremely fortunate to encounter.

Table of Contents

Author’s note

Introduction

PART I: THE SCIENCE OF UNCERTAINTY

Chaos, Chaos Everywhere
The Geometry of Chaos
Noisy, Million-Dollar Butterflies
Quantum Uncertainty: Reality Lost?

PART II: PREDICTING OUR CHAOTIC WORLD

The Two Roads to Monte Carlo
Climate Change—Catastrophe or Just Lukewarm?
Pandemics
Financial Crashes
Deadly Conflict and the Digital Ensemble of Spaceship Earth
Decisions! Decisions!

PART III: UNDERSTANDING THE CHAOTIC UNIVERSE AND OUR PLACE IN IT

Quantum Uncertainty: Reality Regained?
Our Noisy Brains
Free Will, Consciousness, and God

JC’s overview and review

This is a beautifully written book: eloquent while at the same time approachable, spiked with anecdotes and occasional self-deprecating humor.  Some quotes from the Introduction, that give a flavor of the book, including the style of writing.

“I will be elevating the notion of uncertainty, or doubt, to a status it is not usually given: not as an “Oh, I suppose we’d better do a risk analysis” afterthought, but as a matter of primal importance and focus. There are two reasons for doing this. First, there is the practical reason that we are liable to make lousy decisions if we base them on predictions with unreliable estimates of uncertainty. But just as important, at least for me as a scientist, we may be able to understand better the way systems work by focusing on the ways in which they are or can become uncertain.”

“In short, the scope of this book is quite unique. On the one hand we cover the loftiest questions that philosophy has ever addressed and attempt to answer them in novel ways. On the other we’ll describe practical techniques that have transformed the way we go about predicting how our world will evolve over the coming days, years and decades ahead. Some readers, I hope, will be exhilarated by the discussion of long-standing conceptual problems like free will, consciousness and the puzzling nature of quantum physics. Others may be thrilled to see the science of chaos applied for the benefit of society (and some of the poorest parts of society in particular). For yet others, reading this book may help them understand themselves better. Not least they may come to realize that some of our apparent shortcomings are not signs of irrationality or failure, but manifestations of our unique ability to cope with the enormous uncertainties of life. There is, I hope, something for everyone.”

Part I:  “Three important ideas are discussed in Part I. The first is that a type of geometry—what I call the geometry of chaos—explains why some systems can be stable and predictable for much of the time, and yet their future behaviour becomes completely uncertain on occasion. The second idea applies to systems that are so complex, we will never be able to model them exactly. In such situations, adding noise to a model can be a good way to represent some of the missing complexity. In this way, noise is often a positive constructive resource, and not the nuisance we typically think it to be.”

Part I deals with topics that I last encountered in my undergraduate Modern Physics course (in the 1970’s), so I don’t have much of a framework for interpreting this. Historical and philosophical perspectives are combined with the scientific exposition. It is a very interesting read and for the most part understandable with my limited background (my interest flagged a bit in Chapter 4). Part I sets the stage for Parts II and III.

Part I is PopSci’s favorite:  “In terms of this broad enlightening nature, the first of the three sections in the book stands out head and shoulders above the rest. Palmer starts by exploring chaos and gives the best explanation of the behaviour of chaotic systems, state space and attractors I’ve come across. Then he throws in Cantor sets, then shows the relationship of weather forecasts to all this, and introduces p-adic numbers (arguably the only bit that could have been better explained). He then shows graphically (literally, not metaphorically) how the introduction of noise can make models of chaotic systems work better. Finally in this section, he takes on quantum uncertainty, with one of the only explanations of the use of Bell’s inequality I’ve ever seen that is at least vaguely comprehensible.”

Part II: “We apply the ideas discussed in Part I to develop practical tools for predicting complex, inherently uncertain systems. The key idea here is ensemble prediction: running our models multiple times while varying uncertain initial conditions and model equations. When the spread of a reliable ensemble system is small, we can make fairly precise forecasts with confidence. By contrast, when the spread is large, we can make forecasts only using the language of probability. In this way, the geometry of chaos is manifest in the variable spread of the ensemble. Ensemble prediction methods are applied to weather and climate (where the techniques are reasonably mature and well validated), and to disease, economics and conflict (where techniques are still in development).”

Part II is to me the liveliest part of the book, with real world applications of the ideas of ensemble prediction.  Chapter 5 provides a fascinating account of the history of weather forecasts, including numerical weather prediction.  The focus of the chapter is Monte Carlo methods that provide the basis for ensemble weather prediction, providing the basis for assessing forecast uncertainty, including Palmer’s central contributions.  (Chapter 6 on Climate Change is discussed in the final section of this review).  In the other chapters of Part II, Palmer speculates on how ensemble forecast approaches could benefit epidemiological models that forecast pandemics, economic models and their failure to predict financial crashes, and models of wars/conflicts.

From PopSci:  “things really liven up when we get onto economics, and how economists are stuck in the fairly useless state meteorologists were before the great storm of 1987, when they used single-run forecasts, rather than ensembles. He also shows fairly bluntly that economists have failed in the development of the kind of models that can handle a chaotic system like the economy.”

Chapter 10 on decisions illustrates real world examples for using ensemble weather forecasts to support decision making – including Peter Webster’s work on probabilistic flood forecasts for Bangladesh.

Part III: “Two key ideas from Part I are applied to try to understand two of the most puzzling aspects of the universe: the world of quantum physics and ourselves. We start by assuming that the geometry of chaos applies to the universe as a whole. This leads to the crucial idea that certain counterfactual worlds, where we might have done something but didn’t, could actually be inconsistent with the laws of physics. This helps make comprehensible certain long-standing quantum mysteries. Following this we discuss the idea that the human brain makes constructive use of noise to model the world around us, and that this has enabled us to become the creative species we are. I then speculate that the geometry of chaos can help explain two of our most visceral but puzzling experiences: having free will and being conscious.”

Part III is more speculative.  Chapter  11 describes Palmer’s new thinking on quantum uncertainty, incorporating the geometry of chaos.   Pretty interesting.

Chapter 12 “Our Noisy Brains” is one of my favorite chapters of the book.   Palmer hypothesizes that the human brain makes constructive use of noise in trying to create a low-order model of the high-order world around us. He argues that such noise is a key element in making us creative, innovative beings.  Palmer extends Daniel Kahnemann’s thinking fast and slow by considering the brain’s energetics in describing these modal operation of the brain.  In the low power mode (thinking fast), when the brain is multi-tasking and is more susceptible to noise, random new ideas can be generated – eureka moments. In this framing, creativity involves a synergy between the low-power (stochasticity, noise) and the power-intensive (deterministic) modes of thinking.

Chapter 13 extends these ideas to include a role for quantum physics in counterfactual reasoning in the brain in relation to what human’s regard as free will.

Climate Change

Two reasons to highlight Chapter 6 on Climate Change: this is a climate blog; and the first major review of this book focuses on the climate change issue (see the Scientific American review  below).  The chapter provides a good overview of the greenhouse effect, feedbacks, etc.  and also the history of climate modelling.  Palmer describes his concerns about global climate models, and his plan for the next generation of climate models at much higher resolution.

In line with the major themes of the book in terms of uncertainty and ensemble prediction:

“Indeed, it’s simply not possible to describe climate change in an objective, scientific way without explaining the three roles of ensembles: for estimating the uncertain feedback effects in climate science; for estimating the impact of climate policy on climate change; and for separating the effects of natural chaotic variability from human-induced effects. From the first way of using ensembles, we can address how big an impact our emissions of greenhouse gases are having on global temperatures. With the second set of ensembles, we can assess whether mitigating actions can be effective or not. And from the third set of ensembles, we can not only assess the extent to which observed changes in weather and climate are natural (“climate is always changing”), we can try to attribute specific weather events to climate change, at least in a probabilistic way.”

The more interesting part of the Chapter addresses the question in the chapter subtitle:  “Catastrophe or Just Lukewarm?” Palmer addresses how we should view climate change from a perspective that is consistent with the “primacy of doubt,” treading carefully around the “merchants of doubt” meme. Both sides of the argument are reviewed, which are referred to as the “maximalist” and “minimalist” positions.  In Chapter 6 he concludes:

“Let’s summarise. Taking a specific position—minimalist, maximalist or indeed any specific point in between—is simply inconsistent with the science. The key message of this chapter is that one’s attitude to climate change, like to weather prediction, should be framed in terms of risk: Is the risk of undesirable changes to climate high enough to warrant taking precautionary action now?”

Palmer takes on this question more directly in Chapter 10 Decisions! Decisions!  Palmer introduces the concept of a Statistical Life (which, when expressed in terms of GDP per capita) can be applied to both developed and developing country inhabitants.  His conclusion is that it is overwhelmingly worthwhile to take action now to reduce the risk of a 4 degree “hell on earth” warmer world at an assumed 30% probability.

Hard to disagree with that conclusion related to 4C warming, but the devil is in the assumption of 30% probability of 4 C warming.  With RCP8.5 increasingly being regarded as implausible, and with the IPCC AR6 putting the the upper likely bound of equilibrium climate sensitivity at 4C, the chances of 4C warming are now generally regarded as quite small.

Something here for both the maximalists and minimalists to find unsatisfactory.

Scientific American review

As an example of how this is unsatisfactory to the maximalists, the first major review of the book in the Scientific American is entitled Uncertainty can speed up climate action.

The title speaks for itself.  Any book that mentions climate change, particularly one by a renowned scientist, is expected by the climatariat to urge action for reducing fossil fuels.  From the review:

“But Palmer struggles to frame both the uncertainties of climate change and the severity of its effects. He tees up the chapter (subtitled “Catastrophe or Just Lukewarm?”) by defaulting to a both-sides approach: Are the “maximalists” right to suggest we’re in an emergency and should decarbonize as much and as quickly as possible, or are the “minimalists” right in suggesting that uncertainty is grounds for delaying action? The truth, he writes, is somewhere in the middle. Palmer notes that doubling atmospheric carbon dioxide alone would warm the planet by one degree Celsius. (That’s without factoring in feedback loops it might cause, such as the loss of ice cover or more water vapor in the atmosphere, which would further turn up the heat.) This is, he says, “perhaps not something to make a big deal of.”  (JC note:  this statement about 1 C is taken out out of context of Palmer’s overall argument).

Then the reviewer (Brian Kahn, a journalist), clearly with Merchants of Doubt in his brain, seeks to educate Palmer about climate change:

“But look at a planet that is already one degree warmer today than in preindustrial times, and the view is quite alarming. That incremental shift has fueled unprecedented heat waves on every continent, set the American West ablaze with ferocious intensity, and led to deadly deluges in areas that have never experienced such extreme back-to-back rainfall. Further, the most recent IPCC report, which Palmer urges his readers to reference, paints an increasingly dire picture that would seem to support a more maximalist view. Camille Parmesan, an ecologist at the University of Texas at Austin and one of the lead authors on that report, said in February 2022 that “we’re seeing adverse impacts are being much more widespread and being much more negative than expected in prior reports.” The Primacy of Doubt makes a compelling case for either reducing uncertainty or operating with confidence in the “reliability” of the uncertainty that remains. But it can obscure the much bigger picture of climate action. It’s impossible not to ponder how overlooking such nuances might sit with readers prowling for reasons to brush off the urgency of new climate policies.”

JC recommendation to readers:  Don’t waste your time reading this book if all you are looking for is confirmation of your predilection for (or against) urgent reductions of fossil fuel emissions.   Read this book if you have an intellectual curiosity about physics, chaos, uncertainty and their applications across the sciences.  This remarkable book will surely satisfy and stimulate your curiosity.  Further, such understanding will lead to better understanding of how we think and make decisions.  Most refreshingly for a book that is at least partly in the climate space, this book does NOT attempt to tell us what to think or which decisions to make.