How does a computer “see”? This question, which arose as a practical engineering challenge for computer scientists, has, through decades of scholarly work, evolved to open up elegant problems in pure mathematics. Three pivotal books found in the stacks of the Institute’s Mathematics - Natural Sciences Library, the oldest dating from the 1990s and the newest encapsulating the latest pioneering research from IAS Faculty, trace this fascinating intellectual trajectory.
When we look at a digital photograph, whether it’s of a person, a tree, or a banana, our brains effortlessly distinguish the subject from the background, even when the image is distorted by “noise” stemming from a photographer’s shaking hand or poor lighting.
To a computer, however, the same image is merely a grid of numbers representing brightness or color. If one asks a computer to use a standard mathematical filter, such as a Gaussian blur, to remove noise, the result will likely destroy and blur out the important “edges” that mark the subject of the image, and important detail from the photograph will be lost.
Enter Fields Medal-winning mathematician and Member (1962–63, 1981–82) in the School of Mathematics, David Mumford. In the 1980s, Mumford and his colleague Jayant Shah proposed a mathematical formula to solve this “image segmentation” problem. Their Mumford–Shah functional forced the computer to create a simplified version of the image, which is smooth almost everywhere, but contains important “cracks” or “breaks” where the evidence for an edge is strong. It works by balancing three key priorities:
- Fidelity: The simplified image should stay close to the original.
- Smoothness: Within regions, the image should vary gradually, removing noise.
- Economy: The boundaries (or edges) between regions should be as short and simple as possible.
The functional allows a computer to clean up a noisy image without losing the critical information found at important boundaries. A copy of Mumford’s 1993 book, Filtering, Segmentation and Depth, which contains the discussion of the functional, is held in the Institute’s Mathematics - Natural Sciences Library.
Mumford’s book is joined on the shelves by a second title, Functions of Bounded Variation and Free Discontinuity Problems by Luigi Ambrosio, Nicola Fusco, and Diego Pallara, published in the year 2000. It took the Mumford–Shah functional from an intuitive tool and placed it into a rigorous mathematical discipline.
To understand the contributions of this text, it helps to think of the Mumford–Shah functional as something of a mathematical orphan. It offered a window into a new kind of geometry, but it lacked a formal home in the world of rigorous analysis.
That is because calculus traditionally studies smooth functions, namely curves that change gradually. But an image on a computer has sharp edges where, for example, a dark shadow meets a bright wall. The color values jump instantly from one number to another.
Ambrosio et al. provided an entirely new mathematical language to describe surfaces that were smooth in some places, but could crack or break in others. They also proved that within the conditions of a space called “Special Functions of Bounded Variation,” these surfaces could crack or break optimally. A surface that is optimally cracked allows just enough breaks to account for the sharp change in the image, while keeping the length of those cracks as short and clean as possible.
Ambrosio’s former Ph.D. student and current IBM von Neumann Professor in the School of Mathematics, Camillo De Lellis, then took up the mantle. His most recent book, The Regularity Theory for the Mumford–Shah Functional on the Plane, co-authored with Matteo Focardi, provides a comprehensive journey through the existing literature on the functional, revisiting classic results and incorporating new advancements. Most notably, it tackles the most persistent problem left by Mumford and Shah: the so-called Regularity Conjecture.
While Ambrosio and his collaborators proved that an “optimal breaking” of the surfaces of an image is always mathematically possible, the precise nature and shape of those breaks and cracks remained a mystery. Without a proof of so-called “regularity,” the edges of the breaks could theoretically be infinitely jagged, never smoothing out no matter how closely you zoom in. However, the Mumford–Shah Regularity Conjecture proposes that the most efficient way to simplify a complex image should result in clean, predictable geometry. This has yet to be proven in its full generality, but significant progress has been made.
In their publication, De Lellis and Focardi synthesize decades of research to show that, on a two-dimensional plane, cracks and breaks are indeed forced into a state of elegant geometric order. Specifically, they show that these edges can only meet in two ways: at “triple junctions,” where three lines meet at perfect 120-degree angles, or at “crack tips,” where a line simply terminates.1 This 120-degree junction is the same configuration seen in soap bubbles; it is nature’s most efficient way for regions to meet. Ultimately, De Lellis and Focardi’s book shows that even when a mathematical object is allowed to be messy, efficiency forces it into a state of geometric order.2
Taken together, these three books on the Mathematics - Natural Sciences Library shelves, encompassing many years of inquiry, show how hidden order can be teased out of noise.
[1] The foundational results synthesized by De Lellis and Focardi include the work of Guy David, who provided significant progress on the regularity conjecture. The book further details key contributions from mathematicians such as Luigi Ambrosio, John Andersson, Alexis Bonnet, Nicola Fusco, Jean-Christophe Leger, Hayk Mikayelyan, and Diego Pallara, as well as those of De Lellis and Focardi themselves.
[2] The definitive analysis of these structures, particularly the behavior of “crack tips,” builds upon joint research conducted by De Lellis and Focardi alongside Silvia Ghinassi, Visitor (2019–21) in the School of Mathematics.
