Almost every method for fixing catastrophic forgetting treats it as an optimization failure: tune the loss function, add a penalty, and hope the weights cooperate. I want to convince you the problem is closer to geometry than to optimization, and that the difference changes which solutions can possibly work.
Teach a child to recognize cats, then teach her about dogs, and at the end she knows both. This is so ordinary it is almost not worth saying. It is also something neural networks, by default, cannot do.
Train a network to classify cats until it reaches 94% accuracy. Then train that same network on dogs. Now test it on cats again. The accuracy does not ease down to 80%, or settle at some sensible compromise. It falls off a cliff, often to single digits, near random guessing. The network that knew cats no longer knows them. Not partially. Not approximately. The knowledge is simply gone.
This is catastrophic forgetting, and it has been one of the central unsolved problems in machine learning since the late 1980s. It is not a quirk of small networks or bad hyperparameters. Networks with billions of parameters forget just as completely as networks with thousands. The failure is structural, which is exactly what makes it interesting.
The first thing to understand, the thing that took me longer to absorb than I would like to admit, is that nothing is erased when a network forgets. The weights are not zeroed. The training data is not deleted. The capacity does not shrink. What actually happens is subtler, and the simulation below is built to show it to you directly rather than to tell you about it.
That is the whole phenomenon in one picture. The dog training did not reach into the network and delete the cat solution. It simply followed the gradient of the dog loss, ∇L_dog, downhill toward the dog optimum, and dragged the weights along with it. The cat optimum stayed exactly where it was. The network just stopped standing on it.
Forgetting is not erasure. It is displacement. The old solution still exists in weight space; the network has simply moved somewhere else.
This reframing sounds almost pedantic, but it has a sharp consequence. If you believe forgetting is erasure, the natural fix is to protect the important weights, clamping them in place so the next task cannot overwrite them. This is precisely the logic behind Elastic Weight Consolidation and the large family of regularization methods that followed it. They estimate which weights mattered for the old task and add a penalty that resists moving them.
But if forgetting is displacement, that logic answers the wrong question. The issue is not that individual weights got overwritten. The issue is that the entire weight vector slid into a different region of the space. The right question is geometric: along which directions can the network move to learn dogs without sliding off the cat optimum? That question has an answer, and most of this essay is about finding it.
I did not arrive at this framing cleanly. I arrived at it through the result you saw foreshadowed in step seven. I had been comparing a series of increasingly elaborate continual-learning architectures against a deliberately dumb baseline, a frozen backbone, a model that received no gradient updates and learned nothing new at all. The baseline kept winning. A model that could not learn was outperforming models designed specifically to learn continuously, because it had one advantage they could not match: a frozen model cannot forget.
Sitting with that result, the question finally inverted. I had been asking why is my architecture failing to retain knowledge? The better question was what is the retention failure actually made of? The answer, it turns out, is not in the loss function. It is in the shape of the space the weights move through.
Background
It would be easy to read Section I as a story about cats and dogs. It is not. The cat-and-dog setup is the smallest possible version of a problem that every useful learning system runs into the moment it leaves the lab.
A robot trained in a warehouse meets a kitchen. A diagnostic model trained on one hospital's scans sees a new imaging machine. A language assistant learns a fact this week that it did not know last week. In every case the system has to absorb something new from a stream of experience that never stops and never repeats, without throwing away what it already knew. That is the continual learning problem, and it is harder than it sounds for a reason that has nothing to do with computing power.
The reason is a tension built into how gradient learning works. A neural network is a function controlled by a long list of numbers, the weights. Training on a task means nudging those numbers to lower the loss on that task. Training on cats moves the weights toward a setting that is good for cats. Training on dogs then moves them toward a setting that is good for dogs. Unless those two good settings happen to sit in the same place, the second round of training pulls the weights away from the first.
Most of the time they do not sit in the same place, so the pull is real, and the size of the pull decides everything. A small pull leaves the old task mostly intact. A large pull carries the weights out of the region where the old task worked at all. When the pull is large enough to leave that region completely, the forgetting is catastrophic. This is not a bug that better engineering removes. It is a property of moving a single point through a space where two tasks want it in two different places.
The same update that makes the network better at the new task is what makes it worse at the old one. They are not two effects. They are one motion seen from two sides.
There is a hopeful wrinkle, though, and it is the reason any of this is solvable. The two good settings are usually not as far apart as the collapse suggests. Cats and dogs share a great deal: fur, four legs, the rough silhouette of a mammal, the texture of an animal photographed against a background. A network that has learned cats has already built much of what it needs for dogs. Somewhere in the space there is a setting that is good for both at once. The question is whether ordinary training, taking one task at a time, can ever find it. The honest answer is usually no, and the simulation below shows you why.
The shared neurons are where the trouble lives. A neuron used only by cats can be left alone while the network learns dogs. A neuron used only by dogs is free to change. But a neuron used by both has to hold two jobs in one set of weights, and when dog training adjusts it to serve dogs better, it serves cats worse. The damage is concentrated almost entirely in that overlap.
This is why network size alone does not solve the problem. A bigger network has more neurons, so the overlap is a smaller fraction of the whole, which helps a little. But two related tasks will always share their most useful features, and those shared features are exactly the ones both tasks fight over. You cannot make the overlap disappear by adding neurons, because the overlap is not about scarcity. It is about the tasks genuinely needing the same structure.
The obvious response is to keep some old examples and mix them into new training, so the gradient sees both tasks at once. This is replay, and it works in proportion to how much you store. It does not change the geometry, though. The overlap is still contested; you are just paying with memory and compute to keep re-teaching the old task. As the number of tasks grows, the stored fraction shrinks and the patch wears thin.
So the real question sharpens. We know a setting exists that is good for both tasks. We know the conflict is concentrated in the neurons they share. What we need is a way to let the network learn the new task using directions that do not disturb the shared structure the old task depends on. Whether such directions exist, and how many of them there are, turns out to be the whole game. That is where Section III goes.
The dilemma
Section II left us with a request that sounds reasonable: learn the new task using directions that leave the old task's structure alone. Before we go looking for those directions, it is worth asking why the conflict exists in the first place, because the answer reveals that it is not really an engineering problem at all.
The oldest name for the conflict is the stability-plasticity dilemma, and it predates deep learning by decades. The idea is simple to state. A learning system has to be plastic enough to absorb new information, and stable enough to hold on to what it already knows. Those two demands pull in opposite directions. Turn plasticity up and the system learns fast but overwrites its past. Turn it down and the past is safe but nothing new gets in.
Stated like that, it sounds like a knob. Find the right setting between rigid and forgetful, and you are done. That framing is comforting and it is wrong, because stability and plasticity are not two ends of one slider. They are two different requirements that happen to be satisfied by moving the same weights in incompatible ways. To learn the new task, the weights must move. To keep the old task, those same weights must not move. No single value of a learning rate resolves a contradiction like that. It only chooses which side to lose on.
You cannot tune your way out of the dilemma, because the dilemma is not a setting. It is the geometry of asking one set of weights to be in two places at once.
The simulation below makes the trade visible as a landscape. Two of the directions are plasticity and stability. The third, the height, is how well the system does on old and new tasks together. Most systems are stuck on the low ground. The interesting question is whether any arrangement reaches the high ground, and what it would take to get there.
The ridge is the whole point. There exists a region of high joint performance, but it is narrow, and you cannot drift onto it by adjusting a single dial. A frozen model sits at one dead corner, a standard network at the other, and both are far from the ridge. Tuning a learning rate slides you along the floor between those corners. It never lifts you onto the high ground.
So how does anything get onto the ridge? The honest answer is that the brain does, and it does so by refusing to treat stability and plasticity as a single tradeoff at all. Instead of one system forced to be both flexible and stable, it uses two systems with different jobs, running on different clocks.
In 1995, McClelland, McNaughton and O'Reilly proposed that the brain solves this with two memory systems rather than one.[5] The hippocampus is the fast learner. It can record a new experience in a single exposure, using sparse representations that keep new memories from colliding with old ones. The neocortex is the slow learner. It folds knowledge in gradually, over many repetitions, into the dense overlapping structure that supports generalization.
The part that matters most is how they interact. The hippocampus does not write directly into the neocortex in real time. It waits, and during sleep it replays compressed versions of recent experience back to the cortex, interleaved with the cortex's existing knowledge. Because the slow system only ever sees a mixture of old and new, it is never dominated by a single new task. It updates toward a running average, not toward the last thing that happened.
This is not a clever loss function. It is a structural arrangement: two learners, two timescales, and a replay channel between them so that fast learning and slow learning never fight over the same weights at the same moment.
That is the lesson worth carrying forward. The systems that reach the ridge do not get there by being tuned more carefully. They get there by being built so the trade never has to be made. Fast learning and slow learning are kept apart, and the conflict that defines the dilemma simply never arises in the same place.
Which returns us to the request from Section II, now sharper. We want a way for an artificial network to learn a new task without disturbing the structure the old task depends on. The brain achieves the equivalent by separation in time. The question for the rest of this essay is whether we can achieve it by separation in space, by finding directions in weight space where new learning can happen without ever pushing on the old. That is what the nullspace is, and it is where we go next.
The geometry
The brain separates learning in time. We do not have sleep cycles or a hippocampus to work with, so we need the artificial equivalent: a way to separate learning in space. To find it we have to go back to the weight space from Section I and look much more closely at one specific moment, the instant the dog gradient starts to pull.
Here is the assumption hiding inside ordinary training. When we train on dogs, we compute the dog gradient and take a step. We treat that gradient as a single instruction: move this way. But a direction in weight space is not atomic. It can be broken into parts, and those parts can have completely different effects on the old task.
Think about standing on a ridge. Some directions take you downhill, some uphill, and some let you walk along the ridge without changing your elevation at all. The dog gradient is like that. Part of it pushes the weights in a direction that raises the cat loss, and that part is what causes forgetting. But part of it may push along a direction where the cat loss does not change at all. Moving that way costs the cat task nothing.
If we could keep only the harmless part of every update, the network could learn dogs while leaving the cat solution exactly where it is. The simulation below is built to show you that these two parts genuinely exist, and that they are perpendicular to each other.
That is the whole idea, and it is smaller than it sounds. The dog update was never a single instruction. It was a sum of two perpendicular movements: one that climbed the cat loss and one that slid along it. The first is the entire reason the network forgot. The second was free all along.
The harmful part and the safe part were always separable. Standard training simply never bothered to separate them, and paid for the whole update with the old task's knowledge.
The safe directions have a precise name. The set of directions in which the cat loss does not change, to first order, is the nullspace of the cat task. Formally, if the cat task constrains the weights through some matrix of directions it cares about, the nullspace is everything orthogonal to those directions. Projecting the dog update onto that nullspace is a single linear operation, Δw_safe = Δw − (harmful component), and what remains is an update that teaches dogs while leaving cats untouched.
In the flat two-dimensional picture you just explored, this feels almost too convenient, because there is exactly one safe direction to find. The obvious worry is that the safe direction is so narrow there is no room to actually learn anything new in it. If the only way to protect cats is to freeze the network in place along its one free axis, we are back to the frozen model from Section I.
This is the right worry, and the answer to it is the reason any of this works. The two-dimensional drawing is a projection. The real weight space of a vision backbone has hundreds of dimensions, and the cat task only constrains a small fraction of them. The number of genuinely free directions, the size of the nullspace, is large. How large, and what happens when an architecture accidentally throws that room away, is exactly where Section V begins.
The nullspace
Section IV left us with a promise and a worry. The promise: there are safe directions, the nullspace of the old task, where new learning costs the old task nothing. The worry: in the flat two-dimensional drawing there was only one such direction, which seemed far too narrow to learn anything real.
The worry dissolves once we count properly. A vision backbone does not have two weight dimensions. It has hundreds. The representation we have been working with is 768-dimensional, and the cat task, like most single tasks, only really constrains a small number of those dimensions. The directions it genuinely cares about might number in the tens. Everything else, the other six hundred and some directions, is nullspace. Room to move.
This reframes the entire problem. If the nullspace is that large, continual learning should be easy. There is enormous space in which to learn new tasks without disturbing old ones. So the real question is not whether the room exists. It is why, in practice, networks still forget catastrophically when the room is sitting right there.
Part of the answer is that ordinary training never looks for the room. It takes the full gradient and steps, harmful component and all, as we saw in Section IV. But there is a second answer that took me much longer to see, and it is the one that finally explained my own failed experiments. Many architectures throw the room away on purpose, before any learning happens, as a side effect of a design choice that looks completely reasonable.
The simulation below has two halves. First it makes the nullspace literal: the cat optimum is not a point but a long flat trench, and you can watch an update either cut across its walls or glide along its floor. Then it counts the room, and shows what a bottleneck does to it.
The trench is the honest picture of an optimum. A single task pins down only a few directions, the walls, and leaves a long flat floor running through everything it does not care about. Ordinary training steps across the walls because it never asked which directions were floor and which were wall. Projecting the update onto the nullspace keeps only the floor-wise part, and the network glides forward, learning the new task while the old one holds.
The room was always there. The failure was never a shortage of safe directions. It was taking steps without checking which directions were safe, and, worse, building networks that discard the safe directions before training even starts.
That second failure is the one I lived through. For a long stretch of experiments my architecture put a bottleneck early in the network, compressing the 768-dimensional representation down to 128 before the task-specific part of the model ever saw it. This is a standard move. Bottlenecks reduce parameters, they regularize, they often help. But a compression from 768 to 128 does not just shrink the representation. It collapses the directions, and the directions it collapses are mostly nullspace, the very room the new task needed to move through safely.
I was projecting onto a nullspace that my own architecture had already flattened. The safe directions had been thrown out at the door. No projection method, however careful, can recover room that the network deleted before learning began. The fix, when it finally came, was almost embarrassingly simple in hindsight.
Instead of compressing the representation and adapting the small result, keep the full 768-dimensional frozen representation and learn a small correction on top of it: h = normalize(f_frozen + α · δ), with the correction strength α initialized at zero.
At the start of a new task the adapter contributes nothing, so the old representation is preserved exactly. Learning then nudges only the correction δ, and because the full space is intact, δ can live in the nullspace, the directions the frozen features do not already use. The room is never thrown away, so there is somewhere safe to learn.
The first time I ran this, it produced a Pareto improvement on the first attempt: higher accuracy on the new task and less forgetting on the old one, at the same time. Not from a cleverer penalty. From finally giving the network back the geometry it needed.
It is worth being clear about what did and did not cause the improvement, because it is the whole thesis of the essay in miniature. I did not find a better loss function. I did not tune a coefficient. I changed the shape of the space the weights were allowed to move through, so that the safe directions still existed when it came time to use them. The win was geometric.
If the safe room is so large, an obvious question follows, and it is the one a careful reader has probably been holding for a while: why does ordinary training not find the nullspace on its own? If most directions are free, why does gradient descent march straight up the wall instead of drifting along the floor where forgetting is zero? The collapse should be the exception, not the rule.
The answer is that nothing in ordinary training is looking for the floor. Gradient descent minimizes the new task's loss and only that. It computes the direction of steepest descent for dogs and steps, and the steepest direction for dogs has no reason to respect the cat task's walls, because the dog loss contains no term for cats. The optimizer is not ignoring the nullspace out of some failure of nerve. The nullspace is simply invisible to an objective that never mentions the old task.
Worse, several forces push the step actively toward the walls rather than along the floor. The features both tasks find useful, the shared structure from Section II, carry the largest gradients, because they are exactly the directions that move the loss the most, and those shared directions are wall directions for the old task. Curvature compounds it: steep directions produce large gradient components, so a raw step is biased toward precisely the high-sensitivity directions the old task cares about most. The flat nullspace, by construction, produces almost no gradient signal at all, so an optimizer chasing steepest descent has nothing pulling it that way. Left alone, gradient descent does close to the most damaging thing available, not out of malice, but because damage and steepness point in the same direction.
That is the gap every successful method is really filling, whether it knows it or not. Replay reintroduces the old task's gradient so the wall directions get pushed back on.[9] Projection methods strip the wall component out by hand.[10,11] The adapter keeps the full space intact so the floor still exists to be found.[12,13] None of this happens for free, because the floor is not where the gradient naturally points. The information about where the old task is sensitive has to be supplied from outside the new task's loss, and that, more than any particular trick, is what separates a method that forgets from one that does not.
The methods
With the trench picture from Section V in hand, the major methods for fighting forgetting finally come into focus. Each of them is aiming at something real. None of them is foolish. But each one tries to win the fight with force or bookkeeping rather than geometry, and that is why they all start to fail in the same place: when tasks get similar, when capacity gets tight, when the harmful direction and the safe direction are hard to tell apart.
The honest way to judge a method is to ask what it does to a single update standing on the trench floor. Does it keep the step on the floor, where learning is free, or does it let the step climb the wall and then try to pay for the damage some other way? The simulation below runs the three big families of methods through exactly that test.
Lined up against the trench, the pattern is hard to miss. Two of the three methods accept that the step will climb the wall and then spend something to limit the damage. Only the third refuses to climb at all.
EWC estimates which weights mattered for the old task and adds a penalty that resists moving them.[6] The instinct is exactly right: protect what the old task depended on. But the instrument is a per-weight stiffness, and stiffness does not know the difference between a wall direction and a floor direction. It resists movement everywhere, so it slows down the harmful climb and the safe glide alike. Worse, the importance estimate it uses, the diagonal of the Fisher information, assumes the weights are independent. They are not. The directions the old task truly cares about are correlated, a low-rank structure that a per-weight number cannot capture. EWC protects the right neighborhood with the wrong map.
Replay keeps a buffer of old examples and mixes them into new training, so the gradient becomes a blend of cat and dog.[9] The blended step climbs the wall less, because the cat part of the gradient pulls back toward the floor. This genuinely helps, which is why replay is the workhorse of practical continual learning. But it is paying with memory and compute to approximate joint training, and it never actually removes the wall-climbing component. As tasks accumulate, the buffer becomes a thinner and thinner slice of the past, and the pullback weakens. Replay buys time. It does not change the geometry.
The nullspace adapter does the one thing the others do not: it removes the wall-climbing component before the step is taken, so the update runs along the floor by construction.[10] It does not resist motion, so it learns the new task at full speed. It does not store old data, so it does not pay in memory. It protects the old task exactly, not approximately, because a step along the floor changes the old loss by zero, not by a little. The catch, the one from Section V, is that this only works if the floor still exists. Compress the representation through a bottleneck first and the adapter has nowhere safe to go, which is precisely the failure that sent me down this whole path.
Every method here has the correct intuition. The difference is that two of them fight the wall-climb after it happens, and one of them declines to climb the wall at all.
This is why the methods fail in the same regime. When tasks are dissimilar and capacity is loose, the harmful and safe directions are easy to separate, and even a blunt instrument like stiffness or replay does well enough. When tasks get similar and capacity gets tight, the wall and the floor crowd together, the harmful component of every step grows, and force-based methods are overwhelmed by exactly the thing they were never built to see. The geometry was always the load-bearing part. The other methods were working despite not addressing it, not because they did.
The transition
Section VI ended on an observation worth pausing over: the standard methods do not fade gradually as conditions worsen. They work, and work, and then they break, all in a narrow band of difficulty. That pattern is a clue. Gradual causes produce gradual effects. A failure that arrives suddenly is usually a sign that something underneath is not a slope but a cliff.
The word for this in physics is a phase transition. Water cooled from 5 degrees to 1 degree barely changes. Cooled one degree further, past zero, it does something completely different: it freezes. The control parameter, temperature, moved smoothly the whole way, but the state of the system jumped. The same kind of jump shows up in many systems with competing constraints, and continual learning is one of them.
Recall the trench. Forgetting was about whether the harmful, wall-climbing component of an update could be separated cleanly from the safe, floor-wise component. When tasks are dissimilar and capacity is loose, the wall and the floor point in clearly different directions, and the separation is easy. As tasks grow more similar, those directions rotate toward each other. The floor narrows. The wall and the floor crowd together.
As long as the floor has any width, forgetting stays near zero. The moment it pinches shut, there is nowhere safe to move, and forgetting jumps. Not slides. Jumps.
This is the claim of the section, and it is a strong one: catastrophic forgetting is better understood as a phase transition than as a gradual tradeoff. There is a critical threshold in the control parameters, task similarity, available capacity, the size of each update, and on one side of it forgetting is negligible while on the other side it is near total. The simulation lets you drive the system across that threshold and watch the jump happen.
The curve is the heart of it. Retained accuracy does not slope downward as the tasks grow more alike. It holds nearly flat across a wide range of similarity, then drops almost vertically once a critical value is crossed. Plotting forgetting against the control parameter does not give you a ramp. It gives you a cliff, and the cliff edge is sharp.
This shape is the signature of a phase transition, and recognizing it changes how we should think about the whole problem. A gradual tradeoff invites gradual remedies: a slightly better penalty, a slightly larger buffer, a slightly smaller learning rate. Each of those nudges the system a little. But if the underlying behavior is a cliff, nudging only moves you along the flat part or shoves you over the edge. It does not change the fact that an edge exists.
This is the precise sense in which forgetting is not an optimization problem. A loss function, a penalty, a regularizer, all of these reshape the surface the optimizer walks on. They can move the location of the cliff. A good regularizer pushes the critical threshold to higher task similarity, buying real headroom, which is why these methods help. What a loss function cannot do is remove the cliff, because the cliff is not a feature of the loss. It is a feature of the geometry: the point where the safe subspace loses its last dimension. You cannot write a penalty that gives a zero-dimensional space a place to move.
A regularizer can move the edge of the cliff. It cannot turn the cliff into a hill. Topology does not negotiate with loss functions.
This also explains the frozen-model result from the very beginning of the essay. The frozen model sits permanently on the safe side of the transition, because it never moves through the space at all. It pays for that safety with an inability to learn, but it is never at risk of falling off the cliff, which is why, in the difficult regime where everything else collapses, doing nothing could win.
The constructive reading is the one that matters. If forgetting is a transition governed by how much safe room remains, then the way to beat it is to keep that room from collapsing: control the geometry so the floor never pinches shut. Preserve the nullspace, as the adapter does, and you push the cliff edge out so far that the realistic operating regime stays comfortably on the flat side. You do not defeat the transition. You arrange to live on the right side of it.
The close
We started with a network that forgot cats the moment it learned dogs, and an admission that for a long stretch of experiments a frozen model that learned nothing kept beating everything I built. The argument since then has been a single idea pursued through one picture: forgetting is not erasure but displacement, the displacement happens because updates climb the walls of the old task instead of running along its floor, and the floor, the nullspace, is large in high dimensions unless an architecture throws it away.
That picture earns a few things. It explains why forgetting is structural rather than a matter of hyperparameters, since the walls and floor are properties of the geometry, not the learning rate. It explains why the standard methods all weaken in the same regime, since they fight the wall-climb with force or memory rather than refusing to climb. It explains why the failure arrives suddenly rather than gradually, since the safe subspace collapses at a threshold rather than shrinking smoothly. And it gave a concrete, working intervention: preserve the nullspace, learn the correction on top of the full representation, and the operating point stays on the safe side of the transition.
None of that makes the account complete. A good picture is honest about its edges, and this one has several. The rest of this section is the list of things the geometry does not yet explain, written as the questions I am actually carrying into the next stretch of work rather than as a tidy summary.
Forgetting that routes through depth, not direction. The whole essay treated the network as a single weight vector moving in one space. Real networks are deep, and a change in an early layer reshapes the loss landscape that every later layer sees. The nullspace of a deep network is not a fixed subspace; it shifts as the layers below it move. The trench picture is a faithful local story and an incomplete global one, and closing that gap means understanding how the safe directions of one layer interact with the safe directions of the next.
Pinning the transition down precisely enough to predict it. Section VII argued that forgetting is a phase transition, but argument is not measurement. To make the claim useful we need an order parameter that can be computed from a network during training and a way to locate the critical threshold before the network falls off the cliff rather than after. Early-warning signals from dynamical systems, the slowing and increased variance that often precede a tipping point, are the natural place to look. If those signals appear in a network approaching its forgetting threshold, the transition stops being a metaphor and becomes a gauge you can read.
Whether the biology actually works this way. The essay leaned on the brain twice: the two-system argument in Section III and the framing of neuromodulation as a way of controlling which directions are plastic. That is suggestive, not settled. The honest test is whether the geometric quantities we care about, the size of the safe subspace, the angle between task gradients, show up in neural recordings during continual learning in animals, and whether the brain's separation in time produces the same effect as our separation in space. If the two strategies converge on the same geometry, that is a strong sign the geometry is the real object.
There is a fourth question that sits underneath the other three and is harder to phrase as a clean experiment. The nullspace projection protects the old task exactly in the linear regime, the regime where moving along the floor changes the old loss by zero. Networks are not linear, and the floor curves once you move far enough along it. How far you can travel before the curvature matters, and whether the safe region is better described as a curved manifold than a flat subspace, is the question that connects this essay to the harder mathematics I have been circling. I do not have a clean answer yet, which is exactly why it is worth working on.
If there is one thing to carry away from this essay, it is not the adapter or the projection or the phase transition. It is the move that produced all three: refusing to treat catastrophic forgetting as an optimization failure and looking instead at the shape of the space the weights move through.
Every method that struggles with forgetting is, in some sense, asking the optimizer to want two things at once and hoping a clever penalty will keep the peace. The geometric view says the peace was never the optimizer's to keep. The question was always which directions are free, and the answer is decided by the architecture and the relationship between the tasks long before the loss function gets a vote.
The stability-plasticity tradeoff is not a law of nature. It is what happens when you do not control the geometry.
That is a hopeful claim, in the end. A law of nature you accommodate. A geometric accident you can fix, by building networks that keep their safe directions intact and learn along them on purpose. The frozen model that started this whole investigation was not wise. It was simply never in motion, and so never at risk. The work ahead is to build networks that move freely and stay safe at the same time, which the geometry says is not a contradiction at all.
This essay drew the geometry in two dimensions and asked you to trust that the high-dimensional version behaves the same way. The next two essays are where that trust gets put to work.
The geometric account is a hypothesis, not a theorem, and the honest thing is to name the ways it could be wrong. Each of these would weaken or overturn the central claim, and each is something I can imagine a careful experiment settling against me.
I do not think any of these is fatal, or I would not have written the essay. But I hold the thesis the way I think it should be held: as the most useful framing I currently have, not as a settled fact. The point of laying out the failure modes is that they are the experiments worth running next.