As we will argue below proper nesting principles emerge from a cost-benefit analysis having to do with the costs of keeping parts aligned vertically, and the benefits of adding more parts horizontally (specialization) and delegating tasks to lower levels (including the tasks of alignment). It might be that the challenge of how to get to aligned artificial general intelligence, is connected to the challenge of how to structure our economies and governments in order to have more autonomy (less totalitarianism), without more chaos (wars, environmental degradation, a loneliness epidemic, a breakdown of family, community and shared meaning, and psychopaths and other types of free riders rising to power or committing mass violence).
Artificial and Real (aka brain-like) Intelligences
Current state of the art LLMs are not operating like our brains in several important ways:
- They have only one loss function. Our brains seem to have several levels (neurons and other brain cells, nested within neural clusters, nested within brain modules, and possibly other levels) where each level is optimizing some function (as we will see below, some of these optimizations are satisficings). Multi-agent hiearchical reinforcement learning is a possible improvement on supervised learning, but these systems are not state of the art.
- They don’t optimize in real time to a changing environment (unless they are doing reinforcement learning, optimizing a value or Q function, but these are not state of the art and currently not competitive with LLMs). They have a training period during which they optimize to a static environment (except that words have some order, but this is different than dynamics), but after that it’s just output as a function of input, without optimization. Our brains are constantly optimizing.
- They are not good at abstracting generalities. There have been various attempts to argue that deep layers abstract, but I remain unconvinced. They have the wrong architecture for abstraction, one of layers, whereas our brains seem to have agentic, subsidiary levels of abstraction. For simplicity purposes, agency is defined as the ability to optimize or satisfice a task, which involves choosing actions.
- They are prone to the dimensionality curse, where the number of parameters needed to model high dimensional inputs increases exponentially and all parameters are being optimized on at once. Our brains seem to have avoided this curse through abstraction and subsidiarity. Each level has only a few (connection) parameters it is optimizing a utility function on, not all of them. In addition, the lowest level parts are not getting all the input variables, but only a small subset of them (this part is similar in convolutional networks).
- At the highest level of the stack of abstraction levels, the probability distribution is highly peaked at 1 class (with almost certainty, i.e. p~=1 for that class and 0 for all others). The certainty class could be “none of the above”. But even without such a “absential” class, highly noisy inputs at the lowest level, end up as highly certain outputs at the highest level. This seems to happen due to both the nature of the constraints, and the nature of the stacking intermediate functions, something we will hopefully have rigorous results about soon.
Biological and Social Systems
Humans keep oscillating between totalitarianism and chaos. If only we structured our societies and AIs based on how the human brain (speculating here) and multicellular organisms are structured, perhaps we could do better. I am not here saying that whatever nature does should be mimicked. I am only advocating biomimicry when the system to be mimicked is an evolutionary organism and/or has agency, because it is only then that there is (at least one) optimization happening. An ecosystem is not optimizing anything, though it might have some stable attractors, or have species in it that are optimizing their fitness (some claim that it is maximizing resource flows, sustainability and biodiversity, but I can think of ecosystems that don’t do this. Ecosystems don’t have agency and are not evolutionary individuals or organisms ). I am not here advocating mimicking an ecosystem in either understanding the brain or building a better social system. I think people who claim that ecosystems are optimizing something are projecting the human tendency to optimize ecosystems for their own ends (and sometimes also for other species’ benefit if they are liberal).
We postulate that brains and functional social systems (like functional families, within villages or tribes, within federations of these) have a properly nested architecture (PNA) just like multicellular organisms and eusocial insects. This architecture has nodes that specialize at each level, interacting through some connections, whether they be material resources or information. It is convenient to look at nodes at higher levels as interacting directly with each other, though in reality they may interact through some of their specialized parts (usually membranes). Each node down to the lowest level can itself be recursively understood as having sub-nodes with connections, and we shall see that an emergent property of this architecture is that at each level, there are more interactions occurring within that level between the nodes at that level, than interactions outside that level. Multicellular organisms interact through specialized sensory organs with each other and their environments. Organs interact through membranes and signaling molecules with each other, same with cells and organelles. An organelle in a cell interacts more with organelles in that same cell than with organelles in other cells (unless it’s a membrane). Cells in an organ interact more with cells in that organ than with cells in other organs. Organs in a multicellular organism interact more with each other within that organism than with organs in other organisms. We shall see that this is an emergent property of a PNA.
Ants in a colony are less agentic than people in most families or villages. In a multicellular organism, organelles are less agentic than cells, which are less agentic than organs, which are less agentic than the whole organism (it is debatable whether the brain is less or more agentic than the whole body, but it is accepted that the brain evolved to serve the body, acting as a coordinator of other organs and predictor of the environment). So there is a spectrum of possible agency as well, going the opposite way from resource exchange.
Why Proper Nesting in the brain?
If the brain has a PNA, here are 2 possible ways to explain the PNA of the brain:
- PNA allows for delegation of simpler, faster tasks to lower levels, more complex, slower tasks to higher levels.
- PNA allows for subsidiarity, which allows for smaller effective number of parts (each level is only responsible for its parts, not the parts of levels above or below), which allows for better complexity management, including free riding management. Free riding when it comes to information processing is not usually about taking more material resources than are given to others in the collective (that is called cancer). It’s about making errors in classification tasks. Free riding reduction is the same problem as AI alignment, and cancer reduction.
Hypothesis 1
We know that many autonomic (unconscious) processes happen in the brain, and that these happen more quickly than conscious thought. Muscles have to move, sensory input has to be processed, metabolism has to be regulated to respond to different environmental inputs. We also know that there are intermediate, instinctual/emotional levels of thought that are slower than the autonomic processes, but faster than reasoning, as observed by Kahneman and others, and subsequently empirically verified. There might be other intermediate levels of abstraction and subsidiarity before we get to advanced reasoning. Each one of these levels presumably does a task, which means some kind of optimization. But optimization is slow, especially with tasks that have many possible local optima that one can get stuck in, taking a while to find a global optimum.
A possible way out of this conundrum, is to do constraint satisficing instead of general optimization. This is a type of optimization that finds what is acceptably good enough, instead of wasting time finding the best solution. It has three additional pros:
- Tasks at lower levels encode previous learning in their connection variables and the constraint parameters. The constraint parameters come from higher levels and are few in number relative to connection variables. As long as they vary slowly relative to the connection variables, the connection variables can satisfy them in real time (unlike in current AIs, that mostly learn during an initial learning period).
- This accords with our experience of downward causation, where thoughts can influence muscles, words, images, and future plans. The lower levels meet constraints parameterized by higher levels. Information also flows upward, to give feedback to higher levels of how the interaction with the environment is going, so that the highest level can optimize the lower level interaction with the environment.
- Constraints can’t be reduced to lower levels, they are the Achilles’ heel of reductionism.
Whether this PNA is better than the one used in AI neural networks is an empirical question. We need to test the properly nested architecture in silico against standard neural networks, with some specific functions. It is also possible that artificial PNA will not do better for specific functions, but will be more adaptible/generalizable across any set of functions, as in AGI.
Hypothesis 2
Hypothesis 2 predicts that beyond a certain number of parts, it is more cost efficient to add another level than add more parts to the highest existing level. If we assume that these nested levels exist (due to evolutionary history), each with a certain number of parts, simulation shows that the consequence of certain constraints is that the total amount of external interactions is lower than the total amount of internal interactions, for each level. To see this, we introduce a simple model where a bunch of nodes are connected by edges, representing resource flows, both in- and out-going (import and export). These resources could be information, material resources, or both, so they are real numbers, the variables of the model. For concreteness, each individual node belongs to a family of 3 nodes, each family belongs to a village of 3 families, each village belongs to a municipality of 3 villages, and each municipality belongs to a nation of 3 municipalities. These are social system metaphors, to help us visualize the model. An individual is akin to a neuron, a family to a cluster of neurons, a village to a brain module, and so on (perhaps with some yet undiscovered levels. Perhaps the two hemispheres are a level which has not yet properly nested since brain modules are bilateral). Here are the key features (the first 3 are constraints) we assume:
- Each level has certain coherence requirements, meaning the sum of interactions between its parts has to exceed a certain positive real number. For now we also assume that these interactions can be understood in terms of lowest level nodes, an assumption that we are relaxing in current (yet unpublished on the website) research. So each family has a coherence requirement, each village has a coherence requirement, each municipality has a coherence requirement, and the one nation has a coherence requirement. For now, these are not specialized and we assume the coherence requirements are all the same.
- The lowest level interaction between nodes is less than an arbitrary real number due to metabolic constraints. We can assume, without loss of generality, that this number is 1, setting an arbitrary scale.
- The capacity of each lowest level node is limited (also due to metabolic constraints) so that the total outgoing plus incoming interactions with other nodes is less than some “Dunbar” capacity, here a real number greater than 1 (different that the historic Dunbar number which was an integer on the order of 150, but both reflect a limitation on the capacity of individuals).
- At the highest level, there is some function to be optimized, such as the fitness. Here we assume this is the total interactions summed over all the nodes, but other functions lead to the same patterns: more internal than external interactions. If Dunbar capacity was infinite or if there were no level coherence constraints, this would not be the case.
We simulate/optimize a model where the lowest level nodes are 81 individuals, nested within 27 families, nested within 9 villages, nested within 3 municipalities. At the lowest level, we assume (because we don’t want to keep modeling lower levels, the recursion has to stop somewhere) that the self interaction of individuals (the resources that their parts are expending on themselves, coming from the environment) is given by a certain number less than 1. Note also that there is no meaning to distance here, so nodes in say, a family can be far away in 3 dimensional space. However, in a system where proximity is correlated with frequency of interaction (such as the brain), it is expected that spatial proximity will determine levels. For now, all the variables are edge variables at the individual level. These are subjected to a constrained optimization. After the optimization, we consider edges between any 2 families to be the sum of all the individual interactions between any 2 members of these families, and so on for villages and municipalities, so that we can visualize the network connectivity (a simple kind of coarse graining).
We measure both export and import at each level, and we can see that the ratios of external to internal interactions are small. Individuals are interacting more with each other in a family than with individuals external to their family. They are interacting more with each other in a village, than with individuals external to that village. More in a municipality than in other municipalities. The apparent randomness in the graphs below is due to many different equally good optima to the constrained optimization problem (the randomness disappears if the Dunbar constraints are removed). This emergent pattern is robust to changes in the lowest level self interaction (except for the individual ratios), or even random self interactions. The following figures show import and export ratios for each part at each level, and also averages of the import and export ratios across all parts at each level on the right ( 27 families, 9 villages, 3 municipalities). The ratios for individual export (or import) to individual self interactions depend significantly on what is assumed for individual self interaction (for graphs below, self interaction=0.5) and vary between 1 and 4. Also, the level coherence constraints without the Dunbar constraint, are no constraints at all, at least when the utility function to be maximized is the total commerce and the coherence threshold for each part is less than the number of sub-parts minus 1 (the edge variables maximizing the utility will all be 1 in that case). If we remove the level coherence constraints (but keep the Dunbar constraint), the import and export ratios become approximately 2.5 time smaller for individuals, and 3, 9, and 20 times bigger for families, villages, and municipalities, respectively (not shown). Note also that without specialization, there is no reason for import and export to differ except by random chance. But if we keep both constraints, then trade ratios (external to internal) are consistently small and decrease by a factor of about 4 at each level going from family to village to municipality (though the exact ratios vary depending on the constraint parameters and the hyperparameters by a few percent, see figures below for examples of variation of average-across-all-modules trade ratios with Dunbar capacity. The individual ratio does vary depending on the self contribution parameter). This tells us that once modules and levels are established, the lowest level trade mostly happens within the lowest level and decreases as the level number increases. However, other forms of trade can emerge at higher levels. For example, in a brain, the lowest level trade might be electrical signals and certain neurotransmitters between neurons in neural clusters, but other forms of trade (involving other kinds of neurotransmitters) could happen between neural clusters, and yet other kinds can happen between neural modules such as between the PFC and the anterolateral motor cortex (all recursively subject to the decreasing trade rule). This is a potentially empirically testable hypothesis! We would need an in-vivo imaging technology that can distinguish different neurotransmitters, and we would image people doing tasks involving different levels of abstraction, and expect to see differential images depending on the level.
Next we look at the connectivity properties of the nodes at each level. We hypothesize that in order to have agency at each level, there has to be complete connectedness at each level. If the Dunbar and coherence constraints are too slack, some (or all) levels will look like disconnected graphs. For example, here are the graphs for individuals and families when the Dunbar capacity is too high, or the coherence limit is too low (at the limit of 0 coehrence limit, or no coherence constraints, there is no meaning to any of the modules–families, villages, municipalities, and the graphs are connected randomly):
There is a line in the Dunbar capacity/ level coherence (for now assuming all the level coherences are the same) plane, below which at least one of the level graphs is disconnected and above which there are no solutions satisfying the constraints. On that critical line (whose position depends on the self interaction of individuals), the constraints are satisfied and all the graphs are connected. We show the line below for 2 values of the self interaction, and we show a few connected graphs, one for each level, with parameters on the line (and self interaction loops hidden, for clarity). It’s a mystery to me at this point why Dunbarcap is roughly equal to coherence threshold+self interaction on the critical line, i.e. why the slope of the line is 1, or even why it’s a line instead of some other curve.
We can also visualize the connectivity of each level nodes as a matrix, where for each node in the first column we map which nodes it gives resources to by traveling along its row, and for each node in the first row, we map which nodes it receives resources from by traveling along its column. Consistent with the the connectivity graphs, we see that most trade is happening within families, less within villages and less within municipalities
Adapting to/learning the environment
So far in our modeling, we have one global optimization, and constraints that apply to different levels. However, the variables that are being optimized and constrained are still all lowest level variables, there is no subsidiarity: agency exists only at the lowest level. Here is a meta-model of a better system that could have evolved (and we can recreate in software): each level confines its variables internally, and these are not reducible to sums (or other functions) or lower level variables, as was our reductionist model above. It’s better because it allows for delegation of simpler, or already learned in similar environments (either in the course of evolution, or life history) tasks to lower levels, and these can happen at a faster rate than the more complex tasks at higher levels. In addition, the constraint parameters (such as the Dunbar capacity and coherence numbers in our model) can be passed by each level to the level below. The highest level computes those parameters by trying to model its perceived environment (as reported in a condensed manner by the level below), optimizing a loss function. It passes these few parameters to the level below, and this level uses these parameters for satisficing constraints, which is also a kind of optimization, but at a faster rate than the level above, starting from some state of connection variables that worked before and changing them slightly. This continues downward to the lowest level, which actually interacts with the environment, and passes the results of this interaction upwards again, in an ongoing up and down loop. So the environment gets encoded in both the constraint variables, and the connection variables, but in a way that is more generalizable than if there was only one level and one optimization. Encoding environmental states and predictions about the environment in constraints is an idea due to Terrence Deacon. We are not aware of any current neural networks which do this.
We see a trend towards this subsidiary behavior already just by assuming a Dunbar and a level coherence constraint: external interactions (at a higher level) are weaker and get weaker as the level number increases. Coherence now means not just that a level is robust to perturbations and maintains homeostasis, but that the parts at each level act as one, and we don’t need to consider their internal sub-parts to track their interactions. As Erik Hoel has shown, this gives a computational advantage, which he called causal emergence. But unlike Hoel, we propose that the coarse grainings that give this advantage don’t need to be found out by time consuming algorithms computing effective information. They can be inferred from how much information (and other resources) is processed internally vs externally. In addition, they are not mere affordances. The coarse grained levels are agentic, but in a limited way–each level affects and is affected by the level above and the level below only, with the lowest level also interacting with the environment.
There are 3 coherence constraints (family, village and municipality) and one Dunbar constraint in our model parametrized by 4 numbers. These numbers are coding for the state of the environment when there is an environmental function to learn. One can imagine adding other constraints and other parameters associated with them, several per level, but not as many as interactions between nodes. Those edge parameters are also encoding the state of the environment, but they are more specific than the constraint parameters that abstract many different states. Instead of higher level edge parameters being defined in terms of individual level edge parameters (sums of these), they could be considered independent, with each level edge parameters being confined to that level. Whether the brain does this is an empirical question whose testing was mentioned above. We can assume it is so and see if environments can be learned in a more generalizable way, with fewer parameters this way. We have preliminary results.