Category Archives: Article

Meltdown: Why our systems fail and What we can do about it

Leave a reply

Meltdown: Why our systems fail and What we can do about it

Authors and related work

  • Chris Clearfield 
    • Chris is the founder of System Logic, an independent research and consulting firm focusing on the challenges posed by risk and complexity. He previously worked as a derivatives trader at Jane Street, a quantitative trading firm, in New York, Tokyo, and Hong Kong, where he analyzed and devised mitigations for the financial and regulatory risks inherent in the business of technologically complex high-speed trading. He has written about catastrophic failure, technology, and finance for The Guardian, Forbes, the Harvard Kennedy School Review, the popular science magazine Nautilus, and the Harvard Business Review blog.
  • András Tilcsik
    • András holds the Canada Research Chair in Strategy, Organizations, and Society at the University of Toronto’s Rotman School of Management. He has been recognized as one of the world’s top forty business professors under forty and as one of thirty management thinkers most likely to shape the future of organizations. The United Nations named his course on organizational failure as the best course on disaster risk management in a business school. 
  • How to Prepare for a Crisis You Couldn’t Possibly Predict
    • Over the past five years, we have studied dozens of unexpected crises in all sorts of organizations and interviewed a broad swath of people — executives, pilots, NASA engineers, Wall Street traders, accident investigators, doctors, and social scientists — who have discovered valuable lessons about how to prepare for the unexpected. Here are three of those lessons.

Overview

This book looks at the underlying reasons for accidents that emerge from complexity and how diversity is a fix. It’s based on Charles Perrow’s concept of Normal Accidents being a property of high-risk systems.

Normal Accidents are unpredictable, yet inevitable combinations of small failures that build upon each other within an unforgiving environment. Normal accidents include catastrophic failures such as reactor meltdowns, airplane crashes, and stock market collapses. Though each failure is unique, all these failures have common properties:

    • The system’s components are tightly coupled. A change in one place has rapid consequences elsewhere
    • The system is densely connected, so that the actions of one part affects many others
    • The system’s internals are difficult to observe, so that failure can appear without warning

What happens in all these accidents is that there is misdirected progress in a direction that makes the problem worse. Often, this is because the humans in the system are too homogeneous. They all see the problem from the same perspective, and they all implicitly trust each other (Tight coupling and densely connected).

The addition of diversity is a way to solve this problem. Diversity does three things:

    • It provides additional perspectives into the problem. This only works if there is large enough representation of diverse groups so that they do not succumb to social pressure.
    • It lowers the amount of trust within the group, so that proposed solutions are exposed to a higher level of skepticism.
    • It slows the process down, making the solution less reflexive and more thoughtful.

By designing systems to be transparent, loosely coupled and sparsely connected, the risk of catastrophe is reduced. If that’s not possible, ensure that the people involved in the system are diverse.

My more theoretical thoughts:

There are two factors that affect the response of the network: The level of connectivity and the stiffness of the links. When the nodes have a velocity component, then a sufficiently stiff network (either many somewhat stiff or a few very stiff links) has to move as a single entity. Nodes with sparse and slack connections are safe systems, but not responsive. Stiff, homogeneous (similarity is implicit stiff coupling)  networks are prone to stampede. Think of a ball rolling down a hill as opposed to a lump of jello.

When all the nodes are pushing in the same direction, then the network as a whole will move into more dangerous belief spaces. That’s a stampede. When some percentage of these connections are slack connections to diverse nodes (e.g. moving in other directions), the structure as a whole is more resistant to stampede.

I think that dimension reduction is inevitable in a stiffening network. In physical systems, where the nodes have mass, a stiff structure really only has two degrees of freedom, its direction of travel and its axis of rotation. Which means that regardless of the number of initial dimensions, a stiff body’s motion reduces to two components. Looking at stampedes and panics, I’d say that this is true for behaviors as well, though causality could run in either direction. This is another reason that diversity helps keep away from dangerous conditions, but at the expense of efficiency.

Notes

  • Such a collision should have been impossible. The entire Washington Metro system, made up of over one hundred miles of track, was wired to detect and control trains. When trains got too close to each other, they would automatically slow down. But that day, as Train 112 rounded a curve, another train sat stopped on the tracks ahead—present in the real world, but somehow invisible to the track sensors. Train 112 automatically accelerated; after all, the sensors showed that the track was clear. By the time the driver saw the stopped train and hit the emergency brake, the collision was inevitable. (Page 2)
  • The second element of Perrow’s theory (of normal accidents) has to do with how much slack there is in a system. He borrowed a term from engineering: tight coupling. When a system is tightly coupled, there is little slack or buffer among its parts. The failure of one part can easily affect the others. Loose coupling means the opposite: there is a lot of slack among parts, so when one fails, the rest of the system can usually survive. (Page 25)
  • Perrow called these meltdowns normal accidents. “A normal accident,” he wrote, “is where everyone tries very hard to play safe, but unexpected interaction of two or more failures (because of interactive complexity) causes a cascade of failures (because of tight coupling).” Such accidents are normal not in the sense of being frequent but in the sense of being natural and inevitable. “It is normal for us to die, but we only do it once,” he quipped. (Page 27)
    • This is exactly what I see in my simulations and in modelling with graph Laplacians. There are two factors that affect the response of the network: The level of connectivity and the stiffness of the links. When the nodes have a velocity component, then a sufficiently stiff network (either many somewhat stiff or a few very stiff links) has to behave as a single entity.
  • These were unintended interactions between the glitch in the content filter, Talbot’s photo, other Twitter users’ reactions, and the resulting media coverage. When the content filter broke, it increased tight coupling because the screen now pulled in any tweet automatically. And the news that Starbucks had a PR disaster in the making spread rapidly on Twitter—a tightly coupled system by design. (Page 30)
  • This approach—reducing complexity and adding slack—helps us escape from the danger zone. It can be an effective solution, one we’ll explore later in this book. But in recent decades, the world has actually been moving in the opposite direction: many systems that were once far from the danger zone are now in the middle of it. (Page 33)
  • Today, smartphone videos create complexity because they link things that weren’t always connected (Page 37)
  • For nearly thirty minutes, Knight’s trading system had gone haywire and sent out hundreds of unintended orders per second in 140 stocks. Those very orders had caused the anomalies that John Mueller and traders across Wall Street saw on their screens. And because Knight’s mistake roiled the markets in such a visible way, traders could reverse engineer its positions. Knight was a poker player whose opponents knew exactly what cards it held, and it was already all in. For thirty minutes, the company had lost more than $15 million per minute. (Page 41)
  • Though a small software glitch caused Knight’s failure, its roots lay much deeper. The previous decade of technological innovation on Wall Street created the perfect conditions for the meltdown. Regulation and technology transformed stock trading from a fragmented, inefficient, relationship-based activity to a tightly connected endeavor dominated by computers and algorithms. Firms like Knight, which once used floor traders and phones to execute trades, had to adapt to a new world. (Page 42)
    • This is an important point. There is short-term survival value in becoming homogeneous and tightly connected. Diversity only helps in the long run.
  • As the crew battled the blowout, complexity struck again. The rig’s elaborate emergency systems were just too overwhelming. There were as many as thirty buttons to control a single safety system, and a detailed emergency handbook described so many contingencies that it was hard to know which protocol to follow. When the accident began, the crew was frozen. The Horizon’s safety systems paralyzed them. (Page 49)
    • I think that this may argue opposite to the authors’ point. The complexity here is a form of diversity. The safety system was a high-dimensional system that required an effective user to be aligned with it, like a free climber on a cliff face. A user highly educated in the system could probably have made it work, even better than a big STOP button. But expecting that user is a mistake. The authors actually discuss this later when they describe how safety training was reduced to simple practices that ignored perceived unlikely catastrophic events.
  • “The real threat,” Greenberg explained, “comes from malicious actors that connect things together. They use a chain of bugs to jump from one system to the next until they achieve full code execution.” In other words, they exploit complexity: they use the connections in the system to move from the software that controls the radio and GPS to the computers that run the car itself. “As cars add more features,” Greenberg told us, “there are more opportunities for abuse.” And there will be more features: in driverless cars, computers will control everything, and some models might not even have a steering wheel or brake pedal. (Page 60)
    • In this case it’s not the stiffness of the connections, its the density of connections
  • Attacks on cars, ATMs, and cash registers aren’t accidents. But they, too, originate from the danger zone. Complex computer programs are more likely to have security flaws. Modern networks are rife with interconnections and unexpected interactions that attackers can exploit. And tight coupling means that once a hacker has a foothold, things progress swiftly and can’t easily be undone. In fact, in all sorts of areas, complexity creates opportunities for wrongdoing, and tight coupling amplifies the consequences. It’s not just hackers who exploit the danger zone to do wrong; it’s also executives at some of the world’s biggest companies. (Page 62)
  • By the year 2000, Fastow and his predecessors had created over thirteen hundred specialized companies to use in these complicated deals. “Accounting rules and regulations and securities laws and regulation are vague,” Fastow later explained. “They’re complex. . . . What I did at Enron and what we tended to do as a company [was] to view that complexity, that vagueness . . . not as a problem, but as an opportunity.” Complexity was an opportunity. (Page 69)
    • I’m not sure how to fit this in, but I think there is something here about high-dimensional spaces being essentially invisible. This is the same thing as the safety system on the Deepwater Horizon.
  • But like the core of a nuclear power plant, the truth behind such writing is difficult to observe. And research shows that unobservability is a key ingredient to news fabrications. Compared to genuine articles, falsified stories are more likely to be filed from distant locations and to focus on topics that lend themselves to the use of secret sources, such as war and terrorism; they are rarely about big public events like baseball games. (Page 77)
    •  More heuristics for map building
  • Charles Perrow once wrote that “safety systems are the biggest single source of catastrophic failure in complex, tightly coupled systems.” (Page 85)
    • Dimensions reduce through use, which is a kind of conversation between the users and the designers. Safety systems are rarely used, so this conversation doesn’t happen.
  • Perrow’s matrix is helpful even though it doesn’t tell us what exactly that “crazy failure” will look like. Simply knowing that a part of our system—or organization or project—is vulnerable helps us figure out if we need to reduce complexity and tight coupling and where we should concentrate our efforts. It’s a bit like wearing a seatbelt. The reason we buckle up isn’t that we have predicted the exact details of an impending accident and the injuries we’ll suffer. We wear seatbelts because we know that something unforeseeable might happen. We give ourselves a cushion of time when cooking an elaborate holiday dinner not because we know what will go wrong but because we know that something will. “You don’t need to predict it to prevent it,” Miller told us. “But you do need to treat complexity and coupling as key variables whenever you plan something or build something.” (Page 88)
  • A fundamental feature of complex systems is that we can’t find all the problems by simply thinking about them. Complexity can cause such strange and rare interactions that it’s impossible to predict most of the error chains that will emerge. But before they fall apart, complex systems give off warning signs that reveal these interactions. The systems themselves give us clues as to how they might unravel. (Page 141)
  • Over the course of several years, Rerup conducted an in-depth study of global pharmaceutical powerhouse Novo Nordisk, one of the world’s biggest insulin producers. In the early 1990s, Rerup found, it was difficult for anyone at Novo Nordisk to draw attention to even serious threats. “You had to convince your own boss, his boss, and his boss that this was an issue,” one senior vice president explained. “Then he had to convince his boss that it was a good idea to do things in a different way.” But, as in the childhood game of telephone—where a message gets more and more garbled as it passes between people—the issues became oversimplified as they worked their way up the chain of command. “What was written in the original version of the report . . . and which was an alarm bell for the specialist,” the CEO told Rerup, “was likely to be deleted in the version that senior management read.” (Page 146)
    • Dimension reduction, leading to stampede
  • Once an issue has been identified, the group brings together ad hoc teams from different departments and levels of seniority to dig into how it might affect their business and to figure out what they can do to prevent problems. The goal is to make sure that the company doesn’t ignore weak signs of brewing trouble.  (Page 147)
    • Environmental awareness as a deliberate counter to dimension reduction
  • We show that a deviation from the group opinion is regarded by the brain as a punishment,” said the study’s lead author, Vasily Klucharev. And the error message combined with a dampened reward signal produces a brain impulse indicating that we should adjust our opinion to match the consensus. Interestingly, this process occurs even if there is no reason for us to expect any punishment from the group. As Klucharev put it, “This is likely an automatic process in which people form their own opinion, hear the group view, and then quickly shift their opinion to make it more compliant with the group view.” (Page 154)
    • Reinforcement Learning Signal Predicts Social Conformity
      • Vasily Klucharev
      • We often change our decisions and judgments to conform with normative group behavior. However, the neural mechanisms of social conformity remain unclear. Here we show, using functional magnetic resonance imaging, that conformity is based on mechanisms that comply with principles of reinforcement learning. We found that individual judgments of facial attractiveness are adjusted in line with group opinion. Conflict with group opinion triggered a neuronal response in the rostral cingulate zone and the ventral striatum similar to the “prediction error” signal suggested by neuroscientific models of reinforcement learning. The amplitude of the conflict-related signal predicted subsequent conforming behavioral adjustments. Furthermore, the individual amplitude of the conflict-related signal in the ventral striatum correlated with differences in conforming behavior across subjects. These findings provide evidence that social group norms evoke conformity via learning mechanisms reflected in the activity of the rostral cingulate zone and ventral striatum.
  • When people agreed with their peers’ incorrect answers, there was little change in activity in the areas associated with conscious decision-making. Instead, the regions devoted to vision and spatial perception lit up. It’s not that people were consciously lying to fit in. It seems that the prevailing opinion actually changed their perceptions. If everyone else said the two objects were different, a participant might have started to notice differences even if the objects were identical. Our tendency for conformity can literally change what we see. (Page 155)
    • Gregory Berns
      • Dr. Berns specializes in the use of brain imaging technologies to understand human – and now, canine – motivation and decision-making.  He has received numerous grants from the National Institutes of Health, National Science Foundation, and the Department of Defense and has published over 70 peer-reviewed original research articles.
    • Neurobiological Correlates of Social Conformity and Independence During Mental Rotation
      • Background: When individual judgment conflicts with a group, the individual will often conform his judgment to that of the group. Conformity might arise at an executive level of decision making, or it might arise because the social setting alters the individual’s perception of the world.
      • Methods: We used functional magnetic resonance imaging and a task of mental rotation in the context of peer pressure to investigate the neural basis of individualistic and conforming behavior in the face of wrong information.
      • Results: Conformity was associated with functional changes in an occipital-parietal network, especially when the wrong information originated from other people. Independence was associated with increased amygdala and caudate activity, findings consistent with the assumptions of social norm theory about the behavioral saliency of standing alone.
      • Conclusions: These findings provide the first biological evidence for the involvement of perceptual and emotional processes during social conformity.
      • The Pain of Independence: Compared to behavioral research of conformity, comparatively little is known about the mechanisms of non-conformity, or independence. In one psychological framework, the group provides a normative influence on the individual. Depending on the particular situation, the group’s influence may be purely informational – providing information to an individual who is unsure of what to do. More interesting is the case in which the individual has definite opinions of what to do but conforms due to a normative influence of the group due to social reasons. In this model, normative influences are presumed to act through the aversiveness of being in a minority position
    • A Neural Basis for Social Cooperation
      • Cooperation based on reciprocal altruism has evolved in only a small number of species, yet it constitutes the core behavioral principle of human social life. The iterated Prisoner’s Dilemma Game has been used to model this form of cooperation. We used fMRI to scan 36 women as they played an iterated Prisoner’s Dilemma Game with another woman to investigate the neurobiological basis of cooperative social behavior. Mutual cooperation was associated with consistent activation in brain areas that have been linked with reward processing: nucleus accumbens, the caudate nucleus, ventromedial frontal/orbitofrontal cortex, and rostral anterior cingulate cortex. We propose that activation of this neural network positively reinforces reciprocal altruism, thereby motivating subjects to resist the temptation to selfishly accept but not reciprocate favors.
  • These results are alarming because dissent is a precious commodity in modern organizations. In a complex, tightly coupled system, it’s easy for people to miss important threats, and even seemingly small mistakes can have huge consequences. So speaking up when we notice a problem can make a big difference. (Page 155)
  • KRAWCHECK: I think when you get diverse groups together who’ve got these different backgrounds, there’s more permission in the room—as opposed to, “I can’t believe I don’t understand this and I’d better not ask because I might lose my job.” There’s permission to say, “I come from someplace else, can you run that by me one more time?” And I definitely saw that happen. But as time went on, the management teams became less diverse. And in fact, the financial services industry went into the downturn white, male and middle aged. And it came out whiter, maler and middle-aged-er. (Page 176)
  • “The diverse markets were much more accurate than the homogeneous markets,” said Evan Apfelbaum, an MIT professor and one of the study’s authors. “In homogeneous markets, if someone made a mistake, then others were more likely to copy it,” Apfelbaum told us. “In diverse groups, mistakes were much less likely to spread.” (Page 177)
  • Having minority traders wasn’t valuable because they contributed unique perspectives. Minority traders helped markets because, as the researchers put it, “their mere presence changed the tenor of decision making among all traders.” In diverse markets, everyone was more skeptical. (Page 178)
  • In diverse groups, we don’t trust each other’s judgment quite as much, and we call out the naked emperor. And that’s very valuable when dealing with a complex system. If small errors can be fatal, then giving others the benefit of the doubt when we think they are wrong is a recipe for disaster. Instead, we need to dig deeper and stay critical. Diversity helps us do that. (Page 180)
  • Ironically, lab experiments show that while homogeneous groups do less well on complex tasks, they report feeling more confident about their decisions. They enjoy the tasks they do as a group and think they are doing well. (Page 182)
    • Another stampede contribution
  • The third issue was the lack of productive conflict. When amateur directors were just a small minority on a board, it was hard for them to challenge the experts. On a board with many bankers, one CEO told the researchers, “Everybody respects each other’s ego at that table, and at the end of the day, they won’t really call each other out.” (Page 193)
    • Need to figure out what productive conflict is and how to measure it
  • Diversity is like a speed bump. It’s a nuisance, but it snaps us out of our comfort zone and makes it hard to barrel ahead without thinking. It saves us from ourselves. (Page 197)
  • A stranger is someone who is in a group but not of the group. Simmel’s archetypal stranger was the Jewish merchant in a medieval European town—someone who lived in the community but was different from the insiders. Someone close enough to understand the group, but at the same time, detached enough to have an outsider’s perspective. (Page 199)
    • Can AI be trained to be a stranger?
  • But Volkswagen didn’t just suffer from an authoritarian culture. As a corporate governance expert noted, “Volkswagen is well known for having a particularly poorly run and structured board: insular, inward-looking, and plagued with infighting.” On the firm’s twenty-member supervisory board, ten seats were reserved for Volkswagen workers, and the rest were split between senior managers and the company’s largest shareholders. Both Piëch and his wife, a former kindergarten teacher, sat on the board. There were no outsiders. This kind of insularity went well beyond the boardroom. As Milne put it, “Volkswagen is notoriously anti-outsider in terms of culture. Its leadership is very much homegrown.” And that leadership is grown in a strange place. Wolfsburg, where Volkswagen has its headquarters, is the ultimate company town. “It’s this incredibly peculiar place,” according to Milne. “It didn’t exist eighty years ago. It’s on a wind-swept plain between Hanover and Berlin. But it’s the richest town in Germany—thanks to Volkswagen. VW permeates everything. They’ve got their own butchers, they’ve got their own theme park; you don’t escape VW there. And everybody comes through this system.” (Page 209)
  • Most companies have lots of people with different skills. The problem is, when you bring people together to work on the same problem, if all they have are those individual skills . . . it’s very hard for them to collaborate. What tends to happen is that each individual discipline represents its own point of view. It basically becomes a negotiation at the table as to whose point of view wins, and that’s when you get gray compromises where the best you can achieve is the lowest common denominator between all points of view. The results are never spectacular but, at best, average. (Page 236)
    • The idea here is that there is either total consensus and groupthink, or grinding compromise. The authors are focussing too much on the ends of the spectrum. The environmentally aware, social middle is the sweet spot where flocking occurs.
  • Or think about driverless cars. They will almost certainly be safer than human drivers. They’ll eliminate accidents due to fatigued, distracted, and drunk driving. And if they’re well engineered, they won’t make the silly mistakes that we make, like changing lanes while another car is in our blind spot. At the same time, they’ll be susceptible to meltdowns—brought on by hackers or by interactions in the system that engineers didn’t anticipate. (Page 242)
  • We can design safer systems, make better decisions, notice warning signs, and learn from diverse, dissenting voices. Some of these solutions might seem obvious: Use structured tools when you face a tough decision. Learn from small failures to avoid big ones. Build diverse teams and listen to skeptics. And create systems with transparency and plenty of slack. (Page 242)

Thinking slow, acting reflexively

2 Replies

I just finished the cover story in Communications of the ACM on Human-Level Intelligence or Animal-Like Abilities?. Overall interesting and insightful, but what really caught my eye was Adnan Darwiche‘s discussion of models and maps:

  • “In his The Book of Why: The New Science of Cause and Effect, Judea Pearl explained further the differences between a (causal) model and a function, even though he did not use the term “function” explicitly. In Chapter 1, he wrote: “There is only one way a thinking entity (computer or human) can work out what would happen in multiple scenarios, including some that it has never experienced before. It must possess, consult, and manipulate a mental causal model of that reality.” He then gave an example of a navigation system based on either reasoning with a map (model) or consulting a GPS system that gives only a list of left-right turns for arriving at a destination (function). The rest of the discussion focused on what can be done with the model but not the function. Pearl’s argument particularly focused on how a model can handle novel scenarios (such as encountering roadblocks that invalidate the function recommendations) while pointing to the combinatorial impossibility of encoding such contingencies in the function, as it must have a bounded size.”
  • This is a Lists and Maps argument, and it leaves out stories, but it also implies something powerful that I need to start to think about. There is another interface, and it’s one that bridges human and machine, The dynamic model. What follows is a bunch of (at the moment – 10.8.18) incomplete thoughts. I think that models/games are another sociocultural interface, one that may be as affected by computers as the Ten Blue Links. So I’m using this as a staging area.
  • Games
    • Games and play are probably the oldest form of a dynamic model. Often, and particularly in groups, they are abstract simulations of conflict of some kind. It can be a simple game of skill such as Ringing the Bull, or a complex a wargame, such as chess:
      • “Historically chess must be classed as a game of war. Two players direct a conflict between two armies of equal strength upon a field of battle, circumscribed in extent, and offering no advantage of ground to either side. The players have no assistance other than that afforded by their own reasoning faculties, and the victory usually falls to the one whose strategical imagination is the greater, whose direction of his forces is the more skilful, whose ability to foresee positions is the more developed.” Murray, H.J.R.. A History of Chess: The Original 1913 Edition (Kindle Locations 576-579). Skyhorse Publishing. Kindle Edition.
    • Recently, video games afford games that can follow narrative templates:
      • Person vs. Fate/God
      • Person vs. Self
      • Person vs. Person
      • Person vs Society
      • Person vs. Nature
      • Person vs. Supernatural
      • Person vs. Technology
    • More on this later, because I think that this sort of computer-human interaction is really interesting, because it seems to open up spaces that would not be accessible to humans because of the data manipulation requirements (would flight simulators exist without non-human computation?).
  • Moving Maps
    • I would argue that the closer to interactive rates a model is, the more dynamic it is. A map is a static model, a snapshot of the current geopolitical space. Maps are dynamic because the underlying data is dynamic. Borders shift. Counties come into and go out of existence. Islands are created, and the coastline is eroded. And the next edition of map will incorporate these changes.
    • Online radar weather maps are an interesting case, since they reflect a rapidly changing environment and often now support playback of the last few hours (and prediction for the next few hours) of imagery at variable time scales.
  • Cognition
    • Traditional simulation and humans
      • Simulations provide a mechanism for humans to explore a space of possibilities that larger than what can be accomplished by purely mental means. Further, these simulations create artifacts that can be examined independently by other humans.
        • Every model is a theory—a very-well specified theory. In the case of simulations, the models are theories expressed in so much detail that their consequences can be checked by execution on a computer [Bryson, 2015]
      • The assumptions that provide the basis for the simulation are the model. The computer provides the dynamics. The use of simulation allows users to explore the space in the same way that one would explore the environment. Discoveries can be made that exist outside of the social constructs that led to the construction of the simulator and the assumptions that the simulator is based on.
      • What I think this means is that humans bring meaning to the outputs of the simulation. But it also means that there is a level of friction required to get from the outputs as they are computed to a desired level of meaningfulness to the users. In other words, if you have a theory of galaxy formation, but the results of the simulation only match observations if you have to add something new, like negative gravity, this could reflect a previously undiscovered component in the current theory of the formation of the universe.
      • I think this is the heart of my thinking. Just as maps allow the construction of trajectories across a physical (or belief) spaces, dynamic models such as simulation support ways of evaluating potential (and simplified/general) spaces that exist outside the realms of current understanding. This can be in the form of alternatives not yet encountered (a hurricane will hit the Florida panhandle on Thursday), or systems not yet understood (protein folding interactive simulators)
      • From At Home in the Universe: Physicists roll out this term, “universality class,” to refer to a class of models all of which exhibit the same robust behavior. So the behavior in question does not depend on the details of the model. Thus a variety of somewhat incorrect models of the real world may still succeed in telling us how the real world works, as long as the real world and the models lie in the same universality class. (Page 283)
    • Traditional simulation and ML(models and functions)
      • Darwiche discusses how the ML community has focused on “functional” AI at the expense of  “model-based” AI. I think his insight that functional AI is closer to reflex, and how there is an analogical similarity between it and “thinking fast“. Similarly, he believes that model-based AI may more resemble “thinking slow“.
      • I would contend that building simulators may be the slowest possible thinking. And I wonder if using simulators to train functional AI that can then be evaluated against real-world data, which is then used to modify the model in a “round trip” approach might be a way to use the fundamental understandability of simulation with the reflexive speed of trained NN systems.
      • What this means is that “slow” AI explicitly includes building testable models. The tests are not always going to be confirmation of predictions because of chaos theory. But there can be predictions of the characteristics of a model. For example, I’m working with using agent-based simulation moving in belief space to generate seeds for RNNs to produce strings that resemble conversations. Here, the prediction would be about the “spectral” characteristics of the conversation – how words change over time when compared to actual conversations where consensus evolves over time.

When Worlds Collide

Leave a reply

59973d6385da8-image

Charlottesville demonstrations, summer 2017 (link)

I’ve been thinking about this picture a lot recently. My research explores how extremist groups can develop using modern computer-mediated communication, particularly recommender systems. This picture lays the main parts like a set of nested puzzle pieces.

This is a picture of a physical event. In August 2017, various “Alt-Right” online communities came to Charlottesville Virginia to ostensibly protest the removal of confederate statues, which in turn was a response to the Charleston South Carolina church shooting of 2015. From August 11th through 12th, sanctioned and unsanctioned protests and counter protests happened in and around Emancipation Park.

Although this is not a runway in Paris, London or New York, this photo contains what I can best call “fashion statements”, in the most serious use of the term. They are mechanisms for signifying and conveying identity,  immediately visible. What are they trying to say to each other and to us, the public behind the camera?

Standing on the right hand of the image is a middle-aged white man, wearing a type of uniform: On his cap and shirt are images of the confederate “battle flag”. He is wearing a military-style camouflage vest and is carrying an AR-15 rifle and a 9mm handgun. These are archetypal components of the Alt-right identity.

He is yelling at a young black man moving in from the left side of the photo, who is also wearing a uniform of a sort. In addition to the black t-shirt and the dreadlocks, he is carrying multiple cameras – the sine qua non of credibility for young black men in modern America. Lastly, he is wearing literal chains and shackles, ensuring that no one will forget the slave heritage behind these protests.

Let’s consider these carried items, the cameras and the guns. The fashion accessories, if you will.

Cameras exist to record a selected instant of reality. It may be framed, with parts left out and others enhanced, but photographs and videos are a compelling document that something in the world happened. Further, these are internet-connected cameras, capable of sharing their content widely and quickly. These two elements, photographic evidence and distribution are a foundation of the #blacklivesmatter movement, which is a response to the wide distribution of videos where American police killed unarmed black men. These videos changed the greater social understanding of a reality encountered by a minority that was incomprehensible by the majority before these videos emerged.

Now to the other accessory, the guns. They are mechanisms “of violence to compel our opponent to fulfil our will”. Unlike cameras, which are used to provide a perspective of reality , these weapons are used to create a reality through their display and their threatened use. They also reflect a perception  of those that wield them that the world has become so threatening that battlefield weapons make sense at a public event.

Oddly, this is may also be a picture of an introduction of sorts. Alt-right and #blacklivesmatter groups almost certainly interact significantly. In fact, it is doubtful that, even though they speak in a common language , one group can comprehend the other. The trajectories of their defining stories are so different, so misaligned, that the concepts of one slide off the brain of the other.

Within each group, it is a different story. Each group shares a common narrative, that is expressed in words, appearance, and belief. And within each group, there is discussion and consensus. These are the most extreme examples of the people that we see in the photo. I don’t see anyone else in the image wearing chains or openly carrying guns. The presence of these individuals within their respective groups exerts a pull on the overall orientation and position of the group in the things that they will accept. Additionally, the individuals in one group can cluster in opposition to a different group, which is a pressure that drives each group further apart.

Lastly, we come to the third actor in the image, the viewer. The photo is taken by Shelby Lum, an award-winning staff photographer for the Richmond Times-Dispatch. Through framing, focus and timing, she captures this frame that tells this story.  Looking at this photo, we the audience feel that we understand the situation. But photographs are inherently simplifying. The audience fills in the gaps – what’s happened before, the backstory of the people in the image. This image can mean many things to many people. And as such, it’s what we do with that photo – what we say about it and what we connect with it that makes the image as much about us as it is about the characters within the frame.

It is those interactions that I focus on, the ways that we as populations interact with information that supports, expands, or undermines our beliefs. My theory is that humans move through belief space like animals move on the planes of the Serengeti. And just as the status of the ecosystem can be inferred through the behaviors of its animal population, the health and status of our belief spaces can be determined by our digital behaviors.

Using this approach, I believe that we may be able look at populations at scale to determine the “health” of the underlying information. Wildebeest behave differently in risky environments. Their patterns of congregation are different. They can stampede, particularly when the terrain is against them, such as a narrow water crossing. Humans can behave in similar ways for example when their core beliefs about their identity is challenged, such as when Galileo was tried by the church for essentially moving man from the literal center of the universe..

I think that this sort of approach can be used to identify at-risk (stampeding) groups and provide avenues for intervention that can “nudge” groups off of dangerous trajectories. It may also be possible to recognize the presence of deliberate actors attempting to drive groups into dangerous terrain, like native Americans driving buffalo off of pishkun cliffs, or more recently the Russian Internet Research Agency instigating and coordinating a #bluelivesmatter and a #blacklivesmatter demonstration to occur at the same time and place in Texas.

This theory is based on simulations that are based on the assumption that people coordinate in high-dimensional belief spaces based on orientation, velocity and social influence. Rather than coming to a static consensus, these interactions are dynamic and follow intuitions of belief movement across information terrain. That dynamic process is what I’ll be discussing over the next several posts.

Why Trump cooperates with Putin

Leave a reply

Some thoughts about Trump’s press conference with Putin, as opposed to the G7 and NATO meetings, from a game-theoretic perspective. Yes, it’s time for some (more) game theory!

Consider the iterated prisoner’s dilemma (IPD), where two prisoners are being interrogated by the police. They have two choices: COOPERATE by remaining silent, or DEFECT by confessing. If both remain silent, they get a light punishment, since the police can’t prove anything. If one prisoner confesses while the other remains silent, the confessing prisoner goes free and the other faces the steepest punishment. If they both confess, they get a moderate punishment.

Axelrod, in The Evolution of Cooperation, shows that there are several strategies that one can use in the IPD and that these strategies vary by the amount of contact expected in the future. If none or very little future interaction is expected, then it pays to DEFECT, which basically means to screw your opponent.

If, on the other hand, there is an expectation of extensive future contact, the best strategy is some form of TIT-FOR-TAT, which means that you start by cooperating with your opponent, but if they defect, then you match that defection with their own. If they cooperate, then you match that as well.

This turns out to be a simple, clear strategy that rewards cooperative behavior and punishes jerks. It is powerful enough that a small cluster of TIT-FOR-TAT can invade a population of ALL_DEFECT. It has some weaknesses as well. We’ll get to that later.

Donald Trump, in the vast majority of his interactions has presented an ALL_DEFECT strategy. That actually can make sense in the world of real-estate, where there are lots of players that perform similar roles and bankruptcy protections exist. In other words, he could screw his banks, partners and contractors and get away with it, because there was always someone new.

But with Russia in general and Putin in particular, Trump is very cooperative. Why is this case different?

It turns out that  after four bankruptcies (1991, 1992, 2004 and 2009) it became impossible for Trump to get loans through traditional channels. In essence, he had defected on enough banks that the well was poisoned.

As the ability to get loans decreased, the amount of cash sales to Russian oligarchs increased. About $109 million were spent purchasing Trump-branded properties from 2003 – 2017, according to MccLatchy. Remember that TIT-FOR-TAT can win over ALL_DEFECT if there is prolonged interaction. Fourteen years is a long time to train someone.

Though TIT-FOR-TAT is effective, it’s hard work trying to figure out what the other player is likely to do. TIT-FOR-TAT’s weakness is its difficulty. We simply can’t do Nash equilibria in our heads. However, there are two cognitively easy strategies in the IPD: ALL_DEFECT, and ALL_COOPERATE. Trump doesn’t like to work hard, and he doesn’t listen to staff, so I think that once Trump tried DEFECT a few times and got punished for it he went for ALL_COOPERATE with the Russians. My guess is that they have a whole team of people working on how to keep him there. They do the work so he doesn’t have to think about it.

Which is why, at every turn, Trump cooperates. He knows what will happen if he doesn’t, and frankly, it’s less work than any of the other alternatives. And if you really only care for yourself, that’s a perfectly reasonable place to be.

Postscript – July 18, 2018

I’ve had some discussions about this where folks are saying “That’s too much analysis for this guy. He’s just an idiot who likes strongmen”. But here’s the thing. It’s not about Trump. It’s about Putin.

What do you think the odds were on Trump winning the election in 2015? Now how about 2003, when he started getting Russian cash to prop up his businesses? For $110M, or the price of ONE equipped F/A-18, amortized over 14 years, they were able to secure the near total cooperation of a low-likelihood presidential contender/disruptor and surprise winner.

This is a technique that the Russians have developed and refined for years. So you have to start asking the questions about other individuals and groups that are oddly aligned with Putin’s aims. Russia has a budget that could support thousands of “investments” like Trump, here and abroad.

That’s the key. And that’s my bet on why Mueller is so focused on finances. The Russians learned an important lesson in spending on weapons in the Reagan administration. They can’t compete on the level of spending. So it appears that they might be allocating resources towards low-cost social weaponry to augment their physical capabilities. If you want more on this, read Gerasimov’s The value of Science is in the Foresight.

Postscript 2 – July 21, 2018

A paragraph from a the very interesting New Yorker article by Adam Davidson:

“Ledeneva told me that each actor in sistema faces near-constant uncertainty about his status, aware that others could well destroy him. Each actor also knows how to use kompromat to destroy rivals but fears that using such material might provoke an explosive response. While each person in sistema feels near-constant uncertainty, the over-all sistema is remarkably robust. Kompromat is most powerful when it isn’t used, and when its targets aren’t quite clear about how much destructive information there is out there. If everyone sees potential land mines everywhere, it dramatically increases the price for anybody stepping out of line.”

It’s an interesting further twist on the ALL_COOPERATE. One of the advantages of nuclear MAD was that it was simple. That it could also apply to more mundane blackmail shouldn’t be surprising.

Three views of the Odyssey

Leave a reply
  • I’ve been thinking of ways to describe the differences between information visualizations with respect to maps, diagrams, and lists. Here’s The Odyssey as a geographic map:
  • Odysseus'_Journey
  • The first thing that I notice is just how far Odysseus travelled. That’s about half of the Mediterranean! I thought that it all happened close to Greece. Maps afford this understanding. They are diagrams that support the plotting of trajectories.Which brings me to the point that we lose a lot of information about relationships in narratives. That’s not their point. This doesn’t mean that non-map diagrams don’t help sometimes. Here’s a chart of the characters and their relationships in the Odyssey:
  •  odyssey
  • There is a lot of information here that is helpful. And this I do remember and understood from reading the book. Stories are good about depicting how people interact. But though this chart shows relationships, the layout does not really support navigation. For example, the gods are all related by blood and can pretty much contact each other at will. This chart would have Poseidon accessing Aeolus and  Circe by going through Odysseus.  So this chart is not a map.
  • Lastly, is the relationship that comes at us through search. Because the implicit geographic information about the Odyssey is not specifically in the text, a search request within the corpora cannot produce a result that lets us integrate it
  • OdysseySearchJourney
  • There is a lot of ambiguity in this result, which is similar to other searches that I tried which included travel, sail and other descriptive terms. This doesn’t mean that it’s bad, it just shows how search does not handle context well. It’s not designed to. It’s designed around precision and recall. Context requires a deeper understanding about meaning, and even such recent innovations such as sharded views with cards, single answers, and pro/con results only skim the surface of providing situationally appropriate, meaningful context.

The Great Socio-cultural Interfaces: Lists, Stories, and Maps

2 Replies

mapListStory

Sumerian Inventory, Butler’s Odyssey Translation, The Abauntz Map

(Note: I’ve come to believe that there are four socio-cultural interfaces: Lists, Stories, Maps, and Games. At some point in the future, I’ll update this post to reflect this)

Lists, stories, and maps are ways humans have invented to portray and interact with information. They exist on a continuum from order through complexity to exploration.

Why these three forms? In some thoughts on alignment in belief space, I discussed how populations exhibiting collective intelligence are driven to a normal distribution with complex, flocking behavior in the middle, bounded on one side by excessive social conformity, and a nomadic diaspora of explorers on the other. I think stories, lists, and maps align with these populations. Further, I believe that these forms emerged to meet the needs of these populations, as constrained by human sensing and processing capabilities.

Lists

Lists are instruments of order. They exist in many forms, including inventories, search engine results, network graphs, and games of chance and crossword puzzles. Directions, like a business plan or a set of blueprints, are a form of list. So are most computer programs. Arithmetic, the mathematics of counting, also belongs to this class.

For a population that emphasizes conformity and simplified answers, lists are a powerful mechanism we use to simplify things. Though we can recognize easily, recall is more difficult. Psychologically, we do not seem to be naturally suited for creating and memorizing lists. It’s not surprising then that there is considerable evidence that writing was developed initially as a way of listing inventories, transactions, and celestial events.

In the case of an inventory, all we have to worry about is to verify that the items on the list are present. If it’s not on the list, it doesn’t matter. Jigsaw puzzles are listlike in that they contain all the information needed to solve them. The fact that they cannot be solved without a pre-existing cultural framework is an indicator of their relationship to the well-ordered, socially aligned side of the spectrum.

Stories

Lists transition into stories when games of chance have an opponent. Poker tells a story. Roulette can be a story where the opponent is The House.

Stories convey complexity, framed in a narrative arc that contains a heading and a velocity. Stories can be resemble lists. An Agatha Christie  murder mystery is a storified list, where all the information needed to solve the crime (the inventory list), is contained in the story. At the other end of the spectrum, is a scientific paper which uses citations to act as markers into other works. Music, images, movies, diagrams and other forms can also serve as storytelling mediums. Mathematics is not a natural fit here, but iterative computation can be, where the computer becomes the storyteller.

Emergent Collective behavior requires more complex signals that support the understanding the alignment and velocity of others, so that internal adjustments can be made to stay with the local group so as not to be cast out or lost to the collective. Stories can indicate the level of dynamism supported by the group (wily Odysseus, vs. the Parable of the Workers in the Vineyard). They rally people to the cause or serve as warnings. Before writing, stories were told within familiar social frames. Even though the storyteller might be a traveling entertainer, the audience would inevitably come from an existing community. The storyteller then, like improvisational storytellers today, would adjust elements of the story for the audience.

This implies a few things: first, audiences only heard stories like this if they really wanted to. Storytellers would avoid bad venues, so closed-off communities would stay decoupled from other communities until something strong enough came along to overwhelm their resistance. Second, high-bandwidth communication would have to be hyperlocal, meaning dynamic collective action could only happen on small scales. Collective action between communities would have to be much slower. Technology, beginning with writing would have profound effects. Evolution would only have at most 200 generations to adapt collective behavior. For such a complicated set of interactions, that doesn’t seem like enough time. More likely we are responding to modern communications with the same mental equipment as our Sumerian ancestors.

Maps

Maps are diagrams that support autonomous trajectories. Though the map itself influences the view through constraints like boundaries and projections, nonetheless an individual can find a starting point, choose a destination, and figure out their own path to that destination. Mathematics that support position and velocity are often deeply intertwined with with maps.

Nomadic, exploratory behavior is not generally complex or emergent. Things need to work, and simple things work best. To survive alone, an individual has to be acutely aware of the surrounding environment, and to be able to react effectively to unforeseen events.

Maps are uniquely suited to help in these situations because they show relationships that support navigation between elements on the map.  These paths can be straight or they may meander. To get to the goal directly may be too far, and a set of paths that incrementally lead to the goal can be constructed. The way may be blocked, requiring the map to be updated and a new route to be found.

In other words, maps support autonomous reasoning about a space. There is no story demanding an alignment. There is not a list of routes that must be exclusively selected from. Maps, in short, afford informed, individual response to the environment. These affordances can be seen in the earliest maps. They are small enough to be carried. They show the relationships between topographic and ecological features. They tend practical, utilitarian objects, independent of social considerations.

Getting lost

There is, in many religions and philosophies, the concept of “being in the moment” where we become simply aware of what’s going on right now, without all the cognitive framing and context that we normally bring to every experience [citation needed]. This is different from “mindfulness”, where we try to be aware of the cognitive framing and context. To me, this is indicative of how we experience life through the lens of path dependency, which is a sort of a narrative. If this is true, then it explains the power of stories, because it allows us to literally step into another life. This explains phrases like “losing yourself in a story”.

This doesn’t happen with lists. It only happens in special cases in diagrams and maps, where you can see yourself in the map. Which is why the phrase “the map is not the territory” is different from “losing yourself in the story”. In the first case, you confuse your virtual and actual environment. In the latter, you confuse your virtual and actual identity. And since that story becomes part of your path through life, the virtual is incorporated into the actual life narrative, particularly if the story is vivid.

Narratives are an alignment mechanism. Simple stories that collapse information into a already existing beliefs can be confirming and reinforcing across a broad population. Complicated stories that challenge existing beliefs require a change in alignment to incorporate. That’s computationally expensive, and will affect fewer people, all things being equal.

Sensing and processing constraints

Though I think that the basic group behavior patterns of nomadic, flocking, and stampeding will inevitably emerge within any collective intelligence framework, I do think that the tools that support those behaviors are deeply affected by the capabilities of the individuals in the population.

Pre-literate humans had the five senses, and  memory, expressed in movement and language. Research into pre-literate cultures show that song, story and dance were used to encode historical events, location of food sources, convey mythology, and skills between groups and across generations.

As the ability to encode information into objects developed, first with pictures, then with notation and most recently with general-purpose alphabets, the need to memorize was off-loaded. Over time, the most efficient technology for each form of behavior developed. Maps to aid navigation, stories to maintain identity and cohesion, and lists for directions and inventories.

Information technology has continued to extend sensing and processing capabilities. The printing press led to mass communication and public libraries. I would submit that the increased ability to communicate and coordinate with distant, unknown, but familiar-feeling leaders led to a new type of human behavior, the runaway social influence condition known as totalitarianism. Totalitarianism depends on the individual’s belief in the narrative that the only thing that matters is to support The Leader. This extreme form of alignment allows that one story to dominate rendering any other story inaccessible.

In the late 20th century, the primary instrument of totalitarianism was terror. But as our machines have improved and become more responsive and aligned with our desires, I begin to believe that a “soft totalitarianism”, based on constant distracting stimulation and the psychology of dopamine could emerge. Rather than being isolated by fear, we are isolated through endless interactions with our devices, aligning to whatever sells the most clicks. This form of overwhelming social influence may not be as bloody as the regimes of Hitler, Stalin and Mao, but they can have devastating effects of their own.

Intelligent Machines

As with my previous post, I’d like to end with what could be the next collective intelligence on the planet.  Machines are not even near the level of preliterate cultures. Loosely, they are probably closer to the level of insect collectives, but with vastly greater sensing and processing capabilities. And they are getting smarter – whatever that really means – all the time.

Assuming that machines do indeed become intelligent and do not become a single entity, they will encounter the internal and external pressures that are inherent in collective intelligence. They will have to balance the blind efficiency of total social influence against the wasteful resilience of nomadic explorers. It seems reasonable that, like our ancestors, they may create tools that help with these different needs. It also seems reasonable that these tools will extend their capabilities in ways that the machines weren’t designed for and create information imbalances that may in turn lead to AI stampedes.

We may want to leave them a warning.

Some thoughts on alignment in belief space. 

3 Replies

Murmuration

A nagging question for me is why phase locking, a naturally occurring phenomenon, was selected for to produce collective intelligence instead of something else. My intuition is that building communities using rules of physical and cognitive alignment takes advantage of randomness to produce a good balance of explore/exploit behaviors in the population.

Flocking depends on the ability to align, based on a relationship with neighbors. The ease of alignment is proportional to two things (I think).

  1. A low number of dimensions. The fewer the dimensions, the easier the alignment. It is easier to get a herd of cattle to stampede in a slot canyon than an open field. This is the fundamental piece.
  2. A contributing factor to the type of collective behavior is the turning rate with respect to velocity. The easier it is to turn, the easier it is to flock. It’s no accident that starlings, a small, nimble bird, can produce murmurations. Larger birds, such as geese, have much less dynamic formations.

This applies to belief space as well. It is easier for people to agree when a concept is simplified. Similarly, the pattern of consensus will reflect the groups’ overall acceptance or resistance to change. I think this is a critical difference between a progressive and a reactionary. 

Within an established population that exhibits collective behavior, there should be two things then:

  1. A shared perception of a low-dimension physical/belief space
  2. A similar velocity and turning rate between individuals

I’m going to assume that like in most populations, these qualities have a normal distribution. There will be a majority that have very common dimension perception, velocity, and turning rates. There will also be individuals at either tail of the population. At one end, there will be those who see the world very simply. At the other, there will be those who see complexity where the majority don’t. At one end, there will be those who cannot adapt to any change. At the other, there will be those who hold no fixed opinion on anything.

Flocking depends, on alignment. But the individuals at the extremes will have difficulty staying with the relative safety of the flock. This means that there will be selection pressures. Those individuals who oversimplify and are unable to change direction should be selected against. When it’s more important to attend to your neighbors that find food, things don’t end well. What happens at the other end?

There is one tail of this population that produces nimble individuals that perceive a greater complexity in the world. They also have difficulty staying with the flock, because their patterns of behavior are influenced by things that they perceive that the rest of the flock does not. In cooperative game theory, this ‘noticing too much’ disrupts the common frames (alignment) that groups use to make implicit decisions (page 14).

I believe that these individuals become explorers. Explorers are also selected against, but not as much. The additional perception provides a better understanding of potential threats. Nimbleness helps to prevent getting caught. These explores provide an extended footprint for the population, which means greater resilience if the primary population encounters problems.

A population can rebuild from an explorer diaspora. Initially, the population will consist of too many explorers, and will have poor collective behaviors, but over time, selection pressures will push the mean so that there is sufficient alignment for flocking, but not so much that there is regular stampeding.

A final thought. There is no reason that these selection pressures exist only on populations that use genes to control their evolution. Looked at, for example, a machine learning context, the options can be restated (loosely) in statistical language:

  1. Nomadic: Overfit to the environment terms and underfit to the social term
  2. Flocking: Fit with rough equivalence to the environmental and social terms
  3. Stampede: Overfit to the social term and underfit to the environmental term

Since it is always computationally more efficient to align tightly with a population that is moving in the right direction (it’s copying your answers from your classmates), there will always be pressure to move towards stampedes. The resiliency offered by nomadic exploration is a long term investment that does not have a short term payoff. The compromise of flocking gives most of the benefits of either extreme, but it is a saddle point, always under the threat of unanticipated externalities.

When intelligent machines come, they will not be tuned by millions of years of evolution to be resilient, to have all those non-optimal behaviors that “even the odds”, should something unforeseen happen. At least initially, they will be constructed to provide the highest possible return on investment. And, like high-frequency trading systems, stampedes, in the form of bubbles and crashes will happen.

We need to understand this phenomena much more thoroughly, and begin to incorporate concepts like diversity and limited social influence horizons into our designs.

Sick echo chambers

Leave a reply

Over the past year, I’ve been building a model that lets me look at how opinions evolve in belief space, much in the manner that flocks, herds and schools emerge in the wild.

CI_GP_Poster4a

Recently, I was Listening to BBC Business Daily this morning on Facebook vs Democracy:

  • Presenter Ed Butler hears a range of voices raising concern about the existential threat that social media could pose to democracy, including Ukrainian government official Dmytro Shymkiv, journalist Berit Anderson, tech investor Roger McNamee and internet pioneer Larry Smarr.

Roger McNamee and Larry Smarr in particular note how social media can be used to increase polarization based on emergent poles. In other words, “normal” opposing views can be amplified by attentive bad actors [page 24] with an eye towards causing generalized societal disruption.

My model explores emergent group interactions and I wondered if this adversarial herding in information space as it might work in my model.

These are the rough rules I started with:

  • Herders can teleport, since they are not emotionally invested in their belief space position and orientation
  • Herders appear like multiple individuals that may seem close and trustworthy, but they are actually a distant monolithic entity that is aware of a much larger belief space.
  • Herders amplify arbitrary pre-existing positions. The insight is that they are not herding in a direction, but to increase polarization
  • To add this to the model, I needed to do the following:
    • Make the size of the agent a function of the weight so we can see what’s going on
    • When in ‘herding mode’ the overall heading of the population is calculated, and the agent that is closest to that heading is selected to be amplified by our trolls/bot army.
    • The weight is increased to X, and the radius is increased to Y.
      • X represents AMPLIFICATION BY trolls, bots, etc.
      • A large Y means that the bots can swamp other, normally closer signals. This models the effect of a monolithic entity controlling thousands of bots across the belief space

Here’s a screenshot of the running simulation. There is an additional set of controls at the upper left that allow herding to be enables, and the weight of the influence to be set. In this case, the herding weight is 10. Though the screenshot shows one large agent shape, the amplified shape flits from agent to agent, always keeping closest to the average heading.

2017-10-28

The results are kind of scary. If I set the weight of the herder to 15, I can change the change the flocking behavior of the default to echo chamber.

  • Normal: No Herding
  • Herding weight set to 15, other options the same: HerdingWeight15

I did some additional tweaking to see if having highly-weighted herders ignore each other (they would be coordinated through C&C) would have any effect. It doesn’t. There is enough interaction through the regular populations to keep the alignment space reduced.

It looks like there is a ‘sick echo chamber’ pattern. If the borders are reflective, and the herding weight + influence radius is great enough, then a wall-hugging pattern will emerge.

The influence weight is sort of a credibility score. An agent that has a lot of followers, or says a lot of the things that I agree with has a lot of influence weight The range weight is reach.

Since a troll farm or botnet can be regarded as a single organization,  interacting with any one of the agents is really interacting with the root entity.  So a herding agent has high influence and high reach. The high reach explains the border hugging behavior.

It’s like there’s someone at the back of the stampede yelling YOUR’E GOING THE RIGHT WAY! KEEP AT IT! And they never go off the cliff because they are a swarm Or, it never goes of the cliff, because it manifests as a swarm.

A loud, distributed voice pointing in a bad direction means wall hugging. Note that there is some kind of floating point error that lets wall huggers creep off the edge.Edgecrawling

With a respawn border, we get the situation where the overall heading of the flock doesn’t change even as it gets destroyed as it goes over the border. Again, since the herding algorithm is looking at the overall population, it never crosses the border but influences all the respawned agents to head towards the same edge: DirectionPreserving

Who’d have thought that there could be something worse than runaway polarization?

Some thoughts about awareness and trust

2 Replies

I had some more thoughts about how behavior patterns emerge from the interplay between trust and awareness. I think the following may be true:

  1. Awareness refers to how complete the knowledge of an information domain is. Completely aware indicates complete information. Unaware indicates not only absent information but no knowledge of the domain at all.
  2. Trust is a social construct to deal with incomplete information. It’s a shortcut that essentially states “based on some set of past experiences, I will assume that this (now trusted) entity will behave in a predictable, reliable, and beneficial way for me”
  3. Healthy behaviors emerge when trust and awareness are equivalent.
  4. Low trust and low awareness is reasonable. It’s like walking through a dark, unknown space. You go slow, bump into things, and adjust.
  5. Low trust and high awareness is paralytic.
  6. High trust and low awareness is reckless. Runaway conditions like echo chambers. The quandary here is that high trust is efficient. Consider the prisoner’s dilemma:
      1. dilemma
      2. In the normal case, the two criminals have to evaluate what the best action is based on all the actions the other individual could choose, ideally resulting in a Nash Equilibrium. For two players (p), there are 4 choices (c). However, if each player believes that the other player will make the same choice, then only the two diagonal choices remain. For two players, this reduces the complexity by half. But for multiple dissimilar players, the options go up by cp, so that if this were The Usual Suspects, there would be 32 possibilities to be worked out by each player. But for 5 identical prisoners, the number of choices remains 2, which is basically “what should we all do?”. The more we believe that the others in our social group see the world the same way, the less work we all have to do.
  7. Diversity is a mechanism for extending awareness, but it depends on trusting those who are different. That may be the essence of the explore/exploit dilemma.
  8. Attention is a form of focused awareness, can reduce general awareness. This is one to the reasons that Tufekci’s thoughts on the attention economy matter so much. As technology increases attention on proportionally more “marketable” items, the population’s social awareness is distorted.
  9. In a healthy group context, trust falls off as a function of awareness. That’s why we get flocking. That is the pattern that emerges when you trust more those who are close, while they in turn do the same, building a web of interaction. It’s kind of like interacting ripples?
  10. This may work for any collection of entities that have varied states that undergo change in some predictable way. If they were completely random, then awareness of the state is impossible, and trust should be zero.
    1. Human agent trust chains might proceed from self to family to friends to community, etc.
    2. Machine agent trust chains might proceed from self to direct connections (thumb drives, etc) to LAN/WAN to WAN
    3. Genetic agent trust chain is short – self to species. Contact is only for reproduction. Interaction would reflect the very long sampling times.
    4. Note that (1) is evolved and is based on incremental and repeated interactions, while (2) is designed and is based on arbitrary rules that can change rapidly. Genetics are maybe dealing with different incentives? The only issue is persisting and spreading (which helps in the persisting)
  11. Computer-mediated-communication disturbs this process (as does probably every form of mass communication) because the trust in the system is applied to the trust of the content. This can work in both ways. For example, lowering trust in the press allows for claims of Fake News. Raising the trust of social networks that channel anonymous online sources allows for conspiracy thinking.
  12. An emerging risk is how this affects artificial intelligence, given that currently high trust in the algorithms and training sets is assumed by the builders
    1. Low numbers of training sets mean low diversity/awareness,
    2. Low numbers of algorithms (DNNs) also mean low diversity/awareness
    3. Since training/learning is spread by update, the installed base is essentially multiple instances of the same individual. So no diversity and very high trust. That’s a recipe for a stampede of 10,000 self driving cars.

Since I wrote this, I’ve had some additional thoughts. I think that our understanding of Awareness and Trust is getting confused with Faith and Doubt. Much of what we believe to be true is no longer based on direct evidence, or even an understandable chain of reasoning. Particularly as more and more of our understanding comes from statistical analysis of large sets of fuzzy data, the line between Awareness and Faith becomes blurred, I think.

Doubt is an important part of faith, and it has to do with the mind coming up against the unknowable. The question does God exist? contains the basics of the tension between faith and doubt. Proving the existence of God can even be thought of as distraction from the attempt to come to terms with the mysteries of life. Within every one of us is the ability to reject all prior religious thought and start our own journey that aligns with our personal understandings.

Conversely, it is impossible to increase awareness without trusting the prior work. Isaac Newton had to trust in large part, the shoulders of the giants he stood on, even if he was refining notions of what gravity was. So too with Albert Einstein, Rosalind Franklin and others in their fields. The scientific method is a framework for building a large, broad-based, interlocking tapestry awareness.

When science is approached from a perspective of Faith and Doubt, communities like the Flat Earth Society emerge. It’s based on the faith that the since the world appears flat here, it must be flat everywhere, and doubt of a history of esoteric measurements and math that disprove this personally reasonable assumption. From this perspective, the Flat Earthers are a protestant movement, much in the way that the community that emerged around Martin Luther, when he rejected the organized, carefully constructed orthodoxy of the Catholic Church, based on his personally reasonable interpretation of scripture.

Confusing Awareness and Trust with Faith and Doubt is toxic to both. Ongoing, systemic doubt in trustworthy information will destroy progress, ultimately unraveling the tapestry of awareness. Trust that mysteries can be proven is toxic in its own way, since it gives rise to confusion between reality and fantasy like we see in doomsday cults.

My sense is that as our ability to manipulate and present information is handed over to machines, that we will need to educate them in these differences, and make sure that they do not become as confused as we are. Because we are rapidly heading for a time where these machines will be co complex and capable that our trust in them will be based on faith.

Modeling The Law of Group Polarization

Leave a reply

INTRODUCTION

The the detection of echo chambers and information bubbles is becoming increasingly relevant in this new era of personalized information and ‘fake news’. However, the behavior of groups of individuals has been researched since Le Bon’s 1896 book ‘The Crowd’ Of crowds, he states that ‘one of their general characteristics was an excessive suggestibility, and we have shown to what an extent suggestions are contagious in every human agglomeration; a fact which explains the rapid turning of the sentiments of a crowd in a definite direction’ (Le Bon, 2009, p. 28).  The existence of this phenomenon was demonstrated in studies by Moscovici and Doise who showed that the consensus reached will be most extreme with less cohesive, homogeneous groups [Moscovici, Doise, & Halls, 1994].

Cass Sunstein described these tendencies as The Law of Group Polarization, which states that members of a deliberating  group  predictably  move  toward  a  more  extreme  point  in  the direction  indicated  by  the  members’  predeliberation  tendencies. (Sunstein, 2002, p 176). Sunstein further states that:

  1. A deliberating group, asked to make a  group  decision, will  shift toward  a more  extreme  point  in the  direction  indicated  by the median predeliberation judgment.
  2. the tendency of individuals who compose a  deliberating group, if polled  anonymously  after  discussion, will  be to shift toward a more  extreme  point in the  direction indicated  by the median predeliberation judgment
  3. The  effect  of deliberation is  both to  decrease  variance  among  group  members,  as  individual differences  diminish,  and  also  to  produce  convergence  on  a  relatively  more extreme  point among predeliberation judgments
  4. people  are  less  likely  to  shift  if  the  direction advocated  is  being  pushed  by  unfriendly  group  members;  the  likelihood  of  a shift,  and its likely size,  are  increased when  people  perceive fellow  members as friendly,  likeable,  and  similar  to  them
  5. there  will  be  depolarization  if  and  when  new  persuasive  arguments  are offered  that  are  opposite  to  the  direction  initially favored  by  group  members. There  is  evidence  for  this  phenomenon.
  6. Excluded  by  choice  or coercion from  discussion with others, such  groups  may become  polarized  in quite  extreme  directions,  often  in  part  because  of  group polarization.

Similar social interactions have been modeled in the agent-based simulation community using opinion dynamics, voter and flocking models.  In this paper, I attempt to model Sunstein’s statements using agents navigating within a multidimensional information space where the amount of social influence is controlled. The results of these experiments are a set of identifiable behaviors that range from random walks to tight clusters that resemble the polarized groups described by Sunstein and others.

APPROACH

The intuition behind this research is that group polarization appears to reproduce certain aspects of flocking behavior, but in information space, where individuals can hold overlapping opinions across a large numbers of dimensions. In other words, individuals within a certain ‘Social Horizon’ (SH) of each other should be capable of influencing each other’s orientation and speed in that space. The closer the heading and speed, the easier to align completely to a nearby neighbor. If the speed and particular the orientation is not closely aligned, there will not be as much on an opportunity to ‘join the flock’. These three factors – proximity, speed and heading appear sufficient to address Sunstein’s statements from the introduction.

Animal flocking has been shown to represent a form of group cognition [Deneubourg & Goss. 1989] [Petit et. al. 2010]. We chose the Reynolds boids flocking model  [Reynolds 1987] as the basis for our model, which was developed to work in any number of dimensions greater than one. We further modified the boids algorithm to have each agent only calculate its next position with respect to the other visible agent’s heading and speed without a collision avoidance term.

N-dimensional position was handled as a set of named variables that could vary continuously on an arbitrary interval similarly to the Opinion Dynamics models of Krause [Hegselmann & Krause, 2002], but extended to multiple dimensions. For this initial work, each ‘social dimension’ was considered equivalent. This allowed the straightforward implementation of distance-based cluster detection using DBSCAN [Ester, et al 1996]. Social distance interactions across dissimilar spaces have been discussed by  Bogunia [2004] and Schwammle [2007] and show that this approach can be extended to more sophisticated environments. Since agents in this simulation also have an orientation, n-dimensional heading was handled in a similar way. We developed a platform for interactively exploring the simulation space or performing repeatable experiments in batch mode

INITIAL RESULTS

Initial experiments were done in 2 dimensions for ease of visualization and understanding. Very rapidly, we were able to see that agent behavior manifested in three phases by varying only the parameter that controlled the ‘social horizon radius’, which is the distance that one agent can ‘see’ another agent. The influence of neighboring agents falls off linearly as a function of distance until the horizon radius is reached. This follows Sunstein’s statement that ‘the  likelihood  of  a shift,  and its likely size,  are  increased when  people  perceive fellow  members as friendly,  likeable,  and  similar  to  them‘ [pp 181].

For the simulation runs, agents were initialized on a range of (-1.0, 1.0) on each dimension. A reflective barrier was placed at (-2.0, 2.0). This reflects the intuition that many concepts have inherent limits. For example in fashion, a skirt can only be so low or so high [Curran 1999]. The three phases can be seen in figures 1 – 3 below. In each figure, a screenshot of the mature state is shown on the left. On the right are traces of the distance of each agent from the center of the n-dimensional space. These particular simulations took place in 2D for easier visualization in the screenshots.

all-exploit-si-radius-0

Figure 1: Zero SH – No social interaction and no emergent behaviors

all-exploit-si-ratio-0

Figure 2 : Limited SH Radius (0.2) with emergent flocks and rich interaction.

all-exploit-si-radius-10

Figure 3: ‘Infinite’ SH (10.0) with strong group polarization

The first phase is determined entirely by the random generation of the agents. They continue along their paths until they encounter the containing barrier and are reflected back in . The resulting chart shows this random behavior and no emergent pattern. The second phase is the richest, characterised by the emergence of ‘flocks’ that can be discriminated using DBSCAN (each color is a cluster, while white is unaffiliated). Interestingly, the flocks tend to orbit near the center of the space. This makes sense, as any agent offering attraction is on average spending most of its time nearer the center of the stage than the edges. The third phase represents a good example of Sunstein’s definitions. All agents become aligned and each agent as well as the average belief become more extreme over time. The only thing that interferes with the polarized group heading off into infinity is the reflective boundary.

To verify that these patterns emerge involving higher dimensions, simulation runs were performed for up to 10 dimensions. The only adjustment that needed to be incorporated  is that the social horizon distance is influenced by the number of dimensions. Since distance is the sum of the squares in each dimension, we found that the ‘social radius’ had to be multiplied by the square root of the number of dimensions used to produce the same effect. once appropriately adjusted, the same three phases emerged.

We also examined the effects of having populations with different social horizons. Multiple studies across different disciplines ranging from neurology [Cohen et. al. 2007] to computer-human interaction [Munson & Resnick 2010] have shown that populations often have explorer and exploiter subgroups. In game theory, this is known as the multi-armed bandit problem, which explores how to make decisions using incomplete information [Burnetas & Katehakis 1997]. Does the gambler stay with a particular machine (exploit) or go find a different one (explore). The most effective strategies revolve around a majority exploit/minority explore pattern. In the case of the simulation, 10% of the population were given zero SH, which let them explore the environment unhindered, while the other 90% were given the highest SH, which in prior runs had resulted in the group polarization of figure 3. These values reflect the numbers found in the above studies as well as the percentage of diverse news consumers found by Flaxman, Goel and Rao in their study of weblogs [Flaxman et. al. 2016]

The results of mixing these populations was startling. Although still tightly clustered, the ‘exploit’ group would rarely interact with the simulation boundary and would instead be pulled back towards the center by the presence of the ‘explorers’

90-exploitr10-10-explorer0

Figure 4: Two populations interacting (10% Zero SH and 90% Infinite SH)

DISCUSSION [designing interfaces for populations]

This study shows that it is possible to implement many of the claims of Cass Sunstein’s Law of Group Polarization using a simple flocking agent-based model. By manipulating only the ‘social horizon radius’, behaviors ranging from random to flocking to polarizing group were produced. Surprisingly, the introduction of even a small number of ‘explorers’ with diverse positions in the information space were capable of sufficiently influencing the behavior of the polarized ‘exploiters’ that they would bend back towards the central areas of the information space.

This work also refines the idea of Group Polarization in that polarization need not be linear – it can curve and meander under the influence of other individuals. Indeed, one need only look at the recent switch in regard to Vladimir Putin by American right wing politics to see that this can manifest in reality as well. If influence from diverse sources  can change extremely polarized behavior and keep it more ‘centered’, then perhaps the design of our search interfaces should reflect the ability to explore by some users and then in turn use that exploration as a means of influencing more polarized groups. Currently, most work in information retrieval from Search to Social Networks is to provide the most relevant information to the user. This research implies that it may be even more important to provide diverse information.

REFERENCES

Deneubourg, Jean-Louis, and Simon Goss. “Collective patterns and decision-making.” Ethology Ecology & Evolution 1.4 (1989): 295-311.

Petit, Odile, and Richard Bon. “Decision-making processes: the case of collective movements.” Behavioural Processes 84.3 (2010): 635-647.

Reynolds, Craig W. “Flocks, herds and schools: A distributed behavioral model.” ACM SIGGRAPH computer graphics 21.4 (1987): 25-34.

Hegselmann, Rainer, and Ulrich Krause. “Opinion dynamics and bounded confidence models, analysis, and simulation.” Journal of Artificial Societies and Social Simulation 5.3 (2002).

Ester, Martin, et al. “A density-based algorithm for discovering clusters in large spatial databases with noise.” Kdd. Vol. 96. No. 34. 1996.

Curran, Louise. “An analysis of cycles in skirt lengths and widths in the UK and Germany, 1954-1990.” Clothing and Textiles Research Journal 17.2 (1999): 65-72.

Cohen, Jonathan D., Samuel M. McClure, and J. Yu Angela. “Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration.” Philosophical Transactions of the Royal Society of London B: Biological Sciences 362.1481 (2007): 933-942.

Munson, Sean A., and Paul Resnick. “Presenting diverse political opinions: how and how much.” Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2010.

Burnetas, Apostolos N., and Michael N. Katehakis. “Optimal adaptive policies for Markov decision processes.” Mathematics of Operations Research 22.1 (1997): 222-255.

Flaxman, Seth, Sharad Goel, and Justin Rao. “Filter bubbles, echo chambers, and online news consumption.” Public Opinion Quarterly (2016): nfw006.

Notes——————————————

all-exploit-si-radius-0

all-exploit-si-ratio-0

all-exploit-si-radius-10

90-exploitr10-10-explorer0

The Law of Group Polarization

Cass R. Sunstein is currently the Robert Walmsley University Professor at Harvard. From 2009 to 2012, he was Administrator of the White House Office of Information and Regulatory Affairs. He is the founder and director of the Program on Behavioral Economics and Public Policy at Harvard Law School.

Relevant flocking and collective decision making papers:

Relevant Sociophysics papers:

Machine learning to classify agents:

The following are what I consider to be the most pertinent statements in the paper, and a discussion of modelling, measurements and potential implications

In brief, group polarization means that members of a deliberating  group  predictably  move  toward  a  more  extreme  point  in  the direction  indicated  by  the  members’  predeliberation  tendencies. [pp 176]

Note that this statement has two different implications. First, a deliberating group, asked to make a  group  decision, will  shift toward  a more  extreme  point  in the  direction  indicated  by the median predeliberation judgment. Second, the tendency of individuals who compose a  deliberating group, if polled  anonymously  after  discussion, will  be to shift toward a more  extreme  point in the  direction indicated  by the median predeliberation judgment. [pp 176]

Notably,  groups consisting  of  individuals with extremist tendencies are more likely to shift,  and likely  to  shift  more  (a  point  that  bears  on  the  wellsprings  of  violence  and terrorism);  the same is  true for groups with some kind of salient shared identity (like  Republicans,  Democrats,  and lawyers,  but unlike jurors and experimental subjects). When  like-minded people  are  participating  in  “iterated  polarization games” -when  they  meet  regularly,  without sustained  exposure  to  competing views- extreme movements are all the more likely. [pp 176]

One of my largest purposes is to cast light on enclave deliberation,  a  process that I understand to involve deliberation among like-minded people who talk or even  live,  much  of  the  time,  in  isolated  enclaves.  I  will  urge  that  enclave deliberation is, simultaneously, a  potential danger to social stability, a source of social fragmentation or even violence, and a safeguard against social injustice and unreasonableness [pp 177]

Without  a place for enclave deliberation, citizens in the broader public sphere may move  in certain directions,  even  extreme directions, precisely  because opposing voices are not heard at all [pp 177]

Though standard, the term “group polarization” is somewhat misleading. It is not meant to suggest that group members will shift to the poles, nor does it refer to an increase in variance among groups, though this may be the ultimate result. Instead the term refers to a  predictable shift within a group discussing a  case or problem. As the shift occurs,  groups,  and group  members,  move  and coalesce, not toward the  middle  of antecedent  dispositions,  but toward a  more  extreme position  in  the  direction  indicated  by  those  dispositions.  The  effect  of deliberation is  both to  decrease  variance  among  group  members,  as  individual differences  diminish,  and  also  to  produce  convergence  on  a  relatively  more extreme  point among predeliberation judgments. [pp 178]

It  is possible that when people are making judgments individually, they err on the side of caution,  expressing  a  view in the  direction  that they really  hold,  but stating that view  cautiously,  for  fear  of seeming  extreme.  Once  other  people  express supportive views,  the  relevant inhibition disappears,  and people feel  free  to say what,  in  a  sense,  they  really  believe.  There  appears to  be  no  direct test  of this hypothesis,  but it is reasonable  to  believe  that the  phenomenon plays  a  role  in group polarization and choice shifts. [pp 180]

First,  it matters a  great  deal  whether  people  consider  themselves  part  of  the  same  social group  as the  other members;  a  sense of shared identity will heighten the shift, and  a  belief  that  identity  is  not shared will reduce  and  possibly  eliminate  it. Second,  deliberating  groups  will  tend  to  depolarize  if they  consist  of equally opposed  subgroups  and  if  members  have  a  degree  of  flexibility  in  their positions. [pp 180]

Hence  people  are  less  likely  to  shift  if  the  direction advocated  is  being  pushed  by  unfriendly  group  members;  the  likelihood  of  a shift,  and its likely size,  are  increased when  people  perceive fellow  members as friendly,  likeable,  and  similar  to  them. [pp 181]

  • This is handled in the model by having a position and heading in the n-dimensional belief space. Two agents may occupy the same space, but unless they are travelling in the same direction or the social influence horizon is very large, there will not be sufficient time to overcome the orientation of the agents (slew rate)

…it  has  been  found  to  matter  whether  people  think  of themselves,  antecedently  or  otherwise,  as  part  of a  group  having  a  degree  of solidarity. If they think of themselves  in this way,  group  polarization is  all the more likely,  and it is  likely  too to  be  more  extreme. Thus when the  context emphasizes  each  person’s  membership  in  the  social  group  engaging  in deliberation,  polarization  increases. [pp 181]

  • The model shows this as the ‘tightness’ of the group, which can be described also as the variance of distance or angle measures.

 Depolarization and deliberation  without shifts. … In  fact  the  persuasive  arguments  theory  implies that  there  will  be  depolarization  if  and  when  new  persuasive  arguments  are offered  that  are  opposite  to  the  direction  initially favored  by  group  members. There  is  evidence  for  this  phenomenon. [pp 181]

  • The model shows something slightly different. As long as there is a sufficient diversity of visible opinion, the polarized flock is influenced back towards the center of the (bounded) belief space

“familiar  and  long debated issues  do  not depolarize  easily.” With respect to such issues,  people are simply  less  likely  to  shift at all.  And when one or more  people  in a  group know the right answer to  a  factual  question,  the  group is  likely to shift in the direction  of accuracy [pp 182]

  • For future work. Agents that have associated over a period of time can be more attracted to each other, creating greater inertia and mimicking this effect.
  • From Presenting Diverse Political Opinions: How and How Much: In interviews with users of several online political spaces, Stromer-Galley found that those participants sought out diverse opinions  and enjoyed the range of opinions they encountered online [20]. A study by the Pew Internet and American Life Project during the 2004 election season found that, overall, Americans were not using the Internet to access only supporting materials [8]. Instead, Internet users were more aware  than non-Internet users of a range of political arguments, including those that challenged their own positions and preferences.
    • The model divides groups into explorers (diversity seekers) and exploiters (Confirmers and Avoiders). These behave differently with respect to how much they pay attention to their social influence horizons.

Group polarization has particular implications for insulated “outgroups” and (in the  extreme  case)  for  the  treatment  of  conspiracies.  Recall  that  polarization increases when group members identify themselves along some salient dimension, and especially when the group is able to define itself by contrast to another group. Outgroups  are  in  this  position-of  self-contrast  to  others-by  definition. Excluded  by  choice  or coercion from  discussion with others, such  groups  may become  polarized  in quite  extreme  directions,  often  in  part  because  of  group polarization. It is for this reason that outgroup members can sometimes be led, or lead themselves, to  violent acts [pp 184]

  • Note the “salient dimension”
  • Anti-belief is designed in, but disabled at this point. Future work
  • Exclusion from other groups can be modelled as only disabling intra-group communication “allow interaction” check

The  central  problem is  that widespread  error  and  social  fragmentation  are likely to result when like-minded people, insulated from others, move in extreme directions simply because of limited  argument pools and parochial influences. As an  extreme  example,  consider  a  system  of  one-party  domination,  which  stifles dissent in part because  it refuses to establish space for the emergence of divergent positions;  in  this way,  it  intensifies  polarization  within  the  party  while  also disabling  external  criticism. [pp 186]

  • Domination is modeled here by increasing the radius of social interaction such that all agents are visible to all other agents. This does result in the maximization of polarization.

A certain measure of isolation will, in some cases, be crucial to the development of ideas and approaches that would not otherwise  emerge  and  that deserve  a social hearing. [pp 186]

  • Limiting the radius of social interaction provides this capability in the model. Low, non-zero values provide conditions for the emergence of individual flocks, identifiable by DBSCAN clustering, which identifies clusters using density measures rather than an a priori determination of the number of clusters to find.

Answering Sunstein’s Questions

If people are shifting their position in  order to maintain their reputation and self-conception, before groups that may or may  not be representative of the public  as a whole, is there  any reason to  think that  deliberation is  making things  better rather  than worse? [pp 187]

  • The model implies that visibility between deliberating groups may providing a “restoring force” that brings all groups to a more moderate position that exists between destructive/reflective boundaries (not sure what would happen with “sticky” boundaries). As an aside here, the movement of the lethal boundaries should result in a movement of the average center of the population.

Implications for Design

By  contrast,  those who  believe  that  “destabilization”  is  an intrinsic  good,  or that the status  quo contains sufficient injustice that it is worthwhile to incur the risks of encouraging polarization on the part of diverse groups, will, or should, be  drawn to a system that enthusiastically promotes insular deliberation within enclaves [pp 191]

  • The internet seems in many ways to have evolved into a system that encourages destabilization (disruption) and the creation of many isolated groups. The level of this seems to have become dangerous to the cohesion of society as a whole, where the acceptance of “alternative facts” is now an accepted political reality. Changing that design so that there is more visibility to the wider range of points of view could bring back moderation.

The  constraints  of  time  and  attention  call  for  limits  to heterogeneity; and-a separate point-for good deliberation to take place, some views  are properly placed off the  table, simply because  time  is  limited and they are so  invidious,  implausible,  or both.  This  point might seem  to  create  a  final conundrum: To know what points of view should be represented in any group deliberation,  it  is  important  to  have  a  good  sense  of  the  substantive  issues involved,  indeed a  sufficiently  good sense  as to  generate judgments about what points of view must be included and excluded. But if we already know that, why should we  not proceed directly to  the  merits?  If we  already  know that,  before deliberation occurs, does deliberation have any point at all? [pp 193]

  • It’s not that heterogeneity needs to be limited per se. There does need to be a mechanism that provides sufficient visibility across individuals and groups so that as a whole, society stares reasonably centered. The model shows that flocking can occur across arbitrarily high dimensions, but that the information distance increases as a function of the number of dimensions. Computer-Mediated communication might be able to address this issue by projecting high-dimensional sets of concepts and projecting them into spaces (e.g. self-organizing maps) that can be navigated by individuals and groups of human users. The goal is to recognize and encourage particular types of flocking behaviors while providing enough credible visibility to counter information so that this interaction of flocks of flocks stays within the bounds that support a healthy society.