Closer To The Ideal

Clay Shirky wrote a stupid post suggesting women did not know how to promote themselves. Some of the reactions in the comments are very good:

Annalee says:

Obviously women need to speak up more. And men need to shut up more, especially when women are talking.

I’ve definitely taken the “act like a braggart” approach to my career, and honestly it’s not just a matter of speaking up. It’s a matter of speaking up over and over again, even when somebody is interrupting you or telling you that you have no right to speak in the first place.

I wish the problem were just training women to raise their hands more often, but of course it isn’t. It’s training men to notice when the women are raising their hands. And training the men NOT to raise their hands when they clearly do not have the answers.

This is also good:

Patty Zevallos says:

I get real tired of hearing other people telling me what is going on inside my head because I am female. I have no problem promoting myself. I have no problem jumping into something a bit tricky to get ahead. I have been highly successful at doing the work I want to do in media production (see http://www.pbzproductions.com/resume/) for 31 solid years, and don’t have any problem explaining how good I am in my field. I currently am getting calls and emails about high-end web design jobs because you, Clay Shirky, have absolutely no idea what you are talking about.

The problem is that you are so completely out of touch with what is going on. You have no idea that the Green Living movement (www.pbzproductions.com/green/) is up and running and initiated mostly by women, that women are getting organized and helping each other, that women-owned businesses are growing and thriving.

Perhaps you should actually talk to people instead of ranting.

This is also good:

kim sbarcea says:

Frankly, I’m amazed that this “rant” even occurs in 2010. Having said this, I think you are barking up the wrong tree. The fact that organisations or senior management might actually allow men to behave like arrogant self-aggrandizing jerks (to use your words) is the problem. Not whether women can or cannot (or even should) act in the same manner.

Maybe it’s just Australia, but I have worked with many women who can be described as self-aggrandizing, prone to extreme exaggeration, arrogant, nasty, back-stabbing, overly self-confident and so on.

This is also good:

T.T. says:

You know that calling a badly argued essay a “rant” doesn’t make it less badly argued, right?

“…not enough women have what it takes to behave like arrogant self-aggrandizing jerks.”

This statement is a special case of “women are Z”. Women are Z. They just are. You know that. You can make categorical claims because your experience is universal. If only we could all be so special.

This last one hits the angle that I find most surprising, that Shirky would sink to the level of making categorical, universal claims about women. In 2010? Is this some kind joke? Surely it suggests mental laziness if anyone who starts a sentence with “All black people are…” or “All women are…”

I wrote my own thoughts about the essay in the post called The most disappointing essay Clay Shirky has ever written.

I usually like Clay Shirky a lot, though his most recent piece seems poorly reasoned. Shirky suggests that women are not aggressive enough in negotiations, but then he mentions how this hurts them in 2-sided markets, and he lists colleges as one such market. Did he miss the fact that the majority of all college graduates are now women? Nearly half of medical school students nationwide are now female. Young women are out-earning men in some areas and professions:

Women’s pay relative to men’s rose rapidly from 1980 to 1990 (from 60.2% to 71.6%), and less rapidly from 1990 to 2004 (from 71.6% to 76.5%), though young women have started to outearn young men in some large urban centers with young women earning up to 20% more than their male counterparts.

But women with children have less negotiating power:

However, other trends are decidedly negative: a study at Cornell University concluded in 2005 found that women with children were less likely to be hired and if hired would be paid a lower salary than male applicants

These 2 facts suggest that raising kids limits women’s negotiating power. Women don’t have an innate lack of negotiating skills, but the circumstances of raising children imposes some hard constraints, that fall disproportionately on women. Shirky is ranting about the wrong issue.

This is what Shirky says:

This worry isn’t about psychology; I’m not concerned that women don’t engage in enough building of self-confidence or self-esteem. I’m worried about something much simpler: not enough women have what it takes to behave like arrogant self-aggrandizing jerks.

Remember David Hampton, the con artist immortalized in “Six Degrees of Separation”, who pretended he was Sydney Poitier’s son? He lied his way into restaurants and clubs, managed to borrow money, and crashed in celebrity guest rooms. He didn’t miss the fact that he was taking a risk, or that he might suffer. He just didn’t care.

It’s not that women will be better off being con artists; a lot of con artists aren’t better off being con artists either. It’s just that until women have role models who are willing to risk incarceration to get ahead, they’ll miss out on channelling smaller amounts of self-promoting con artistry to get what they want, and if they can’t do that, they’ll get less of what they want than they want.

…And it looks to me like women in general, and the women whose educations I am responsible for in particular, are often lousy at those kinds of behaviors, even when the situation calls for it. They aren’t just bad at behaving like arrogant self-aggrandizing jerks. They are bad at behaving like self-promoting narcissists, anti-social obsessives, or pompous blowhards, even a little bit, even temporarily, even when it would be in their best interests to do so. Whatever bad things you can say about those behaviors, you can’t say they are underrepresented among people who have changed the world.

Again, it is tough to reconcile the career success that young women are having with Shirky’s narrative of “They are bad at behaving like self-promoting narcissists”.

This is the dumbest thing I’ve ever read in any of Shirky’s essays:

Now this is asking women to behave more like men, but so what? We ask people to cross gender lines all the time. We’re in the middle of a generations-long project to encourage men to be better listeners and more sensitive partners, to take more account of others’ feelings and to let out our own feelings more.

Maybe “we ask people to cross gender lines all the time” but usually the goal is to make the world a better place. For instance, “encourage men to be better listeners and more sensitive partners” probably makes the world a better place, whereas encouraging women to be “anti-social obsessives, or pompous blowhards” does not. It is really disappointing to be reading stuff like this in an essay by Shirky.

However, even in an ideal future, self-promotion will be a skill that produces disproportionate rewards, and if skill at self-promotion remains disproportionately male, those rewards will as well. This isn’t because of oppression, it’s because of freedom.

If professional women in their 20s continue to pass by professional men in their 20s, then clearly women know how to promote themselves. If women with children continue to be handicapped in their careers, then we are dealing with oppression, not freedom. At the very least, we are dealing with work practices and family practices that are in need of innovation.

In these circumstances, people who don’t raise their hands don’t get called on, and people who raise their hands timidly get called on less.

…It’s tempting to imagine that women could be forceful and self-confident without being arrogant or jerky, but that’s a false hope, because it’s other people who get to decide when they think you’re a jerk, and trying to stay under that threshold means giving those people veto power over your actions.

Surely someone hacked into Shirky’s blog and is trying to discredit him by publishing an idiotic essay? Where is the nuance and subtlety of thought that gave us such classics as Power Laws, Weblogs, and Inequality? I feel like I’m reading a fundamentalist version of Shirky, a stripped down version of Shirky, lacking any of the careful qualifiers that grace his previous work.

My main problem with his essay is the broad-brush nature of his description of the problem. It is pointless to talk as if women are doing poorly in every profession, when nearly 50% of new doctors are female.

Since Shirky works in and around the tech industry, I suspect that a lot of his remarks were aimed at the tech industry. Had he explicitly said “What I’m saying here only applies to the tech industry” then he would have been on slightly safer ground. There are certainly some odd gender imbalances in the tech industry, which I just wrote about in my last post.

I feel like I know where Shirky is coming from. I’ve come close to writing a similar essay. I have seen talented women sabotage their own careers. For some reason, this happens more in the tech industry than anywhere else. I have felt a frustration similar to the one that I think Shirky was trying to express.

The title of this blog post is “The most disappointing essay Clay Shirky has ever written”. Why am I disappointed? Mostly because I have come to expect a great deal of emotional honesty from Shirky. In posts such as “The Failure of #amazonfail” he does a rare thing: he admits that he made mistakes of judgement due to being caught up in the emotion of the moment, and he also talks himself back to sanity, all the while being candid about the emotions he is experiencing. That kind of emotional transparency is missing from “A rant about women.” I understand the frustration of seeing talented women sabotage their own careers. On the one hand, I know it is damn tempting to rant about that frustration. On the other hand, I think it is important that I, and Shirky, keep ourselves from expressing that frustration in untruthful ways.

Shirky has been a supporter of many politically progressive causes. So have I. Most progressive activists occassionally suffer some moment of burn out, during which time, they may say some damn reactionary things. They lean on their reputation at such times. I’ve done it. Shirky is doing it here. His essay amounts to “I will say some unqualified, harsh things here, to vent my frustration, and surely people will cut me some slack, because my previous progressive efforts have surely earned me some good will.” There is some truth in that – no one will change their opinion of Shirky simply because he wrote one bad essay. But we will all feel disappointed.

I’ve spent the last 10 years working as a computer programmer, but now that Darren Hoyt and I are trying to launch WP Questions I find myself reading a lot more about marketing.

I very much like this bit in Fast Company, where Duncan Watts argues that which songs emerge as hits is a largely random process:

Watts wanted to find out whether the success of a hot trend was reproducible. For example, we know that Madonna became a breakout star in 1983. But if you rewound the world back to 1982, would Madonna break out again? To find out, Watts built a world populated with real live music fans picking real music, then hit rewind, over and over again. Working with two colleagues, Watts designed an online music-downloading service. They filled it with 48 songs by new, unknown, and unsigned bands. Then they recruited roughly 14,000 people to log in. Some were asked to rank the songs based on their own personal preference, without regard to what other people thought. They were picking songs purely on each song’s merit. But the other participants were put into eight groups that had “social influence”: Each could see how other members of the group were ranking the songs.

Watts predicted that word of mouth would take over. And sure enough, that’s what happened. In the merit group, the songs were ranked mostly equitably, with a small handful of songs drifting slightly lower or higher in popularity. But in the social worlds, as participants reacted to one another’s opinions, huge waves took shape. A small, elite bunch of songs became enormously popular, rising above the pack, while another cluster fell into relative obscurity.

But here’s the thing: In each of the eight social worlds, the top songs–and the bottom ones–were completely different. For example, the song “Lockdown,” by 52metro, was the No. 1 song in one world, yet finished 40 out of 48 in another. Nor did there seem to be any compelling correlation between merit and success. In fact, Watts explains, only about half of a song’s success seemed to be due to merit. “In general, the ‘best’ songs never do very badly, and the ‘worst’ songs never do extremely well, but almost any other result is possible,” he says. Why? Because the first band to snag a few thumbs-ups in the social world tended overwhelmingly to get many more. Yet who received those crucial first votes seemed to be mostly a matter of luck.

Word of mouth and social contagion made big hits bigger. But they also made success more unpredictable. (And it’s worth noting, no one in the social worlds had any more influence than anyone else.) So yes, Watts figures, if you rewound the world to 1982, Madonna would likely remain a total unknown–and someone else would have slipped into her steel-tipped corset. “You cannot predict in advance whether a band gets this huge cascade of popularity, because the social network is liable to throw up almost any result,” he marvels.

Predictably, the music industry received the analysis–”Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market,” published in Science in 2006–with a cocked eyebrow. When Watts presented his findings to executives at a major record label last spring, the younger among them were reasonably receptive. They’re accustomed to the unpredictability of hit-making online, so they can grasp the terrifying randomness of success.

But the older execs?

Watts laughs. “They were all like, ‘I think it’s bullshit. I’m still going to go with my gut,’” he recalls. “And I’m like, Okay, good luck to you. You’re going to need it.”

He is going over ground that Clay Shirky examined in 2003, in such essays as “The FCC, Weblogs, and Inequality“:

Yesterday, the FCC adjusted the restrictions on media ownership, allowing newspapers to own TV stations, and raising the ownership limitations on broadcast TV networks by 10%, to 45% from 35%. It’s not clear whether the effects of the ruling will be catastrophic or relatively unimportant, and there are smart people on both sides of that question. It is also unclear what effect the internet had on the FCC’s ruling, or what role it will play now.

What is clear, however, is a lesson from the weblog world: inequality is a natural component of media. For people arguing about an ideal media landscape, the tradeoffs are clear: Diverse. Free. Equal. Pick two.

He talked about the issue at even greater length in Power Laws, Weblogs, and Inequality:

Freedom of Choice Makes Stars Inevitable #

To see how freedom of choice could create such unequal distributions, consider a hypothetical population of a thousand people, each picking their 10 favorite blogs. One way to model such a system is simply to assume that each person has an equal chance of liking each blog. This distribution would be basically flat – most blogs will have the same number of people listing it as a favorite. A few blogs will be more popular than average and a few less, of course, but that will be statistical noise. The bulk of the blogs will be of average popularity, and the highs and lows will not be too far different from this average. In this model, neither the quality of the writing nor other people’s choices have any effect; there are no shared tastes, no preferred genres, no effects from marketing or recommendations from friends.

But people’s choices do affect one another. If we assume that any blog chosen by one user is more likely, by even a fractional amount, to be chosen by another user, the system changes dramatically. Alice, the first user, chooses her blogs unaffected by anyone else, but Bob has a slightly higher chance of liking Alice’s blogs than the others. When Bob is done, any blog that both he and Alice like has a higher chance of being picked by Carmen, and so on, with a small number of blogs becoming increasingly likely to be chosen in the future because they were chosen in the past.

Think of this positive feedback as a preference premium. The system assumes that later users come into an environment shaped by earlier users; the thousand-and-first user will not be selecting blogs at random, but will rather be affected, even if unconsciously, by the preference premiums built up in the system previously.

Note that this model is absolutely mute as to why one blog might be preferred over another. Perhaps some writing is simply better than average (a preference for quality), perhaps people want the recommendations of others (a preference for marketing), perhaps there is value in reading the same blogs as your friends (a preference for “solidarity goods”, things best enjoyed by a group). It could be all three, or some other effect entirely, and it could be different for different readers and different writers. What matters is that any tendency towards agreement in diverse and free systems, however small and for whatever reason, can create power law distributions.

Because it arises naturally, changing this distribution would mean forcing hundreds of thousands of bloggers to link to certain blogs and to de-link others, which would require both global oversight and the application of force. Reversing the star system would mean destroying the village in order to save it.

Clay Shirky is writing about algorithmic authority.

Algorithmic authority is the decision to regard as authoritative an unmanaged process of extracting value from diverse, untrustworthy sources, without any human standing beside the result saying “Trust this because you trust me.” This model of authority differs from personal or institutional authority, and has, I think, three critical characteristics.

First, it takes in material from multiple sources, which sources themselves are not universally vetted for their trustworthiness, and it combines those sources in a way that doesn’t rely on any human manager to sign off on the results before they are published. This is how Google’s PageRank algorithm works, it’s how Twitscoop’s zeitgeist measurement works, it’s how Wikipedia’s post hoc peer review works. At this point, its just an information tool.

Second, it produces good results, and as a consequence people come to trust it. At this point, it’s become a valuable information tool, but not yet anything more.

The third characteristic is when people become aware not just of their own trust but of the trust of others: “I use Wikipedia all the time, and other members of my group do as well.” Once everyone in the group has this realization, checking Wikipedia is tantamount to answering the kinds of questions Wikipedia purports to answer, for that group. This is the transition to algorithmic authority.

As the philosopher John Searle describes social facts, they rely on the formulation X counts as Y in C — in this case, Wikipedia comes to count as an acceptable source of answers for a particular group.

There’s a spectrum of authority from “Good enough to settle a bar bet” to “Evidence to include in a dissertation defense”, and most uses of algorithmic authority right now cluster around the inebriated end of that spectrum, but the important thing is that it is a spectrum, that algorithmic authority is on it, and that current forces seem set to push it further up the spectrum to an increasing number and variety of groups that regard these kinds of sources as authoritative.

There are people horrified by this prospect, but the criticism that Wikipedia, say, is not an “authoritative source” is an attempt to end the debate by hiding the fact that authority is a social agreement, not a culturally independent fact. Authority is as a authority does.

Odd. Here is a blog post, nominally written in 2005, that uses the phrase “irrational exuberance” and links to a page that explains that the phrase “irrational exuberance” comes from a speech in December of 2006. I wonder how Gene Smith, in April of 2005, knew that Allan Greenspan would coin this phrase 20 months later?

Turning a blind eye to those problems is what turns strange zeitgeist into irrational exuberance.

Posted by Gene Smith on Apr 20, 2005

Whimsley points out that the Internet (and recommendation systems, in particular) expand choices for individuals, but potentially lower the overall range of things that are likely (emphasis on the word “likely”) to be experienced. The focus is especially on recommendation algorithms, such as the one on Amazon.com:

You can see that [in the graph above] on the left, in Internet World, a few products were chosen a lot, especially the one centred on about (-0.2, -0.2). In Offline World there are many more medium-sized dots, showing that the consumption of products is more equal. In Internet World one product has “gone viral” and gets chosen over 1500 times out of the total of 3600, while 26 products languish in the obscurity of being sampled fewer than ten times. In Offline World no single product is chosen more than 10% of the time, and only 14 products are sampled fewer than ten times. In short, niche products do better in Offline World than in Internet World.

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

The point here is similar to the point Clay Shirky made back in 2003:

A persistent theme among people writing about the social aspects of weblogging is to note (and usually lament) the rise of an A-list, a small set of webloggers who account for a majority of the traffic in the weblog world. This complaint follows a common pattern we’ve seen with MUDs, BBSes, and online communities like Echo and the WELL. A new social system starts, and seems delightfully free of the elitism and cliquishness of the existing systems. Then, as the new system grows, problems of scale set in. Not everyone can participate in every conversation. Not everyone gets to be heard. Some core group seems more connected than the rest of us, and so on.

Prior to recent theoretical work on social networks, the usual explanations invoked individual behaviors: some members of the community had sold out, the spirit of the early days was being diluted by the newcomers, et cetera. We now know that these explanations are wrong, or at least beside the point. What matters is this: Diversity plus freedom of choice creates inequality, and the greater the diversity, the more extreme the inequality.

Shirky is writing about weblogs, but what he says applies to books, CDs, videos – anything where people can influence the choices that other people make. Shirky’s point applies to the point that Whimsley is making about recommendation engines. This part applies equally to webloggers or videos:

We also know that as the number of options rise, the curve becomes more extreme. This is a counter-intuitive finding – most of us would expect a rising number of choices to flatten the curve, but in fact, increasing the size of the system increases the gap between the #1 spot and the median spot.

Shirky offered a hypothetical example of how this works:

To see how freedom of choice could create such unequal distributions, consider a hypothetical population of a thousand people, each picking their 10 favorite blogs. One way to model such a system is simply to assume that each person has an equal chance of liking each blog. This distribution would be basically flat – most blogs will have the same number of people listing it as a favorite. A few blogs will be more popular than average and a few less, of course, but that will be statistical noise. The bulk of the blogs will be of average popularity, and the highs and lows will not be too far different from this average. In this model, neither the quality of the writing nor other people’s choices have any effect; there are no shared tastes, no preferred genres, no effects from marketing or recommendations from friends.

But people’s choices do affect one another. If we assume that any blog chosen by one user is more likely, by even a fractional amount, to be chosen by another user, the system changes dramatically. Alice, the first user, chooses her blogs unaffected by anyone else, but Bob has a slightly higher chance of liking Alice’s blogs than the others. When Bob is done, any blog that both he and Alice like has a higher chance of being picked by Carmen, and so on, with a small number of blogs becoming increasingly likely to be chosen in the future because they were chosen in the past.

Think of this positive feedback as a preference premium. The system assumes that later users come into an environment shaped by earlier users; the thousand-and-first user will not be selecting blogs at random, but will rather be affected, even if unconsciously, by the preference premiums built up in the system previously.