Thursday, January 24, 2013

We're not the media if we depend on Google (or Amazon)

Update: 25 January 2013. This morning when I typed Catalonia into the Google Search box, and then clicked the News tab, all of the stories about the Declaration of Sovereignty were hidden behind an even smaller "+ Show more" link below the generic AP post carried by the Huffington Post. If I click on "All 88 news sources" I get this complete fail:

Google can't find Catalonia

Maybe there is a conspiracy!

I've also changed the headline of this article from "We're not the media if we don't appear on Google (or Amazon)" to "We're not the media if we depend on Google (or Amazon)". We need an alternative.

Yesterday there was a news story I was particularly interested in. But searching for it on the internets made me rethink a lot of assumptions about the idea that "we are the media". I first heard that expression used by Amanda Palmer, a singer/songwriter/artist who ran an incredibly successful crowdsourcing campaign a few months ago—raising over $1 million—to fund a new record. One of her most intoxicating claims was that she didn't need to sign with a studio, and she didn't need to follow a studio's rules and directives. With the support of individual people—me included—she could put out her record and go on tour. In short, she and "we are the media".

I've been writing books for the last 20 or so years with the same idea in mind. My HTML books helped people realize that they could publish their ideas, their writings, their photographs, their lives without having to ask for anyone's permission. At first, I published a gallery of these brand new websites, and I loved going through them and seeing democracy in action: all these people sharing their thoughts with the world, with no gatekeepers to stand in the way.

The gallery quick grew to multiple pages of lists, and the web to billions, and so I left the job of cataloging these new creations to the brand new Google. I'll get to that in a moment.

A few years ago, when I discovered EPUB, I had the same feeling of democratic excitement: here was a technology that could help people publish and otherwise share their writings. We humans yearn to share our stories with others. EPUB, an open format, facilitates such an interchange.

But yesterday, it really hit home how far we are from complete democracy of content. And that's because though we may have the technology and even free tools for creating webpages and ebooks, there are massive economic and political powers that stand between us and our readers. We cannot ignore them. Let me show you yesterday's example.

First, let me admit that the word I typed into Google was "Catalonia". But the ramifications go way beyond this particular news item. So, please bear with me as I use my favorite example to illustrate something much larger.

When I type in "Catalonia", Google says there are 83 news items. When I ask Google to show me "all 83 news sources", it gives me a list of only 16. You can try this at home.

Down at the bottom of the page, Google has a little disclaimer that says it omits some entries that are "very similar to those already displayed". And this is probably not the right time to discuss wire services and the sad fact that a single mediocre story with no byline about a small, new far away country might be reproduced verbatim, with no fact chacking, in 80 or so American newspapers. But I digress. But if you click "repeat the search with omitted results included" Google still only displays 25 items. There is no way to see all 83.

Google's "Full Coverage"

I would very much like to know which other 58 news sources are reporting on my topic. Why won't Google show me? And how does it decide which 25 items to show? And isn't it a problem that I am necessarily limited by Google's criteria?

Do I think there's a conspiracy to keep the other 58 news sources from being revealed? Probably not, but regardless, it bothers me quite a bit that there's no way to get to them. Like it or not, we are not the media, if we cannot be Googled.

And it doesn't stop there. Amazon recently decided that authors of the same genre books cannot review other authors' books. Amazon became successful because of its review system. I am not the only person who used to look at Amazon reviews in order to decide which book to find from the library. Amazon's reviews were an amazing crowdsourcing adventure, now run amok by enterprising and back-stabbing authors on the one hand, and Amazon wanting to watch the gates, on the other.

And don't even get me started on Amazon search. How much publisher money is behind Amazon's search results? I have no idea, but I'm guessing it's not insignificant. We are not the media if we don't come up on Amazon.

The Amazon review issue brings up yet another aspect of this whole story, which I touched on in an essay titled "If you know it already, don't tell". The basic gist of that article is that we believe people more when we don't think they have an axe to grind, even if, ironically, as in the case I describe, the person isn't particularly knowledgeable at all about the topic.

This is the same conundrum that Amazon's new policy throws at us, and that Porter Anderson confronted in his recent post in On the Ether. Where is the line between enough knowledge about a subject in order to give a reasoned opinion and so much knowledge about the subject that your opinion is biased? In this day and age when every one's past opinions are documented publicly, can there even be unbiased opinions? I think not. I believe that the important thing is to know what the opinion is, so we can use that information to assess the validity of the opinion. Suppressing all the opinions that Amazon has decided fall into a particular category of user solves nothing.

In these heady times in which self-publishers, and the web designers that came before them, are gleeful with democratic information sharing, we must be absolutely aware that the gatekeepers have simply moved the gates. Whereas in earlier eras we couldn't even publish without their help, now we can publish, only to find our sites and books hidden in a sea of information. Even if your book or website is in there, if Google or Amazon or whatever search product of the moment won't bring it up, it's as if it weren't there.

All is not lost. Twitter, and to some degree Facebook as well, are still amazingly democratic, allowing us to connect directly with kindred spirits all over the world. While we can't forget that social media still generates only a fraction of web traffic (on the order of 5-6%), I think its influence goes far beyond those numbers, if only in its ability to connect people and movements. But will they last? And Kickstarter (and for example, the Catalan crowdsourcing site Verkami) are great at helping us harness the power of hundreds (or thousands if you're Amanda Palmer) to offer alternatives to these gatekeepers or, when need be, attract their attention. Technology-wise, the use of accurate metadata can also help make a book or website "findable".

Ironically, it is the very birth of Google that gives me hope. Way back then, Altavista was the most popular search engine, and I remember being increasingly frustrated with its ads, its promoted links, and the seemingly deliberate obstacles that it placed between me and what I was looking for. This clean, lithe little ad-free Google came along and all of a sudden we could find things again. It felt like the whole world changed over to this little search engine that actually worked in a matter of weeks. Perhaps it's time for some new disruption.

And while we wait (or develop) the tools to maintain our democratic web, we must be careful to be aware that right now there is no unbiased opinion, and no unbiased search.


  1. Regarding Amazon and the reviews posted there, this chilling article from the New York Times, "Swarming a Book Online," shows another way that Amazon reviews can be used to discredit an author's work and essentially make it invisible to or unavailable to those who may want to read it.

  2. With regards to searching things up on Google and Amazon, I think there's a lot to be said about interfaces also - the ways we interact with the web pages in question.

    The layouts that are currently used for searches on these two websites cannot possibly be democratic because they are in list format. Using your example, even if Google did show all the search results on one page, due to their list format the person searching will see some results before others. And people are more likely to look at the first links that appear rather than scroll right down to the bottom of the list.

    So then you ask, why should some search results be listed above others? As services, Google and Amazon want as many of their users to be happy with the service they receive as possible. Therefore they list the most 'popular' search results first, i.e. the results that the majority of people are most likely to be after. If they randomized the list order instead of using popularity, people would get frustrated with the service they've received.

    There's also the physical limitation of a computer screen - okay, Google could list all possible results on the same page and the user could scroll down...but this is awkward and it would also take much more time to load.

    Perhaps a new form of interface is required that presents all search results equally, but also allows the user to find what they want as fast as possible. Or, if there is not something specific they are looking for, gives them an equal chance of finding something from any of the results of the search.

  3. The up-side to this trend is that every time a service limits info or gate-keeps for profits, it is usually vulnerable to a competitor who's gates are more wide open.
    I remember how excited I was about rich-media epub, only to realize that B&N Nook won't allow independent titles to have embedded media. Now they are struggling because of it (and other factors).
    I am about to upload my 7th rich-media title to Itunes, but it is a sad fact that marketing and publicity is such a hurdle. My sales are minimal and it is because it is hard to get people to even know these rich-media books exist...
    A great outlet would be some sort of collective marketing platform for indy, ebook publishers (other than smashwords), one for truly independent content creators who don't go through smashwords. Is there something like this that I don't already know about?

  4. Amazon does (or at least did a few years ago) alter its search result to maximize Amazon's potential income. One of Amazon's lawyers admitted precisely that to me in a phone conversation.

    Her only defense was that, even though an inexpensive version of a book might be invisible in the search results, it could be reached in some convoluted fashion by following by following one of the links and then clicking on a link there. In my case, potential buyers could get to my Across Asia on a Bicycle in paperback by clicking on the more expensive hardback, which was displayed, and then find a link to the paperback there.

    And when I brought that travesty up with an Amazon programmer I'd met, his terse advice was to "never trust Amazon search results." Too many games are being played. For books, the best way to search is by ISBN. For other items, you can try the manufacturers stock number.

    That problem seems to come and go. At times, it has been so bad with some of my titles that I've thought of raising the price of hardback editions into the $50 range just to drive them to the top of search lists.

    Ironically, at that time, while Amazon wasn't even listing my edition of Across Asia, on Google my version on Amazon was the top hit for the term.

    I gave up on Google's news service years ago when there was a 'runaway bride' story going on in Atlanta. I didn't care to follow it myself, but I was disgusted by the fact that Google was giving prominent display to the official Chinese government news stories on the topic and ignoring Atlanta papers.

    In fact, I noticed that Google was obviously giving that propaganda arm of the Chinese dictatorship a higher ranking than almost any other source. That was in the era when Google was pandering to China to open doors there.

    That's when I wrote off Google's "integrity." I still use them for politically neutral searches. But I don't trust them and inch beyond that.

    In the case of Catalonia, I suspect those high up in Google are 'internationalists' in the nasty sense that they sneer at national identities, particularly emerging ones. That'd also explain why for years their decorated search pages puffed trivial stuff while ignoring important national dates such as the Fourth of July.

    One suggestion. If you got some ideas about what news sources might be getting blocked from display, try using Google's advanced search with their web address. As with Amazon, there are sometimes ways to work around a wall.

  5. It's funny. I live in Brazil and I search for Catalonia on google it shows a total of 40,600,000 results.

    1. Try clicking the News tab so that you can see the sites that Google considers "news" organizations, generally newspapers and TV stations.

  6. I wonder if an initiative like Common Crawl's ( can help against this remiss Google attitude in searches.

    But only, really, a posteriori, I suppose?

    anyway, I've thought it most interesting.


  7. I live in Argentina and get 33,900,000 results for Catalonia.
    If I search for Catalonia Press I get 5,310,000 results and the first result is
    Not strange at all.
    Google takes into account (among other things) where you live (your IP tells them where you are) and your previous searching habits.

    1. Try clicking the News tab so that you can see the sites that Google considers "news" organizations, generally newspapers and TV stations.

      Also, I've tried looking at different Googles (e.g., and get the same results.

  8. Sorry to burst your bubble, but Twitter is also a gatekeeper. I have a friend with a relatively unpopular Twitter account, and if I search with my account for a post from my friend using very specific keywords, it does not show up in results. It's as if my friend's account simply doesn't exist.

    1. It's true I'm awfully enthusiastic about Twitter these days, there are certainly plenty of signs that its democratic days are waning, including your example. Thanks for sharing.

  9. Search engines are our eyes and ears onto everything beyond our personal physical space. Radio, newspapers, even books!, hardly count anymore. Nobody says, "Gee, I don't know. I'll Encyclopedia Brittanica that."

    We've handed that vast power to private companies with no obligation to tell us where they've set up gates, how they're using them, what they get out of it, or how they decide what we can see.

    If there was ever a case for a public utility, run transparently, with accountability, and with user control of the gates, search engines are it.

    It boggles my mind that so many people have so happily handed both perception and understanding over to for-profit companies to monetize however they like.

  10. Thanks for the post. It IS frustrating, but I wonder how much of your frustration is the result of your filter bubble. Have you followed Eli Pariser's work on this, by chance? For example, is the duckduckgo search engine available to you? I'd be interested to know what the comparison of the same searches across different engines results in.

    And this?
    "If there was ever a case for a public utility, run transparently, with accountability, and with user control of the gates, search engines are it."


  11. Could explain also why YouTube is always going through interface changes. Which in my opinion have been getting progressively more restricting in nature.

  12. If you use a Mac, get DEVONagent Pro from DEVONthink. You get all 83. That's just with their fast web search. You can select their deep search option, leave it on all night and get every mention in every database in the world. And it doesn't track you.

    Google's first Panda algorithm change in Feb 2011 screwed everyone, IMO.

  13. I’m not sure what’s up with Liz and all the rest of you, but, when I searched Google News today for “Catalonia” I got “About 5,670 results” on the first of many, many pages. They were from all kinds of news sources from all around the world.

  14. Peter, it's not just searching on Catalonia, (or it may be, who knows if those 5,670 results are actually there), it's clicking the News tab so you get the **newspaper** articles. Then you'll see one or more stories, each with maybe three summaries. And a little "+ all n news sources", which was 83 when I wrote this article, but today, the top news item only has 8 sources. Nonetheless when I click to expand, it says there are NO sources of news articles about Catalonia. So there's no way to see the other stories that supposedly exist, but are not summarized on that front page.


More of my books