The Birth Of The NewsMaster: The Network Starts To Organize Itself
NewsMasters: RSS And The Opportunity For Sustainable Filtering And Aggregation Of Online Content Into Niche Websites And Dedicated Information Channels
An independent publisher opportunity and a socially valuable way of filtering and organizing news information and the Web at large.
- Introduction
- What Is The Discovery?
- Where Is The Real Value?
- Summary: Critical Issues/ Problems At Hand
- The Filtering Mechanisms Available To Us
- A Learning Voyage Into RSS
- Possible Solutions: RSS Aggregators?
- How Do I Want To Find Information Relevant To Me
Emerging Technologies - A New Digital Professional Role: The NewsMaster
- Are Newsmasters Sustainable?
- Conclusions
- Special Note From the Author
- Appendix
- Would be NewsMasters Here Is Your Toolkit
Introduction
Yesterday, at the latest DEMO 2004 Conference in Scottsdale, Arizona, Feedster
“Announced a new tool called FeedPaper, which combines syndicated Internet content (culled from the 500,000 newsfeeds continuously searched by Feedster), such as The New York Times, Washington Post, USA Today, Amazon.com, many IT publications, PRNewswire, and hundreds of thousands of blogs.
A Feedpaper is a web-based and RSS-enabled micropublication on any topic of its creator's choosing. Creating a Feedpaper enables its publisher to blend, track, and share information on any topic.
Online publishers and e-commerce sites no longer need to see search engines simply as sources of traffic and advertising networks.
For the first time, publishers can use a search engine to filter and deliver relevant content to their readers within their own pages, including customized topic search and syndication’’
Scott Rafer, Feedster’s CEO, said (https://web.archive.org/web/20060704230507/http://www.internetnews.com/dev-news/article.php/3313991)
Indeed, the key problem we are confronted with is this: how to navigate, filter, and reduce to humanly manageable dimensions the gigantic flow of information coming at us in increasing amounts.
The entry of the RSS publishing and distribution format into the fray of media and technologies available for communication has only added to the tidal wave of news and information that has gradually been submerging us.
The proliferation of many alternative online independent news sources and the epidemic expansion of the blogger community have created a sharp increase in information channels available to us outside the mainstream, established sources.
Further, the Web keeps growing at a steady pace, and the amount of content being made available on a daily basis is simply too much to be managed by our every day Google and RSS aggregators.
We need something of an entirely new order to manage this.
An army of newsmasters.
(And a special Toolkit to start working on it right now.)
What Is The Discovery?
The discovery is the unlimited and yet untapped power we now have to search, filter, aggregate, and create focused news/information channels with the only support of our know-how, culture, experience, and a little unknown-free technology called RSS.
Where Is The Real Value?
The real value is in the fact that the effective widespread application of this filtering and redistribution process would be a highly valuable social endeavor and one that not only can enrich our rapidly evolving global culture but can also give sustainability to those who have the will and skill to ride it.
Summary: Critical Issues/ Problems At Hand
· Too much information coming in
· Too many sources to scan
· General categorization of sources is too limited. By author or publishing house is too limiting. I can’t scan 100 authors a day to see if there is something interesting for me to read or write about
· Too many new sources coming online daily. Which ones do I pick?
· Difficulty in separating relevant news from press releases, personal rants, and job vacancy announcements
· Dubious quality of many news sources
· Our search for relevant content and news is an individual activity, in general, with no direct benefit for the community at large
· Searching for information is mostly carried out as a separate, un-contextual, just-in-time quick personal search.
The Filtering Mechanisms Available To Us
As Stephen Downes mentions in his beautiful post about networks in his last issue of OL Daily, the network of information needs to organize itself automatically, and we must help it do so.
We must learn to perceive and use the internet in exactly the same way as a self-organizing network.
Up until now, we have organized and searched for information mostly by:
a) Typical major search engines (Google, Alltheweb, Teoma, Wisenut, etc.)
b) Directories (ODP, Yahoo, etc.)
c) Metasearch tools (Copernic Meta, Vivissimo, Surfwax, Ixquick)
d) Data Visualization tools (Grokker, Kartoo, TouchGraph, Musicplasma)
We have also built internal directories inside our companies and organizations, databases, and knowledgebases of all kinds. But of these last ones, most of them remain like isolated islands of which few search tools can report the size, content, and updated map.
More recently, the army of individual reporters, writers, journalists, academics, and researchers who have adopted independent publishing systems like CMSs, weblogs, and other database-based content management solutions has been proactively weaving, organizing, and filtering, at an initial level of refinement the news flow and content coming at us daily.
Bloggers, foremost, have taken a natural role in filtering and aggregating mechanisms of the newsflows running through the Internet.
Bloggers have been the first to scout new independent sources and alternative voices and have pointed links to new ways of looking at any issue.
They, more than any other group, have brought, among much useless noise, the true emergence of effective meta-news sources that originate, filter, and aggregate valuable content online.
So, what appears to the many superficial onlookers as a universe of mindless writings (blogs) is nothing less than the initial phase of a complex and orderly process whereby humanity at large takes control of filtering, gathering, and re-organizing his own know-how and discoveries.
The world has become a global village, and as such, it has started taking care of itself in a global, dispersed, self-sufficient, synchronized fashion.
A Google query is an individualistic just-in-time search with no benefit for society.
A Google query is a one-time shot.
You type the query, you get the results, and it is over.
You probably don’t re-use that search anymore. You will retype it someday when you will need it again.
Nobody else outside of you benefits from that search.
Again, it is a one-time shot.
A Learning Voyage Into RSS
To better understand what this new RSS-based universe really entails and may likely involve, I have dived, soul and body, into it to understand in first-person how much the new information publishing and distribution protocol could be effectively revolutionizing our immediate future.
The process has taken quite a few months, but let me tell you that the view derived from this open training has been pretty amazing, to say the least.
In short order and for the skeptics out there, here is what I have done to “understand” RSS and to develop a good feel for where it is headed:
I have converted all of my online sites into content management systems that would be able to generate RSS feeds. I have organized content into categories and created individual RSS feeds for each one of them.
I have started using RSS newsreaders and aggregators and have reported extensively about my findings.
I have tested tools to convert RSS feeds into email and to convert emails into RSS feeds.
I have started analyzing sites and services syndicating other RSS feed content.
I have extensively delved into using all of the RSS monitoring tools like Technorati, DayPop, Blogdex, Popdex, Feedster, Blogstreet, and many others.
Inspired by the work of Ari Paparo https://web.archive.org/web/20070306234603/http://www.aripaparo.com/archive/000632.html and by the growing number of valuable directories and search engines aggregating and indexing valuable RSS-based content I went on to preparing the publicly accessible RSSTop55, a best and most recommended list of top RSS-based submission directories.
Early in this discovery voyage, I got interested and fascinated by tools like MyRSS which would allow you to theoretically create an RSS feed out of anyone's website page updating its content out there.
A love for “scraping” has followed right after. Scraping is the art of deciphering the underlying content structure of a Web page content in order to be able to create a framework or container that can “extract” the content of that site at will in an organized fashion.
Scraping a Web site allows anyone to create an RSS feed for any Web page. The feed is fuelled by the new content appearing on that page, and therefore, not all pages lend themselves well to be scraped and to be sources for useful feeds.
I then started syndicating RSS feeds generated by my different sites across the sites themselves, and then later, I started creating valuable and unique news channels by aggregating RSS news feeds available out there.
Two good examples of this are Kolabora (N/A), where you can see the home page populated by news coming from different RSS news feeds I have selected, and MasterViews, where, once again, the news page is completely generated by carefully filtering and aggregating selected RSS-based news feeds.
Possible Solutions: RSS Aggregators?
Many have touted the advent of RSS aggregators as one possible solution to the issue of information overflow without yet realizing the intrinsic limitations of this approach when applied literally.
You can subscribe easily to hundreds of RSS feeds in no time at all. And there are a great number of people I like to read about.
And what I really want to read is what these people say about the things that interest me.
Jon Udell is a great guy. Of his posts, though, only one in thirty is something that matches my level of interest and competence. The rest is too “geeky” for me.
So, it is impossible for me to feel motivated to look at Jon’s feed, though I know that from time to time, he will bring me some wonderful, unique gems.
So, I give in, and I tend to scan only those sources that are more directly in tune with me. And by accepting this compromise, I often lose all of the opportunity to discover Jon’s magic realizations when they do really happen.
I have seen and heard of people subscribing to hundreds, if not thousands, of feeds inside their RSS aggregators.
Is that manageable?
Do these people get better and more information than everyone else?
It is not.
They don’t.
They don’t because it is just not possible and not enjoyable at all to go through so many posts in which the greatest majority is not even of relevance to their interest.
It is just like doing a one-keyword search on Google and then going through all of the results pages that come up. Yes, you can come up with some interesting discoveries, but would you really do that?
As for Google, where the enormous amount of relevant results obscures the view of most of what could be of interest to me, we do need some new, more effective way of filtering the river of news coming at us.
Visualization is one solution, and you will see many new fascinating applications of interface and information design to search mechanisms in the near future. Visualization is the quantum leap needed to perceive the forest from the trees before wanting to dive deeper into it.
How Do I Want To Find Information Relevant To Me
Overloaded and disappointed, I have spent some thinking time testing many RSS tools and aggregators only to gradually increase my particular appreciation for Bloglines.
Though Blogline features are not completely unique and are available in parts across several other resources online, this is the service that allowed me to come closest to the idea of news and content filtering that I was starting to build up in my head.
Bloglines, in fact, not only allow you to add as many feeds as you like inside a neat set of organized content folders, but they also allow you to subscribe to specific search queries you create.
That is, Bloglines allows you to record any search you do across the RSS-based content it has indexed and to automatically rerun that search for you each time you check for new content.
As simple as it sounds, this facility provides a terrific resource of fresh news on any topic you may select!
The first time you experience the use of this, you will feel an exhilarating vibe. The crafting of your own little search query becomes a key to generating valuable content on almost any topic you can think of.
Unfortunately, Bloglines indexes only less than one hundred thousand blogs (!), but think if you could do the same search query on Technorati, where over one million blogs are tracked daily. Would that be interesting?
With the help of Bloglines’ “subscribed searches” feature, I am now a happy scout reading the “topics” I am interested in and free from NOT having to read as a robot the individual feeds of one hundred and more authors and sites I am interested in.
That brings by itself another important corollary. When creating a saved search, you are, in fact, creating a very custom and personal news feed tailored to your own specific request.
If you could only save that Blogline feed as an actual RSS feed, you would start appreciating more directly the feeling of becoming an independent newsmaster yourself. (While Bloglines doesn’t yet offer this option, other online services, Blogdigger, have started to do so.)
Emerging Technologies
The whole idea I am presenting in this essay is made possible by the rapid emergence of a new group of tools and technologies that allow scraping, search, filtering, aggregation, and syndication of RSS-based content at costs and with ease of use, making them accessible to the layman.
My ideal future RSS aggregator gives me the possibility of not only subscribing to as many selected feeds as I like, but it also allows me to pick out my preferred content by filtering out through sophisticated queries and inclusion/exclusion features what is not relevant to my interests.
If you can select sources, formulate advance queries, create search feeds, and mix this combination in newly created information/news feeds, you have just visualized the primitive profile of the professional role I have been visualizing.
A New Digital Professional Role: The NewsMaster
Figure now the ability to concert, orchestrate, edit, and refine quality search formulas that tap into the whole RSS universe and beyond and that filter out relevant content based on selected keywords, sources, type of content, ranking, and many other possible criteria.
Moreover, iSyndicate (today, Yellowbrix) and others have done this before, and many sites still benefit from this simple automated mechanism.
But imagine now this mechanism being made available to each one who wants to use it.
Imagine having the power to select, aggregate, and filter content while creating a new RSS feed that is as unique as the person who has built it.
And this set of “magic search formulas” that anyone can now build is not a one-time shot anymore.
This is a “hose” from which useful information around a tightly defined topic can be extracted every day or as soon as there is something of value out there.
It is a hose from which other people can quench their knowledge thirst.
My unique personal filtering effort becomes useful for everyone.
I have a new profession for information seekers, digital librarians, and knowledge management evangelists: the NewsMaster.
You may probably find a better term for this, but the guy I am envisioning is a new type of webmaster who specializes in crafting uniquely powerful magic search formulas generating continuous RSS feeds on narrowly selected topics by:
a) Selecting and aggregating valuable resources (like a normal aggregator does)
b) Creating advanced search queries in the blogosphere ( a la Bloglines)
c) Creating advanced search queries on the traditional Web content at large
d) Creating advanced search queries on the overall pot of content derived at points a), b), and c) and generating new highly filtered RSS feeds matching specific content and quality criteria.
Searching and identifying key valuable sources and complex filtering formulas will be the outstanding job of the new information gatherers and publishers online: the NewsMasters.
Their ability and craft will be in identifying the query formulas that, when cleverly combined and refined by trial and error, will allow for powerful automatic news-gathering formulas.
These formulas need not only to identify the keywords/keyphrases that generate the best and most relevant set of results but also to grow through the intelligent use of multiple filtering criteria and precisely targeted inclusion/exclusion mechanisms.
The creation of dedicated information channels, originated by independent publishers and not by vested commercial interests or mainstream media conglomerates, may create the opportunity for a true renaissance of culture learning and a multiplication of our abilities to manage large amounts of rapidly changing information.
Search specialists and librarians who will craft with time investments appropriate queries to get the information they were looking for will be generously rewarded with an eternal fountain of relevant info for the time to come.
The act of search evolves from a mere set of one-time shots at finding something just in time to the start of a collective refined meta-filtering process that generates better information for everyone.
Are Newsmasters Sustainable?
In the wake of all this, I am sure some of you have also seen the opportunity to create niche Websites and information channels on very specific topics and with a possible significant rewarding economy.
The opportunities are indeed in abundance, and one needs only to explore some of the first rough ideas I have listed here below to get some directions on where to start.
Here is an initial list of possible revenue streams that could be ethically generated around the NewsMasters activity.
1. Excerpt-only RSS feeds bring readers back online to check for interesting stories and allow for online-only contextual advertising (Google AdSense, Blogads)
2. Low subscription rate to receive premium or “full content” feeds.
3. Editorial promotion inside RSS feeds. Ethical review and inclusion of commercial reviews by independent publishers.
4. RSSAds and similar initiatives.
5. Sponsorships! - This is one of the best ways to make specialized RSS newsfeeds pay back. Have the feed description and reference image carry a valuable sponsorship.
6. Donations
7. Micropayments
8. More traffic to Web sites where other products/services are offered.
9. Custom newsfeed creation for third parties.
10. Provision of high-quality custom feeds to major news syndication companies and organizations.
11. Syndication of content on commercial sites.
12. Premium subscription of feeds from a specific author or group of associates.
13. More (please suggest some in the bottom comments area)
Conclusions
RSS puts a large quantity of content de facto in the public domain, and it offers an easy way to reuse, syndicate, and aggregate such content in an unlimited number of ways.
If someone puts an RSS feed on the Internet, it is most likely that she is doing this for others to read and possibly further syndicate. So it is in the DNA of RSS to want to be free, to be further reused, personalized, and syndicated.
The above-described ability in the above essay to have many individual users work as Newsmasters in organizing the very own content we are generating provides us all with a new horizon of access to news in ways and fashion that we had not even conceived of until yesterday.
This is not Google News at the nth power. This is much, much more.
Specialized, focused information niches generated by the dedicated work of individual technology-enabled NewsMasters are something totally new for us to consider and reflect upon.
The power of many people at work seems to me irresistible and capable of creating profound changes in the way we operate and inform ourselves.
With this new search and filtering approach, we can all start generating better and more refined information channels as we become enabled to aggregate, select, and filter the most relevant content according to extremely sophisticated criteria and long-nurtured selected source lists.
The simplification of these new gathering and filtering tools, of which FeedPaper may be the first actual working example, is just around the corner. There is no way of avoiding this. It is just coming.
Tools like FeedPaper will allow capable thinkers to dedicate themselves to this new, highly valuable socio-alchemical effort and to create fountains of information that will hopefully be useful for much time to come.
With these technologies and with our applied intelligence, many alternative new sources of info can be effectively created. This will naturally filter out the blog noise from the valuable blog/RSS info. True diversity will bloom as an unlimited number of views and perspectives can be made available in such a possible information system.
So, it does look to me as a renaissance of sorts is truly taking place. And if can draw any conclusion from the first-hand feelings I have had, I must say that this is going to be a fascinating, revolutionary turn.
Special Note From the Author:
While we are only seeing the birth of the first primitive group of such tools, I expect many to become available soon with an array of features and controls that will make the independent news reporters' work a truly fascinating one.
Feed paper from Feedster may be the first one to become available.
For now, if you wish to pioneer this new, exciting, vast land, I have prepared a special toolkit just for this purpose.
It is called the NewsMasters Toolkit, and it contains all of the tools and resources you can start using now to build and syndicate your own selected news information channels.
Appendix:
The NewsMaster dream tool
(advanced aggregation/filtering tool - specification) Feb.17, 2004
What I would want to have in the ideal tool I yet have to meet.
Characteristics:
· Search according to multiple criteria
· Ability to group multiple queries together
· Full Boolean expression control integration
· Ability to filter out selected keywords/ keyphrases
· Ability to pre-select predetermined sources
· Ability to ban selected news sources
· Ability to filter news sources based on several criteria:
- Technorati Cosmos indicators
- Google PageRank
- Alexa absolute ranking
- Google linkbacks (or other equivalent)
- More to be defined
· Ability to filter news items based on article length
· Ability to filter news items based on link exclusion and inclusion rules
· Control of frequency of updates
· Generation of RSS feed
· Generation Of OPML of news sources utilized
· Query formula export
· Ability to include search results for specific queries on major search engines
· Export to multiple RSS formats
Would be NewsMasters Here Is Your Toolkit:
If you want to learn more about the advanced tools that allow you to start experimenting now with becoming a novel NewsMaster, please see my NewsMasters Toolkit for 32 nifty resources.
Click here to find out more.
Originally written by Robin Good for MasterNewMedia and first published on Thursday, February 19, 2004, as “The Birth Of The NewsMaster: The Network Starts To Organize Itself.”
Robin Good
