Finding and Writing Fresh Blog Content

Often I'm asked: How do you find all the content for your blog? Over the years, I've developed different techniques and approaches to find interesting things.

Adapt your tools to find the hidden gems

First of all, you need to be a bit of a dynamic researcher. That means your tools need to adapt to a changing landscape. This ability to change your approaches also means that, well, there is no secret to it per se -- or if there was one today, it might be outdated next week. When a new resource arises a few days from now, and you're one of the early adopters, it will give your blog content the edge. The worst thing, on the opposite, is to only rely on big, well-known resources.

For example, when you write a tech blog, you should not rely too heavily on Slashdot, Digg, Del.icio.us, CNet, or Wired. But that doesn't mean you should ignore them. It's just that there's a good chance people already saw your story if it has been covered in any of those. On the other hand, if you see a story in a blog that you believe is not read by a large crowd, re-blogging this may be of interest to your readers.

Customize RSS feeds by keyword

Some news sources offer you a customizable feed. This saves you from going through standards routines like "visit site, enter keyword, press search, check which items are new." Here are some examples which return RSS (or Atom) feeds for you to subscribe to, and I will just use the keyword "Performancing" here -- the actual keyword of choice could be "boats", "soccer", "PSP", and whatever else your blog is about:

• Google News Search for "Performancing" (RSS): These will be mainstream sources, like Wired, or more well-known blogs.
• Del.icio.us results for the tag "Performancing" (RSS): Here you have the findings of a social bookmarking service, which combines both older links and fresh ones.
• Digg search results for "Performancing" (RSS): This is a similar to Del.icio.us in that it's a social linking page (the more "diggs" a story gets, the more likely it is promoted to the front-page of the site). I have to say the site has a much fresher look than Del.icio.us even considering this week's Del.icio.us redesign -- but hey, we don't care about that in layout-less RSS.
• Furl "Performancing" links (RSS): Yet another social bookmarking community.
• Technorati's tag results for "Performancing" (RSS): Technorati is the most well-known blog search engine, and allows you to grab their results via RSS readers.
• Yahoo search results for "Performancing" (RSS): This is useless to find new articles only, but can be used for monitoring a search result.
• Moreover links feed for "Performancing" (RSS): Here you will find even more news headlines.

Screen-scraping, an alternative to RSS

Some pages you want to keep an eye on don't offer you any kind of RSS or Atom feed. You have two choices: ask the site owner to release an RSS feed to then patiently wait for that to happen. Or, and I kinda like this better, you simply go and screen-scrape the content you want (go and read the site's ToS and check if that seems to be OK with the site, we don't want to suggest anything unlawful here). Screen-scraping consists of setting up a script to run every 30 minutes or so which will read the site's content. You can then alert yourself of changes.

If watching the page for mere changes isn't powerful enough, you can go a step further and convert the HTML to XML for refined tracking. I found PHP5 to be one of the languages doing this job best because you can read the HTML into an XML structure with little more than a single line ("@$dom->loadHTMLFile($url);") -- and for that, it doesn't matter if the site is using any kind of valid HTML (let alone XHTML). You can then use XPath to access only the headlines, or only the links, and so on.

For my topic of choice, I've set up a tracking program to monitor a certain company's press releases; its help pages; its option pages; and more. Whenever something happens, I'll get an email. I'm storing a copy of the old page on my server to compare old and new to spot differences, if necessary.

Your readers!

Without getting tips from my readers, I would be practically lost. It can be one of the most important information source for you, and often contains not only hidden gems, but also scoops -- especially if the source asks to be kept anonymous on your blog (you might still know her name, which is important for judging the story's authority).

When you want your readers to help you, first of all make it easy for them to find your contact information. Have either a link reading "Email", or "Contact" or similar as one of the visible navigation items (don't bury this into a long blogroll -- in fact, avoid a long blogroll, as it will only clutter your navigation). On my blog, I actively ask everyone to submit their stories.

It's good style to ask your readers how they want to be named as source. Some would prefer their full name to be used for credits. Often with a link to their blog or homepage. Others may want only their first name, or no name at all to be named. I suggest you obfuscate their name like "John D." upon first publishing, reply asking if the full name should be named, and then act accordingly (e.g. rewrite the credits to "John Doe").

Another way to have the community create the story is to set up a discussion forum. I'm not talking about comments only, because usually a comment belongs to a post. I'm talking about an independent (or semi-independent) system for everyone to create articles. This way, the community can talk amongst each other and also quickly alert everyone to fresh headlines. Even if you're away from the computer and your blog, this can be a tool for your readers to keep track of breaking news. And remember once you hit a certain size, every important breaking news you miss out on has the potential to hurt your blog.

Subscribe to blogs in your niche

You should know your blog niche's neighborhood if you want to keep up-to-date. Blogs with a similar focus as your own should become your number one resource. After a while you will have understood which blogs contain the most original stories, which blogs have an almost natural talent to find fresh and former hidden news gems. Go visit those blogs daily. Maybe it will only be around 5 blogs or less, and checking them for news does not take much of your time; but it can be incredibly valuable.

Meatspace

With meatspace I'm referring to everything that's not found online. Like books you read, newspapers, magazines. Talks you have with people. Telephone interviews. Conferences, or other meetings. Everywhere, there's content, and you should even take a camera in case something unusual comes along (depending on the quality of your cell phone camera, that may be sufficient for the blog). I run a tech blog with a focus on an internet company but even for me, the occasional public billboard blue screen is a gift from heaven! It's content, because your visitors might find it interesting. And the special thing about meatspace is that often, more original content comes out of it because it's harder to locate news (well, unless you're the 50th blogger to report on last weekend's convention).

Searching Google for content

Sometimes, I simply enter a few keywords into Google since I suspect there might be interesting content out there somewhere. Because not always will only fresh, news-style links be of interest. Often, finding an interesting, weird, amusing or forgotten website can be of interest to your readers as well. There's no "magic" in this approach, because you need to determine the keywords you search for (and it should not be anything obvious).

If you want to re-use the content you'll find via Google, you might want to restrict your search to cover only copyright-free material. This is possible using the advanced Google settings. Also, Yahoo has a stand-alone Creative Commons search.

Top Post Lists

If your blog doesn't fit any specific niche, you can also use the "most popular posts" lists for inspiration. Go to Technorati, Blogpulse, Daypop and similar, but I suggest to not use too much from there. After all, you don't want to become an echo chamber for other blogs -- and whatever you find here has become popular before you blogged it.

Think... and report on your thoughts!

Not all of your articles must contain news -- or differently put, you can create the news yourself by posting something highly original and fresh. When you spend large parts of your time cracking your head on a given niche topic, you will become an expert in this field. And when you then post large pieces of analysis, this analysis in itself can be a valuable resource to others. In that case, it's worthwhile for you to sometimes stop crawling for related fresh links only.

Stop, and take a time-out to think. Maybe you're reporting on Company X. Stop, and think about where Company X might be 10 years from now. Maybe you're blogging about the FooBar product line. Stop, and think about the meaning of FooBar and its impact on our lives. Sometimes, you become a better blogger -- when, for a while, you stop blogging.