The Search Engine Duplicate Content Penalty – What is it?

A debate is raging over whether search engines penalized websites for providing duplicate content. Search engines keep their algorithms secret (along with just about everything else they do) better than Alan Greenspan and the Federal Reserve. With all this secrecy the only way to the truth is from observation and a few ‘off the cuff’ quips from search engine principals.

Investigation reveals that Google, AltaVista and others have filed patent applications concerning duplicate content filters. These filings are public record and open for public scrutiny. Examination of the filings shows that the search engines are indeed interested in filtering duplicate pages and penalizing sites that use duplicate content. Traffic studies on this site (www.profitbooks.com) and others clearly show there is not only a page by page duplicate content penalty but also a site wide penalty. Additionally, articles created for redistribution show a search engine duplicate content filter works over a period of a few weeks to remove pages duplicating the the article.

It is worthwhile to note that there are at least two types of duplicate content. There are duplicate content pages that contain similar or duplicate content that appears on other pages of the same website and there duplicate content pages that contain similar or duplicate content that appears on pages of a different website. The former being the considered the most objectionable and often classified as search engine spam. We’ll confine our discussion here to the latter, that is, content pages that are similar to pages on other websites.

Duplicate content pages would include pages of content “scraped” from copyrighted material or other websites, rss, syndicated and other content feeds, the reprinting of articles intended for redistribution and public domain content. Are all forms of duplicate content penalized?

The answer seems to be yes, if you use too much duplicate content on a page. The key to avoiding a duplicate content penalty is to mix the duplicate content with original content on a page and re-organize the content duplicated. Try to use your own page title and description meta tags and keep the duplicated content to no more than 70% of the content on a page.

Next time we’ll discuss some possible ways you can still use articles and other duplicate content without running afoul of the search engines. In the mean time, don’t go overboard filling your website with copied articles, feeds and syndicated content.