Download Robots.txt parser

Home : All Categories : Web Search : Robots.txt parser



		Robots.txt parser Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details). Visit site to download...



Sponsored links

SearchGooSimple PHP Google API Search
SearchGooSimple is perhaps the simplest PHP Google API Search Script there is. W...

Multi-Search (PHP)
This is a PHP script for interfacing multiple search engines using only one form...

Free PHP Google Site Search Script
This free Google website search script is easy to install, free to use, and offe...

CyberCrawl Meta Edition
CyberCrawl Meta Edition is a fully modular MetaSearch-Solution. The Script (writ...

Adbie Keywords Catalog - Adsense Website
New unique Adsense website. AdBie Keywords Catalog script its a very profitable ...

Aruntx Msea
The Aruntx Msea - Microsoft Internet Explorer 7 Search Listing Add-on. Add Searc...

DeadMans Redirect
DMR is a portal page creation system. DMR acts as a jack of all trades form, whi...

SmartPPC Pro Search Engine Script
Need to serve Pay-per-Click Ads? Want to pull feeds in many formats from top pro...

Google block
You can search inside you page or in the whole Internet.The results page is conf...

Adbie Search Engine Lite - Adsense Website
AdBie Search Engine Lite verson script its a very profitable search engine scrip...

Robots.txt parser script free download