SEARCH:
 
Home : All categories : Web Search : Robots.txt parser

Robots.txt parser



Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).


Visit site to download...

Multi-Search (PHP)
This is a PHP script for interfacing multiple search engines using only one form. It allows you to p
Web & Site search engine php script, Free
This is just a basic WEB search engine, like google but not powered by Google. You can limit it to
CyberCrawl Meta Edition
CyberCrawl Meta Edition is a fully modular MetaSearch-Solution. The Script (written in PHP3) queries
PPC Directory
Free script for SEO masters creates you own search engine and directory with thousands of SE-optimiz
SearchGooSimple PHP Google API Search
SearchGooSimple is perhaps the simplest PHP Google API Search Script there is. With just two files t
DeadMans Redirect
DMR is a portal page creation system. DMR acts as a jack of all trades form, which allows for URL re
Adbie Search Engine Lite - Adsense Website
AdBie Search Engine Lite verson script its a very profitable search engine script with integrated Go
Google API Search
With this script you can be up and running your own Google search engine in just seconds! The Google
RevenuePilot Pay Per Click Search Script
With this script you can be running your own pay per click site in just a few minutes. Best of all i
K-Search
K-Search is very fast and highly customizable meta-search engine. It queries Google, AltaVista, MSN,