Paramètres .htaccess pour robots indélicats

vince — 2009-05-20T22:15:22Z

Peut se dépose en .htaccess la racine du site ou plus directement dans la configuration du serveur apache, en
/etc/apache2/httpd.conf par exemble.

La sélection de l'indélicat s'opère par l'user-agent du "visiteur.

 
 RewriteEngine on
 RewriteCond %{HTTP_USER_AGENT} ^Alexibot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^asterias [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BackDoorBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Black.Hole [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BlowFish [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BotALot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BuiltBotTough [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Bullseye [OR]
 RewriteCond %{HTTP_USER_AGENT} ^BunnySlippers [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Cegbfeieh [OR]
 RewriteCond %{HTTP_USER_AGENT} ^CheeseBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
 RewriteCond %{HTTP_USER_AGENT} ^CopyRightCheck [OR]
 RewriteCond %{HTTP_USER_AGENT} ^cosmos [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
 RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
 RewriteCond %{HTTP_USER_AGENT} ^DittoSpyder [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
 RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EroCrawler [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
 RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
 RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Foobot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^FrontPage [NC,OR]
 RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
 RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
 RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Harvest [OR]
 RewriteCond %{HTTP_USER_AGENT} ^hloader [OR]
 RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
 RewriteCond %{HTTP_USER_AGENT} ^httplib [OR]
 RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]
 RewriteCond %{HTTP_USER_AGENT} ^humanlinks [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]
 RewriteCond %{HTTP_USER_AGENT} ^InfoNaviRobot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
 RewriteCond %{HTTP_USER_AGENT} ^JennyBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
 RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Kenjin.Spider [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Keyword.Density [OR]
 RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
 RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
 RewriteCond %{HTTP_USER_AGENT} ^LexiBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^libWeb/clsHTTP [OR]
 RewriteCond %{HTTP_USER_AGENT} ^LinkextractorPro [OR]
 RewriteCond %{HTTP_USER_AGENT} ^LinkScan/8.1a.Unix [OR]
 RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mata.Hari [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
 RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
 RewriteCond %{HTTP_USER_AGENT} ^MIIxpc [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mister.PiX [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
 RewriteCond %{HTTP_USER_AGENT} ^moget [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mozilla/2 [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mozilla/3.Mozilla/2.01 [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
 RewriteCond %{HTTP_USER_AGENT} ^NPBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Offline.Explorer [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Openfind [OR]
 RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
 RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
 RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ProPowerBot/2.14 [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^QueryN.Metasearch [OR]
 RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
 RewriteCond %{HTTP_USER_AGENT} ^RepoMonkey [OR]
 RewriteCond %{HTTP_USER_AGENT} ^RMA [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SlySearch [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SpankBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^spanner [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^suzuran [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Szukacz/1.4 [OR]
 RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Telesoft [OR]
 RewriteCond %{HTTP_USER_AGENT} ^The.Intraformant [OR]
 RewriteCond %{HTTP_USER_AGENT} ^TheNomad [OR]
 RewriteCond %{HTTP_USER_AGENT} ^TightTwatBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Titan [OR]
 RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [OR]
 RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [OR]
 RewriteCond %{HTTP_USER_AGENT} ^True_Robot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^turingos [OR]
 RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot/1.5 [OR]
 RewriteCond %{HTTP_USER_AGENT} ^URLy.Warning [OR]
 RewriteCond %{HTTP_USER_AGENT} ^VCI [OR]
 RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebBandit [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebEnhancer [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Web.Image.Collector [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebmasterWorldForumBot [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Website.Quester [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Webster.Pro [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WebZip [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
 RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WWW-Collector-E [OR]
 RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Xenu's [OR]
 RewriteCond %{HTTP_USER_AGENT} ^Zeus
 RewriteRule ^(.*)$ http://www.robotstxt.org/

Redirection de http:// vers http://www.

vince — 2008-08-01T22:06:17Z

Surtout pour des facilités d'indexation, une page identique pourrait être indexée à deux reprises avec et sans www.

Le mode rewrite doit être activé.

Deux solutions :

créer un fichier .htaccess directement à la racine du domaine.
paramétrer la redirection dans la configuration du serveur.

Réécriture d'urls par .htaccess

Insérer, dans un .htaccess à la racine du site le code suivant :

RewriteEngine On
 RewriteBase /
 RewriteCond %{HTTP_HOST} ^vosinformations\.com [NC]
 RewriteRule (.*) http://www.vosinformations.com/$1 [QSA,R=301,L]

Réécriture d'urls dans la configuration du serveur

Modifier la configuration du VirtualHost de la façon suivante :


 ...
 ServerName www.vosinformations.com
 ServerAlias vosinformations.com
 
 ....
 
 ...
 RewriteEngine On
 RewriteCond %{HTTP_HOST} ^ashasumputh\.com [NC]
 RewriteRule (.*) http://www.ashasumputh.com/$1 [QSA,R=301,L]
 ...

Toujours préférable (pratique) de configurer directement au niveau du serveur.