robots分析報告

京東:User-agent: * ? ? ? ? ? ? ??

Disallow: /?* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?禁止訪問網站中的動態頁面

Disallow: /pop/*.html ? ? ? ? ? ? ? ? ? 禁止訪問/pop/目錄下的所有以".html"為后綴的URL

Disallow: /pinpai/*.html?*? ? ? ? ? ? 禁止訪問/pinpai/目錄下的所有以".html"為后綴的動態頁面

User-agent: EtaoSpider

Disallow: / ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?禁止EtaoSpider訪問網站

User-agent: HuihuiSpider

Disallow: / ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?禁止HuihuiSpider訪問網站

User-agent: GwdangSpider

Disallow: / ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

User-agent: WochachaSpider

Disallow: / ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?禁止GwdangSpider、WochachaSpider訪問網站


天貓:

User-agent: *

Disallow: / ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 禁止所有搜索引擎訪問網站的任何部分

亞馬遜:

User-agent: *

Disallow: /buycar

Disallow: /cart

Disallow: /checkout

Disallow: /class

Disallow: /com

Disallow: /common

Disallow: /css

Disallow: /dll

Disallow: /doc

Disallow: /dp/e-mail-friend/

Disallow: /dp/manual-submit/

Disallow: /dp/product-availability/

Disallow: /dp/rate-this-item/

Disallow: /dp/shipping/

Disallow: /dp/twister-update/

Disallow: /gp/aws/ssop

Disallow: /gp/cart

Disallow: /gp/css/homepage.html

Disallow: /gp/customer-reviews/common/du

Disallow: /gp/flex

Disallow: /gp/gfix

Disallow: /gp/history

Disallow: /gp/item-dispatch

Disallow: /gp/music/clipserve

Disallow: /gp/music/wma-pop-up

Disallow: /gp/offer-listing

Disallow: /gp/product/e-mail-friend

Disallow: /gp/product/product-availability

Disallow: /gp/product/rate-this-item

Disallow: /gp/recsradio

Disallow: /gp/slredirect

Disallow: /gp/twitter/

Disallow: /gp/vote

Disallow: /gp/voting/

Disallow: /gp/yourstore

Disallow: /inc

Disallow: /js

Disallow: /lib

Disallow: /mn/bookLookInsideApp

Disallow: /mn/checkInitApp

Disallow: /mn/checkoutAlertMsgApp

Disallow: /mn/checkoutredirectApp

Disallow: /mn/giftCardApp

Disallow: /mn/loginApplication

Disallow: /mn/loyaltyApp

Disallow: /mn/orderAddrApp

Disallow: /mn/orderCfmApp

Disallow: /mn/orderDetailApp

Disallow: /mn/orderFailApp

Disallow: /mn/orderHistoryApp

Disallow: /mn/orderModifyApp

Disallow: /mn/orderSummaryApp

Disallow: /mn/paymentRedriveApp

Disallow: /mn/recommendReviewApp

Disallow: /mn/releaseReviewApp

Disallow: /mn/reviewVoteApplication

Disallow: /mn/selectPaymentMethodApp

Disallow: /mn/selectShippingOpptionApplication

Disallow: /mn/shipmentTraceApp

Disallow: /mn/shoppingCartApplication

Disallow: /mn/tellFriend

Disallow: /mn/thankYouApplication

Disallow: /mn/virtualAccountApp

Disallow: /mn/yourAccountApp

Disallow: /paper

Disallow: /xml

Disallow: /youraccount

Disallow: /ap/signin ? ? ? ? ? ? ? ? ? ? ? ? ?禁止訪問以上目錄下的內容

Disallow: /gp/registry/wishlist/

Disallow: /wishlist/ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??

Allow: /wishlist/universal*

Allow: /wishlist/vendor-button*

Allow: /wishlist/get-button*

Disallow: /gp/wishlist/

Allow: /gp/wishlist/universal*

Allow: /gp/wishlist/vendor-button*

Allow: /gp/wishlist/ipad-install*

Disallow: /registry/wishlist/ ? ? ? ? ? ? ? ? ? ? ?除了Allow里面*模糊匹配的網址之外該文件夾下其它內容都不可訪問

Disallow: /gp/help/contact-us/general-questions.html*?type&email&skip=true

Disallow: /gp/help/customer/accessibility?ie=UTF8&initialIssue=forgotpw&skip=true

Disallow: /gp/registry/search.html

Disallow: /gp/orc/rml/

Disallow: /gp/digital/fiona/manage

Disallow: /gp/entity-alert/external

Disallow: /gp/customer-reviews/dynamic/sims-box

Disallow: /review/dynamic/sims-box

Disallow: /gp/redirect.html

Disallow: /gp/customer-media/upload/

Disallow: /gp/customer-media/actions/delete/

Disallow: /gp/customer-media/actions/edit-caption/

Disallow: /gp/dmusic/

Disallow: /registry

Disallow: /*/wishlist

Disallow: /gp/registry

Disallow: /gp/aag

Disallow: /gp/socialmedia/giveaways

Disallow: /gp/aw/so.html

Disallow: /gp/pdp/profile/

Disallow: /gp/help/customer/display.html*nodeId=200843370

Disallow: /gp/help/customer/display.html*nodeId=200877580

Disallow: /gp/help/customer/display.html*nodeId=200877590

Disallow: /gp/help/customer/display.html*nodeId=200879080

Disallow: /gp/help/customer/display.html*nodeId=200879100

Disallow: /gp/help/customer/display.html*nodeId=200879120

Disallow: /gp/help/customer/display.html*nodeId=200879160

Disallow: /gp/help/customer/display.html*nodeId=200879140

Disallow: /gp/help/customer/display.html*nodeId=200877610

Disallow: /gp/help/customer/display.html*nodeId=200878960

Disallow: /gp/help/customer/display.html*nodeId=200878980

Disallow: /gp/help/customer/display.html*nodeId=200879000

Disallow: /gp/help/customer/display.html*nodeId=200879040

Disallow: /gp/help/customer/display.html*nodeId=200879020

Disallow: /gp/help/customer/display.html*nodeId=200877630

Disallow: /gp/help/customer/display.html*nodeId=200879200

Disallow: /gp/help/customer/display.html*nodeId=200879220

Disallow: /gp/help/customer/display.html*nodeId=200879240

Disallow: /gp/help/customer/display.html*nodeId=200879280

Disallow: /gp/help/customer/display.html*nodeId=200879260

Disallow: /gp/help/customer/display.html*nodeId=200877650

Disallow: /gp/help/customer/display.html*nodeId=200879320

Disallow: /gp/help/customer/display.html*nodeId=200879340

Disallow: /gp/help/customer/display.html*nodeId=200879360

Disallow: /gp/help/customer/display.html*nodeId=200879400

Disallow: /gp/help/customer/display.html*nodeId=200879380

Disallow: /gp/help/customer/display.html*nodeId=200877560

Disallow: /gp/help/customer/display.html*nodeId=200843460

Disallow: /gp/help/customer/display.html*nodeId=200843440

Disallow: /gp/help/customer/display.html*nodeId=200899270

Disallow: /gp/help/customer/display.html*nodeId=200879440

Disallow: /gp/help/customer/display.html*nodeId=200899330

Disallow: /gp/help/customer/display.html*nodeId=200899350

Disallow: /gp/help/customer/display.html*nodeId=200899390

Disallow: /gp/help/customer/display.html*nodeId=200899410

Disallow: /gp/help/customer/display.html*nodeId=200899430

Disallow: /gp/help/customer/display.html*nodeId=200899220

Disallow: /gp/help/customer/display.html*nodeId=200899450

Disallow: /gp/help/customer/display.html*nodeId=200899670

Disallow: /gp/help/customer/display.html*nodeId=200899530

Disallow: /gp/help/customer/display.html*nodeId=200899470

Disallow: /gp/help/customer/display.html*nodeId=200899550

Disallow: /gp/help/customer/display.html*nodeId=200899570

Disallow: /gp/help/customer/display.html*nodeId=200899510

Disallow: /gp/help/customer/display.html*nodeId=200899610

Disallow: /gp/help/customer/display.html*nodeId=200899630

Disallow: /gp/help/customer/display.html*nodeId=200899650

Disallow: /gp/help/customer/display.html*nodeId=200879180

Disallow: /gp/help/customer/display.html*nodeId=200879060

Disallow: /gp/help/customer/display.html*nodeId=200879300

Disallow: /gp/help/customer/display.html*nodeId=200879420

Disallow: /gp/help/customer/display.html*nodeId=200899290

Disallow: /gp/help/customer/display.html*nodeId=200899310

Disallow: /gp/help/customer/display.html*nodeId=200843380

Disallow: /gp/help/customer/display.html*nodeId=200843420

Disallow: /gp/help/customer/display.html*nodeId=200899230

Disallow: /gp/help/customer/display.html*nodeId=200899250

Disallow: /gp/help/customer/display.html*nodeId=200899370

Disallow: /reviews/iframe

Disallow: /gp/help/reports/infringement/jquery/handle-notice-submit.html

Disallow: /gp/help/customer/handler/handle-email-submit.html ?禁止訪問以上的html...


亞馬遜具體的有點看不下去,也沒有找到相關的解釋,只能稍微看一下學習學習了...

最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容