python - scrapy ne suit pas les liens

Question

J'essaye d'analyser le site avec Scrapy. Les URLs que j'ai besoin d'analyser se sont formées comme ceci: http://example.com/productID/1234/. Ces liens peuvent être trouvés sur des pages avec l'adresse comme: http://example.com/categoryID/1234/. Le fait est que mon robot récupère la première page categoryID (http://www.example.com/categoryID/79/, comme vous pouvez le voir sur la trace ci-dessous), mais rien de plus. Qu'est-ce que je fais mal? Je vous remercie.

Voici mon code Scrapy:

# -*- coding: UTF-8 -*- 

#THIRD-PARTY MODULES 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.selector import HtmlXPathSelector 

class ExampleComSpider(CrawlSpider): 
    name = "example.com" 
    allowed_domains = ["http://www.example.com/"] 

    start_urls = [ 
     "http://www.example.com/" 
    ] 

    rules = (
     # Extract links matching 'categoryID/xxx' 
     # and follow links from them (since no callback means follow=True by default). 
     Rule(SgmlLinkExtractor(allow=('/categoryID/(\d*)/',),)), 

     # Extract links matching 'productID/xxx' and parse them with the spider's method parse_item 
     Rule(SgmlLinkExtractor(allow=('/productID/(\d*)/',)), callback='parse_item'), 
    ) 

    def parse_item(self, response): 

     self.log('Hi, this is an item page! %s' % response.url) 

Voici une trace de Scrapy:

2012-01-31 12:38:56+0000 [scrapy] INFO: Scrapy 0.14.1 started (bot: parsers) 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Enabled item pipelines: 
2012-01-31 12:38:57+0000 [example.com] INFO: Spider opened 
2012-01-31 12:38:57+0000 [example.com] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023 
2012-01-31 12:38:57+0000 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080 
2012-01-31 12:38:58+0000 [example.com] DEBUG: Crawled (200) <GET http://www.example.com/> (referer: None) 
2012-01-31 12:38:58+0000 [example.com] DEBUG: Filtered offsite request to 'www.example.com': <GET http://www.example.com/categoryID/79/> 
2012-01-31 12:38:58+0000 [example.com] INFO: Closing spider (finished) 
2012-01-31 12:38:58+0000 [example.com] INFO: Dumping spider stats: 
    {'downloader/request_bytes': 199, 
    'downloader/request_count': 1, 
    'downloader/request_method_count/GET': 1, 
    'downloader/response_bytes': 121288, 
    'downloader/response_count': 1, 
    'downloader/response_status_count/200': 1, 
    'finish_reason': 'finished', 
    'finish_time': datetime.datetime(2012, 1, 31, 12, 38, 58, 409806), 
    'request_depth_max': 1, 
    'scheduler/memory_enqueued': 1, 
    'start_time': datetime.datetime(2012, 1, 31, 12, 38, 57, 127805)} 
2012-01-31 12:38:58+0000 [example.com] INFO: Spider closed (finished) 
2012-01-31 12:38:58+0000 [scrapy] INFO: Dumping global stats: 
    {'memusage/max': 26992640, 'memusage/startup': 26992640} 

Answer 1

Il peut y avoir une différence entre "www.example.com" et "example.com" . Si elle aide, vous pouvez les utiliser à la fois de cette façon

allowed_domains = ["www.example.com", "example.com"] 

Answer 2

Remplacer:

allowed_domains = ["http://www.example.com/"] 

avec:

allowed_domains = ["example.com"] 

Cela devrait faire l'affaire.