共计 4470 个字符,预计需要花费 12 分钟才能阅读完成。
导读 | 在日常爬虫练习中,我们爬取到的数据需要进行保存操作,在 scrapy 中我们可以使用 ImagesPipeline 这个类来进行相关操作,这个类是 scrapy 已经封装好的了,我们直接拿来用即可。 |
在使用 ImagesPipeline 下载图片数据时,我们需要对其中的三个管道类方法进行重写,其中 — get_media_request 是对图片地址发起请求
— file path 是返回图片名称
— item_completed 返回 item, 将其返回给下一个即将被执行的管道类
那具体代码是什么样的呢,首先我们需要在 pipelines.py 文件中,导入 ImagesPipeline 类,然后重写上述所说的 3 个方法:
from scrapy.pipelines.images import ImagesPipeline
import scrapy
import os
class ImgsPipLine(ImagesPipeline):
def get_media_requests(self, item, info):
yield scrapy.Request(url = item['img_src'],meta={'item':item})
#返回图片名称即可
def file_path(self, request, response=None, info=None):
item = request.meta['item']
print('########',item)
filePath = item['img_name']
return filePath
def item_completed(self, results, item, info):
return item
方法定义好后,我们需要在 settings.py 配置文件中进行设置,一个是指定图片保存的位置 IMAGES_STORE = ‘D:\\ImgPro’,然后就是启用“ImgsPipLine”管道,
ITEM_PIPELINES = {'imgPro.pipelines.ImgsPipLine': 300, #300 代表优先级,数字越小优先级越高}
设置完成后,我们运行程序后就可以看到“D:\\ImgPro”下保存成功的图片。
完整代码如下:
spider 文件代码:
# -*- coding: utf-8 -*-
import scrapy
from imgPro.items import ImgproItem
class ImgSpider(scrapy.Spider):
name = 'img'
allowed_domains = ['www.521609.com']
start_urls = ['http://www.521609.com/daxuemeinv/']
def parse(self, response):
#解析图片地址和图片名称
li_list = response.xpath('//div[@class="index_img list_center"]/ul/li')
for li in li_list:
item = ImgproItem()
item['img_src'] = 'http://www.521609.com/' + li.xpath('./a[1]/img/@src').extract_first()
item['img_name'] = li.xpath('./a[1]/img/@alt').extract_first() + '.jpg'
# print('***********')
# print(item)
yield item
items.py 文件
import scrapy
class ImgproItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
img_src = scrapy.Field()
img_name = scrapy.Field()
pipelines.py 文件
from scrapy.pipelines.images import ImagesPipeline
import scrapy
import os
from imgPro.settings import IMAGES_STORE as IMGS
class ImgsPipLine(ImagesPipeline):
def get_media_requests(self, item, info):
yield scrapy.Request(url = item['img_src'],meta={'item':item})
#返回图片名称即可
def file_path(self, request, response=None, info=None):
item = request.meta['item']
print('########',item)
filePath = item['img_name']
return filePath
def item_completed(self, results, item, info):
return item
settings.py 文件
import random
BOT_NAME = 'imgPro'
SPIDER_MODULES = ['imgPro.spiders']
NEWSPIDER_MODULE = 'imgPro.spiders'
IMAGES_STORE = 'D:\\ImgPro' #文件保存路径
LOG_LEVEL = "WARNING"
ROBOTSTXT_OBEY = False
#设置 user-agent
USER_AGENTS_LIST = ["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
USER_AGENT = random.choice(USER_AGENTS_LIST)
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
# 'User-Agent':"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
'User-Agent':USER_AGENT
}
#启动 pipeline 管道
ITEM_PIPELINES = {'imgPro.pipelines.ImgsPipLine': 300,}
以上即是使用 ImagesPipeline 下载保存图片的方法,今天突生一个疑惑,爬虫爬的好,真的是牢饭吃的饱吗?还请各位大佬解答!
正文完
星哥说事-微信公众号
![post-qrcode](https://www.xgss.net/wp-content/uploads/2023/07/二维码-微信公众号256x256.jpg)