作者热门文章
- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
我有以下问题:我需要在网络抓取期间将图像保存到 mongodb。我有一个图像链接。我试过这个:
images_binaries = [] # this will store all images data before saving it to mongodb
# save as file on hard disc
urllib.urlretrieve(url, self.album_path + '/' + photo_file_name)
images_binaries.append(open(self.album_path + '/' + photo_file, 'r').read())
....
# after that I append this array of images raw data to Item
post = WaralbumPost()
post['images_binary'] = images_binaries
....
Waralbum 商品代码:
from scrapy.item import Item, Field
class WaralbumPost(Item):
images_binary = Field()
但这在保存到 mongo 时会导致错误:bson.errors.InvalidStringData: strings in documents must be valid UTF-8: '\xff\.....
执行此操作的更好方法是什么?转换原始图像数据是否可以解决这个问题?也许,scrapy 有一种保存图像的好方法?感谢您的回答
解决方案:我删除了这一行: images_binaries.append(open(self.album_path + '/' + photo_file, 'r').read()) post['images_binary'] = images_binaries在我的 WaralbumPost 中,我还保存了图像 url。然后,在 pipelines.py 中我得到这个 url 并将图像保存在 mongo 中。 pipelines.py代码:
class WarAlbum(object):
def __init__(self):
connection = pymongo.Connection(settings['MONGODB_SERVER'], settings['MONGODB_PORT'])
db = connection[settings['MONGODB_DB']]
self.collection = db[settings['MONGODB_COLLECTION']]
self.grid_fs = gridfs.GridFS(getattr(connection, settings['MONGODB_DB']))
def process_item(self, item, spider):
links = item['img_links']
ids = []
for i, link in enumerate(links):
mime_type = mimetypes.guess_type(link)[0]
request = requests.get(link, stream=True)
_id = self.grid_fs.put(request.raw, contentType=mime_type, filename=item['local_images'][i])
ids.append(_id)
item['data_chunk_id'] = ids
self.collection.insert(dict(item))
log.msg("Item wrote to MongoDB database %s/%s" %
(settings['MONGODB_DB'], settings['MONGODB_COLLECTION']),
level=log.DEBUG, spider=spider)
return item
希望这对某人有帮助
最佳答案
使用 GridFS。 Example:
String newFileName = "my-image";
File imageFile = new File("/users/victor/images/image.png");
GridFS gfsPhoto = new GridFS(db, "photo");
GridFSInputFile gfsFile = gfsPhoto.createFile(imageFile);
gfsFile.setFilename(newFileName);
gfsFile.save();
关于python - 如何将图像保存到具有图像 url 的 mongodb?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/25334125/
我是一名优秀的程序员,十分优秀!