python - 提取子字符串的上下文 URL-6ren

python - 提取子字符串的上下文 URL

转载作者：太空宇宙更新时间：2023-11-03 16:43:09

我正在构建一个 scrapy 应用程序，如果 URL 中的子字符串匹配，我需要提取完整的 URL。

例如:

假设某个页面具有我感兴趣的以下 URL:

/public/flag?cat=Computers/Programming/Languages/Python/Books&url=http://www.pearsonhighered.com/educator/academic/product/0,,0130260363,00%2Ben-USS_01DBC.html
/public/flag?cat=Computers/Programming/Languages/Python/Books&url=http://www.brpreiss.com/books/opus7/html/book.html
/public/flag?cat=Computers/Programming/Languages/Python/Books&url=http://www.diveintopython.net/
/public/flag?cat=Computers/Programming/Languages/Python/Books&url=http://rhodesmill.org/brandon/2011/foundations-of-python-network-programming/
[还有 18 个]

但我的搜索字符串是 flag?cat=Computers/Programming/Languages/Python/Books

仅返回 URL 的匹配部分，而不返回完整的 URL。如何获取上面列出的完整 URL？

这是一个基于示例的简单 scrapy 测试用例:

from scrapy.spiders import Spider
from scrapy.selector import Selector
import scrapy

class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
    ]

    def parse(self, response):
        #scrapy.shell.inspect_response( response, self )
        results = response.xpath('//body').re('(flag\?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks)')
        print results

输出:

[
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks',
    u'flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks'
]

预期输出:

[
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.pearsonhighered.com%2Feducator%2Facademic%2Fproduct%2F0%2C%2C0130260363%2C00%252Ben-USS_01DBC.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.brpreiss.com%2Fbooks%2Fopus7%2Fhtml%2Fbook.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.diveintopython.net%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Frhodesmill.org%2Fbrandon%2F2011%2Ffoundations-of-python-network-programming%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.techbooksforfree.com%2Fperlpython.shtml"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.freetechbooks.com%2Fpython-f6.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fgreenteapress.com%2Fthinkpython%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.network-theory.co.uk%2Fpython%2Fintro%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.freenetpages.co.uk%2Fhp%2Falan.gauld%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.wiley.com%2FWileyCDA%2FWileyTitle%2FproductCd-0471219754.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fhetland.org%2Fwriting%2Fpractical-python%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fsysadminpy.com%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.qtrac.eu%2Fpy3book.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.wiley.com%2FWileyCDA%2FWileyTitle%2FproductCd-0764548077.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=https%3A%2F%2Fwww.packtpub.com%2Fpython-3-object-oriented-programming%2Fbook"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.network-theory.co.uk%2Fpython%2Flanguage%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.pearsonhighered.com%2Feducator%2Facademic%2Fproduct%2F0%2C%2C0130409561%2C00%252Ben-USS_01DBC.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.informit.com%2Fstore%2Fproduct.aspx%3Fisbn%3D0201616165%26redir%3D1"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.pearsonhighered.com%2Feducator%2Facademic%2Fproduct%2F0%2C%2C0201748843%2C00%252Ben-USS_01DBC.html"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.informit.com%2Fstore%2Fproduct.aspx%3Fisbn%3D0672317354"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fgnosis.cx%2FTPiP%2F"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&amp;url=http%3A%2F%2Fwww.informit.com%2Fstore%2Fproduct.aspx%3Fisbn%3D0130211192"><img src="/img/flag.png" alt="[!]" title="report an issue with this listing'
]

最佳答案

问题是 .re() 只会返回与表达式匹配的部分。相反，如果您想继续使用正则表达式检查，请使用 re:test()钩子(Hook):

response.xpath('//body//a/@href[re:test(., "flag\?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks")]').extract()

在我这边产生以下内容:

[
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&url=http%3A%2F%2Fwww.pearsonhighered.com%2Feducator%2Facademic%2Fproduct%2F0%2C%2C0130260363%2C00%252Ben-USS_01DBC.html', 
    u'/public/flag?cat=Computers%2FProgramming%2FLanguages%2FPython%2FBooks&url=http%3A%2F%2Fwww.brpreiss.com%2Fbooks%2Fopus7%2Fhtml%2Fbook.html',
    ...
]

关于python - 提取子字符串的上下文 URL，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/36604507/

文章推荐： python - 为 Sublime Text 3 安装 numpy

文章推荐： javascript - 使用 JavaScript 将日期时间类型发布到数据库

文章推荐： arrays - Ruby:将 int 和存储转换为数组

jQuery 子>父>子
我有这个 html 代码: HELLO WORLD! X V HELLO WORLD! X V 我想按 X(类关闭)将父 div 的高度更改为 20px 并显示 V(类打开)，但在每个 d
database-design - 在数据库中存储(子)日志和(子)分类帐以用于会计应用程序
在会计应用程序的许多不同实现中，有两种主要的数据库设计方法来保存日志和分类帐数据。只保留 Journal 信息，然后 Ledger 只是 Journal 的一个 View (因为 journal 总
子里面的 Perl 子
我想在另一个子里面有一个子， sub a { sub b { } } 我想为每次调用 sub b 创建一个新的 sub a 实例。有没有办法在 Perl 中做到这一点？当我运行上面的
excel - 查找重复项和重命名主/子
我有一些代码正在查找重复项并突出显示单元格: Private Sub cmdDups_Click() Dim Rng As Range Dim cel As Range Set Rng = ThisW
delphi - 子表中具有替代链接字段的主/子
可能有一个简单的解决方案，但我很难过。我有一个包含一个 ID 字段的主表。在两个可能的字段中有一个具有该 ID 的子表。想象一个由选手 A 和选手 B 组成的 double 队。Master 表将有
javascript - 将相关元素嵌套在一起父/子
假设我有一个包含对象的数组: [ { "id": "5a97e047f826a0111b754beb", "name": "Hogwarts", "parentId": "
mysql - 同时批量插入父/子
我正在尝试对 MySQL 数据库表执行一对父/子模型的批量插入，但似乎无法使用标准的 ActiveRecord 功能来完成。所以，我尝试了 activerecord-import gem，但它也不支持
c# - 子/父事件引发
我有一个带有多个子类的父抽象类。最终，我希望通过 GUI 中的进度条显示子类中完成的进度。我目前所做的，我意识到这是行不通的，是在父类中声明为每个子类将覆盖的虚拟方法的事件方法定义。所以像: pub
Javascript(子)对象通过键数组访问
是否可以通过键数组在对象中设置变量？例如我有这个对象: var obj = {'outer': {'inner': 'value'} }; 并希望设置由键数组选择的值: var keys = ['ou
mysql - 具有多个级别的多重关系父/子
我有一个名为 companies 的 MySQL 表，如下所示: +---------+-----------+-----------+ | id_comp | comp_name | id_pare
linux - 子。命令在终端上不起作用
我正在尝试使用 sublime text 在 sublime text 上的 ionic 上打开我的第一个应用程序。它给了我一个“找不到命令”的错误。如何修复？我试过这些命令: sudo rm -r
Python 共享属性父/子
不好意思问，但我正在使用 webapp2，我正在设计一个解决方案，以便更容易定义路由 based on this google webapp2 route function .但这完全取决于能够在子级
c++ - 获取用数字字符串表示的树中的所有直接父/子
我有代表树的数字字符串(我不知道是否有官方名称): 012323301212 上面的例子代表了 2 棵树。根用 0 表示。根的直接子代为“1”，“1”的直接子代为“2”，依此类推。我需要将它们分组到由
Android ==> 子 Activity ？
是否可以在当前 Activity 之上添加 Activity 。例如，假设我单击一个按钮，然后它将第二个 Activity 添加到当前 Activity 。而第二个 Activity 只覆盖了我当前
REST 子/子资源单个实体
我很难思考如何为子资源建模。以作者的书籍为例。你可以有 N 本书，每本书只有一位作者。 /books GET /books POST /books/id PUT /books/id DELETE 到
Python:(子)字符串等价与列表快速成员资格测试
有人可以向我解释以下内容(python 2.7) 来自已解析文件的两个字符串数字: '410.9''410.9 '(注意尾随空格) A_LIST = ['410.9 '] '410.9' in '41
.net - 是否存在指定的(子)索引分隔符？
背景在 PowerShell 中构建 hash table 是很常见的通过特定属性快速访问对象，例如以 LastName 为基础建立索引: $List = ConvertFrom-Csv @' I
polymer - 子 Web 组件的调用方法
我真的很难弄清楚如何调用嵌套 Polymer Web 组件的函数。这是标记: rise-distribution组件有 canPlay我想从 rise-playlist
graphviz - 具有大(子)集群的图形的隐点错误消息
我写了一个小工具转储(以 dot 格式)一个项目的依赖关系图，其中所有位于同一目录中的文件都聚集在一个集群中。当我尝试生成包含相应图形的 pdf 时，dot开始哭: 命令 dot -Tpdf trim
perl - 如何通过指定其解析树来创建 perl 子？
给定一个 CODE ref，是否可以: 访问该 CODE ref 的解析树通过指定 CODE ref 的解析树来创建一个新的 CODE ref，该解析树可以包含在 1 中返回的解析树的元素通常我们

太空宇宙

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

python - 提取子字符串的上下文 URL