gpt4 book ai didi

scrapy - Apscheduler+scrapy 信号仅适用于主线程

转载 作者:行者123 更新时间:2023-12-03 21:16:58 30 4
gpt4 key购买 nike

我想将 apscheduler 与 scrapy.but 我的代码是错误的。
我应该如何修改它?

settings = get_project_settings()
configure_logging(settings)
runner = CrawlerRunner(settings)

@defer.inlineCallbacks
def crawl():
reactor.run()
yield runner.crawl(Jobaispider)#this is my spider
yield runner.crawl(Jobpythonspider)#this is my spider
reactor.stop()

sched = BlockingScheduler()
sched.add_job(crawl, 'date', run_date=datetime(2018, 12, 4, 10, 45, 10))
sched.start()

错误:builtins.ValueError:信号仅适用于主线程

最佳答案

这个问题在这里得到了很好的回答:How to integrate Flask & Scrapy?它涵盖了各种用例和想法。我还发现该线程中的一个链接非常有用:https://github.com/notoriousno/scrapy-flask
要更直接地回答您的问题,请试试这个。它使用了上述两个链接的解决方案,特别是它使用了钩针库。

import crochet
crochet.setup()

settings = get_project_settings()
configure_logging(settings)
runner = CrawlerRunner(settings)

# Note: Removing defer here for the example
#@defer.inlineCallbacks

@crochet.run_in_reactor
def crawl():
runner.crawl(Jobaispider)#this is my spider
runner.crawl(Jobpythonspider)#this is my spider

sched = BlockingScheduler()
sched.add_job(crawl, 'date', run_date=datetime(2018, 12, 4, 10, 45, 10))
sched.start()

关于scrapy - Apscheduler+scrapy 信号仅适用于主线程,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/53605039/

30 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com