java - new IndexWriter() 处的 LockObtainFailedException

Question

我正在使用 Crawler Controller 来抓取中型网站的所有页面。它随机抓取 2-3 个页面，然后导致 IndexWriter 上的锁定

Directory dir = FSDirectory.open(new File(index));
IndexWriterConfig conf = new IndexWriterConfig(org.apache.lucene.util.Version.LUCENE_41,new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_41));
writer = new IndexWriter(dir, conf);  // line which throws lock exception. 

日志:

From: SiteSearch.KCCrawlerController.(80): Lock obtain timed out: NativeFSLock@D:\Websites\ccc\WEB-INF\lucene-index\en\write.lock: 05/08/2014 10:57:55 org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@D:\Websites\ccc\WEB-INF\lucene-index\en\write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:636) at SiteSearch.KCCrawlerController.(KCCrawlerController.java:80) at org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:66) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.tridion.ambientdata.web.AmbientDataServletFilter.doFilter(AmbientDataServletFilter.java:255) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at adminV3.ugc.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:429) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Unknown Source)

Adding jsp: http://example.com/en/consulting/diagnostics.jsp?crawler=yes
From: SiteSearch.KCCrawler.visit(95): Stream closed: 05/08/2014 10:57:55 java.io.IOException: Stream closed at org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204) at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:312) at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342) at SiteSearch.KCCrawler.visit(KCCrawler.java:95) at edu.uci.ics.crawler4j.crawler.WebCrawler.processPage(WebCrawler.java:306) at edu.uci.ics.crawler4j.crawler.WebCrawler.run(WebCrawler.java:189) at java.lang.Thread.run(Unknown Source)

为什么我会收到此异常？任何帮助。

更新:2014 年 8 月 17 日:

当我第一次运行索引器时，它成功完成并抛出以下异常。如果我对此进行搜索，我会成功获得结果。但是，如果我再次运行索引器，它会抛出上面提到的锁定异常。它还显示我的 Controller 类被调用了两次。

org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet jsp threw exception java.io.IOException: Stream closed at org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:115) at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:188) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:118) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:77) at org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:82) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386)

at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

Answer 1

引用 Javadocs -

“打开 IndexWriter 会为正在使用的目录创建一个锁定文件。尝试在同一目录上打开另一个 IndexWriter 将导致 LockObtainFailedException。如果使用同一目录上的 IndexReader 从索引中删除文档，也会引发 LockObtainFailedException。”

“IndexWriter 实例是完全线程安全的，这意味着多个线程可以同时调用其任何方法。如果您的应用程序需要外部同步，则不应在 IndexWriter 实例上进行同步，因为这可能会导致死锁；请改用您自己的(非 Lucene)对象。”

https://lucene.apache.org/core/4_1_0/core/org/apache/lucene/index/IndexWriter.html

您是否为正在抓取的每个页面创建新的 IndexWriter 实例？