java - Solr 导入命令不起作用

Question

我需要你的帮助来解决一个大问题。我必须使用 solr 对来自 EMC xDB 源的一些数据进行索引。我将 xDB 配置为像 RDBMS 一样被请求。然后我配置了 Squirrel 来请求 xDB。它有效，例如我尝试时:

 Select * from books

我使用与 squirrel 相同的驱动程序和连接字符串配置了 solr。

这是我的 data-config.xml:

<dataConfig>
<dataSource type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>
<document name="books">
    <entity name="book" query="select * from books">
        <field column="author" name="author" />
        <field column="price" name="price" />
        <field column="name" name="name" />
        <field column="description" name="description" />
        <field column="title" name="title" />
        <field column="publish_date" name="publish_date" />         
    </entity>
</document>
</dataConfig>

solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
        <str name="config">C:\solr\solr-5.2.1\server\solr\jcg\conf\data-config.xml</str>
    </lst>
</requestHandler>

我的 xDB BOOKS-MetaData.xml:

<?xml version="1.0" encoding="UTF-16"?>
<TABLEMETADATA TABLENAME="books">
  <RECORD-COUNT>2</RECORD-COUNT>
  <ROOT-ELEMENT>books</ROOT-ELEMENT>
  <ROW-ELEMENT>book</ROW-ELEMENT>
  <DATATYPES>
     <BOOKS-TYPE>
      <NAME>author</NAME>
      <COLNO>1</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>title</NAME>
      <COLNO>2</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
    </BOOKS-TYPE>
    <BOOKS-TYPE>
      <NAME>genre</NAME>
      <COLNO>3</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>price</NAME>
      <COLNO>4</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>publish_date</NAME>
      <COLNO>5</COLNO>
      <DATATYPE>DATETIME</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>description</NAME>
      <COLNO>6</COLNO>
      <DATATYPE>VARCHAR2(350)</DATATYPE>
    </BOOKS-TYPE>
  </DATATYPES>
</TABLEMETADATA>

solr 响应的一部分:

"verbose-output": [
"entity:book",
[
  "document#1",
  [
    "query",
    "select * from books",
    "time-taken",
    "0:0:0.114",
    "EXCEPTION",
    "org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1

当我检查 solr 日志时，有 3 个异常。第 1 号:

getNext() failed for query 'select * from books':org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

数字2:

Exception while processing: book document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

最后一个:

Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
... 29 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 31 more
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

提前感谢您的帮助，因为我真的不明白为什么它适用于 squirrel 而不适用于 solr。

编辑问题以回复@Cheffe

这确实是由于 getObject() 方法导致的 xDB 驱动程序问题。我查看了松鼠代码，它有效，因为它使用列的索引而不是其名称。

int idx = _columnIndices != null ? _columnIndices[i] : i + 1;
        try
        {
            final int columnType = _rsmd.getColumnType(idx);
            //final String columnClassName = _rsmd.getColumnClassName(idx);
            switch (columnType)
            {
                case Types.NULL:
                    row[i] = null;
                    break;

                // TODO: When JDK1.4 is the earliest JDK supported
                // by Squirrel then remove the hardcoding of the
                // boolean data type.
                case Types.BIT:
                case 16:

//案例类型.BOOLEAN: row[i] = _rs.getObject(idx); 。 。 .

无论如何，它可以与

convertType="true"

Answer 1

检查 Solr 源后:该数据库的驱动程序可能不适合与 Solr 的 DIH 一起运行。不幸的是，该数据库是专有的，因此我无法尝试。

异常是从您的供应商的驱动程序内部引发的。所有三个根本原因都在那里。当您向下滚动到堆栈跟踪的最底部时，您可以看到这一点。这些堆栈跟踪的所有内容都告诉您

Method xxx.getObject-columnLabel is not yet implemented.

这意味着:您的数据库供应商 EMC 没有在其 JDBC 驱动程序中实现此方法。如果您不想亲自动手，您应该向他们提交错误报告或功能请求，表明您需要实现此功能。

<小时/>

您可能会看到堆栈跟踪中记录 Solr 来源的最后一行指向 the internal getARow method

private Map<String, Object> getARow() {
  if (resultSet == null)
    return null;

  Map<String, Object> result = new HashMap<>();
  for (String colName : colNames) {
    try {
      if (!convertType) {
      Object value = resultSet.getObject(colName); // <-- here the exception takes of
      if (value instanceof BigDecimal || value instanceof BigInteger) {
        result.put(colName, value.toString());
      } else {
        result.put(colName, value);
      }
      continue;
    }
// more code there, but left out

您可以通过设置 dataimporthandler 的配置参数来绕过 resultSet.getObject 的调用。也就是说这是

convertType (default: false) – Applies an additional conversion from the field type returned by the database to the field type defined in the schema.xml. The default value seems to be safer, because it does not cause extra, magical conversion. However, in special cases (eg BLOB fields), that conversion is one of the ways of solving the problem. [Reference]

您需要在 data-config.xml 中执行此操作

<dataConfig>
<dataSource convertType="true" type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>

但是我无法告诉您您的数据库及其驱动程序是否成功。可能这只会导致另一个异常。在这种情况下，您将需要回退到解决方案

1. 要求您的供应商这样做
2. 编写自定义数据导入例程