mongodb - org.apache.solr.common.SolrException : TransactionLog doesn't know how to serialize class org. bson.types.ObjectId;尝试实现 ObjectResolver？

Question

从 mongodb 执行数据导入时，Solr 抛出以下错误:

org.apache.solr.common.SolrException: TransactionLog doesn't know how to serialize class org.bson.types.ObjectId; try implementing ObjectResolver?
at org.apache.solr.update.TransactionLog$1.resolve(TransactionLog.java:100)
at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:234)
at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:589)
at org.apache.solr.update.TransactionLog.write(TransactionLog.java:395)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:320)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80)
at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:254)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:526)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
at java.lang.Thread.run(Thread.java:748)

我的 Solr 版本是 6.6.0。错误的原因可能是什么？如何解决？

Answer 1

我在尝试从 mongoDB 中的多个集合导入数据时遇到了这个问题。

假设你没有使用 mongo-connector，我使用以下导入数据。

• Solr-6.6.0
• solr-dataimporthandler-6.6.0
• mongo-java-driver-3.5.0
• Solr Mongo importer

由于返回的“_id”是 ObjectId 类型，我的解决方法是在将“_id”索引到 solr 之前将其转换为字符串，并在查询“_id”时将其转换为 ObjectId 类型，然后再运行查询。

下载 solr mongo 导入器并进行以下更改。

MongoMapperTransformer.java

public class MongoMapperTransformer extends Transformer {

@Override
public Object transformRow(Map<String, Object> row, Context context) {

    for (Map<String, String> map : context.getAllEntityFields()) {
        String mongoFieldName = map.get(MONGO_FIELD);
        String mongoId = map.get(MONGO_ID);
        if (mongoFieldName == null)
            continue;

        String columnFieldName = map.get(DataImporter.COLUMN);

        //If the field is ObjectId convert it into String
        if (mongoId != null && Boolean.parseBoolean(mongoId)) {
            Object srcId = row.get(columnFieldName);
            row.put(columnFieldName, srcId.toString());
        }
        else{
            row.put(columnFieldName, row.get(mongoFieldName));
        }
    }

    return row;
}


public static final String MONGO_FIELD = "mongoField";

//To identify the _id field
public static final String MONGO_ID = "objectIdToString";

}

接下来，替换函数

public Iterator <Map<String, Object>> getData(String query){...} 

在 MongoDataSource.java 中包含以下内容:

@Override
public Iterator<Map<String, Object>> getData(String query) {

    DBObject queryObject = new BasicDBObject();

    /* If querying by _id, since the id is a string now, 
     * it has to be converted back to type ObjectId() using the 
     * constructor 
     */ 
    if(query.contains("_id")){
        @SuppressWarnings("unchecked")
        Map<String, String> queryWithId = (Map<String, String>) JSON.parse(query);
        String id = queryWithId.get("_id");
        queryObject = new BasicDBObject("_id", new ObjectId(id));
    }
    else{
        queryObject = (DBObject) JSON.parse(query);
    }

    LOG.debug("Executing MongoQuery: " + query.toString());

    long start = System.currentTimeMillis();
    mongoCursor = this.mongoCollection.find(queryObject);
    LOG.trace("Time taken for mongo :"
            + (System.currentTimeMillis() - start));

    ResultSetIterator resultSet = new ResultSetIterator(mongoCursor);
    return resultSet.getIterator();
}

完成这些更改后，您可以使用 ant 构建 jar。

将 jars(solr mongo importer 和 mongo-java-driver)复制到 lib 目录中。我将它们复制到 ${solr-install-dir}/contrib/dataimport-handler/lib

在 solr-config.xml 中为上述 jars 添加 lib 指令:

<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />

最后，这是一个 mongo 集合和 data-config.xml 的示例

User collection
{
    "_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
    "name" : "User1",
    "phone" : "123456789"
}

Address collection
{
    "_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
    "address" : "#666, Maiden street"
}

数据配置.xml

不要忘记为 _id 字段提及 objectIdToString="true"，以便 MongoMapperTransformer 可以将 id 字符串化。

<dataConfig>
   <dataSource name="MyMongo"
           type="MongoDataSource"
           database="test"
            />
   <document name="UserDetails">
   <!-- if query="" then it imports everything -->
      <entity name="users"
          processor="MongoEntityProcessor"
          query=""
          collection="user"
          datasource="MyMongo"
          transformer="MongoMapperTransformer">
              <field column="_id"  name="id" mongoField="_id" objectIdToString="true" />
              <field column="phone" name="phone" mongoField="phone"/>

          <entity name="address"
                processor="MongoEntityProcessor"
                query="{_id:'${users._id}'}"
                collection="address"
                datasource="MyMongo"
                transformer="MongoMapperTransformer">
                <field column="address" name="adress" mongoField="address"/>
          </entity>
   </entity>
   </document>
</dataConfig>

托管模式将 id 字段作为字符串。此外，如果您在 mongodb 中有嵌套对象，则必须使用脚本转换器在 solr 中对它们进行索引。

希望这会有所帮助，祝你好运!