hadoop - Druid hadoop批处理主管:无法将类型id 'index.hadoop'解析为SupervisorSpec的子项

Question

我正在尝试启动一个Druid主管来摄取存储在hadoop中的PArqurt数据。但是，我收到以下错误，但找不到有关它的任何信息:

"error":"Could not resolve type id 'index_hadoop' as a subtype of [simple type, class io.druid.indexing.overlord.supervisor.SupervisorSpec]: known type ids = [NoopSupervisorSpec, kafka]\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP)

我试图修复它，将hadoop深度存储， Parquet 和avro扩展加载到扩展加载列表中，但这没有用。

这是我的主管JSON配置:

{
  "type" : "index_hadoop",
  "spec" : {
    "dataSchema" : {
      "dataSource" : "hadoop-batch-timeseries",
      "parser" : {
        "type": "parquet",
        "parseSpec" : {
            "format" : "parquet",
            "flattenSpec": {
                "useFieldDiscovery": true,
                "fields": [
                ]
            },
            "timestampSpec" : {
                "column" : "timestamp",
                "format" : "auto"
            },
            "dimensionsSpec" : {
                "dimensions": [ "installation", "var_id", "value" ],
                "dimensionExclusions" : [],
                "spatialDimensions" : []
            }
        }
      },
      "metricsSpec" : [
        {
          "type" : "count",
          "name" : "count"
        }
      ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "DAY",
        "queryGranularity" : "NONE",
        "intervals" : [ "2018-10-01/2018-11-30" ]
      }
    },
    "ioConfig": {
      "type": "hadoop",
      "inputSpec": {
        "type": "granularity",
        "dataGranularity": "day",
        "inputFormat": "org.apache.druid.data.input.parquet.DruidParquetInputFormat",
        "inputPath": "/warehouse/tablespace/external/hive/demo.db/integers",
        "filePattern": "*.parquet",
        "pathFormat": "'year'=yyy/'month'=MM/'day'=dd"
      },
    },
    "tuningConfig" : {
      "type": "hadoop"
    }
  },
  "hadoopDependencyCoordinates": "3.1.0"
}

Answer 1

我遇到了同样的问题。通过将其作为任务提交而不是作为主管提交来解决:

curl -X POST -H 'Content-Type: application/json' -d @my-spec.json http://my-druid-coordinator-url:8081/druid/indexer/v1/task