- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
我最近在 3 节点 MapR 集群上安装了 Impala。当我运行一个简单的查询时。性能不如 Impala + HDFS。这是查询:
SELECT *
FROM ft_test, ft_wafer
WHERE ft_test_parquet.id = ft_wafer_parquet.id
and month = 1
and day = 8
and param = 2913;
大约用了 3 秒。但是当使用相同的查询但使用 HDFS 时。 30Gb 的表大小需要不到 1 秒的时间。
这是查询配置文件:
Query Runtime Profile:
Query (id=dc4c084615fbf9bb:4261466f00000000):
Summary:
Session ID: 5d4edbf63653cdf6:1a59ff5354c9d4bd
Session Type: BEESWAX
Start Time: 2017-05-25 16:31:25.121391000
End Time: 2017-05-25 16:31:28.584404000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.7.0 RELEASE (build a535b583202c4a81080098a10f952d377af1949d)
User: root
Connected User: root
Delegated User:
Network Address: ::ffff:127.0.0.1:58546
Default Db: inspex
Sql Statement: select *
FROM ft_test_partition, ft_wafer_parquet
WHERE ft_test_partition.file = ft_wafer_parquet.file
and month = 1
and day = 8
and param = 2913 limit 100
Coordinator: mapr1:22000
Query Options (non default):
Plan:
----------------
Estimated Per-Host Requirements: Memory=704.67MB VCores=2
04:EXCHANGE [UNPARTITIONED]
| limit: 100
| hosts=1 per-host-mem=unavailable
| tuple-ids=1,0 row-size=800B cardinality=1
|
02:HASH JOIN [INNER JOIN, BROADCAST]
| hash predicates: ft_wafer_parquet.file = ft_test_partition.file
| runtime filters: RF000 <- ft_test_partition.file
| limit: 100
| hosts=1 per-host-mem=690.00KB
| tuple-ids=1,0 row-size=800B cardinality=1
|
|--03:EXCHANGE [BROADCAST]
| | hosts=1 per-host-mem=0B
| | tuple-ids=0 row-size=78B cardinality=8235
| |
| 00:SCAN HDFS [inspex.ft_test_partition, RANDOM]
| partitions=1/29 files=1 size=171.69MB
| predicates: param = 2913
| table stats: 813365826 rows total
| column stats: all
| hosts=1 per-host-mem=704.00MB
| tuple-ids=0 row-size=78B cardinality=8235
|
01:SCAN HDFS [inspex.ft_wafer_parquet, RANDOM]
partitions=1/1 files=1 size=66.83KB
runtime filters: RF000 -> ft_wafer_parquet.file
table stats: 1500 rows total
column stats: all
hosts=1 per-host-mem=192.00MB
tuple-ids=1 row-size=722B cardinality=1500
----------------
Estimated Per-Host Mem: 738904067
Estimated Per-Host VCores: 2
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
----------------------------------------------------------------------------------------------------------------------
04:EXCHANGE 1 0.000ns 0.000ns 100 1 0 -1.00 B UNPARTITIONED
02:HASH JOIN 1 42.999ms 42.999ms 0 1 3.29 MB 690.00 KB INNER JOIN, BROADCAST
|--03:EXCHANGE 1 999.990us 999.990us 9.15K 8.23K 0 0 BROADCAST
| 00:SCAN HDFS 1 2s973ms 2s973ms 9.15K 8.23K 106.05 MB 704.00 MB inspex.ft_test_partition
01:SCAN HDFS 1 16.999ms 16.999ms 1.02K 1.50K 1.78 MB 192.00 MB inspex.ft_wafer_parquet
Planner Timeline: 15.315ms
- Analysis finished: 5.081ms (5.081ms)
- Equivalence classes computed: 5.601ms (519.374us)
- Single node plan created: 9.054ms (3.453ms)
- Runtime filters computed: 9.409ms (354.377us)
- Distributed plan created: 11.507ms (2.098ms)
- Planning finished: 15.315ms (3.808ms)
Query Timeline: 3s463ms
- Start execution: 0.000ns (0.000ns)
- Planning finished: 17.999ms (17.999ms)
- Submit for admission: 17.999ms (0.000ns)
- Completed admission: 17.999ms (0.000ns)
- Ready to start 2 remote fragments: 18.999ms (999.990us)
- All 2 remote fragments started: 19.999ms (999.990us)
- Rows available: 3s246ms (3s226ms)
- First row fetched: 3s346ms (99.999ms)
- Unregister query: 3s462ms (115.998ms)
- ComputeScanRangeAssignmentTimer: 0.000ns
ImpalaServer:
- ClientFetchWaitTimer: 214.997ms
- RowMaterializationTimer: 999.990us
Execution Profile dc4c084615fbf9bb:4261466f00000000:(Total: 3s228ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 1
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed Enabled
----------------------------------------------------------------------------------------------------------------------------
0 2 1 1 LOCAL false 0 (1) N/A N/A true
Fragment start latencies: Count: 2, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
Final filter table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed Enabled
----------------------------------------------------------------------------------------------------------------------------
0 2 1 1 LOCAL false 0 (1) N/A N/A true
Per Node Peak Memory Usage: mapr1:22000(108.65 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F02:(Total: 3s226ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(500.000ms): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 209.83 KB (214864)
- PerHostPeakMemUsage: 0
- PrepareTime: 0.000ns
- RowsProduced: 0 (0)
- TotalCpuTime: 101.999ms
- TotalNetworkReceiveTime: 3s226ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 48 (48)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 12.21 GB (13111148544)
- PeakMemoryUsage: 256.00 KB (262144)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 0.000ns
EXCHANGE_NODE (id=4):(Total: 3s226ms, non-child: 0.000ns, % non-child: 0.00%)
BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 61.05 KB (62513)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 0.000ns
- FirstBatchArrivalWaitTime: 3s226ms
- PeakMemoryUsage: 0
- RowsReturned: 100 (100)
- RowsReturnedRate: 30.00 /sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F00:(Total: 3s001ms, non-child: 0.000ns, % non-child: 0.00%)
split sizes: min: 66.83 KB, max: 66.83 KB, avg: 66.83 KB, stddev: 0
completion times: min:3s227ms max:3s227ms mean: 3s227ms stddev:0.000ns
execution rates: min:20.70 KB/sec max:20.70 KB/sec mean:20.70 KB/sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.86
- BloomFilterBytes: 1.00 MB (1048576)
- PeakMemoryUsage: 5.07 MB (5320864)
- PerHostPeakMemUsage: 108.65 MB (113924736)
- PrepareTime: 38.999ms
- RowsProduced: 1.02K (1024)
- TotalCpuTime: 3s232ms
- TotalNetworkReceiveTime: 2s940ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 13.999ms
CodeGen:(Total: 262.997ms, non-child: 262.997ms, % non-child: 100.00%)
- CodegenTime: 999.990us
- CompileTime: 73.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1953028)
- NumFunctions: 85 (85)
- NumInstructions: 2.86K (2857)
- OptimizationTime: 151.998ms
- PrepareTime: 36.999ms
DataStreamSender (dst_id=4):(Total: 999.990us, non-child: 999.990us, % non-child: 100.00%)
- BytesSent: 61.05 KB (62513)
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 59.62 MB/sec
- RowsReturned: 1.02K (1024)
- SerializeBatchTime: 999.990us
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 185.83 KB (190290)
HASH_JOIN_NODE (id=2):(Total: 3s001ms, non-child: 42.999ms, % non-child: 1.43%)
- BuildPartitionTime: 1.999ms
- BuildRows: 9.15K (9153)
- BuildRowsPartitioned: 9.15K (9153)
- BuildTime: 0.000ns
- GetNewBlockTime: 0.000ns
- HashBuckets: 16.38K (16384)
- HashCollisions: 0 (0)
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.29 MB (3445888)
- PinTime: 0.000ns
- ProbeRows: 1.02K (1024)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 0.000ns
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 0.000ns
EXCHANGE_NODE (id=3):(Total: 2s941ms, non-child: 2s941ms, % non-child: 100.00%)
- BytesReceived: 314.85 KB (322407)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 0.000ns
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 9.15K (9153)
- RowsReturnedRate: 3.11 K/sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
HDFS_SCAN_NODE (id=1):(Total: 16.999ms, non-child: 16.999ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 128.51 KB (131593)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 49 (49)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 1 (1)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 1.78 MB (1866400)
- PerReadThreadRawHdfsThroughput: 4.18 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 1.50K (1500)
- RowsReturned: 1.02K (1024)
- RowsReturnedRate: 60.23 K/sec
- ScanRangesComplete: 1 (1)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 14.999ms
- MaterializeTupleTime(*): 999.990us
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 2.216ms
- ScannerThreadsVoluntaryContextSwitches: 27 (27)
- TotalRawHdfsReadTime(*): 29.999ms
- TotalReadThroughput: 0.00 /sec
Filter 0 (1.00 MB):
- Rows processed: 1.50K (1500)
- Rows rejected: 3 (3)
- Rows total: 1.50K (1500)
Averaged Fragment F01:(Total: 3s191ms, non-child: 160.998ms, % non-child: 5.04%)
split sizes: min: 171.69 MB, max: 171.69 MB, avg: 171.69 MB, stddev: 0
completion times: min:3s210ms max:3s210ms mean: 3s210ms stddev:0.000ns
execution rates: min:53.47 MB/sec max:53.47 MB/sec mean:53.47 MB/sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.86
- BloomFilterBytes: 0
- PeakMemoryUsage: 106.05 MB (111206232)
- PerHostPeakMemUsage: 108.65 MB (113924736)
- PrepareTime: 33.999ms
- RowsProduced: 9.15K (9153)
- TotalCpuTime: 6s330ms
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 36.999ms
CodeGen:(Total: 51.999ms, non-child: 51.999ms, % non-child: 100.00%)
- CodegenTime: 999.990us
- CompileTime: 5.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1953028)
- NumFunctions: 13 (13)
- NumInstructions: 228 (228)
- OptimizationTime: 11.999ms
- PrepareTime: 33.999ms
DataStreamSender (dst_id=3):(Total: 4.999ms, non-child: 4.999ms, % non-child: 100.00%)
- BytesSent: 314.85 KB (322407)
- NetworkThroughput(*): 153.74 MB/sec
- OverallThroughput: 61.49 MB/sec
- RowsReturned: 9.15K (9153)
- SerializeBatchTime: 2.999ms
- TransmitDataRPCTime: 1.999ms
- UncompressedRowBatchSize: 769.68 KB (788150)
HDFS_SCAN_NODE (id=0):(Total: 2s973ms, non-child: 2s973ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 1.00
- BytesRead: 171.79 MB (180132958)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- DecompressionTime: 130.998ms
- MaxCompressedTextFileLength: 0
- NumColumns: 8 (8)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 1 (1)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 106.05 MB (111196408)
- PerReadThreadRawHdfsThroughput: 434.91 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 28.05M (28047320)
- RowsReturned: 9.15K (9153)
- RowsReturnedRate: 3.08 K/sec
- ScanRangesComplete: 1 (1)
- ScannerThreadsInvoluntaryContextSwitches: 112 (112)
- ScannerThreadsTotalWallClockTime: 3s157ms
- MaterializeTupleTime(*): 2s977ms
- ScannerThreadsSysTime: 566.243ms
- ScannerThreadsUserTime: 2s525ms
- ScannerThreadsVoluntaryContextSwitches: 100 (100)
- TotalRawHdfsReadTime(*): 394.996ms
- TotalReadThroughput: 57.11 MB/sec
Fragment F00:
Instance dc4c084615fbf9bb:4261466f00000001 (host=mapr1:22000):(Total: 3s001ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:1/66.83 KB
Filter 0 arrival: 3s210ms
MemoryUsage(500.000ms): 0, 3.05 MB, 3.14 MB, 3.13 MB, 3.09 MB, 3.12 MB, 3.19 MB
ThreadUsage(500.000ms): 1, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 1.86
- BloomFilterBytes: 1.00 MB (1048576)
- PeakMemoryUsage: 5.07 MB (5320864)
- PerHostPeakMemUsage: 108.65 MB (113924736)
- PrepareTime: 38.999ms
- RowsProduced: 1.02K (1024)
- TotalCpuTime: 3s232ms
- TotalNetworkReceiveTime: 2s940ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 13.999ms
CodeGen:(Total: 262.997ms, non-child: 262.997ms, % non-child: 100.00%)
- CodegenTime: 999.990us
- CompileTime: 73.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1953028)
- NumFunctions: 85 (85)
- NumInstructions: 2.86K (2857)
- OptimizationTime: 151.998ms
- PrepareTime: 36.999ms
DataStreamSender (dst_id=4):(Total: 999.990us, non-child: 999.990us, % non-child: 100.00%)
- BytesSent: 61.05 KB (62513)
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 59.62 MB/sec
- RowsReturned: 1.02K (1024)
- SerializeBatchTime: 999.990us
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 185.83 KB (190290)
HASH_JOIN_NODE (id=2):(Total: 3s001ms, non-child: 42.999ms, % non-child: 1.43%)
ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Hash Table Construction Codegen Enabled, Join Build-Side Prepared Asynchronously, 1 of 1 Runtime Filter Published
- BuildPartitionTime: 1.999ms
- BuildRows: 9.15K (9153)
- BuildRowsPartitioned: 9.15K (9153)
- BuildTime: 0.000ns
- GetNewBlockTime: 0.000ns
- HashBuckets: 16.38K (16384)
- HashCollisions: 0 (0)
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.29 MB (3445888)
- PinTime: 0.000ns
- ProbeRows: 1.02K (1024)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 0.000ns
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 0.000ns
EXCHANGE_NODE (id=3):(Total: 2s941ms, non-child: 999.990us, % non-child: 0.03%)
BytesReceived(500.000ms): 70.08 KB, 127.46 KB, 162.66 KB, 230.08 KB, 301.42 KB, 312.19 KB
- BytesReceived: 314.85 KB (322407)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 0.000ns
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 9.15K (9153)
- RowsReturnedRate: 3.11 K/sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
HDFS_SCAN_NODE (id=1):(Total: 16.999ms, non-child: 16.999ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Disabled, PARQUET Codegen Enabled
Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:1/66.83 KB
Runtime filters: All filters arrived. Waited 0
BytesRead(500.000ms): 0, 0, 0, 0, 0, 0
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 128.51 KB (131593)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 49 (49)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 1 (1)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 1.78 MB (1866400)
- PerReadThreadRawHdfsThroughput: 4.18 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 1.50K (1500)
- RowsReturned: 1.02K (1024)
- RowsReturnedRate: 60.23 K/sec
- ScanRangesComplete: 1 (1)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 14.999ms
- MaterializeTupleTime(*): 999.990us
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 2.216ms
- ScannerThreadsVoluntaryContextSwitches: 27 (27)
- TotalRawHdfsReadTime(*): 29.999ms
- TotalReadThroughput: 0.00 /sec
Filter 0 (1.00 MB):
- Rows processed: 1.50K (1500)
- Rows rejected: 3 (3)
- Rows total: 1.50K (1500)
Fragment F01:
Instance dc4c084615fbf9bb:4261466f00000002 (host=mapr1:22000):(Total: 3s191ms, non-child: 160.998ms, % non-child: 5.04%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:1/171.69 MB
MemoryUsage(500.000ms): 0, 91.50 MB, 82.91 MB, 86.63 MB, 67.68 MB, 66.67 MB, 53.51 MB
ThreadUsage(500.000ms): 1, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 1.86
- BloomFilterBytes: 0
- PeakMemoryUsage: 106.05 MB (111206232)
- PerHostPeakMemUsage: 108.65 MB (113924736)
- PrepareTime: 33.999ms
- RowsProduced: 9.15K (9153)
- TotalCpuTime: 6s330ms
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 36.999ms
CodeGen:(Total: 51.999ms, non-child: 51.999ms, % non-child: 100.00%)
- CodegenTime: 999.990us
- CompileTime: 5.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1953028)
- NumFunctions: 13 (13)
- NumInstructions: 228 (228)
- OptimizationTime: 11.999ms
- PrepareTime: 33.999ms
DataStreamSender (dst_id=3):(Total: 4.999ms, non-child: 4.999ms, % non-child: 100.00%)
- BytesSent: 314.85 KB (322407)
- NetworkThroughput(*): 153.74 MB/sec
- OverallThroughput: 61.49 MB/sec
- RowsReturned: 9.15K (9153)
- SerializeBatchTime: 2.999ms
- TransmitDataRPCTime: 1.999ms
- UncompressedRowBatchSize: 769.68 KB (788150)
HDFS_SCAN_NODE (id=0):(Total: 2s973ms, non-child: 2s973ms, % non-child: 100.00%)
ExecOption: Expr Evaluation Codegen Enabled, PARQUET Codegen Enabled, Codegen enabled: 1 out of 1
Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:1/171.69 MB
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0%
File Formats: PARQUET/SNAPPY:8
BytesRead(500.000ms): 88.45 MB, 112.45 MB, 136.45 MB, 152.66 MB, 168.66 MB, 171.79 MB
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 1.00
- BytesRead: 171.79 MB (180132958)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- DecompressionTime: 130.998ms
- MaxCompressedTextFileLength: 0
- NumColumns: 8 (8)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 1 (1)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 106.05 MB (111196408)
- PerReadThreadRawHdfsThroughput: 434.91 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 28.05M (28047320)
- RowsReturned: 9.15K (9153)
- RowsReturnedRate: 3.08 K/sec
- ScanRangesComplete: 1 (1)
- ScannerThreadsInvoluntaryContextSwitches: 112 (112)
- ScannerThreadsTotalWallClockTime: 3s157ms
- MaterializeTupleTime(*): 2s977ms
- ScannerThreadsSysTime: 566.243ms
- ScannerThreadsUserTime: 2s525ms
- ScannerThreadsVoluntaryContextSwitches: 100 (100)
- TotalRawHdfsReadTime(*): 394.996ms
- TotalReadThroughput: 57.11 MB/sec
我已经做过的是:使用 Parquet 、分区、计算统计数据。但是仍然无法获得与之前相同的时间。
据我所见。大部分时间花在扫描 HDFS 上,这很奇怪,因为这通常不是一个耗时的部分。请看一下。任何输入都会很好。谢谢。
最佳答案
这可能是因为节点 0 上的 HDFS SCAN 花费了更多时间:-HDFS_SCAN_NODE(id=0):(总计:2s973ms,非子:2s973ms,% 非子:100.00%)文件格式:PARQUET/SNAPPY:8MapR Converge 社区提出了一个确切的问题: https://community.mapr.com/message/59777-impala-scan-mapr-fs-slow
关于hadoop - Impala 扫描 MapR-FS 慢,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44189503/
安装 MapR 时遇到问题安装程序。它无法从存储库中读取。请检查以下链接以了解详细错误。让我知道如何克服这一点。 http://justpaste.it/q6vl 最佳答案 您可以尝试清理 repo
我不小心删除了 hadoop.tmp.dir,在我的例子中是/tmp/{user.name}/*。现在每次当我从 CLI 运行配置单元查询时,mapred 作业将在任务尝试时失败,如下所示: Erro
大家好, 我在 Centos 6.4, 64 位上使用 Mapr M7 版本。我正在 MapR 上测试 MapR-tables,但单击 MapR-tables --> Error Home direc
我没有对 mapred-site.xml 中 mapred.local.dir 指定的目录的写入权限(也没有对 mapred-site.xml 的写入权限)有没有一种方法可以在每个 session 基
我正在使用 Hadoop 版本 0.20.2(Cloudera 发行版 cdh3u6)并发现问题。据我了解,如果我在 /etc/hadoop/conf/mapred-site.xml 中设置一个值,它
我正在尝试从 mapr fs origin 进行简单的数据移动到 mapr fs destination (这不是我的用例,只是为了测试目的做这个简单的 Action )。尝试 validate 时这
我在MapR DB中创建了一个表,并使用hbase shell将一些原始数据加载到该表中。我在哪里可以在MapR文件系统上找到这些数据?像 hive 数据一样存储在仓库目录中,MapR DB如何工作?
MapR 自带了一种不同于HDFS 方式的新架构(Container Architecture)。有什么区别?这如何存储数据?有什么优点和缺点? 最佳答案 参见 https://www.youtube
我无法找到向我解释元数据如何在 MAPR(文件元数据)中分布的特定链接。当我查看 cloudera/hortonworks/apache hadoop 时,我知道元数据存储在 namenode 的内存
用于MapR DB的MapR REST API似乎无法正常工作。我只是尝试将列系列的列表命令用作过滤器,但它没有给我适当的输出,但它与maprcli选项完美配合。这是我所做的操作列表。 使用maprc
嗨,我正在尝试在 ubuntu 12.04 上精确安装 MapR,但面临一些问题。我按照下面的链接进行安装 http://doc.mapr.com/display/MapR/Quick+Install
我在 mapr-clusters.conf 中指定了两个集群 cluster1 secure=true cldb1:7222 cldb2:7222 cldb3:7222 cluster2 secure
是否正在进行将 Hadoop 管道从 mapred 移植到 mapreduce 包的工作? 谢谢,梅格 最佳答案 JIRA 问题 MAPREDUCE-1362解决了将管道升级到新的 mapreduce
我正在玩 Mapr Sandbox,我不明白 Mapr Warden 应用程序的作用是什么。 Mapr 网站包含对配置文件的引用,但没有程序本身的描述。 最佳答案 Warden 是一个轻型 Java
我使用 MapR,我想从 LDAP 创建用户。 我可以创建我的用户,但 Hue 不想创建关联文件夹,我收到此消息:“获取当前用户的用户信息时出错,***(错误 500)” 我尝试使用本地用户,但这是同
我现在使用 CDH 5.1。它通过 YARN 启动正常的 Hadoop 作业,但 hive 仍然可以使用 mapred。有时一个大查询会挂起很长时间,我想杀死它。 我可以通过 JobTracker W
本文整理了Java中org.apache.hadoop.mapred.YARNRunner类的一些代码示例,展示了YARNRunner类的具体用法。这些代码示例主要来源于Github/Stackove
本文整理了Java中org.apache.hadoop.mapred.YarnChild类的一些代码示例,展示了YarnChild类的具体用法。这些代码示例主要来源于Github/Stackoverf
本文整理了Java中org.apache.hadoop.mapred.YarnOutputFiles类的一些代码示例,展示了YarnOutputFiles类的具体用法。这些代码示例主要来源于Githu
本文整理了Java中org.apache.hadoop.mapred.YarnClientProtocolProvider类的一些代码示例,展示了YarnClientProtocolProvider类
我是一名优秀的程序员,十分优秀!