作者热门文章
- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我正在尝试在 中插入 ~ 56,249,000 个项目berkeleydb-JE .我跑了数据库缓存大小 获取有关我的数据库的一些统计信息:
java -jar je-5.0.34.jar DbCacheSize -records 56248699 -key 8 -data 20
=== Environment Cache Overhead ===
3,155,957 minimum bytes
To account for JE daemon operation and record locks,
a significantly larger amount is needed in practice.
=== Database Cache Size ===
Minimum Bytes Maximum Bytes Description
--------------- --------------- -----------
1,287,110,736 1,614,375,504 Internal nodes only
4,330,861,264 4,658,126,032 Internal nodes and leaf nodes
=== Internal Node Usage by Btree Level ===
Minimum Bytes Maximum Bytes Nodes Level
--------------- --------------- ---------- -----
1,269,072,064 1,592,660,160 632,008 1
17,837,712 21,473,424 7,101 2
198,448 238,896 79 3
2,512 3,024 1 4
EnvironmentConfig cfg=(...)
cfg.setTransactional(true);
cfg.setAllowCreate(true);
cfg.setReadOnly(false);
cfg.setCachePercent(80);
cfg.setConfigParam(EnvironmentConfig.LOG_FILE_MAX,"250000000");
cfg.setAllowCreate(true);
cfg.setTransactional(true);
cfg.setReadOnly(false);
Transaction txn= env.beginTransaction(null, null);
//open db with transaction 'txn'
Database db=env.open(...txn)
Transaction txn2=this.getEnvironment().beginTransaction(null, null);
long record_id=0L;
while((item=readNextItem(input))!=null)
{
(...)
++record_id;
db.put(...); //insert record_id/item into db
/** every 100000 items commit and create a new transaction.
I found it was the only way to avoid an outOfMemory exception */
if(record_id%100000==0)
{
txn2.commit();
System.gc();
txn2=this.getEnvironment().beginTransaction(null, null);
}
}
txn2.commit();
txn.commit();
100000 / 56248699 ( 0.2 %). 13694.9 records/seconds. Time remaining:68.3 m Disk Usage: 23.4 Mb. Expect Disk Usage: 12.8 Gb Free Memory : 318.5 Mb.
200000 / 56248699 ( 0.4 %). 16680.6 records/seconds. Time remaining:56.0 m Disk Usage: 49.5 Mb. Expect Disk Usage: 13.6 Gb Free Memory : 338.3 Mb.
(...)
6600000 / 56248699 (11.7 %). 9658.2 records/seconds. Time remaining:85.7 m Disk Usage: 2.9 Gb. Expect Disk Usage: 24.6 Gb Free Memory : 165.0 Mb.
6700000 / 56248699 (11.9 %). 9474.5 records/seconds. Time remaining:87.2 m Disk Usage: 2.9 Gb. Expect Disk Usage: 24.7 Gb Free Memory : 164.8 Mb.
6800000 / 56248699 (12.1 %). 9322.6 records/seconds. Time remaining:88.4 m Disk Usage: 3.0 Gb. Expect Disk Usage: 24.8 Gb Free Memory : 164.8 Mb.
(Ctrl-C... abort...)
MemTotal: 4021708 kB
MemFree: 253580 kB
Buffers: 89360 kB
Cached: 1389272 kB
SwapCached: 56 kB
Active: 2228712 kB
Inactive: 1449096 kB
Active(anon): 1793592 kB
Inactive(anon): 596852 kB
Active(file): 435120 kB
Inactive(file): 852244 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 3174028 kB
HighFree: 57412 kB
LowTotal: 847680 kB
LowFree: 196168 kB
SwapTotal: 4085756 kB
SwapFree: 4068224 kB
Dirty: 16320 kB
Writeback: 0 kB
AnonPages: 2199056 kB
Mapped: 111280 kB
Shmem: 191272 kB
Slab: 58664 kB
SReclaimable: 41448 kB
SUnreclaim: 17216 kB
KernelStack: 3792 kB
PageTables: 11328 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6096608 kB
Committed_AS: 5069728 kB
VmallocTotal: 122880 kB
VmallocUsed: 18476 kB
VmallocChunk: 81572 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10232 kB
DirectMap2M: 903168 kB
Max. Heap Size (Estimated): 872.94M
Ergonomics Machine Class: server
Using VM: Java HotSpot(TM) Server VM
(...)
6800000 / 56248699 (12.1 %). 13144.8 records/seconds. Time remaining:62.7 m Disk Usage: 1.8 Gb. Expect Disk Usage: 14.6 Gb Free Memory : 95.5 Mb.
(...)
6800000 / 56248699 (12.1 %). 9322.6 records/seconds. Time remaining:88.4 m Disk Usage: 3.0 Gb. Expect Disk Usage: 24.8 Gb Free Memory : 164.8 Mb.
最佳答案
首先,我会删除您对 System.gc(); 的显式调用。
如果您注意到这有助于提高性能,请考虑使用不同的 GC 算法。例如,当 bdb/je 缓存使用率始终接近可用堆的 70% 时,G1GC 的性能会更好。
其次,在某些时候,B+ 索引更新将是 n log n 性能并且会减少插入时间。
不使用事务会更快。特别是,如果您可以在导入失败时从头开始重新启动导入。
只记得在最后做一个 environment.sync() 和一个检查点。在执行此导入时,您可能希望禁用 BDB/je 检查点和 BDB/je GC 线程。
config.setConfigParam(EnvironmentConfig.ENV_RUN_CLEANER, "false");
config.setConfigParam(EnvironmentConfig.ENV_RUN_CHECKPOINTER, "false);
config.setConfigParam(EnvironmentConfig.ENV_RUN_IN_COMPRESSOR, "false");
public void checkpointAndSync()
throws ObjectStoreException
{
env.sync();
CheckpointConfig force = new CheckpointConfig();
force.setForce(true);
try
{
env.checkpoint(force);
} catch (DatabaseException e)
{
log.error("Can not chekpoint db " + path.getAbsolutePath(), e);
throw new ObjectStoreException(e);
}
}
关于optimization - 向 BerkeleyDB-JE 插入数据越来越慢,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/15065538/
我是一名优秀的程序员,十分优秀!