linux 2.6.43，ext3，10K RPM SAS 磁盘，在不同文件上进行 2 个顺序写入(直接 io)，就像随机写入一样

Question

我最近在这个问题上停滞不前:

"2 sequential write(direct io 4KB alignemnt block) on different file acting like random write, which yield poor write performance in 10K RPM SAS disk".

最让我困惑的是:我有一批服务器，都配备了相同类型的磁盘(raid 1 配有 2 300GB 10K RPM 磁盘)，但响应不同。

• several servers seams ok with this kind of write pattern, disk happy accepted up to 50+MB/s; (same kernel version, same filesystem, with different lib (libc 2.4))
• others not so much, 100 op/s seams reach the limit of underlying disk, which confirm the random write performance of disk; ((same kernel version, same filesystem, with different lib (libc 2.12)))

[注意:我检查了不同 libc 的“pwrite”代码，它只告诉我们简单的“系统调用”]

我已经设法排除了以下可能性:1.我自己的程序中的软件bug； 通过一个简单的守护进程(无需动态链接进行编译)，进行顺序直接io写入；2.磁盘问题； 在一台测试机上切换 2 个不同版本的 Linux 系统，它们在我的直接 io 写入模式上表现良好，但在切换到旧的 lib 版本几天后，随机写入很糟糕；

我尝试比较:

1. /sys/block/sda/queue/*, which may different in both way;
2. filefrag show nothing but two different file interleaved sequenctial grow physical block id;

一定有某种写入策略导致这个问题，但我不知道从哪里开始:

1. different kernel setting ?, may be related to how ext3 allocate disk block ?
2. raid cache(write back) or disk cache write strategy?
3. or underlying disk strategy to mapping logical block into real physical block ?

非常感谢

Answer 1

答案是:这是因为/sys/block/sda/queue/schedule 设置:

1. MACHINE A: display schedule: cfq, but undlying, it's deadline;
2. MACHINE B: the schedule is consistent with cfq; //=> SINCE my server is db svr, deadline is my best option;