gpt4 book ai didi

java - FileChannel 和 ByteBuffer 写入额外数据

转载 作者:太空宇宙 更新时间:2023-11-04 09:31:53 52 4
gpt4 key购买 nike

我正在创建一个方法,该方法将接收一个文件并将其拆分为 shardCount block 并生成奇偶校验文件。

当我运行此方法时,似乎我正在将额外的数据写入奇偶校验文件中。这是我第一次使用 FileChannel 和 ByteBuffers,所以尽管我盯着文档看了大约 8 个小时,但我不确定我是否完全理解如何使用它们。

此代码是奇偶校验部分的简化版本。

public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
RandomAccessFile rin = new RandomAccessFile(file, "r");
FileChannel fcin = rin.getChannel();

//Create parity files
File parity = new File(fileID + "_parity");
if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
FileChannel parityOut = parityRAF.getChannel();

long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);

//Make buffers for each section of the file we will be reading from
for (int i = 0; i < shardCount; i++) {
ByteBuffer bb = ByteBuffer.allocate(1024);
shardBuffers.add(bb);
}

ByteBuffer parityBuffer = ByteBuffer.allocate(1024);

//Generate parity
boolean isParityBufferEmpty = true;
for (long i = 0; i < bytesPerFile; i++) {
isParityBufferEmpty = false;
int pos = (int) (i % 1024);
byte p = 0;

if (pos == 0) {
//Read chunk of file into each buffer
for (int j = 0; j < shardCount; j++) {
ByteBuffer bb = shardBuffers.get(j);
bb.clear();
fcin.read(bb, bytesPerFile * j + i);
bb.rewind();
}
//Dump parity buffer
if (i > 0) {
parityBuffer.rewind();
parityOut.write(parityBuffer);
parityBuffer.clear();
isParityBufferEmpty = true;
}
}

//Get parity
for (ByteBuffer bb : shardBuffers) {
if (pos >= bb.limit()) break;
p ^= bb.get(pos);
}

//Put parity in buffer
parityBuffer.put(pos, p);
}

if (!isParityBufferEmpty) {
parityBuffer.rewind();
parityOut.write(parityBuffer);
parityBuffer.clear();
}

fcin.close();
rin.close();
parityOut.close();
parityRAF.close();
}

如果奇偶校验算法或文件 IO 有任何问题,或者我可以做些什么来优化它,请告诉我。我很高兴听到其他(更好)的文件 IO 方法。

最佳答案

这是我找到的解决方案(尽管可能需要更多调整):

public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
int BUFFER_SIZE = 4 * 1024 * 1024;
RandomAccessFile rin = new RandomAccessFile(file, "r");
FileChannel fcin = rin.getChannel();

//Create parity files
File parity = new File(fileID + "_parity");
if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
FileChannel parityOut = parityRAF.getChannel();

//Create shard files
ArrayList<File> shards = new ArrayList<>(shardCount);
for (int i = 0; i < shardCount; i++) {
File f = new File(fileID + "_part_" + i);
if (f.exists()) throw new FileAlreadyExistsException("Could not create shard file! File already exists!");
shards.add(f);
}

long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);

ArrayList<ByteBuffer> shardBuffers = new ArrayList<>(shardCount);

//Make buffers for each section of the file we will be reading from
for (int i = 0; i < shardCount; i++) {
ByteBuffer bb = ByteBuffer.allocate(BUFFER_SIZE);
shardBuffers.add(bb);
}

ByteBuffer parityBuffer = ByteBuffer.allocate(BUFFER_SIZE);

//Generate parity
boolean isParityBufferEmpty = true;
for (long i = 0; i < bytesPerFile; i++) {
isParityBufferEmpty = false;
int pos = (int) (i % BUFFER_SIZE);
byte p = 0;

if (pos == 0) {
//Read chunk of file into each buffer
for (int j = 0; j < shardCount; j++) {
ByteBuffer bb = shardBuffers.get(j);
bb.clear();
fcin.position(bytesPerFile * j + i);
fcin.read(bb);
bb.flip();
}

//Dump parity buffer
if (i > 0) {
parityBuffer.flip();
while (parityBuffer.hasRemaining()) {
parityOut.write(parityBuffer);
}
parityBuffer.clear();
isParityBufferEmpty = true;
}
}

//Get parity
for (ByteBuffer bb : shardBuffers) {
if (!bb.hasRemaining()) break;
p ^= bb.get();
}

//Put parity in buffer
parityBuffer.put(p);
}

if (!isParityBufferEmpty) {
parityBuffer.flip();
parityOut.write(parityBuffer);
parityBuffer.clear();
}

fcin.close();
rin.close();
parityOut.close();
parityRAF.close();
}

按照 VGR 的建议,我用 flip() 替换了 rewind()。我还切换到相对操作而不是绝对操作。我不认为绝对方法会调整光标位置或限制,因此这可能是错误的原因。我还将缓冲区大小更改为 4MB,因为我有兴趣为大文件生成奇偶校验。

关于java - FileChannel 和 ByteBuffer 写入额外数据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/57020609/

52 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com