gpt4 book ai didi

java - 使用 Hive UDF 解压缩列数据

转载 作者:太空宇宙 更新时间:2023-11-04 11:28:41 26 4
gpt4 key购买 nike

上下文:使用Hive UDF的evaluate()方法解压列数据

异常(exception):

Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public static org.apache.hadoop.io.Text Test.UDFDecompressor.evaluate(java.lang.String) throws org.apache.hadoop.hive.ql.metadata.HiveException on object Test.UDFDecompressor@1008df1e of class Test.UDFDecompressor with arguments {x��}kw⸲�_a�����֤�\��a-B�i�@`�����"�nc3�I����$_�E�� } of size 1

源代码:

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.zip.DataFormatException;
import java.util.zip.InflaterInputStream;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;

public class Decompress extends UDF{
public static String evaluate(String data1) throws IOException, DataFormatException{
ByteArrayInputStream bao=new ByteArrayInputStream(data1.getBytes());
InflaterInputStream iis= new InflaterInputStream(bao);
String out="";
byte[] bt=new byte[1024];
int len=-1;
while ((len =iis.read(bt))!=-1){
out += new String(Arrays.copyOf(bt, len));
}
JavaStringObjectInspector stringInspector;
stringInspector = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
String ip = stringInspector.getPrimitiveJavaObject(out);

//return new String(ip.getBytes(Charset.forName("UTF-8")));
//return new String(ip.getBytes(Charset.forName("UTF-8")));
return ip;
}
}

我尝试了多种使用 gZib、zLIb Java Api 解压缩的方法,但遇到了相同的错误。任何人都可以帮我解决这个问题,并建议使用 Hive UDF 解压缩列数据的正确方法

提前致谢。

最佳答案

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.InflaterInputStream;

public class Decompress extends UDF {

private final Text r = new Text();

public Text evaluate(BytesWritable bw) throws IOException {
ByteArrayInputStream zipped = new ByteArrayInputStream(bw.getBytes());
InflaterInputStream inflater = new InflaterInputStream(zipped);
ByteArrayOutputStream unzipped = new ByteArrayOutputStream();
byte[] bt = new byte[1024];
int len;
while ((len = inflater.read(bt)) != -1) {
unzipped.write(bt, 0, len);
}

r.clear();
r.set(unzipped.toByteArray());
return r;
}
}

关于java - 使用 Hive UDF 解压缩列数据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44061561/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com