java - 下载文件时内存不足

Question

我正在尝试在 android 中下载 CSV 文件。下载后，我将使用 Jackson 进行映射，然后保存到 sqlite3 数据库。我面临的问题是，当我进行下载文件的调用时，我收到 OutOfMemoryError。文件大小约为 12MB。当我在浏览器中尝试相同的操作时，我能够下载该文件，因此我认为这不是端点的问题。

为了下载，我使用 AsyncTask。

class DownloadCSVTask extends AsyncTask<Void, String, Void> {

    private final Call<ResponseBody> csvDump;
    private final Context context;

    public DownloadCSVTask(Call<ResponseBody> csvDump, Context context) {
        this.csvDump = csvDump;
        this.context = context;
    }

    @Override
    protected Void doInBackground(Void... voids) {
        try {
            Response<ResponseBody> response = csvDump.execute();
            File fileFromResponse = createFileFromResponse(response);
            mapFileContentsToModelAndPersist(fileFromResponse);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "CSV pull failed");
        }
        return null;
    }

    private File createFileFromResponse(Response<ResponseBody> response) throws IOException {
        File path = context.getFilesDir();
        File csvFile = File.createTempFile(TEMP_PREFIX, CSV_EXTENSION, path);
        FileOutputStream fileOutputStream = context.openFileOutput(csvFile.getName(), Context.MODE_PRIVATE);
        fileOutputStream.write(response.body().bytes());
        Log.i(getClass().getCanonicalName(), "Writing file completed" + csvFile.getAbsolutePath());
        fileOutputStream.close();
        return csvFile;
    }

    private void mapFileContentsToModelAndPersist(File fileFromResponse) {
        try {
            MappingIterator<City> cityIter = new CsvMapper().readerWithTypedSchemaFor(City.class).readValues(fileFromResponse);
            List<City> cities = cityIter.readAll();
            Stream.of(cities).forEach(City::persist);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "Failed to map contents of file to City Class");
        }
    }
}

我收到的错误日志:

 java.lang.RuntimeException: An error occured while executing doInBackground()
                         E      at android.os.AsyncTask$3.done(AsyncTask.java:300)
                         E      at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
                         E      at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:242)
                         E      at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
                         E      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                         E      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                         E      at java.lang.Thread.run(Thread.java:841)
                         E  Caused by: java.lang.OutOfMemoryError
                         E      at java.lang.String.<init>(String.java:354)
                         E      at java.lang.String.<init>(String.java:393)
                         E      at okio.Buffer.readString(Buffer.java:616)
                         E      at okio.Buffer.readString(Buffer.java:599)
                         E      at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:263)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:42)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.NetworkInterceptor.intercept(NetworkInterceptor.java:23)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
                         E      at okhttp3.RealCall.execute(RealCall.java:60)
                         E      at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
                         E      at retrofit2.ExecutorCallAdapterFactory$ExecutorCallbackCall.execute(ExecutorCallAdapterFactory.java:89)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:60)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:47)
                         E      at android.os.AsyncTask$2.call(AsyncTask.java:288)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:237)

我在这里做错了什么吗？还有其他方法吗？

Answer 1

Android 上 Java 代码的内存量限制为 16 Mb 左右(因设备而异)。因此，您要么需要将此部分移至 native NDK 代码，在其中您可以 malloc() 与设备拥有的内存一样多的内存，要么将文件直接保存到磁盘，然后处理它，而不将整个文件加载到 Java 缓冲区中。在具有 16Kb 字节数组的循环中使用 URLConnection.getInputStream() 或 URLConnection.read()，而不是将整个文件复制到 response 数组中。或者，您可以使用 BufferedReader.readLine() 直接从 HTTP 流中逐行读取它，并逐行填充数据库。