java - 安卓并行(simultaneous)图片下载

Question

请大家帮忙，这很重要。

我需要在不到 5 分钟的时间内下载大约 2000 张图像。所以我决定进行并行图片下载。

如果有人可以建议我进行并行下载的更好方法或告诉我哪里出错了，我将不胜感激。

在错误日志中，已完成任务的数量总是不同的...

请帮忙。

我正在使用 Artem Zinnatullin 的异步任务执行器

public class AsyncTaskExecutor {

private static final int CORE_POOL_SIZE;
private static final int MAXIMUM_POOL_SIZE;
private static final int KEEP_ALIVE;
private static final TimeUnit TIME_UNIT;

private static final BlockingQueue<Runnable> concurrentPoolWorkQueue;
private static final ThreadFactory concurrentThreadFactory;
    private static final ThreadPoolExecutor concurrentExecutor;

private AsyncTaskExecutor() {}

static {
 CORE_POOL_SIZE = 5;
 MAXIMUM_POOL_SIZE = 128;
 KEEP_ALIVE = 1;
 TIME_UNIT = TimeUnit.SECONDS;

 concurrentPoolWorkQueue = new LinkedBlockingQueue<Runnable>(10);
 concurrentThreadFactory = new AsyncTaskThreadFactory();
 concurrentExecutor = new ThreadPoolExecutor(CORE_POOL_SIZE,MAXIMUM_POOL_SIZE, KEEP_ALIVE, TIME_UNIT, concurrentPoolWorkQueue, concurrentThreadFactory);
} 
/**
* Concurrently executes AsyncTask on any Android version
* @param task to execute
* @param params for task
* @return executing AsyncTask
*/
@SuppressLint("NewApi")
public static <Params, Progress, Result> AsyncTask<Params, Progress, Result>
 executeConcurrently(AsyncTask<Params, Progress, Result> task,
 Params... params) {
 if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
 task.executeOnExecutor(concurrentExecutor, params);
 } else {
 task.execute(params);
 }
 return task;
}

    /**
* Thread factory for AsyncTaskExecutor
* @author Artem Zinnatullin
*
*/
private static class AsyncTaskThreadFactory implements ThreadFactory {
 private final AtomicInteger count;

 {
 count = new AtomicInteger(1);
 }

@Override
public Thread newThread(Runnable r) {
return new Thread(r, "AsyncTask #" + count.getAndIncrement());
}
}

然后我使用循环下载我所有的图像。

for (PhotoToDBHelper photoToDBHelper : notDownloaded) {
            try {
                photoToDBHelper.download(m_context);
            } catch (Exception e) {
                // TODO: handle exception
            }

        }

下载方法如下所示:

public void download(Context context) throws Exception {
    HttpClient httpClient = new DefaultHttpClient();
    HttpGet httpGet = new HttpGet(new URL(getPhotosUrl() + m_fileName).toString());
    HttpResponse response = httpClient.execute(httpGet);
    HttpEntity entity = response.getEntity();

    InputStream inputStream = entity.getContent();
    filePath = Environment.getExternalStorageDirectory().toString() + getFoldeRelativePath() + m_fileName;

    Utils.createFileOnExternalStorage(getFoldeRelativePath(), filePath);
    FileOutputStream fos = new FileOutputStream(filePath, false);

    byte[] buffer = new byte[20000];
    int byteRead = 0;

    while ((byteRead = inputStream.read(buffer)) != -1) {
        fos.write(buffer, 0, byteRead);
    }
    fos.close();
}

当启动我的 android 应用程序时，我得到了这个错误

02-25 01:08:30.133: E/AndroidRuntime(32357): FATAL EXCEPTION: main
02-25 01:08:30.133: E/AndroidRuntime(32357): java.lang.RuntimeException: Unable to start activity ComponentInfo{"package".TabHostActivity}: java.util.concurrent.RejectedExecutionException: Task android.os.AsyncTask$3@2c0809b0 rejected from java.util.concurrent.ThreadPoolExecutor@2be18f48[Running, pool size = 128, active threads = 1, queued tasks = 10, completed tasks = 27]
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1967)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1992)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.access$600(ActivityThread.java:127)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1158)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.Handler.dispatchMessage(Handler.java:99)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.Looper.loop(Looper.java:137)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.main(ActivityThread.java:4441)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.lang.reflect.Method.invokeNative(Native Method)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.lang.reflect.Method.invoke(Method.java:511)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at dalvik.system.NativeStart.main(Native Method)
02-25 01:08:30.133: E/AndroidRuntime(32357): Caused by: java.util.concurrent.RejectedExecutionException: Task android.os.AsyncTask$3@2c0809b0 rejected from java.util.concurrent.ThreadPoolExecutor@2be18f48[Running, pool size = 128, active threads = 1, queued tasks = 10, completed tasks = 1032]
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1967)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:782)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1303)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.AsyncTask.executeOnExecutor(AsyncTask.java:564)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".data.AsyncTaskExecutor.executeConcurrently(AsyncTaskExecutor.java:58)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".TabHostActivity.checkUpdateState(TabHostActivity.java:130)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".TabHostActivity.onCreate(TabHostActivity.java:88)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.Activity.performCreate(Activity.java:4465)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1049)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1931)
02-25 01:08:30.133: E/AndroidRuntime(32357):    ... 11 more

Answer 1

有几个问题需要解决:

1. 5 分钟内 2000 张图像大约是每秒 7 张图像。根据图像大小、网络带宽、处理器和存储速度等...这可能对任何代码或并发量都不可行。吞吐量的数学公式为:7 * N 字节每秒，其中 N 是平均图像大小。如果有大图像，则可能是死点。
2. 异常表明线程池已达到排队任务的最大数量 (10)。这可以基于线程池管理器实现进行配置。总体而言，尝试同时启动 2000 个线程可能会加剧这种情况。建议分组排队任务(使用线程)，监控它们的完成情况，然后开始更多任务，所有任务都分成小组，每次五 (5) 个。
3. 如果有任何服务器控件，请考虑使用 sprite 模型，其中多个图像存储在一个图像(文件/资源​​)中，按图像内的相对 x/y 坐标分解。这会将 HTTP 或任何网络请求的数量从 2000 减少到 1(可能)，从而显着提高性能。