java - 如何在很短的时间内检查数千个节点的 url 是否有效？

Question

我需要检查特定网址“http://16.100.106.4/xmldata?item=all”是否有效？现在的事情是，如果 url 不起作用，那么连接超时会等待大约 20 秒，然后才会给出异常，我使用下面的代码。现在，我必须检查大约 20000 个 IP 的 URL，我不能等待那么长时间。线程可以是一种选择，但我也不确定我应该使用多少线程。我希望整个操作在几秒钟内完成。

public static boolean exists(String URLName){
     boolean available = false;

          try{
                final  URLConnection connection = (URLConnection) new URL(URLName).openConnection();
                connection.connect();

                System.out.println("Service " + URLName + " available, yeah!");
                available = true;
            } catch(final MalformedURLException e){
                throw new IllegalStateException("Bad URL: " + available, e);
            } catch(final Exception e){
                // System.out.print("Service " + available + " unavailable, oh no!", e);
                available = false;
            }
            return available;
  } 

Answer 1

我会像这样解决生产者/消费者问题:

public class URLValidator {

  private final CompletionService<URLValidationResult> service;

  public URLValidator(ExecutorService exec) {
    this.service = new CompletionService<URLValidationResult>(exec);
  }

  // submits a url for validation
  public void submit(String url) {
     service.submit(new Callable<URLValidationResult>() {
         public URLValidationResult call() {
           return validate(url);
         }          
     });
  }

  // retrieves next available result. this method blocks
  // if no results are available and is responsive to interruption.
  public Future<URLValidationResult> next() {
     try {
       return service.take();
     } catch (InterruptedException e) {
       Thread.currentThread().interrupt();
     }
  }

  private URLValidationResult validate(String url) {
      // Apply your url validation logic here (i.e open a timed connection
      // to the url using setConnectTimeout(millis)) and return a 
      // URLValidationResult instance encapsulating the url and
      // validation status
  }

}

需要提交 url 进行验证的线程将使用 submit(String url) 方法。该方法将依次向完成服务提交异步执行的任务。处理验证结果的线程将使用 next() 方法。该方法返回代表已提交任务的 Future。您可以按如下方式处理返回的 future:

URLValidator validator = // the validator instance....

// retrieve the next available result
Future<URLValidationResult> future = validator.next();

URLValidationResult result = null;
try {
  result = future.get();
} catch (ExecutionException e) {
  // you submitted validation task has thrown an error,
  // handle it here.
}

// do something useful with the result
process(result);