java - 无法使用 POST 请求检索第 2 页

Question

我正在尝试检索 this 的第 2 页的 HTML网址。我已经添加了所需的帖子表单数据的值，如 __EVENTTARGET 和 __EVENTARGUMENT，但这仍然只返回第一页。知道我可能遗漏了什么吗？

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.LinkedHashMap;
import java.util.Map;

public class HttpURLConnectionExample {
public static void main(String[] args) throws Exception {
    URL url = new URL(
            "http://www.themetaldirectory.com/?featured=0&country=USA");
    Map<String, Object> params = new LinkedHashMap<>();
    params.put("__EVENTTARGET", "ctl00%24ctl00%24ContentPlaceHolderDefault%24DirectoryDisplay_12%24GridView1");
    params.put("__EVENTARGUMENT", "Page$2");

    StringBuilder postData = new StringBuilder();
    for (Map.Entry<String, Object> param : params.entrySet()) {
        if (postData.length() != 0)
            postData.append('&');
        postData.append(URLEncoder.encode(param.getKey(), "UTF-8"));
        postData.append('=');
        postData.append(URLEncoder.encode(String.valueOf(param.getValue()),
                "UTF-8"));
    }
    byte[] postDataBytes = postData.toString().getBytes("UTF-8");

    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    conn.setRequestMethod("POST");
    conn.setRequestProperty("Content-Type",
            "application/x-www-form-urlencoded");
    conn.setRequestProperty("Content-Length",
            String.valueOf(postDataBytes.length));
    conn.setDoOutput(true);
    conn.getOutputStream().write(postDataBytes);

    Reader in = new BufferedReader(new InputStreamReader(
            conn.getInputStream(), "UTF-8"));
    for (int c; (c = in.read()) >= 0; System.out.print((char) c))
        ;
}

Answer 1

您为 __EVENTTARGET 参数发送了错误的值。

期望值为

ctl00$ctl00$ContentPlaceHolderDefault$DirectoryDisplay_12$GridView1

但是您发送的值作为 URL 编码了两次，第一次是在您设置参数值时

params.put("__EVENTTARGET",
"ctl00%24ctl00%24ContentPlaceHolderDefault%24DirectoryDisplay_12%24GridView1");

第二个，当你编码参数值在

postData.append(URLEncoder.encode(String.valueOf(param.getValue()), "UTF-8"));

所以你真的发送了一个双重 URL 编码值

ctl00%2524ctl00%2524ContentPlaceHolderDefault%2524DirectoryDisplay_12%2524GridView1

首先设置不带 URL 编码的参数值，应该可以工作

 params.put("__EVENTTARGET",
 "ctl00$ctl00$ContentPlaceHolderDefault$DirectoryDisplay_12$GridView1");

编辑

如果你想将结果 InputStream 保存到一个字符串，你有很多可能的方法，其中之一如下:

StringBuilder inputStringBuilder = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));

String line;
while ((line = br.readLine()) != null) {
    inputStringBuilder.append(line);
}
String htmlString = inputStringBuilder.toString();