php - 请求页面返回 403 Bad Behavior

Question

我写了一个小脚本来验证 url 是否存在。我正在使用 get_headers 来检索标题。问题是对于某些 url，例如这个:https://forum.obviousidea.com响应是 403 Bad Behavior，而如果我用浏览器打开页面，它就可以正常工作。

示例输出:

$headers = get_headers(https://forum.obviousidea.com);
print_r($headers);

(
    [0] => HTTP/1.1 403 Bad Behavior
    [Server] => nginx/1.6.2
    [Date] => Tue, 04 Jun 2019 21:56:27 GMT
    [Content-Type] => text/html; charset=ISO-8859-1
    [Content-Length] => 913
    [Connection] => close
    [Set-Cookie] => Array
        (
            [0] => bb_lastvisit=1559685385; expires=Wed, 03-Jun-2020 21:56:25 GMT; Max-Age=31536000; path=/; secure
            [1] => bb_lastactivity=0; expires=Wed, 03-Jun-2020 21:56:25 GMT; Max-Age=31536000; path=/; secure
            [2] => PHPSESSID=cqtkdcfpm0k2s8hl4cup6epa37; path=/
        )

    [Expires] => Thu, 19 Nov 1981 08:52:00 GMT
    [Cache-Control] => private
    [Pragma] => private
    [Status] => 403 Bad Behavior
)

如何使用 get_headers 获取正确的状态代码？

请注意，现在该网站可以使用答案中建议的用户代理。

但是例如这个 url 仍然不起作用:https://filezilla-project.org/download.php?type=client

Answer 1

您可能已经在 php.ini 中或通过 ini_set 更改了 UserAgent header

检查它或像下面的示例一样设置 UserAgent

ini_set('user_agent', '');
$headers = get_headers('https://forum.obviousidea.com');

我更喜欢使用 bellow curl 函数:

 /**
 * @param string $url
 * @param array  $headers
 * @return array
 * @throws Exception
 */
function curlGetHeaders(string $url, array $headers = [])
{
    $url = trim($url);
    if (!filter_var($url, FILTER_VALIDATE_URL)) {
        throw new Exception("$url is not a valid URL", 422);
    }
    $url  = explode('?', $url);
    $curl = curl_init();
    curl_setopt_array($curl, [
        CURLOPT_URL            => $url[0],
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_NOBODY         => true,
        CURLOPT_HEADER         => true,
        CURLOPT_HTTP_VERSION   => CURL_HTTP_VERSION_1_1,
        CURLOPT_CUSTOMREQUEST  => "GET",
    ]);

    if (isset($url[1])) {
        curl_setopt($curl, CURLOPT_POSTFIELDS, $url[0]);
    }

    if (!empty($headers)) {
        foreach($headers as $key => $header) {
            $curlHeaders[] = "$key:$header";
        }
        curl_setopt($curl, CURLOPT_HTTPHEADER, $curlHeaders);
    }


    $response     = rtrim(curl_exec($curl));
    $responseCode = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
    curl_error($curl);
    curl_close($curl);
    $headers                  = [];
    $data                     = explode("\r\n", $response);
    $headers['Response-Code'] = $responseCode;
    $headers['Status']        = $data[0];
    array_shift($data);
    foreach($data as $part) {
        $middle = explode(":", $part, 2);
        if (!isset($middle[1])) {
            $middle[1] = null;
        }
        $headers[trim($middle[0])] = trim($middle[1]);
    }

    return $headers;
}