java - 根据位置过滤 Shodan 查询结果

Question

我正在尝试解析 Shodan 查询结果并编写一个新的 JSON 文件，其中仅包含与我设置的条件相匹配的结果。

JSON 条目示例:

{
  "matches": [
    {
      "product": "Microsoft IIS httpd",
      "hostnames": [],
      "hash": -1722221328,
      "ip": 1261462342,
      "isp": "AT&T Internet Services",
      "transport": "tcp",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5",
        "cpe:/o:microsoft:windows"
      ],
      "data": "",
      "asn": "AS7018",
      "port": 631,
      "version": "7.5",
      "link": "Ethernet or modem",
      "location": {
        "city": null,
        "region_code": null,
        "area_code": null,
        "longitude": -97.822,
        "country_code3": "USA",
        "latitude": 37.751000000000005,
        "postal_code": null,
        "dma_code": null,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:55:24.306344",
      "domains": [],
      "org": "AT&T Internet Services",
      "os": null,
      "_shodan": {
        "crawler": "122dd688b363c3b45b0e7582622da1e725444808",
        "id": null,
        "module": "http-simple-new",
        "options": {}
      },
      "ip_str": "75.48.99.70"
    },
    {
      "hash": 605323305,
      "ip": 1757819678,
      "isp": "Google Cloud",
      "transport": "tcp",
      "data": "",
      "asn": "AS15169",
      "port": 9000,
      "hostnames": [
        "30.51.198.104.bc.googleusercontent.com"
      ],
      "location": {
        "city": "Mountain View",
        "region_code": "CA",
        "area_code": 650,
        "longitude": -122.0574,
        "country_code3": "USA",
        "latitude": 37.41919999999999,
        "postal_code": "94043",
        "dma_code": 807,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:51:35.997036",
      "domains": [
        "googleusercontent.com"
      ],
      "org": "Google Cloud",
      "os": null,
      "_shodan": {
        "crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
        "id": null,
        "module": "https-simple-new",
        "options": {}
      },
      "ip_str": "104.198.51.30"
    }
  ],
  "total": 2
}

我的希望是加载 JSON 文件并迭代元素集，如果元素不符合位置国家/地区代码为“US”的条件，则删除该元素。

我的代码(由 https://gist.github.com/madonnelly 和 Iterate over JsonObject properties 提供)如下:

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;

import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

public class ParseJSON {

    public static void main(String[] args) {
        JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");

        Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();

        for (Map.Entry<String, JsonElement> queryResult : queryResults) {
            JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
            for (JsonElement locData : locArray) {
                if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
                    System.out.println(locData.getAsString());
                }
            }
        }
    }

    public static JsonObject convertFileToJSON(String fileName) {

        // Read from File to String
        JsonObject jsonObject = new JsonObject();

        try {
            JsonParser parser = new JsonParser();
            JsonElement jsonElement = parser.parse(new FileReader(fileName));
            jsonObject = jsonElement.getAsJsonObject();
        } catch (FileNotFoundException e) {

        }
        return jsonObject;
    }
}

当我运行代码时，我收到错误

Exception in thread "main" java.lang.IllegalStateException: Not a JSON Object: [{"product":"Microsoft IIS httpd","hostnames":[],"hash":-1722221328,"ip":1261462342,"isp":"AT&T Internet Services","transport":...}] at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:90) at com.cti.shodan.ParseJSON.main(ParseJSON.java:22)

我确信我犯了很多错误，希望有人能指出我所犯的错误。提前致谢!

Answer 1

您对具体的 JSON 文档解析有一些不符合其真实结构的假设。我假设您将显示过滤掉其子属性值 ($.matches.*.location.country_code) 的匹配结果 ($.matches)。

    for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
        final JsonElement value = queryResult.getValue();
        // This is necessary to skip <"total": 2>
        if ( value.isJsonArray() ) {
            // Here comes an array, and should be iterated, rather than taken as an object
            for ( final JsonElement match : value.getAsJsonArray() ) {
                // This was the root cause, not an array
                final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
                // Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
                if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
                    // Previously getAsString() -- it requires a JSON string literal, just remove it
                    System.out.println(match);
                }
            }
        }
    }

使用 Java 8 可能会更简单:

shodanJsonObject.entrySet()
        .stream()
        .map(Entry::getValue)
        .filter(JsonElement::isJsonArray)
        .map(JsonElement::getAsJsonArray)
        .flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
        .peek(System.out::println)
        .map(JsonElement::getAsJsonObject)
        .map(jsonObject -> jsonObject.getAsJsonObject("location"))
        .filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
        .forEach(jsonObject -> {
        }); // forEach is a terminal operation and it "pushes" the entire chain above

如果可以使用像 JsonPath 这样的查询库，这可能是最具表现力的方式:

final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
    System.out.println(match);
}