string - 算法分析 URL 列表和黑名单 URL 与黑名单词

Question

假设文本文件中有一个 URL 列表(以百万为单位)，文本文件中还有另一个包含黑名单单词的列表。

我愿意对 URL 列表进行如下处理。

- Parse the URLs and store them in some DS
- Process the URLs and blacklist those URLs which contain atleast one of the 
  blacklisted words.
- If there exists a URL containing 50% or more blacklisted words, add the other 
  words of that URL in the list of blacklisted words.
- Since now the blacklisted words list has been modified then it's probable 
  that the URLs which were not blacklisted earlier can get blacklisted now. So, 
  the algorithm should handle this case as well and mark the earlier whitelisted 
  URLs as blacklisted if they contain these newly added blacklisted words.

最后我应该有一个列入白名单的 URL 列表

有什么建议可以用来实现最有效的时间和空间复杂度解决方案的最佳算法和 DS？

Answer 1

使用矩阵来存储 URL。

1. 首先，将每个 URL 按 Porter Stemmer 拆分为单词，并将它们放入矩阵中(一行对应一个 URL，一项对应一个词)。

2. 然后使用TFIDF对矩阵中的每个词进行评分，并删除低分词(它们将是像“a”、“the”等流行词，它们对判断垃圾邮件没有信息)。

3. 手动初始化黑名单(放入一些常用的黑词)。

4. 按照您给定的方式运行流程。