java - 使用 Web 驱动程序 Selenium 和 JSoup 进行分页

Question

我正在开发一个应用程序，它使用 JSoup 从网站获取数据。我能够得到正常的数据。

但现在我需要对其实现分页。有人告诉我它必须与 Web 驱动程序、Selenium 一起使用。但我不知道如何与他合作，有人可以告诉我该怎么做吗？

public class MainActivity extends AppCompatActivity {

   private String url = "http://www.yudiz.com/blog/";
   private ArrayList<String> mAuthorNameList = new ArrayList<>();
   private ArrayList<String> mBlogUploadDateList = new ArrayList<>();
   private ArrayList<String> mBlogTitleList = new ArrayList<>();

   @Override
   protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.activity_main);
       new Description().execute();

   }

   private class Description extends AsyncTask<Void, Void, Void> {

       @Override
       protected Void doInBackground(Void... params) {
           try {
               // Connect to the web site
               Document mBlogDocument = Jsoup.connect(url).get();
               // Using Elements to get the Meta data
               Elements mElementDataSize = mBlogDocument.select("div[class=author-date]");
               // Locate the content attribute
               int mElementSize = mElementDataSize.size();

               for (int i = 0; i < mElementSize; i++) {
                   Elements mElementAuthorName = mBlogDocument.select("span[class=vcard author post-author test]").select("a").eq(i);
                   String mAuthorName = mElementAuthorName.text();

                   Elements mElementBlogUploadDate = mBlogDocument.select("span[class=post-date updated]").eq(i);
                   String mBlogUploadDate = mElementBlogUploadDate.text();

                   Elements mElementBlogTitle = mBlogDocument.select("h2[class=entry-title]").select("a").eq(i);
                   String mBlogTitle = mElementBlogTitle.text();

                   mAuthorNameList.add(mAuthorName);
                   mBlogUploadDateList.add(mBlogUploadDate);
                   mBlogTitleList.add(mBlogTitle);
               }
           } catch (IOException e) {
               e.printStackTrace();
           }
           return null;
       }

       @Override
       protected void onPostExecute(Void result) {
           // Set description into TextView

           RecyclerView mRecyclerView = (RecyclerView)findViewById(R.id.act_recyclerview);

           DataAdapter mDataAdapter = new DataAdapter(MainActivity.this, mBlogTitleList, mAuthorNameList, mBlogUploadDateList);
           RecyclerView.LayoutManager mLayoutManager = new LinearLayoutManager(getApplicationContext());
           mRecyclerView.setLayoutManager(mLayoutManager);
           mRecyclerView.setAdapter(mDataAdapter);

       }
   }
}

Answer 1

问题陈述(根据我的理解):Scraper 应该能够转到下一页，直到使用博客页面末尾可用的分页选项完成所有页面。

现在，如果我们检查分页中的下一个按钮，我们可以看到以下 html。 a class="next_page"href="http://www.yudiz.com/blog/page/2/"

现在我们需要指示 Jsoup 在下一次循环迭代中获取这个动态 url 来废弃数据。这可以使用以下方法来完成:

        String url = "http://www.yudiz.com/blog/";
        while (url!=null){
            try {
                Document doc = Jsoup.connect(url).get();
                url = null;
                System.out.println(doc.getElementsByTag("title").text());
                for (Element urls : doc.getElementsByClass("next_page")){
                    //perform your data extractions here.
                    url = urls != null ? urls.absUrl("href") : null;
                }               
            } catch (IOException e) {
                e.printStackTrace();
            }
        }