php - 是否可以在 php 循环中重复 file_get_html()？

Question

我在使用一个相当简单的抓取程序时遇到了一些问题。希望这些工作部分可以帮助感兴趣的人建立新闻聚合器。我喜欢获取特定的内容，所以我发现抓取内容并按关键字过滤可以帮助我避免关于与我生活无关的事情的头条新闻这一恼人的问题。

如果您能提供任何帮助或建议，我将不胜感激。这个问题困扰了我几天，我还没有找到好的解决方案。

基本上，该程序从一个表中获取一组项目以移动到另一个表，在此过程中应该从当前正在处理的表 A 中获取表条目的 URL 组件，并使用 include 调用另一个程序.程序 B 使用它从该条目中获取内容并将其保存到一个文件中，该文件是根据表 A 中当前项目的其他属性命名的。

然后该文件被保存到一个文件夹中，程序 B 中的 dom 变量被清除，以便循环可以继续。

这是程序A:

<?php
$db = 'newsfeed';
$zeta = 0;
$beta = 0;
// connect to RDS instance MySQL Database Newsfeed
include_once('/var/www/dbfunctions/mysqli_connectdb.php');

// set content source table 
$sourcetable = 'feedsources';
$mastertable = 'mastertable';

// set date to remove results older than
date_default_timezone_set("UTC");
$datenow = date_timestamp_get(date_create());
$offset = "86400";
$deldate = $datenow - $offset;

//begin cycling through content data

//delete all "old" entries from the mastertable

//get number of source items present
$itemquery = "SELECT id,name FROM $sourcetable";
$itemresult = mysqli_query($conn, $itemquery);

while ($row = mysqli_fetch_assoc($itemresult)) {
$sourceid = $row['id'];
$sourcename = $row['name'];

        // cycle throught the data tables 
        $dataquery = "SELECT * FROM $sourcetable WHERE id = $sourceid;";

        $dataresult = mysqli_query($conn, $dataquery);

        while ($row = mysqli_fetch_assoc($dataresult)) {
        $table = $row['datatable'];
        }

        // copy all data from the targetted table into the master table 

        //loop through the targetted table and copy to mysql
        $getdata = "SELECT * FROM ".$table.";";
     $datareturn = mysqli_query($conn, $getdata);

        while ($row = mysqli_fetch_assoc($datareturn)) {
                $date = $row['datecreated'];
                $title = addslashes($row['title']);
                $url = addslashes($row['url']);
                $tags = addslashes($row['tags']);
                $titleid = $row['id'];

        //get content and place in html file in /var/www/html/nuzr/content/



                //check whether the item already exists in the table
                $checkquery = "select id from ".$mastertable." where title = '".$title."';";

                $checkcheck = mysqli_query($conn, $checkquery);

                if(mysqli_num_rows($checkcheck) > 0){

                }else{

        require_once("getcontent.php");

                $copy = "INSERT INTO ".$mastertable." VALUES ('NULL','$table','$sourcename','$date','$title','$url','$tags','$filename');";
                mysqli_query($conn, $copy);

                echo "Beta is ".$beta;
                $beta = $beta + 1;
                        }
                }
        // clean the master table
        $delquery = 'DELETE FROM '.$mastertable.' WHERE datecreated < '.$deldate.';';

        mysqli_query($conn, $delquery);

}

function clear()
{
    $this->dom = null;
    $this->parent = null;
    $this->parent = null;
    $this->children = null;
}

?>

程序B

<?php
//Check Start
//echo "Program Starts";

// Include the library
include('/var/www/tools/dom/simple_html_dom.php');

$source = $url;
$content = array();
$header1 = array();
$header2 = array();
$i = 0; $y = 0;

// Retrieve the DOM from a given URL
$html = file_get_html($source);

        //grab headers in case initial title is a header
        foreach($html->find('h1') as $e){

                $header1[$i] = $e->outertext;

                //echo $e->outertext;

                $i = $i + 1;
        }

        $i = 0;

        foreach($html->find('h2') as $e){

                $header2[$i] = $e->outertext;

                //echo $e->outertext;

                $i = $i + 1;
        }

        //reset counter
        $i = 0;

        // Find all paragraph tags and print their content into a text file
        foreach($html->find('p') as $e){

                $content[$i] = $e->outertext;

                //echo $e->outertext;

                $i = $i + 1;
        }

//create the content storage file
$filename = "/var/www/html/nuzr/content/".$table.$titleid.".html";
echo "The filename is".$filename;
        $file = fopen($filename,"a");

// write header and link to original article
$titleblurb = "<b>Original article courtesy of <a href='".$url."'>".$sourcename."</a></b>";
fwrite($file, $titleblurb);

// set site specific parameters based on header / footer size
if($sourcename == "The Globe and Mail"){

//Set indexing parameters
$z = $i - 13; $y = 2;

        //Add Header content
        $text = $header1[0];
        fwrite($file, $text);
        $text = $header2[1];
        fwrite($file, $text);

}elseif($sourcename == "CNN Money"){

//Set indexing parameters
$z = $i - 3; $y = 1;

        //Add header content
        $text = $header1[0];
        fwrite($file, $text);
        $text = $header2[1];
        fwrite($file, $text);



}elseif($sourcename == "CNN Markets"){

//Set indexing parameters
$z = $i - 3; $y = 1;

        //Add header content
        $text = $header1[0];
        fwrite($file, $text);
        //$text = $header2[1];
        //fwrite($file, $text);

}elseif($sourcename == "BBC Business"){

//Set indexing parameters
$z = $i - 9; $y = 1;

        //Add header content
        $text = $header1[0];
        fwrite($file, $text);
        //$text = $header2[1];
        //fwrite($file, $text);

}elseif($sourcename == "BBC Politics"){

//Set indexing parameters
$z = $i - 0; $y = 1;

        //Add header content
        $text = $header1[0];
        fwrite($file, $text);
        //$text = $header2[1];
        //fwrite($file, $text);

}else{
echo $sourcename;
}

        do{

        $text = $content[$y];
        fwrite($file, $text);
        $y = $y +1;

        }while($y<$z);

echo "Zeta is".$zeta;
$zeta = $zeta +1;

//close the content file
fclose($file);

//echo "File end.";

$html->clear();
unset($html);





?>

长话短说，当我将所有这些输出 echo 作为更新点运行时，似乎包含(程序 B)仅运行第一次迭代，然后停止运行。在我添加清除之前，我遇到了 file_get_html() 的问题。

Answer 1

你需要把程序B的代码放在里面

    while ($row = mysqli_fetch_assoc($datareturn)) {
        fetchURL($url);
    }

这样对于每个条目，当您有 $url 时，您可以调用程序 B。我建议将程序 B 放置为函数并在 while 循环中调用该函数。

像这样

function fetchURL($url) {
    // Place PROGRAM B here. You can make the `include` be part of the program A at the top.
}