linux - 使用 SED 进行网页抓取

Question

让我先声明一下，我知道在有这么多更好的工具的情况下，在 sed 中进行网页抓取是一个愚蠢的想法。我试着把这件事告诉我的老师，但他坚持要使用 sed 和 awk。

我的目标是抓取一个 Twitter 页面并返回所有推文。使用此代码:

sed -n '/<p class=\"TweetTextSize  js-tweet-text tweet-text\" lang=\"en\" data-aria-label-part=\"0\">/,/<\/p>/p' $targ | sed 's/<[^>]\+>/ /g;s/^[ \t]*//;/^$/d'

On this URL到目前为止，我一直没有得到我想要的东西。

这是围绕推文的代码:

<p class="TweetTextSize js-tweet-text tweet-text" lang="en" data-aria-label-part="0">Where is our universe headed? Answer to it&#39;s death, not to worry <a href="/PaulMattSutter" class="twitter-atreply pretty-link js-nav" dir="ltr" data-mentioned-user-id="556263627" ><s>@</s><b>PaulMattSutter</b></a> explains. <a href="https://t.co/ig8KtQzwOI" rel="nofollow" dir="ltr" data-expanded-url="http://oak.ctx.ly/r/42bnj" class="twitter-timeline-link" target="_blank" title="http://oak.ctx.ly/r/42bnj" ><span class="tco-ellipsis"></span><span class="invisible">http://</span><span class="js-display-url">oak.ctx.ly/r/42bnj</span><span class="invisible"></span><span class="tco-ellipsis"><span class="invisible">&nbsp;</span></span></a> <a href="https://t.co/kOy4jLCogk" class="twitter-timeline-link u-hidden" data-pre-embedded="true" dir="ltr" >pic.twitter.com/kOy4jLCogk</a></p>

现在，我只想从 <p class="TweetTextSize js-tweet-text tweet-text" lang="en" data-aria-label-part="0"> 之间的页面抓取内容及其对应的</p>然后我想过滤它，以便删除中间的所有标签(打开和关闭)，这样它看起来像这样:

Where is our universe headed? Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj pic.twitter.com/kOy4jLCogk

Answer 1

试试这个:

 sed -n '/<p class="TweetTextSize  js-tweet-text tweet-text" lang="en" data-aria-label-part="0">.*<\/p>/{s/<[^>]*>//g;p}' file

输出:

        You’ve never seen Pluto like this! @NASANewHorizons returns sharpest views yet! Take a look: http://go.nasa.gov/1NxLViz  pic.twitter.com/iK4OJKeLUE        The world is filled with those who do nothing yet criticize. I love this @rickygervais motto: Go create! pic.twitter.com/m7zB8ucclT        Can you see me waving? How to spot #Mars in the night sky: https://youtu.be/hv8hVvJlcJQ  https://amp.twimg.com/v/9b945202-5a89-4efa-80ab-514a0c17a965 …        We knew the first high-res images of Pluto would be good, but we didn't expect THIS good: http://ow.ly/Vvrvm  pic.twitter.com/r8ZX4Z5tVt        Toddler wears the most precious 'Monsters, Inc.' costume this world has ever seen http://on.mash.to/1HLrcMk  pic.twitter.com/2PyuDW9yms        Are you on your way to your @TMobile store, @ATT customers?! GO!! #UncarrierUnwrapped http://t-mo.co/1SzhOfb  pic.twitter.com/0vNJbD2J4k        1 year ago today, @USNavy @ussanchorage recovered Orion in Pacific ocean after successful flight. pic.twitter.com/DKW00LzlC3        How Fallout 4 mastermind Todd Howard builds his epic dream worlds http://wrd.cm/1IILf8N  pic.twitter.com/WUV98e6U62        23-year-old’s design collaboration tool Figma launches with $14M to fight Adobe http://tcrn.ch/1YPW8Pw  pic.twitter.com/wVrZbfgBOc        Here’s a list of questions investors will ask themselves after you pitch a company to them. http://kpcb.cc/b466b5b  pic.twitter.com/zOJDOrq0na        Blow your mind with this--ENJOY! (And check the SWEET video too)! http://www.nasa.gov/feature/new-horizons-returns-first-best-images-of-pluto … #PlutoFlyby pic.t        Human gene editing is OK, but no designer babies—for now, summit concludes: http://ow.ly/VscSg  #GeneEditSummit pic.twitter.com/UW6DKtQX4h        NASA releases the highest resolution images ever taken of Pluto http://www.theverge.com/2015/12/4/9851162/nasa-new-horizons-pluto-photo-new-high-resolution?utm_campai        A drone that finds you a parking space and then hovers in it to stop anyone else taking it first.        The ubiquity of guns poses a much greater threat to Americans than terrorism http://bit.ly/1lbvPoQ  By @Horganism pic.twitter.com/pdxvVqGS59        Scientist disinvited from speaking at conference because of her pregnancy http://ow.ly/Vvyam  pic.twitter.com/FUAjaycIom        Samsung agrees to pay Apple $548 million in settlement http://engt.co/1jEN28t  pic.twitter.com/3DmmccJUgW        World's first in-human gene-editing treatment will tackle hemophilia http://www.engadget.com/2015/12/03/worlds-first-in-human-CRISPR-hemophilia/ … pic.twitte        Where is our universe headed?  Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj  pic.twitter.com/kOy4jLCogk        New Horizons just sent back the sharpest images possible of Pluto's surface. Drink it in http://www.wired.com/2015/12/new-horizons-just-sent-its-highest-res-image        Get people to do what you want with these 11 clever psychological tricks: http://lifehac.kr/GZaSWhN  pic.twitter.com/5rhWlVL9t0