Python : Web Scraping Specific Keywords
<p>My Question shouldn't be too hard to answer, The problem im having is im not sure how to scrape a website for specific keywords.. I'm quite new to Python.. So i know i need to add in some more details , Firstly what i dont want to do is use Beautiful Soup or any of those libs, im using lxml and requests, What i do want to do is ask the user for an input for a website and once its provided , Send a request to the provided URL, once the request is made i want it to grab all the html which i believe ive done using html.fromstring(site.content) so all thats been done the problem im having is i want it to find any link or text with the ending '.swf' and print it below that.. Anyone know any way of doing this?</p>

<pre><code>def ScrapeSwf():
     flashSite = raw_input('Please Provide Web URL : ')
     print 'Sending Requests...'
     flashReq = requests.get(flashSite)
     print 'Scraping...'
     flashTree = html.fromstring(flashReq.content)
     print ' Now i want to search the html for the swf link in the html'
     print ' And Display them using print probablly with a while condition'
</code></pre>

<p>Something like that .. Any help is highly appreciated</p>

<p>You're using <code>lxhtml</code> to build the HTML into an object model, so you probably want to use <code>flashTree.xpath</code> to search the DOM using XML Path Language. Find the path you want in the source DOM and then write an xpath that extracts it, your web browser's developer tools and <a href="http://www.w3schools.com/xsl/xpath_intro.asp" rel="nofollow">w3schools</a> can help you.</p>

<p>I personally wouldn't bother, I'd just extract the text I needed using a regular expression (<code>re.find(pattern, flashReq.content)</code>) because it's quicker. If I didn't know regex, wasn't comfortable with them, or I wanted raw speed then I'd use a crude string extraction like so:</p>

<pre><code>start = flashReq.content.find(text_before_it) + len(text_before_it)
end = flashReq.content.find(text_after_it, start)
text_you_want = flashReq.content[start:end]
</code></pre>

python web web-crawler screen-scraping scrape