Scraping google news with BeautifulSoup returns empty results
<p>I am trying to scrape google news using the following code:</p>

<pre><code>from bs4 import BeautifulSoup
import requests
import time
from random import randint


def scrape_news_summaries(s):
    time.sleep(randint(0, 2))  # relax and don't let google be angry
    r = requests.get("http://www.google.co.uk/search?q="+s+"&amp;tbm=nws")
    content = r.text
    news_summaries = []
    soup = BeautifulSoup(content, "html.parser")
    st_divs = soup.findAll("div", {"class": "st"})
    for st_div in st_divs:
        news_summaries.append(st_div.text)
    return news_summaries


l = scrape_news_summaries("T-Notes")
#l = scrape_news_summaries("""T-Notes""")
for n in l:
    print(n)
</code></pre>

<p>Even though this bit of code was working before, I now can't figure out why it's not working anymore. Is it possible that I've been banned by google since I only ran the code 3 or four times? (I tried using Bing News with unfortunate empty results too...)</p>

<p>Thanks. </p>

<p>I tried running the code and it works fine on my computer.</p>

<p>You could try printing the status code for the request, and see if it's anything other than 200.</p>

<pre><code>from bs4 import BeautifulSoup
import requests
import time
from random import randint


def scrape_news_summaries(s):
    time.sleep(randint(0, 2))  # relax and don't let google be angry
    r = requests.get("http://www.google.co.uk/search?q="+s+"&amp;tbm=nws")
    print(r.status_code)  # Print the status code
    content = r.text
    news_summaries = []
    soup = BeautifulSoup(content, "html.parser")
    st_divs = soup.findAll("div", {"class": "st"})
    for st_div in st_divs:
        news_summaries.append(st_div.text)
    return news_summaries


l = scrape_news_summaries("T-Notes")
#l = scrape_news_summaries("""T-Notes""")
for n in l:
    print(n)
</code></pre>

<p><a href="https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/" rel="nofollow">https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/</a> for a list of status code that's a sign you have been banned.</p>

python web-scraping beautifulsoup google-news