How to have beautiful soup fetch emails?
<p>Im working with beautiful soup and would like to grab emails to a depth of my choosing in my web scraper. Currently however I am unsure why my web scraping tool is not working. Everytime I run it, it does not populate the email list. </p>

<pre><code>#!/usr/bin/python
from bs4 import BeautifulSoup, SoupStrainer
import re
import urllib
import threading

def step2():
    file = open('output.html', 'w+')
    file.close()
    # links already added
    visited = set()
    visited_emails = set()
    scrape_page(visited, visited_emails, 'https://www.google.com', 2)

    print('Webpages \n')
    for w in visited:
        print(w)

    print('Emails \n')
    for e in visited_emails:
        print(e)

# Run recursively
def scrape_page(visited, visited_emails, url, depth):

    if depth == 0:
    return

    website = urllib.urlopen(url)
    soup = BeautifulSoup(website, parseOnlyThese=SoupStrainer('a', email=False))

    emails = re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}", str(website))

    first = str(website).split('mailto:')
    for i in range(1, len(first)):
        print(first.split('&gt;')[0])

    for email in emails:
        if email not in visited_emails:
            print('- got email ' + email)
            visited_emails.add(email)


    for link in soup:
        if link.has_attr('href'):
            if link['href'] not in visited:
                if link['href'].startswith('https://www.google.com'):
                    visited.add(link['href'])
                    scrape_page(visited, visited_emails, link['href'], depth - 1)

def main():
    step2()

main()
</code></pre>

<p>for some reason im unsure how to fix my code to add emails to the list. if you could give me some advice it would be greatly appreciated. thanks</p>

<p>You just need to look for the href's with mailto:</p>

<pre><code>emails = [a["href"] for a in soup.select('a[href^=mailto:]')]
</code></pre>

<p>I presume  <a href="https://www.google.com" rel="nofollow">https://www.google.com</a> is a placeholder for the actual site you are scraping as there are no mailto's to scrape on the google page. If there are mailto's in the source you are scraping then this will find them.</p>

python email beautifulsoup