Python - Problems creating a list of URLs using BeautifulSoup
<p>I am trying to make a Python Crawler using BeautifulSoup, but I receive an error that I am trying to write a non-String or other character buffer type to a file.  From examining the program output, I found that my list contains many items that are None.  In addition to having None, I also have a lot of images and things that are not links but are image links inside of my list.  How can I only add the URLs to my list?</p>

<pre><code>    import urllib
    from BeautifulSoup import *

    try:
        with open('url_file', 'r') as f:
            url_list = [line.rstrip('\n') for line in f]
            f.close()
        with open('old_file', 'r') as x:
            old_list = [line.rstrip('\n') for line in f]
            f.close()
    except:
        url_list = list()
        old_list = list()
        #for Testing
        url_list.append("http://www.dinamalar.com/")


    count = 0


    for item in url_list:
        try:
            count = count + 1
            if count &gt; 5:
                break

            html = urllib.urlopen(item).read()
            soup = BeautifulSoup(html)
            tags = soup('a')

            for tag in tags:

                if tag in old_list:
                    continue
                else:
                    url_list.append(tag.get('href', None))


            old_list.append(item)
            #for testing
            print url_list
        except:
            continue

    with open('url_file', 'w') as f:
        for s in url_list:
            f.write(s)
            f.write('\n')


    with open('old_file', 'w') as f:
        for s in old_list:
            f.write(s)
</code></pre>

<p>First off, use <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow">bs4</a> not the no longer maintained <em>BeautifulSoup3</em>, your error is because not all anchors have hrefs so you are trying to write <em>None</em> which causes your error, use <em>find_all</em> and set <em>href=True</em> so you only find anchor tags that have a href attribute:</p>

<pre><code>soup = BeautifulSoup(html)
tags = soup.find_all("a", href=True)
</code></pre>

<p>Also never use blanket except statements, always catch the errors you expect and at least print them when they do occur.  As far as <em>I also have a lot of images and things that are not links</em> goes, if you want to filter for certain links then you have to be more specific, either look for the tags that contain what you are interested in if possible, use a regex <code>href=re.compile("some_pattern")</code> or use a css selectors:</p>

<pre><code># hrefs starting with something
"a[href^=something]"

# hrefs that contain something
"a[href*=something]"

# hrefs ending with  something
"a[href$=something]"
</code></pre>

<p>Only you know the structure of the html and what you want so what you use is completely up to you to decide.</p>

python list beautifulsoup