My Code failed with error "AttributeError: 'NoneType' object has no attribute 'add'"
<p>I have this pieces of code that crawled a site, extract all the needed url from the site, format the url into required format, then at a point where i am suppose to add it to a set for further proccessing. I encounter the following error, "AttributeError: 'NoneType' object has no attribute 'add'"
The part of the code is as follow</p>

<pre><code>class Finder(bs4.BeautifulSoup):

   def __init__(self, m, page_url):
      super().__init__(m, 'html.parser')
      self.page_url = page_url
      self.pdf_url_links = set()

  def handle_starttag(self, name, namespace, nsprefix, attrs):

    if name == 'a':
        for (attributes, value) in attrs.items():
            if ('.pdf&amp;') not in value: pass
            else:
                list_of_links = search_queue_url(value)
                print(list_of_links)
</code></pre>

<p>I got the following url on my screen when I print the variable 'list_of_links' above:</p>

<pre><code>https://julianoliver.com/share/free-science-books/basic_math_and_algebra.pdf
https://www.math.ksu.edu/~dbski/writings/further.pdf
http://www.math.harvard.edu/~shlomo/docs/Advanced_Calculus.pdf
http://www.textbooksonline.tn.nic.in/Books/Std10/Std10-Maths-EM-1.pdf
http://www.corestandards.org/wp-content/uploads/Math_Standards.pdf
https://www.ets.org/s/gre/pdf/gre_math_review.pdf
https://www.math.ust.hk/~machas/differential-equations.pdf
</code></pre>

<p>However, when I attempt to add each of the above url to my set with below code, </p>

<pre><code>self.pdf_url_links.add(list_of_links)
</code></pre>

<p>I got the following error bellow,</p>

<pre><code>AttributeError: 'NoneType' object has no attribute 'add'
</code></pre>

<p>Traceback (most recent call last):</p>

<pre><code> File "C:\Projects\BookScapie\BookScrapie\BookScrapie\link_finders.py", line 7, in __init__
super().__init__(m, 'html.parser')
 File "C:\python-3.5.1.amd64\lib\site-packages\bs4\__init__.py", line 228, in __init__
self._feed()
 File "C:\python-3.5.1.amd64\lib\site-packages\bs4\__init__.py", line 289, in _feed
self.builder.feed(self.markup)
 File "C:\python-3.5.1.amd64\lib\site-packages\bs4\builder\_htmlparser.py", line 167, in feed
parser.feed(markup)
 File "C:\python-3.5.1.amd64\lib\html\parser.py", line 111, in feed
self.goahead(0)
 File "C:\python-3.5.1.amd64\lib\html\parser.py", line 171, in goahead
k = self.parse_starttag(i)
 File "C:\python-3.5.1.amd64\lib\html\parser.py", line 345, in parse_starttag
self.handle_starttag(tag, attrs)
 File "C:\python-3.5.1.amd64\lib\site-packages\bs4\builder\_htmlparser.py", line 65, in handle_starttag
self.soup.handle_starttag(name, None, None, attr_dict)
 File "C:\Projects\BookScapie\BookScrapie\BookScrapie\link_finders.py", line 22, in handle_starttag
self.pdf_url_links.add(list_of_links)
</code></pre>

<p>AttributeError: 'NoneType' object has no attribute 'add'</p>

<p>I will appreciate any good ideas on what I am doing right.
I am using python 3.5</p>

<p>The problem was caused by the way BeautifulSoup constructor works.</p>

<pre><code>class Finder(bs4.BeautifulSoup):

   def __init__(self, m, page_url):
      super().__init__(m, 'html.parser')
      self.page_url = page_url
      self.pdf_url_links = set()
</code></pre>

<p>As soon as <code>BeautifulSoup.__init__</code> is called, the parsing begins, which eventually calls <code>handle_starttag</code> method.</p>

<p>Here, the overridden version of <code>handle_starttag</code> tried to access <code>self.pdf_url_links</code>, which was not initialized.</p>

<p>The solution is to initialize everything that is needed for parsing before calling the super constructor:</p>

<pre><code>   def __init__(self, m, page_url):
      self.page_url = page_url
      self.pdf_url_links = set()
      super().__init__(m, 'html.parser')
</code></pre>

python