Python 2.7 BeautifulSoup, website addresses scraping
<p>Hope you are all well. I'm new in Python and using python 2.7.  </p>

<p>I'm trying to extract only the websites from this public website business directory: <a href="https://www.dmcc.ae/business-directory" rel="nofollow">https://www.dmcc.ae/business-directory</a><br>
the websites i'm looking for are the websites mentioned in every widget.
This directory does not have an API unfortunately.<br>
I'm using BeautifulSoup, but with no success so far.<br>
here is mycode: </p>

<pre><code>import urllib
from bs4 import BeautifulSoup
website = raw_input("Type Website:&gt;\n")
html = urllib.urlopen('https://'+ website).read()
soup = BeautifulSoup(html)
tags = soup('a')
for tag in tags:
    print tag.get('href', None)
</code></pre>

<p>what i get is just the website of the actual website , like <a href="http://portal.dmcc.ae" rel="nofollow">http://portal.dmcc.ae</a> along with other href rather then the websites in the widgets. i also tried replacing soup('a') with soup ('class'), but no luck!
Can anybody help me please? </p>

<p>As somebody has written into comments:</p>

<p>The web page is build by client using JavaScript. What your Python script gets is only a (relatively) short HTML website, that loads other scripts to load the right data. Unfortunately, BeautifulSoup can't execute the website so let it load all of its resources and then parse it.</p>

<p>You should use another technology, which can actually run the site and then parse it. I know about <a href="http://www.seleniumhq.org" rel="nofollow">Selenium</a>, but I have no experiences with it</p>

python python-2.7 web-scraping beautifulsoup