Appending links to new rows in pandas df after using beautifulsoup
<p>I'm attempting to extract some links from a chunk of beautiful soup html and append them to rows of a new pandas dataframe. </p>

<p>So far, I have this code:</p>

<pre><code>url = "http://www.reed.co.uk/jobs
datecreatedoffset=Today&amp;isnewjobssearch=True&amp;pagesize=100"
r = ur.urlopen(url).read()
soup = BShtml(r, "html.parser")
adcount = soup.find_all("div", class_="pages")
print(adcount)
</code></pre>

<p>From my output I then want to take every link, identified by href="" and store each one in a new row of a pandas dataframe.</p>

<p>Using the above snippet I would end up with 6 rows in my new dataset.</p>

<p>Any help would be appreciated!</p>

<p>Your links gives a 404 but the logic should be the same as below. You just need to extract the anchor tags with the <em>page</em> class and join them to the base url:</p>

<pre><code>import  pandas as pd
from urlparse import urljoin
import requests 

base = "http://www.reed.co.uk/jobs"

url = "http://www.reed.co.uk/jobs?keywords=&amp;location=&amp;jobtitleonly=false"
r = requests.get(url).content
soup = BeautifulSoup(r, "html.parser")

df = pd.DataFrame(columns=["links"], data=[urljoin(base, a["href"]) for a in  soup.select("div.pages a.page")])

print(df)
</code></pre>

<p>Which gives you:</p>

<pre><code>                                            links
0    http://www.reed.co.uk/jobs?cached=True&amp;pageno=2
1    http://www.reed.co.uk/jobs?cached=True&amp;pageno=3
2    http://www.reed.co.uk/jobs?cached=True&amp;pageno=4
3    http://www.reed.co.uk/jobs?cached=True&amp;pageno=5
4  http://www.reed.co.uk/jobs?cached=True&amp;pageno=...
5    http://www.reed.co.uk/jobs?cached=True&amp;pageno=2
</code></pre>

python pandas beautifulsoup append