Using python requests and beautiful soup to pull text
<p>thanks for taking a look at my problem. i would like to know if there is any way to pull the data-sitekey from this text... here is the url to the page <a href="https://e-com.secure.force.com/adidasUSContact/" rel="nofollow">https://e-com.secure.force.com/adidasUSContact/</a></p>

<pre><code>&lt;div class="g-recaptcha" data-sitekey="6LfI8hoTAAAAAMax5_MTl3N-5bDxVNdQ6Gx6BcKX" data-type="image" id="ncaptchaRecaptchaId"&gt;&lt;div style="width: 304px; height: 78px;"&gt;&lt;div&gt;&lt;iframe src="https://www.google.com/recaptcha/api2/anchor?k=6LfI8hoTAAAAAMax5_MTl3N-5bDxVNdQ6Gx6BcKX&amp;amp;co=aHR0cHM6Ly9lLWNvbS5zZWN1cmUuZm9yY2UuY29tOjQ0Mw..&amp;amp;hl=en&amp;amp;type=image&amp;amp;v=r20160921114513&amp;amp;size=normal&amp;amp;cb=ei2ddcb6rl03" title="recaptcha widget" width="304" height="78" role="presentation" frameborder="0" scrolling="no" name="undefined"&gt;&lt;/iframe&gt;&lt;/div&gt;&lt;textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="width: 250px; height: 40px; border: 1px solid #c1c1c1; margin: 10px 25px; padding: 0px; resize: none;  display: none; "&gt;&lt;/t
</code></pre>

<p>here is my current code </p>

<pre><code>    import requests 
from bs4 import BeautifulSoup

headers = {
    'Host' : 'e-com.secure.force.com',
    'Connection' : 'keep-alive',
    'Upgrade-Insecure-Requests' : '1',
    'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64)',
    'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding' : 'gzip, deflate, sdch',
    'Accept-Language' : 'en-US,en;q=0.8'
}
url = 'https://e-com.secure.force.com/adidasUSContact/'
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r, 'html.parser')
c = soup.find_all('div', attrs={"class": "data-sitekey"})
print c
</code></pre>

<p>Ok now we have code, it is as simple as:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


soup = BeautifulSoup(requests.get("https://e-com.secure.force.com/adidasUSContact/").content, "html.parser")

key = soup.select_one("#ncaptchaRecaptchaId")["data-sitekey"]
</code></pre>

<p><em>data-sitekey</em> is an <em>attribute</em>, <strong>not</strong> a <em>css</em> class so you just need to extract it from the element, you can find the element by it's <em>id</em> as above.</p>

<p>You could also use the class name:</p>

<pre><code># css selector
key = soup.select_one("div.g-recaptcha")["data-sitekey"]
# regular find using class name
key = soup.find("div",class_="g-recaptcha")["data-sitekey"]
</code></pre>

python beautifulsoup python-requests bs4