Fetching live data from website's with continiously updating data
<p>I can easily get the data when I put <strong>html = urllib.request.urlopen(req)</strong> inside a while loop, but it takes about 3 seconds to get the data. So I thought, maybe if I put that outside, I can get it faster as it won't have to open the URL everytime, but this throws up an <strong>AttributeError: 'str' object has no attribute 'read'</strong>. Maybe it doesn't recognize HTML variable name. How can I speed the processing ?</p>

<pre><code>def soup():
url = "http://www.investing.com/indices/major-indices"
req = urllib.request.Request(
url, 
data=None, 
headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36',
    'Connection': 'keep-alive'    }
       )
global Ltp
global html
html = urllib.request.urlopen(req)
while True:
    html = html.read().decode('utf-8')
    bsobj = BeautifulSoup(html, "lxml")   

    Ltp = bsobj.find("td", {"class":"pid-169-last"} )
    Ltp = (Ltp.text)
    Ltp = Ltp.replace(',' , '');
    os.system('cls')     
    Ltp = float(Ltp)
    print (Ltp, datetime.datetime.now())    

soup()
</code></pre>

<p>you reassign <code>html</code> to equal the UTF-8 string response then keep calling it like its an <code>IO</code> ... this code does not fetch new data from the server on every loop, <code>read</code> simply reads the bytes from the <code>IO</code> object, it doesnt make a new request.</p>

<p>you can speed up the processing with the Requests library and utilise persistent connections (or urllib3 directly)</p>

<p>Try this (you will need to <code>pip install requests</code>)</p>

<pre><code>import os
import datetime

from requests import Request, Session
from bs4 import BeautifulSoup

s = Session()

while True:
  resp = s.get("http://www.investing.com/indices/major-indices")
  bsobj = BeautifulSoup(resp.text, "html.parser")   
  Ltp = bsobj.find("td", {"class":"pid-169-last"} )
  Ltp = (Ltp.text)
  Ltp = Ltp.replace(',' , '');
  os.system('cls')     
  Ltp = float(Ltp)
  print (Ltp, datetime.datetime.now())   
</code></pre>

python web-scraping beautifulsoup