How to scrape real time streaming data with Python?
<p>I was trying to scrape the number of flights for this webpage <a href="https://www.flightradar24.com/56.16,-49.51" rel="nofollow">https://www.flightradar24.com/56.16,-49.51</a></p>

<p>The number is highlighted in the picture below:
<a href="http://i.stack.imgur.com/Zvsmf.png" rel="nofollow"><img src="http://i.stack.imgur.com/Zvsmf.png" alt="enter image description here"></a></p>

<p>The number is updated every 8 seconds.</p>

<p>This is what I tried with BeautifulSoup:</p>

<pre><code>import requests
from bs4 import BeautifulSoup
import time

r=requests.get("https://www.flightradar24.com/56.16,-49.51")
c=r.content
soup=BeautifulSoup(c,"html.parser")
value=soup.find_all("span",{"class":"choiceValue"})
print(value)
</code></pre>

<p>But that always returns 0:</p>

<pre><code>[&lt;span class="choiceValue" id="menuPlanesValue"&gt;0&lt;/span&gt;]
</code></pre>

<p>View source also shows 0, so I understand why BeautifulSoup returns 0 too.</p>

<p>Anyone know any other method to get the current value?</p>

<p>So based on what @Andre has found out, I wrote this code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup
import time

def get_count():
    url = "https://data-live.flightradar24.com/zones/fcgi/feed.js?bounds=59.09,52.64,-58.77,-47.71&amp;faa=1&amp;mlat=1&amp;flarm=1&amp;adsb=1&amp;gnd=1&amp;air=1&amp;vehicles=1&amp;estimated=1&amp;maxage=7200&amp;gliders=1&amp;stats=1"

    # Request with fake header, otherwise you will get an 403 HTTP error
    r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

    # Parse the JSON
    data = r.json()
    counter = 0

    # Iterate over the elements to get the number of total flights
    for element in data["stats"]["total"]:
        counter += data["stats"]["total"][element]

    return counter

while True:
    print(get_count())
    time.sleep(8)
</code></pre>

<p>The code should be self explaining, everything it does is printing the actual flight count every 8 seconds :)</p>

<p><strong>Note:</strong> <em>The values are similar to the ones on the website, but not the same. This is because it's unlikely, that the Python script and the website are sending a request at the same time. If you want to get more accurate results, just make a request every 4 seconds for example.</em></p>

<p>Use this code as you want, extend it or whatever. Hope this helps!</p>

python web-scraping beautifulsoup