How to scrape real time streaming data with Python?
<p>I was trying to scrape the number of flights for this webpage <a href="https://www.flightradar24.com/56.16,-49.51" rel="nofollow">https://www.flightradar24.com/56.16,-49.51</a></p>

<p>The number is highlighted in the picture below:
<a href="http://i.stack.imgur.com/Zvsmf.png" rel="nofollow"><img src="http://i.stack.imgur.com/Zvsmf.png" alt="enter image description here"></a></p>

<p>The number is updated every 8 seconds.</p>

<p>This is what I tried with BeautifulSoup:</p>

<pre><code>import requests
from bs4 import BeautifulSoup
import time

r=requests.get("https://www.flightradar24.com/56.16,-49.51")
c=r.content
soup=BeautifulSoup(c,"html.parser")
value=soup.find_all("span",{"class":"choiceValue"})
print(value)
</code></pre>

<p>But that always returns 0:</p>

<pre><code>[&lt;span class="choiceValue" id="menuPlanesValue"&gt;0&lt;/span&gt;]
</code></pre>

<p>View source also shows 0, so I understand why BeautifulSoup returns 0 too.</p>

<p>Anyone know any other method to get the current value?</p>

<p>The problem with your approach is that the page first loads a view, then performs regular requests to refresh the page. If you look at the network tab in the developer console in Chrome (for example), you'll see the requests to <a href="https://data-live.flightradar24.com/zones/fcgi/feed.js?bounds=59.09,52.64,-58.77,-47.71&amp;faa=1&amp;mlat=1&amp;flarm=1&amp;adsb=1&amp;gnd=1&amp;air=1&amp;vehicles=1&amp;estimated=1&amp;maxage=7200&amp;gliders=1&amp;stats=1">https://data-live.flightradar24.com/zones/fcgi/feed.js?bounds=59.09,52.64,-58.77,-47.71&amp;faa=1&amp;mlat=1&amp;flarm=1&amp;adsb=1&amp;gnd=1&amp;air=1&amp;vehicles=1&amp;estimated=1&amp;maxage=7200&amp;gliders=1&amp;stats=1</a></p>

<p>The response is regular json:</p>

<pre><code>{
  "full_count": 11879,
  "version": 4,
  "afefdca": [
    "A86AB5",
    56.4288,
    -56.0721,
    233,
    38000,
    420,
    "0000",
    "T-F5M",
    "B763",
    "N641UA",
    1473852497,
    "LHR",
    "ORD",
    "UA929",
    0,
    0,
    "UAL929",
    0
  ],
  ...
  "aff19d9": [
    "A12F78",
    56.3235,
    -49.3597,
    251,
    36000,
    436,
    "0000",
    "F-EST",
    "B752",
    "N176AA",
    1473852497,
    "DUB",
    "JFK",
    "AA291",
    0,
    0,
    "AAL291",
    0
  ],
  "stats": {
    "total": {
      "ads-b": 8521,
      "mlat": 2045,
      "faa": 598,
      "flarm": 152,
      "estimated": 464
    },
    "visible": {
      "ads-b": 0,
      "mlat": 0,
      "faa": 6,
      "flarm": 0,
      "estimated": 3
    }
  }
}
</code></pre>

<p>I'm not sure if this API is protected in any way, but it seems like I can access it without any issues using curl.</p>

<p>More info:</p>

<ul>
<li><a href="http://aviation.stackexchange.com/questions/3052/is-there-an-api-to-get-real-time-faa-flight-data">aviation.stackexchange - Is there an API to get real-time FAA flight data?</a></li>
<li><a href="http://forum.flightradar24.com/threads/24-API-access/page3">Flightradar24 Forum - API access</a> (meaning your use case is probably discouraged)</li>
</ul>

python web-scraping beautifulsoup