Scraping a webpage that is using a firebase database
<p><strong>DISCLAIMER: I'm just learning by doing, I have no bad intentions</strong></p>

<p>So, I would like to fetch the list of the applications listed on this website: <a href="http://roaringapps.com/apps" rel="nofollow">http://roaringapps.com/apps</a></p>

<p>I've done similar things in the past, but with simpler websites; this time I'm having problems getting my hands on the data behind this webpage. </p>

<p>The scrolling from page to page is blazing fast so, to understand how the webpage works, I've fired up a packet sniffer and analyzed the traffic. I've noticed that, after the initial loading, no traffic is exchanged between the server and my client, even if I scroll over 2500 records in the browser. How is that possible?</p>

<p>Anyhow. My understanding is that the website is loading the data from a stream of some sort, and render it via Javascript. Am I correct?</p>

<p>So, I've fired up chromium devtools a looked at the "network" tab, and saw that a WebSocket request is made to the following address: wss://s-usc1c-nss-123.firebaseio.com</p>

<p><a href="https://i.stack.imgur.com/XpR63.png" rel="nofollow"><img src="https://i.stack.imgur.com/XpR63.png" alt="chromium devtool"></a></p>

<p>At this point, after googling a bit, I've tried to query the very same server, using the "v=5&amp;ns=roaringapps" query I saw on the devtools window:</p>

<pre><code>from websocket import create_connection
ws = create_connection('wss://s-usc1c-nss-123.firebaseio.com')
ws.send('v=5&amp;ns=roaringapps')
print json.loads(ws.recv())
</code></pre>

<p>And got this reply:</p>

<pre><code>{u't': u'c', u'd': {u't': u'h', u'd': {u'h': u's-usc1c-nss-123.firebaseio.com', u's': u'JUL5t1nC2SXfGaIjwecB6G13j1OsmMVv', u'ts': 1476799051047L, u'v': u'5'}}}
</code></pre>

<p>I was expecting to see a json response with the raw data about applications &amp; so on. What I'm doing wrong? </p>

<p>Thanks a lot!</p>

<p><strong>UPDATE</strong></p>

<p>Actually, I just found out that the website <em>is</em> using json to load its data. I was not seeing it in iterated requests probably because of caching - but disabling it in chromium did the trick.</p>

<p>While the Firebase Database allows you to read/write JSON data. But its SDKs don't simply transfer the raw JSON data, they do many tricks on top of that to ensure an efficient and smooth experience. W</p>

<p>hat you're getting there is Firebase's wire protocol. The protocol is not publicly documented and (if you're new to it) trying to unravel it is going to give you an unpleasant time.</p>

<p>To retrieve the actual JSON at a location, it's easiest to use <a href="https://firebase.google.com/docs/database/rest/start" rel="nofollow">Firebase's REST API</a>. You can get that by simply appending <code>.json</code> to the URL and firing a HTTP GET request against that.</p>

<p>So if the initial data is being loaded from:</p>

<pre><code>https://mynamespace.firebaseio.com/path/to/data
</code></pre>

<p>You'd get the raw JSON by firing a HTTP GET against:</p>

<pre><code>https://mynamespace.firebaseio.com/path/to/data.json
</code></pre>

javascript python web firebase firebase-database