How to print a div data-reactid?
<p>I'm doing a project in my spare time where I have hit a problem with getting data from a webpage into the program.</p>

<p>This is my current code:</p>

<pre><code>import urllib
import re

htmlfile = urllib.urlopen("http://www.superliga.dk/klub/aab?sub=squad")

htmltext = htmlfile.read()

regex = r'&lt;div data-reactid=".3.$squad content.0.$=11:0.0.0.0.1:0.2.0.0"&gt;([^&lt;]*)&lt;/div&gt;'

pattern = re.compile(regex)

goal = re.findall(pattern,htmltext)

print goal
</code></pre>

<p>And it's working okay except this part:</p>

<pre><code>regex = r'&lt;div data-reactid=".3.$squad content.0.$=11:0.0.0.0.1:0.2.0.0"&gt;([^&lt;]*)&lt;/div&gt;'
</code></pre>

<p>I can't make it display all values on the webpage with this <code>reactid</code>, and I can't find any solution to this problem.
Any suggestions to how I can get Python to print it?</p>

<p>You are trying to match a tag you saw on the on the developer console of you browser, right?
Unfortunately the html you saw is only the "final form" of a dynamic page: what you did download with <code>urlopen</code> is only the skeleton of the webpage, which in the browser is then dynamically filled with other elements by the javascript using data fetched from some backend server.</p>

<p>If you try to print the actual value stored in <code>htmltest</code> you will find nothing like what you are trying to match with the regex, and that's because it missed all the further processing normally performed by by the javascript.</p>

<p>What you can try to do is to monitor (through the dev console) the fetched resources and reverse-engineer the API call in order to recover the desired info. Chances are the response of these API call is in JSON format or has a structure way more easily parsable than the html body.</p>

<p><strong>UPDATE</strong>: for example, in Chrome's dev tools I can see async calls like:</p>

<p><code>http://ss2.tjekscores.dk/pro-stats/tournaments/46/top-players?sortBy=eventsStats.goals&amp;limit=5&amp;skip=0&amp;positionId=&amp;q=&amp;seasonId=10392&amp;teamId[]=8470</code></p>

<p>Maybe this returns the info you are looking for.</p>

python html regex