Printing html like contents form a textarea using beautifulsoup
<p>We can get the text inside a tag using the get_text() function in BeautifulSoup. But what if the text area contains some html like code.</p>

<p>Example:</p>

<pre><code>from bs4 import BeautifulSoup
html = "&lt;html&gt;&lt;h1&gt;#include &lt;stdio.h&gt;&lt;/h1&gt;&lt;/html&gt;"
soup = BeautifulSoup(html,"lxml")
print soup.h1.get_text() 
</code></pre>

<p>The above program prints "#include" but I wanted it to be full text inside h1.
This is just a small example. I am working with with scraping a c++ code from web. I have navigated to the text area in which code is present but when I print it it doesn't print the header files.
Textarea:</p>

<pre><code>&lt;textarea rows="20" cols="70" name="file" id="file" style="width: 100%;" data-input-file="1"&gt;#include &lt;bits/stdc++.h&gt;
using namespace std;
struct points
{
    int x;
    int count;
};
points a[105];
int main()
{
    int n;
    int k;
    int t;
    int i;
    int j;

    scanf("%d",&amp;t);

    while(t--) {
        scanf("%d%d",&amp;n,&amp;k);
        for(i = 1; i &lt;= k; i++) {
            scanf("%d",&amp;a[i].x);
            a[i].count = 1;
            if(a[i].x == -1) {
                a[i].x = 1000000000;
            }
        }

        for(i = 2; i &lt;= k; i++) {
            for(j = 1; j &lt; i; j++) {
                if((a[i-j].x + a[j].x) &lt; a[i].x &amp;&amp; (a[i-j].count + a[j].count) &lt;= n) {
                    a[i].x = a[i-j].x + a[j].x;
                    a[i].count = a[i-j].count + a[j].count;
                }
            }
        }

        if(a[k].x == 1000000000) {
            printf("-1\n");
        }
        else {
            printf("%d\n",a[k].x);
        }
    }



}
&lt;/textarea&gt;
</code></pre>

<p>My code for scraping:</p>

<pre><code>from robobrowser import RoboBrowser
browser = RoboBrowser(parser = "lxml")
browser.open('http://www.spoj.com/')
form = browser.get_form(id='login-form')
form['login_user'].value = username
form['password'].value = password
browser.submit_form(form)
browser.open('http://www.spoj.com/myaccount')
l = browser.find(id = "user-profile-tables").find_all('td')
link = l[0].a['href']
link = "http://www.spoj.com" + link
browser.open(link)
codelink = browser.find(title = 'Edit source code')['href']
codelang = browser.find(class_ = 'slang text-center').get_text()
codelink = "http://www.spoj.com" + codelink
browser.open(codelink)
print browser.find(id = 'submit_form').textarea.get_text()
</code></pre>

<p>Is there any way to achieve it?</p>

<p>The problem is the lt and gt signs should be escaped like below:</p>

<pre><code>from bs4 import BeautifulSoup
html = "&lt;html&gt;&lt;h1&gt;#include  &amp;lt;stdio.h&amp;gt;&lt;/h1&gt;&lt;/html&gt;"
soup = BeautifulSoup(html,"lxml")
print(soup.h1.get_text())
</code></pre>

<p>Which would then give you:</p>

<pre><code>#include  &lt;stdio.h&gt;
</code></pre>

<p>None of the parsers are going to consider that text unless they are escaped. each one will give you <code>#include  &lt;stdio.h&gt;&lt;/stdio.h&gt;</code></p>

<p>You may just have to resort to a regex to extract the include statements from the source itself.</p>

<pre><code> patt = re.compile(r"&lt;h1&gt;(\s+)?(#include\s+.*?)(\s+)?&lt;/h1&gt;")
</code></pre>

python html beautifulsoup