Store Scraped WebPage with Python SqlAlchemy
<p>I do many look ups and scrapes of webpages and I would like to store the response to those lookups in a database to avoid having to request them again (to improve speed but also to not be a nuisance).  I am using sqlalchemy and python, and I've created an ORM class like below:</p>

<pre><code>class MyClass(Base):
    __tablename__ = "myclass"
    url = Column(String, primary_key=True)
    article = Column(String)
</code></pre>

<p>When creating a new article and adding it to my session, i.e.:</p>

<pre><code>session.add(MyClass(url=url,article=requests.get(url).text)
</code></pre>

<p>I am able to add it to the database.  However, when I query the url's in the database, I receive this message back,</p>

<pre><code>OperationalError: (sqlite3.OperationalError) unrecognized token: ':' [SQL: u'SELECT http://[mywebsite]]
</code></pre>

<p>I understand that it is having trouble with the ':' in http:// but am wondering if that is the case, is there any way to use a full web url as the primary key? </p>

<p>Furthermore, what would be the recommended data types, both in terms of storage efficiency and look up efficiency to use for storing a webpage and its corresponding content?</p>

<p>I think, escape the url before store it will resolve the problem.</p>

<p>eg:</p>

<pre><code>article = requests.get(url).text
url = url.replace(":", "COLON")
session.add(MyClass(url=url,article=article)
</code></pre>

<p>In a similar way, you can restore it when you take data out</p>

python web-scraping sqlalchemy