Self learning url filter
<p>I need to classify given urls as porn or non porn via python script (not by visiting them in person and watching videos) and I thought about calculating porn probability for each url by classifying words it contains, e.g. if url contains words 'bang' and '18' there is high probability its porn site, I tried implementing it, but it isnt very accurate, are there any python libraries than can help me classify those urls? I'm looking for libraries which can learn from test data, like smart anti-spam filters, like:</p>

<pre><code>    data = {
    'google.com':0,
    'superxxx.com':1,
    'bigbangtheory.com':0,
    'hot18bangbang.com':1,
     ...
     ...
}
</code></pre>

<p>and so on, I've pretty big collection of 'bad' urls, so I think I could train some AI classifier. If this is bad idea, could you recommend me any way of filtering out 'bad' urls from 'good' urls?</p>

<p>The modern approach to do this is to use a character level LSTM sequence classifier. It requires a fairly large amount of data though, but it shouldn't be too hard to find, by getting examples of family filter black lists for example. </p>

<p>Here are some examples of the concept: </p>

<ul>
<li>I would start here, a cool article on character level LSTMs: <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow">The Unreasonable Effectiveness of Recurrent Neural Networks
</a> </li>
<li><a href="https://cs224d.stanford.edu/reports/EugeneLouis.pdf" rel="nofollow">Making a Manageable Email Experience with Deep
Learning</a> </li>
<li><a href="http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/" rel="nofollow">Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras
</a></li>
</ul>

<p>Recurrent neural networks are neural networks that take their own output as input for the next step, or that learn to output state vectors that are passed to their own cell at the next step to represent short term memory.</p>

<p>Basically, your features are sequences of sub sequences of letters (aka, friendship becomes <code>[frie, frien, riend, iends, endsh, ...]</code> in one hot representation), and you have a neural net that has a state that evolves with subsequence it sees, and gives you a judgement at the end.</p>

python machine-learning artificial-intelligence