Python: find all urls which contain string
<p>I am trying to find all pages that contain a certain string in the name, on a certain domain. For example: </p>

<pre><code>www.example.com/section/subsection/406751371-some-string 
www.example.com/section/subsection/235824297-some-string 
www.example.com/section/subsection/146783214-some-string
</code></pre>

<p>What would be the best way to do it?</p>

<p>The numbers before "-some-string" can be any 9-digit number. I can write a script that loops through all possible 9-digit numbers and tries to access the resulting url, but I keep thinking that there should be a more efficient way to do this, especially since I know that overall there are only about 1000 possible pages that end with that string. </p>

<p>I understood your situation, the numeric value before -some-string is a kind of object id for that web site (for example, this question has a id 39594926, and the url is stackoverflow.com/questions/<strong>39594926</strong>/python-find-all-urls-which-contain-string)</p>

<p>I don't think there is a way to find all valid numbers, unless you have a listing (or parent) page from that website lists all these numbers. Take Stackoverflow as an example again, in the question list page, you will see all these question ids.</p>

<p>If you can provide me the website, I could have a look try to find the 'pattern' of these numbers. For some simple website, that number is just a increment to identify objects (could be user, question or anything else).</p>

python url