How to use python to interpret a url
<p>I'm writing code that is attempting to extract the text from the <a href="https://libraryofbabel.info" rel="nofollow">Library of Babel</a>.</p>

<p>They basically use a system of Hexes, Walls, Shelfs, Volumes and Pages to split up their library of randomly generated text files. Here is an example (<a href="https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1" rel="nofollow">https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1</a>)
Here we have Hex: 2, Wall: 1, Shelf: 2, Volume: 22, Page: 1.</p>

<p>I would ideally like to randomly generate a page across all these variables to extract text from, however I am not getting the output I would imagine. </p>

<p>Here is my code:</p>

<pre><code>import requests
from bs4 import BeautifulSoup
from urlparse import urlparse
import random

hex = str(random.randint(0, 6))
wall = str(random.randint(1, 4))
shelf = str(random.randint(1, 5))
vol = str(random.randint(1, 32))
page = str(random.randint(1, 410))

print("Fetching: " + " Hex: " + hex + ", Wall: " + wall + ", Shelf: " + shelf + ", Vol: " + vol + ", Page: " + page)
babel_url = str("https://libraryofbabel.info/browse.cgi?" + hex + "-w" + wall + "-s" + shelf + "-v" + vol + ":" + page)
r = requests.get(babel_url)
soup = BeautifulSoup(r.text)
print(soup.get_text())
</code></pre>

<p>My output would be identical to that if I changed the url to be <a href="https://libraryofbabel.info/browse.cgi" rel="nofollow">https://libraryofbabel.info/browse.cgi</a>. print(babel_url) shows me that the way I wrote the url is fine but something isn't interpreting what I have written in the way I want.</p>

<p>I've found that just pasting <a href="https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1" rel="nofollow">https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1</a> into chrome drops me at <a href="https://libraryofbabel.info/book.cgi" rel="nofollow">https://libraryofbabel.info/book.cgi</a>. But if I navigate to <a href="https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1" rel="nofollow">https://libraryofbabel.info/book.cgi?2-w1-s2-v22:1</a> (or any other page) I can move between pages at will.</p>

<p>The only thing I get in the output worth mentioning is:</p>

<blockquote>
  <p>It appears your browser has javascript disabled. Follow this link to browse without javascript.</p>
</blockquote>

<p>Put on you glasses :<br>
You are requesting <code>browse.cgi</code> instead of <code>book.cgi</code><br><br>
<a href="https://libraryofbabel.info/browse.cgi?2-w2-s1-v10:72" rel="nofollow">https://libraryofbabel.info/browse.cgi?2-w2-s1-v10:72</a><br>
instead of<br>
<a href="https://libraryofbabel.info/book.cgi?2-w2-s1-v10:72" rel="nofollow">https://libraryofbabel.info/book.cgi?2-w2-s1-v10:72</a></p>

python beautifulsoup