Gathering the default landing page URL for a large ip hostlist on a single port
<h2>So here is the requirement breakdown:</h2>

<ul>
<li>Parse NMAP output to define a hostlist that is serving web services.</li>
<li>Use a hostlist to define the default landing page URL for each listening protocol.</li>
<li>Compile a master web services list to include IP:URL.</li>
</ul>

<h2>What I have done so far:</h2>

<ul>
<li>Generated a hostlist that includes all hosts listening for HTTP/S traffic. Each hostlist is separated by port/ssl usage.</li>
</ul>

<h2>What I am having trouble with:</h2>

<ul>
<li><p>It seems logical to be able to do the following: Use a list with the IP of every host listening for http over port 80 to run a query as if we are typing <em><a href="http://test.ip" rel="nofollow">http://test.ip</a></em> into a browser and capturing the default URL it lands on. <em><a href="http://test.url/index.html" rel="nofollow">http://test.url/index.html</a></em></p></li>
<li><p>I have tried to use cURL and Wget and worked a bit with Scrapy but I am not able to capture just the default landing page URL in a scripted fashion. All I need is a means to take a single IP and resolve only the default landing page URL for a single port. I can then integrate this function into a broadened script to run against each port/prot.  </p></li>
</ul>

<p>Why bother with all the extra parsing and tools if you're already scanning with Nmap? The <a href="https://nmap.org/book/nse.html" rel="nofollow">Nmap Scripting Engine</a> can do what you want easily for each HTTP port discovered. You can even use an existing script: <a href="https://nmap.org/nsedoc/scripts/http-title.html" rel="nofollow">http-title</a> will request the URL <code>/</code> and follow up to 5 redirects in order to retrieve the HTML title of the landing page. If a redirect is followed, it includes "Requested resource was <em>/path</em>" to show what the actual landing page URL was.</p>

python linux bash nmap penetration-testing