How to get a list of two different elements with their parents being the same
<p>As an exercise I am trying to <strong>print</strong> all the titles of posts with more than 200 comments from the site reddit.com.</p>

<p>What I tried:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


url1 = "https://www.reddit.com/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

res = requests.get(url1, headers=headers)
res.raise_for_status()
soup = BeautifulSoup(res.content, "html5lib")

g = soup.select('ul &gt; li.first')
j = soup.select('#siteTable div.entry.unvoted &gt; p.title &gt; a ')
list1 = []
for t in j:
    list.append(t.text)

list2=[]
for s in g:
    for p in s.text.split(" "):
        if p.isdigit():
            p = int(p)
            if p &gt; 100:
                list2.append(p)

for q,l in zip(list1, list2):
    if l &gt; 200:
        print(q,l)
</code></pre>

<p>Problem:</p>

<p>It works halfway until there is a <strong>hiccup somewhere</strong> and the lists don't match anymore. As a result, I get titles that have less than 200 comments.</p>

<p>Output:</p>

<pre><code>What the F David Blaine!! 789
So NYC MTA (subway) banned all dogs unless the owner carries them in a bag. I think this owner nailed it. 1075
Bad to the bone 307
TIL there is a "white man" cafÃ© in Tokyo, where Japanese ladies ring a bell to summon tuxedo-wearing caucasians who respond with "yes, princess?" and serve them cake 2145
Earthquake Warning Issued in California 1410
Man impersonating officer busted for attempting to pull over unmarked cruiser 1022
Use of body-worn cameras sees complaints against police âvirtually vanishâ, study finds 2477
Amazing one handed interception 759
A purrfectly executed leap 518
"This bed has a fur pillow, I'll lay here." 792
Back in 'Nam, 1969. Guy on the left is a good friend of mine's dad. He's in hospice now and not doing well but he'll live on in photos. 264
Nintendo Entertainment System: NES Classic Edition - with 30 games - Available in US 11/11/16 290
A scenic view ruined by a drunk driver (Star Wars: Battlefront) 2737
Clouds battling a sunset over Olympic National Park, WA, USA (1334x750) [OC] 2222
What company is totally guilty of false advertising and why? 2746
South Korean President Park Geun-hye has called on North Koreans to abandon their country and defect, just a day after a soldier walked across the heavily fortified border into the South 410
TIFU by underestimating the stupidity of multiple people 334
Special Trump burger at a burger chain in South Africa 311
This Special Ed Teacher Had All of Her Students in Her Wedding 984
</code></pre>

<p>The messup starts after <em>"A scenic view ruined by..."</em></p>

<p>Can someone point me to what is the exact problem here or an alternative way?</p>

<p>Instead of saving it first to lists and hoping that both lists would match up (list1[0]~~list2[0])....I tried to find the most common denominator (parent) and applied the class selection (beautifulsoup) a second time to delve further down the dom (children) and printing it instantly. While scraping a heavily used website such as reddit, it is a good possibility to change even if its split seconds apart and a viable cause for hiccups when saving data to lists and comparing them. </p>

<p>Solution:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


url1 = "https://www.reddit.com/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

res = requests.get(url1, headers=headers)
res.raise_for_status()
soup = BeautifulSoup(res.content, "html5lib")

k= soup.select('#siteTable div.entry.unvoted') # partent

for v in k:
    d = v.select('ul &gt; li.first') #comment
    o = v.select('p.title &gt; a') #title
    for z,x in zip(d,o):
        for p in z.text.split(" "): # convert "351 comments" to integer "351" and compare with 200 
            if p.isdigit():
                p = int(p)
                if p &gt; 200:
                    print(z.text, x.text) #print comments first then title
</code></pre>

python web-scraping beautifulsoup