BeautifulSoup to return div class content
<pre><code>import requests
from bs4 import BeautifulSoup

url = 'https://weather.com/weather/today/l/90006:4:US'
r = requests.get(url)
html_content = r.text
soup = BeautifulSoup(html_content, 'lxml')

weather_row= soup.find('div', {"class" : "today_nowcard-hili"})

print weather_row
</code></pre>

<p>I found the class <code>today_nowcard-hili</code> that returns weather highs and lows: i.e <code>H 84Â°  / L 61Â°</code>. But the above code keeps giving me <code>None</code></p>

<p>It is <em>hilo</em> not <em>hili</em>:</p>

<pre><code>soup.find('div', {"class" : "today_nowcard-hilo"})
</code></pre>

<p>But you can see from what it returns that the data is inserted using Js:</p>

<pre><code>&lt;div class="today_nowcard-hilo"&gt;
&lt;span class="btn-text" data-ng-bind="::'H' | pfTranslate: {context: 'today_nowcard'}"&gt;&lt;/span&gt;
&lt;span class="deg-hilo-nowcard" data-gm-wx-temperature="::todayWxcardVm.forecast.items[0].day.daytemp" data-text-to-replace="{{ todayWxcardVm.getProperValue('[[ forecast.day.temperature[0] ]]', '--') }}"&gt;[[ forecast.day.temperature[0] || '--' ]]&lt;/span&gt;
&lt;span data-ng-if="::todayWxcardVm.forecast.items[0].day.daytemp"&gt; /&lt;/span&gt;
&lt;span class="btn-text" data-ng-bind="::'L' | pfTranslate: {context: 'today_nowcard'}"&gt;&lt;/span&gt;
&lt;span class="deg-hilo-nowcard" data-gm-wx-temperature="::todayWxcardVm.forecast.items[0].day.nighttemp" data-text-to-replace="{{ todayWxcardVm.getProperValue('[[ forecast.night.temperature[0] ]]', '--') }}"&gt;[[ forecast.night.temperature[0] || '--' ]]&lt;/span&gt;
&lt;div&gt;
&lt;span class="btn-text" data-ng-bind="::('UV Index' | pfTranslate: { context: 'weather_terms'})"&gt;&lt;/span&gt;
&lt;span data-gm-wx-uv-index="::todayWxcardVm.obs.uvIndex"&gt;&lt;/span&gt;
&lt;/div&gt;
&lt;/div&gt;
</code></pre>

<p>The weather info is retrieved through an ajax request:</p>

<pre><code>r = requests.get("https://api.weather.com/v2/turbo/vt1precipitation;vt1currentdatetime;vt1pollenforecast;vt1dailyForecast;vt1observation?units=e&amp;language=en-US&amp;geocode=34.05,-118.29&amp;format=json&amp;apiKey=c1ea9f47f6a88b9acb43aba7faf389d4")
print(r.json())
</code></pre>

<p>Which would give you:</p>

<pre><code>{u'vt1currentdatetime': {u'tmZnAbbr': u'PDT', u'datetime': u'2016-10-10T10:54:44.496-07:00'}, u'vt1dailyForecast': {u'dayOfWeek': [u'Monday', u'Tuesday', u'Wednesday', u'Thursday', u'Friday', u'Saturday', u'Sunday', u'Monday', u'Tuesday', u'Wednesday', u'Thursday', u'Friday', u'Saturday', u'Sunday', u'Monday'], u'moonrise': [u'2016-10-10T14:56:44-0700', u'2016-10-11T15:39:14-0700', u'2016-10-12T16:20:52-0700', u'2016-10-13T17:01:06-0700', u'2016-10-14T17:41:27-0700', u'2016-10-15T18:23:24-0700', u'2016-10-16T19:06:47-0700', u'2016-10-17T19:53:38-0700', u'2016-10-18T20:44:27-0700', u'2016-10-19T21:38:21-0700', u'2016-10-20T22:36:08-0700', u'2016-10-21T23:34:45-0700', None, u'2016-10-23T00:34:37-0700', u'2016-10-24T01:32:49-0700'], u'moonset': [u'2016-10-10T00:57:11-0700', u'2016-10-11T01:55:40-0700', u'2016-10-12T02:57:31-0700', u'2016-10-13T04:01:56-0700', u'2016-10-14T05:08:27-0700', u'2016-10-15T06:17:17-0700', u'2016-10-16T07:26:40-0700', u'2016-10-17T08:37:23-0700', u'2016-10-18T09:46:17-0700', u'2016-10-19T10:53:04-0700', u'2016-10-20T11:54:49-0700', u'2016-10-21T12:51:05-0700', u'2016-10-22T13:40:38-0700', u'2016-10-23T14:25:15-0700', u'2016-10-24T15:04:29-0700'], u'validDate': [u'2016-10-10T07:00:00-0700', u'2016-10-11T07:00:00-0700', u'2016-10-12T07:00:00-0700', u'2016-10-13T07:00:00-0700', u'2016-10-14T07:00:00-0700', u'2016-10-15T07:00:00-0700', u'2016-10-16T07:00:00-0700', u'2016-10-17T07:00:00-0700', u'2016-10-18T07:00:00-0700', u'2016-10-19T07:00:00-0700', u'2016-10-20T07:00:00-0700', u'2016-10-21T07:00:00-0700', u'2016-10-22T07:00:00-0700', u'2016-10-23T07:00:00-0700', u'2016-10-24T07:00:00-0700'], u'moonPhrase': [u'Waxing Gibbous', u'Waxing Gibbous', u'Waxing Gibbous', u'Waxing Gibbous', u'Waxing Gibbous', u'Full Moon', u'Full Moon', u'Waning Gibbous', u'Waning Gibbous', u'Waning Gibbous', u'Waning Gibbous', u'Waning Gibbous', u'Last Quarter', u'Waning Crescent', u'Waning Crescent'], u'sunset': [u'2016-10-10T18:24:49-0700', u'2016-10-11T18:23:32-0700', u'2016-10-12T18:22:15-0700', u'2016-10-13T18:20:59-0700', u'2016-10-14T18:19:43-0700', u'2016-10-15T18:18:29-0700', u'2016-10-16T18:17:15-0700', u'2016-10-17T18:16:02-0700', u'2016-10-18T18:14:49-0700', u'2016-10-19T18:13:38-0700', u'2016-10-20T18:12:27-0700', u'2016-10-21T18:11:18-0700', u'2016-10-22T18:10:09-0700', u'2016-10-23T18:09:02-0700', u'2016-10-24T18:07:55-0700'], u'night': {u'iconExtended': [3100, 3100, 3100, 2900, 2900, 2700, 2700, 3100, 3100, 3100, 3100, 3100, 3100, 3100, 3100], u'windDirCompass': [u'SSW', u'SSW', u'SE', u'SSE', u'S', u'S', u'SSW', u'NNW', u'N', u'NNE', u'NE', u'NNE', u'NE', u'SE', u'ESE'], u'humidityPct': [75, 87, 92, 97, 98, 99, 100, 56, 33, 35, 36, 39, 55, 65, 67], u'temperature': [61, 58, 56, 59, 62, 62, 62, 60, 60, 62, 62, 61, 59, 58, 56], u'precipType': [u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain'], u'uvDescription': [u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low', u'Low'], u'cloudPct': [1, 8, 5, 38, 53, 67, 66, 5, 4, 8, 7, 10, 4, 4, 6], u'narrative': [u'A clear sky. Low 61F. Winds light and variable.', u'Clear skies. Low 58F. Winds light and variable.', u'Clear. Low 56F. Winds light and variable.', u'Clear during the evening followed by cloudy skies overnight. Low 59F. Winds light and variable.', u'Clear skies in the evening then becoming cloudy overnight. Low 62F. Winds light and variable.', u'Partly cloudy skies early will become overcast later during the night. Low 62F. Winds light and variable.', u'Partly cloudy skies early will become overcast later during the night. Low 62F. Winds light and variable.', u'Clear skies. Low around 60F. Winds light and variable.', u'Clear. Low around 60F. Winds light and variable.', u'Clear skies. Low 62F. Winds NNE at 5 to 10 mph.', u'Clear skies. Low 62F. Winds NE at 10 to 15 mph.', u'Clear. Low 61F. Winds NNE at 5 to 10 mph.', u'Clear. Low 59F. Winds light and variable.', u'Clear skies. Low 58F. Winds light and variable.', u'Clear skies. Low 56F. Winds light and variable.'], u'thunderEnum': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'qualifier': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], u'windSpeed': [3, 2, 2, 2, 3, 3, 4, 5, 5, 9, 11, 7, 2, 2, 1], u'thunderEnumPhrase': [u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder'], u'windDirDegrees': [193, 207, 144, 152, 171, 174, 197, 332, 351, 13, 36, 33, 51, 132, 116], u'snowRange': [u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u''], u'precipPct': [10, 10, 10, 10, 10, 10, 20, 0, 0, 0, 0, 0, 0, 0, 0], u'precipAmt': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], u'phrase': [u'Clear', u'Clear', u'Clear', u'Partly Cloudy', u'Partly Cloudy', u'Mostly Cloudy', u'Mostly Cloudy', u'Clear', u'Clear', u'Clear', u'Clear', u'Clear', u'Clear', u'Clear', u'Clear'], u'dayPartName': [u'Tonight', u'Tomorrow night', u'Wednesday night', u'Thursday night', u'Friday night', u'Saturday night', u'Sunday night', u'Monday night', u'Tuesday night', u'Wednesday night', u'Thursday night', u'Friday night', u'Saturday night', u'Sunday night', u'Monday night'], u'uvIndex': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'icon': [31, 31, 31, 29, 29, 27, 27, 31, 31, 31, 31, 31, 31, 31, 31]}, u'moonIcon': [u'WXG', u'WXG', u'WXG', u'WXG', u'WXG', u'F', u'F', u'WNG', u'WNG', u'WNG', u'WNG', u'WNG', u'LQ', u'WNC', u'WNC'], u'day': {u'iconExtended': [3200, 3200, 3200, 3200, 3000, 9003, 9003, 9003, 3200, 3200, 3200, 3200, 3200, 3200, 3200], u'windDirCompass': [u'SW', u'SSW', u'SSW', u'SSW', u'SSW', u'SW', u'SW', u'W', u'NW', u'N', u'NE', u'NNE', u'SSE', u'S', u'S'], u'humidityPct': [40, 66, 73, 71, 75, 84, 87, 68, 29, 21, 25, 25, 32, 40, 51], u'temperature': [82, 75, 71, 74, 75, 73, 73, 75, 80, 84, 84, 84, 82, 79, 76], u'precipType': [u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain', u'rain'], u'uvDescription': [u'High', u'High', u'High', u'High', u'High', u'High', u'High', u'High', u'High', u'High', u'High', u'Moderate', u'Moderate', u'Moderate', u'Moderate'], u'cloudPct': [3, 6, 4, 7, 41, 46, 60, 45, 5, 5, 6, 9, 8, 4, 8], u'narrative': [u'Sunny. High 82F. Winds SW at 5 to 10 mph.', u'Sunny skies. High near 75F. Winds light and variable.', u'Sunny skies. High 71F. Winds light and variable.', u'Sunny. High 74F. Winds light and variable.', u'Sunshine and clouds mixed. High around 75F. Winds SSW at 5 to 10 mph.', u'Cloudy early, becoming mostly sunny in the afternoon. High 73F. Winds light and variable.', u'Cloudy skies early, then partly cloudy in the afternoon. High 73F. Winds SW at 5 to 10 mph.', u'Cloudy early, becoming mostly sunny in the afternoon. High around 75F. Winds W at 5 to 10 mph.', u'Mainly sunny. High around 80F. Winds NW at 5 to 10 mph.', u'Mainly sunny. High 84F. Winds light and variable.', u'Sunny skies. High 84F. Winds NE at 10 to 20 mph.', u'Sunny skies. High 84F. Winds NNE at 5 to 10 mph.', u'A mainly sunny sky. High 82F. Winds SSE at 5 to 10 mph.', u'Mainly sunny. High 79F. Winds light and variable.', u'Sunny skies. High 76F. Winds light and variable.'], u'thunderEnum': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'qualifier': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], u'windSpeed': [6, 5, 4, 4, 6, 5, 6, 6, 9, 5, 15, 9, 6, 4, 4], u'thunderEnumPhrase': [u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder', u'No thunder'], u'windDirDegrees': [234, 213, 205, 203, 204, 217, 220, 281, 321, 8, 35, 25, 153, 180, 189], u'snowRange': [u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u'', u''], u'precipPct': [0, 10, 10, 10, 10, 10, 10, 10, 0, 0, 0, 0, 0, 0, 0], u'precipAmt': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], u'phrase': [u'Sunny', u'Sunny', u'Sunny', u'Sunny', u'Partly Cloudy', u'AM Clouds/PM Sun', u'AM Clouds/PM Sun', u'AM Clouds/PM Sun', u'Sunny', u'Sunny', u'Sunny', u'Sunny', u'Sunny', u'Sunny', u'Sunny'], u'dayPartName': [u'Today', u'Tomorrow', u'Wednesday', u'Thursday', u'Friday', u'Saturday', u'Sunday', u'Monday', u'Tuesday', u'Wednesday', u'Thursday', u'Friday', u'Saturday', u'Sunday', u'Monday'], u'uvIndex': [6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 5], u'icon': [32, 32, 32, 32, 30, 30, 30, 30, 32, 32, 32, 32, 32, 32, 32]}, u'sunrise': [u'2016-10-10T06:54:58-0700', u'2016-10-11T06:55:43-0700', u'2016-10-12T06:56:30-0700', u'2016-10-13T06:57:16-0700', u'2016-10-14T06:58:03-0700', u'2016-10-15T06:58:50-0700', u'2016-10-16T06:59:37-0700', u'2016-10-17T07:00:25-0700', u'2016-10-18T07:01:13-0700', u'2016-10-19T07:02:01-0700', u'2016-10-20T07:02:50-0700', u'2016-10-21T07:03:40-0700', u'2016-10-22T07:04:29-0700', u'2016-10-23T07:05:20-0700', u'2016-10-24T07:06:10-0700']}, u'vt1precipitation': {u'severity': [1], u'characteristic': [0], u'forecastedRainAmount': [0.0], u'intensity': [0], u'imminence': [0], u'startTime': [u'2016-10-10T11:00:00-0700'], u'eventType': [0], u'endTime': [u'2016-10-10T18:00:00-0700'], u'forecastedSnowAmount': [0.0]}, u'vt1observation': {u'icon': 32, u'phrase': u'Sunny', u'barometerCode': 1, u'precip24Hour': 0.0, u'temperatureMaxSince7am': 79, u'dewPoint': 49, u'barometerTrend': u'Rising', u'snowDepth': 0.0, u'windDirCompass': u'WSW', u'visibility': 10.0, u'feelsLike': 79, u'altimeter': 30.0, u'temperature': 79, u'uvDescription': u'Moderate', u'gust': None, u'humidity': 35, u'windSpeed': 4, u'windDirDegrees': 250, u'obsQualifierCode': None, u'observationTime': u'2016-10-10T10:25:00-0700', u'obsQualifierSeverity': None, u'uvIndex': 4, u'barometerChange': 0.03}, u'vt1pollenforecast': {u'reportDate': [u'2016-10-10T07:00:00.000-07:00', u'2016-10-11T07:00:00.000-07:00', u'2016-10-12T07:00:00.000-07:00', u'2016-10-13T07:00:00.000-07:00'], u'grass': [0, 0, 0, 0], u'tree': [0, 0, 0, 0], u'weed': [0, 0, 0, 0]}, u'id': u'34.05,-118.29'}
</code></pre>

<p>You can see all the parameters passed including the <code>geocode=34.05,-118.29</code> so you would need to get the coordinates for each location, that is available in the source in one of the scripts as json under <code>"lat":34.05,"long":-118.29</code>:</p>

<pre><code>  window.explicit_location = "90006:4:US";
      window.explicit_location_obj = {"zipCd":"90006","cntryCd":"US","procTm":"20160905130833","locId":"90006","cityNm":"LOS ANGELES","stCd":"CA","prsntNm":"Los Angeles, CA (90006)","coopId":"72295023","lat":34.05,"long":-118.29,"obsStn":"KCQT","secObsStn":"KHHR","tertObsStn":"KSMO","gmtDiff":-8.0,"regSat":"sw","cntyId":"CAC037","cntyNm":"LOS ANGELES","zoneId":"CAZ041","zoneNm":"Los Angeles County Coast including Downtown Los Angeles","cntyFips":"06037","active":1,"dySTInd":"Y","dmaCd":803,"elev":197,"cliStn":"045115","tmZnNm":"Pacific Daylight Time","tmZnAbbr":"PDT","dySTAct":"Y","clsRad":"LAX","ultRad":"LAX","ssRad":"sw","lsRad":"we","siteId":"US","idxId":"KCQT","primTecci":"T72295023","arptId":"LAX","mrnZoneId":"PZZ655","pllnId":"SAN","skiId":"267","tideId":"W9410777","epaId":"ca131","_arptNear":["BUR","LAX","LGB"],"_arptNearDist":[{"key":"BUR:9:US","dist":9},{"key":"LAX:9:US","dist":9},{"key":"LGB:9:US","dist":15}],"_skiNear":[{"key":"261:11:US","tLifts":14,"dist":37},{"key":"267:11:US","tLifts":4,"dist":41},{"key":"402:11:US","tLifts":12,"dist":75},{"key":"43:11:US","tLifts":12,"dist":83},{"key":"400:11:US","tLifts":14,"dist":84},{"key":"361:11:US","tLifts":11,"dist":106},{"key":"227:11:US","tLifts":28,"dist":111},{"key":"194:11:US","tLifts":7,"dist":117},{"key":"29:11:US","tLifts":5,"dist":133},{"key":"134:11:US","tLifts":12,"dist":154}],"_gprId":"NAM","_dstDates":{"startDate":"2016-03-13T10:00:00.000Z","endDate":"2016-11-06T09:00:00.000Z"},"wmId":"SMO","PollenIds":{"tree":"KSNA","grass":"KSNA","ragweed":"KSNA"},"isBoatBeach":true,"locType":4,"stNm":"California","_country":"United States Of America"};
</code></pre>

<p>But you would be using the sites api key which I would not recommend. There are various api's lots which offer free plans you can use to get the weather details like <a href="https://openweathermap.org/appid#get" rel="nofollow">https://openweathermap.org/appid#get</a> or <a href="https://www.wunderground.com/weather/api/d/docs" rel="nofollow">https://www.wunderground.com/weather/api/d/docs</a>.</p>

python beautifulsoup