Scraping a website that requires authentication
<p>I know this question might seem quite straight forward, but I have tried every suggestion and none has worked.</p>

<p>I want to build a Python script that checks my school website to see if new grades have been put up. However I cannot for the life of me figure out how to scrape it.</p>

<p>The website redirects to a different page to login. I have tried all the scripts and answers I could find but I am lost.</p>

<p>I use Python 3, the website is in a <code>https://blah.schooldomate.state.edu.country/website/grades/summary.aspx</code> 
format</p>

<p>The username section contains the following:</p>

<p><code>&lt;input class="txt" id="username" name="username" type="text" autocomplete="off" style="cursor: auto;"&gt;</code></p>

<p>The password is the name except it contains an <code>onfocus</code> HTML element.</p>

<p>One successfully authenticated, I am automatically redirected to the correct page.</p>

<p>I have tried:</p>

<p>using Python 2's cookielib and Mechanize</p>

<p>Using HTTPBasicAuth</p>

<p>Passing the information as a dict to a requests.get()</p>

<p>Trying out many different peoples code including answers I found on this site</p>

<p>Maybe you can use Selenium library.</p>

<p>I let you my code example:</p>

<pre><code>from selenium import webdriver

def loging():
    browser = webdriver.Firefox()
    browser.get("www.your_url.com")

    #Edit the XPATH of Loging INPUT username
    xpath_username = "//input[@class='username']"  

    #Edit the XPATH of Loging INPUT password
    xpath_password = "//input[@class='password']"  

    #THIS will write the YOUR_USERNAME/pass in the xpath (Custom function)
    click_xpath(browser, xpath_username, "YOUR_USERNAME")  
    click_xpath(browser, xpath_username, "YOUR_PASSWORD")  

    #THEN SCRAPE WHAT YOU NEED

#Here is the custom function
#If NO input, will only click on the element (on a button for example)
def click_xpath(self, browser, xpath, input="", time_wait=10):
    try:
        browser.implicitly_wait(time_wait)
        wait = WebDriverWait(browser, time_wait)
        search = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
        search.click()
        sleep(1)
        #Write in the element
        if input:
            search.send_keys(str(input) + Keys.RETURN)
        return search
    except Exception as e:
        #print("ERROR-click_xpath: "+xpath)
        return False
</code></pre>

python