Using Scrapy with authenticated (logged in) user session

In the code above, the FormRequest that is being used to authenticate has the after_login function set as its callback. This means that the after_login function will be called and passed the page that the login attempt got as a response. It is then checking that you are successfully logged in by searching the page for a specific string, in this case "authentication failed". If it finds it, the spi