You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here's the started code used for starting up on writing a new scraper:
from ..utilsimportScraperfrombs4importBeautifulSoupfromcollectionsimportOrderedDictfromdatetimeimportdatetime, dateimportjsonimportrequestsimportpytzclassScraperName:
"""A scraper for <scraper description>."""host='<scraper website>'@staticmethoddefscrape(location='.'):
Scraper.logger.info('ScraperName initialized.')
# do the magic hereScraper.logger.info('ScraperName completed.')
Remember that output goes to JSON files in the given location parameter as the path. Also, the dictionary used to dump the JSON should be an OrderedDict to preserve order.
Testing scrapers
I test the scrapers using a test.py script like the following:
importuoftscrapersimportloggingimportsys# Set up logging so it prints to standard outputlogger=logging.getLogger("uoftscrapers")
logger.setLevel(logging.INFO)
ch=logging.StreamHandler(sys.stdout)
ch.setLevel(logging.INFO)
logger.addHandler(ch)
# Run scraperuoftscrapers.ScraperName.scrape(location='./data')
Placed at the root of the repo, this will import the uoftscrapers from the source and not the one pip3 is managing.