SLS Doctor Directory Data Scraping

Starting URL: [login to view URL] I’d like to collect some information on doctors listed on the website above in three CSV files. Visit the URL above. For “Country,” select “United States.” For this project, we will only scrape data for United States doctors. For “Specialty,” select “GenSurg,” and click Continue. You will receive 25 results per page up to a maximum of 1,000. You will need to scrape results for each specialty. No specialty will have more than 1,000 results. There are approximately 2,000 records in total. For each result, clicking on the doctor’s name will open a profile page that contains address, phone, fax, and--rarely--email. For each doctor, we want to collect the following information, when available, in a CSV file with one row for each doctor: id (found in the URL as “id=...”) email (seldom available but should always be collected if so) firstname middlename lastname suffix (Jr., etc.) title (MD, DO, etc.) phone (should be found in professional information section, not personal information) fax addr1 addr2 city state zip country affiliation (i.e. current employer or other professional affiliation) description (i.e. job title “Program Directory, General Surgery Residency”) member_type (“SLS Physician Member”, etc.) website_url This file should be named scrape_SLS_{date} The first row should contain column names exactly as listed above. See example file [login to view URL] =============== Each doctor has one or more specialties: GenSurg, Thoracic, etc. We want to collect a CSV file with one row for each specialty in the following format: id (incrementing numeric id assigned by freelancer) scrape_SLS_{date}_id (the profile id for this doctor, found in the profile URL ) specialty This file should be named scrape_SLS_{date}_specialty The first row should contain column names exactly as listed above. See example file [login to view URL] ===================== Each doctor may or may not be a member of one or more “Special Interest Group (SIG) Committees.” As an example, this doctor is on several committees. [login to view URL] We want to collect a CSV file with one row for each committee that each doctor is on in the following format: id (incrementing numeric id assigned by the freelancer) scrape_SLS_{date}_id (the profile id for this doctor, found in the profile URL) clinical_interest (name of special interest group committee) This file should be named scrape_SLS_{date}_clinical_interest The first row should contain column names exactly as listed above. See example file [login to view URL]