Website crawler and extractor

Please make sure you read this job description completely before you apply. There are many details that you must know.

I am looking for someone who can create a desktop software that will extract and recreate a website archived on [login to view URL]

The main steps the software must follow are:

• Crawl the specific website to find all of its URL available in [login to view URL]

• Download all pages and documents including video, audio, PDF, images, CSS, javascript etc

• Modify the URL structure (all pages and documents must be linked together correctly), so that the website can be uploaded to another domain/directory.

• Save the results in a ZIP file

Important:

• It MUST work for ALL websites, including websites that use server side programmation, such as PHP, ASP and databases. (I know this sounds challenging, but if you apply for this job or contact me, I will show you a shortcut for this).

• All dead links must be redirected to the root directory, using a permanent 301 redirection (this is a simple line of code to add in a htaccess file)

• Some content will need to be included in a [login to view URL] file in the root directory (I will provide this content).

• External links must work

• The sites extracted from [login to view URL] must work on a different domain/directory than the original one

• You need to extract the content up to 10 pages deep in the website.

• There must be no [login to view URL] footprint left in the source code when I upload a website to a domain.

• The URLs that appear in the navigation bar of a web browser must be the same as they were on the original site. (if uploaded to the same domain of course, otherwise only the domain name is different in the url)

Keep reading, there is more important information below.

The input for the software are:

• The url from the wayback machine, ex: [login to view URL]://[login to view URL]

• The path where I want to save the ZIP file on my computer

The oupput:

• A zip file that contains all the files (html, image, video, wave, etc) that I can upload to the domain/directory of my choice.

To summarize, I want a software that allows me to extract a website from [login to view URL] in a ZIP file so that I then upload and extract this ZIP on a domain and the website is immediately working perfectly without any work required from me.

Keep reading...

If you apply for this job I will show you a couple of websites that were extracted using this method, so you can see how it was done.

IMPORTANT: When you apply for this job, please have "ARCHIVE" in the title of your job application or cover letter.

If I don't see ARCHIVE in the title I will not open it and you don't get the job. I have to do this to make sure freelancers read the job description before they apply.

Additionnal note:

I will be the only owner of the resulting software and I request to receive all source files.

Pengembangan Perangkat Lunak Web Scraping

ID Proyek: #6359986

Tentang proyek