In your existing script, you get the source file using HTTP. In order to be able to download all files, the HTTP server must allow a directory listing; otherwise there is no way to know the existing file names. Also, the web server must implement the DELETE verb (not so common) to be able to delete files remotely. In short, HTTP is not a good option.
If you control the remote computer, set up a FTP server instead. (On Windows with IIS, just add the FTP service; on Linux, probably it's already there). Listing files and directories, downloading, and deleting them is a lot easier with FTP. It may be an old protocol, but it has been designed exactly for that purpose.
If you go for FTP, I can provide a script for transfering a complete tree, with folders and subfolders. Other solutions exists too, we can discuss them by PM.