Wget is a popular command-line utility in Linux that is used to download files from the internet. It supports various protocols such as HTTP, HTTPS, and FTP, making it a versatile tool for downloading files. In this article, we will take a deep dive into the Wget command and explore its various options and how to use them with examples.
Table of Contents show
Installing Wget Command in Linux:
Wget is a command-line utility and is typically pre-installed on most Linux distributions. However, if it is not installed, you can install it using your distribution’s package manager.
For Ubuntu and Debian-based distributions, use the following command:
sudo apt-get install wgetIn Fedora and Red Hat-based distributions, use the following command:
sudo yum install wgetFor Arch Linux-based distributions, use the following command:
sudo pacman -S wgetBasic Syntax of wget Command:
The basic syntax of the Wget command is as follows:
wget [options] [URL]Where options are the various flags that can be used to modify the behavior of the Wget command and URL is the address of the file that needs to be downloaded.
wget Command Options in Linux:
Here is a list of some of the options available for the Wget command:
- -o or –output-file: specify the file name to which the downloaded file should be saved
- -c or –continue: continue the download of a file that was previously interrupted
- -r or –recursive: download all files in a directory recursively
- -nc or –no-clobber: prevent overwriting of existing files with the same name
- -np or –no-parent: prevent going to the parent directory when downloading a file
- -q or –quiet: run in quiet mode, suppressing output messages
- -t or –tries: specify the number of times to try downloading a file before giving up
- -T or –timeout: specify the maximum time to wait for a connection before giving up
- -A or –accept: specify the file types to download
- -N or –timestamping: only download files that are newer than the local copy
- -nc or –no-clobber: prevent overwriting of existing files
- -p or –page-requisites: download all necessary files to display a HTML page offline
- -k or –convert-links: convert links in the document to point at the locally downloaded files
- -P or –directory-prefix: specify the directory where files should be saved
- –limit-rate: specify the download rate limit
- –no-check-certificate: don’t check the server certificate against the available certificate authorities
- -U or –user-agent: specify the user agent string to use when making requests
- -nv or –no-verbose: run in non-verbose mode
- -i or –input-file: specify a file containing a list of URLs to download
- -B or –base: specify the base URL to which the -i option URLs are relative
- -e or –execute: specify a command to run after the download is complete
- -nd or –no-directories: don’t create directories when saving files
- -nH or –no-host-directories: don’t create host directories when saving files
- -x or –force-directories: force creation of directories
- -S or –server-response: display the server response headers
- -O or –output-document: specify the file name to which the downloaded file should be saved.
This is not a complete list of all the options available for the Wget command, but it includes some of the most commonly used options. You can find the complete list of options by running the command wget --help in your terminal or click here.
How to Download Files with Wget Command in Linux
To download a file with Wget, you can use the following command format:
wget [options] [URL]For example, to download a file located at “https://example.com/downloads/file.zip” and save it to the current directory, you can use the following command:
wget https://example.com/downloads/file.zipYou can also specify a different file name to save the downloaded file as using the -O or –output-document option. For example, to save the file as “myfile.zip” instead of “file.zip”, you can use the following command:
wget -O myfile.zip https://example.com/downloads/file.zipIf the file you are downloading is large, you can use the -c or –continue option to continue the download from where it left off in case of a connection interruption. For example:
wget -c https://example.com/downloads/largefile.zipYou can also limit the download speed using –limit-rate option. For example, if you want to limit the download speed to 100KB/s you can use the following command:
wget --limit-rate=100k https://example.com/downloads/largefile.zipYou can also use -nc option if you want to prevent overwriting of existing files with the same name:
wget -nc https://example.com/downloads/largefile.zipYou can also specify multiple options together to customize your download. For example:
wget -nc -c --limit-rate=100k -O myfile.zip https://example.com/downloads/largefile.zipDownloading a File to a Specific Directory using Wget Command
By default, Wget will save the downloaded files to the current directory. However, you can specify a different directory to save the files using the -P or –directory-prefix option.
For example, to download a file located at “https://example.com/downloads/file.zip” and save it to the directory “~/Downloads”, you can use the following command:
wget -P ~/Downloads https://example.com/downloads/file.zipYou can also create nested directories while saving the file using -P option. For example, to download a file and save it in a directory called ‘new_folder’ inside the ‘Downloads’ directory, you can use the following command:
wget -P ~/Downloads/new_folder https://example.com/downloads/file.zipIt is important to note that if the specified directory does not exist, Wget will create it for you. However, if you do not have the necessary permissions to create a directory in the specified location, Wget will not be able to save the file there and will instead save it to the current directory.
In addition, you can also use -nc option if you want to prevent overwriting of existing files with the same name in the specified directory:
wget -nc -P ~/Downloads/new_folder https://example.com/downloads/file.zipIn this way, you can easily download files to a specific directory using the Wget command.
Downloading in Background with Wget Command
Wget is a command-line utility, and by default, it runs in the foreground, meaning the terminal will be occupied while the download is in progress. This can be inconvenient if you want to continue working in the terminal while the download is happening.
To download a file in the background, you can use the -b or –background option. For example, to download a file located at “https://example.com/downloads/file.zip” in the background, you can use the following command:
wget -b https://example.com/downloads/file.zipWhen you run this command, Wget will start the download in the background and return control to the terminal immediately. You can check the progress of the download by looking at the log file (wget-log by default) or using the pgrep and ps command.
You can also specify the log file name using -o or –output-file option. For example, to save the log information in a file called “mylog.log”, you can use the following command:
wget -b -o mylog.log https://example.com/downloads/file.zipBy downloading in background, you can continue to work in the terminal while the download is happening and also you can check the download progress using the log file.
It is also possible to send the download to background and detach from terminal using the nohup command. For example,
nohup wget https://example.com/downloads/file.zip &This will start the download in the background and return control to the terminal immediately, and it will also continue running even if you log out of the terminal.
Changing the Wget User-Agent
The User-Agent is a string that is sent by the web browser or application to the web server with each request. It is used by the server to identify the client and provide appropriate content. By default, Wget sends “Wget/version” as the User-Agent. However, you can change it to any string you want using the -U or –user-agent option.
For example, to change the User-Agent to “MyApp/1.0”, you can use the following command:
wget --user-agent="MyApp/1.0" https://example.com/downloads/file.zipIt is also possible to use different User-Agent for different sites. For example, to use the User-Agent “MyApp/1.0” for “example.com” and “MyApp/2.0” for “example2.com”, you can use the following command:
wget --user-agent="MyApp/1.0" https://example.com/downloads/file.zipwget --user-agent="MyApp/2.0" https://example2.com/downloads/file.zipIt is important to note that not all servers will accept any User-Agent string. Some servers are configured to only accept requests from specific User-Agent strings. In such cases, if you use an invalid User-Agent, the server may return an error or provide you with incorrect content.
Changing the User-Agent can also be useful if you want to download content that is restricted to certain browsers or devices. For example, some websites may only allow access from specific User-Agent strings, such as those from mobile devices. In such cases, you can change the Wget User-Agent to mimic that of a mobile device to gain access to the content.
It is also important to note that falsely identifying yourself by changing the User-Agent may be illegal in certain jurisdictions. So, you should be aware of the laws and regulations in your area before changing the User-Agent.
Downloading Multiple Files with Wget Command
Wget is a powerful command-line tool that can be used to download multiple files at once. There are several ways to accomplish this, depending on your needs.
Using a File with a List of URLs
If you have a list of URLs that you want to download, you can put them in a text file, one URL per line. Then, you can use the -i or –input-file option to tell Wget to read the file and download the URLs.
For example, if you have a file called “urls.txt” with the following content:
https://example.com/downloads/file1.ziphttps://example.com/downloads/file2.ziphttps://example.com/downloads/file3.zipYou can use the following command to download all the files:
wget -i urls.txtUsing a Wildcard
Another way to download multiple files is to use a wildcard in the URL. For example, if you want to download all the files with the extension “.zip” from a specific directory, you can use the following command:
wget https://example.com/downloads/*.zipThis will download all the files in the “downloads” directory that have the “.zip” extension.
Using the -nc option
The -nc or –no-clobber option tells Wget not to overwrite existing files. This can be useful when you want to download multiple files, but you don’t want to overwrite any files that you’ve already downloaded.
For example, the following command will download all the “.zip” files in the “downloads” directory, but it will not overwrite any files that already exist:
wget -nc https://example.com/downloads/*.zipDownloading files in parallel
Wget provides an option to download files in parallel using the -nc or –no-clobber option and using the & at the end of the command. This can significantly speed up the download process, especially when downloading large files or multiple files at once.
For example, to download three files in parallel, you can use the following command:
wget -nc https://example.com/downloads/file1.zip & \wget -nc https://example.com/downloads/file2.zip & \wget -nc https://example.com/downloads/file3.zip &Each wget command is executed in the background by appending & at the end. The -nc option ensures that files are not overwritten, as each file will be downloaded with a unique name.
It is important to note that downloading files in parallel can put a strain on the server and your network. So, it is best to use this feature with caution and only when necessary. Additionally, some servers may not support downloading files in parallel. In such cases, you may need to download the files one at a time.
Downloading via FTP with Wget Command
Wget is not just limited to downloading files via HTTP and HTTPS, it also supports downloading files via FTP (File Transfer Protocol). FTP is a standard network protocol used to transfer files from one host to another over a TCP-based network, such as the Internet.
To download a file via FTP, you need to provide the FTP URL of the file, which typically starts with “ftp://”. You can also use the –ftp-user and –ftp-password options to provide the username and password for the FTP server.
For example, to download a file called “file.zip” from the “downloads” directory on an FTP server, you can use the following command:
wget ftp://ftp.example.com/downloads/file.zipIf the FTP server requires a username and password, you can use the following command:
wget --ftp-user=username --ftp-password=password ftp://ftp.example.com/downloads/file.zipIt’s important to note that using the –ftp-user and –ftp-password options in this way is not secure as the credentials will be visible to other users on the system and may be recorded in shell history files. Instead, you can use the .netrc file to store your FTP credentials.
Wget also supports the use of FTP proxies. If you need to download files via FTP through a proxy server, you can use the –proxy option to specify the proxy server’s address and port. For example, to download a file through a proxy server at “proxy.example.com” on port 8080, you can use the following command:
wget --proxy=http://proxy.example.com:8080 ftp://ftp.example.com/downloads/file.zipIt’s also possible to download an entire directory from an FTP server using Wget by using the -r or –recursive option. This tells Wget to recursively download all the files in the directory, as well as any subdirectories. For example, to download the entire “downloads” directory from an FTP server, you can use the following command:
wget -r ftp://ftp.example.com/downloadsIt’s important to note that when using the -r option, Wget will download all the files and directories recursively, including hidden files and directories. So, it is best to be careful when using this option to avoid downloading unnecessary files.
Creating a Mirror of a Website with Wget
One of the most powerful features of Wget is its ability to create a mirror of a website. This means that Wget can download all the files of a website, including HTML, images, CSS, and JavaScript, and save them to your local machine. This can be useful for creating a backup of a website, or for offline browsing.
To create a mirror of a website, you can use the -m or –mirror option along with the -p or –page-requisites option. The -m option tells Wget to mirror the website, and the -p option tells Wget to download all the necessary files to display the HTML pages, such as images and CSS files.
For example, to create a mirror of the website “https://example.com“, you can use the following command:
wget --mirror -p https://example.comThe above command will download all the files of the website, including the HTML pages, images, CSS, and JavaScript, and save them to your local machine. The files will be saved in a directory structure that mirrors the structure of the website.
You can also use the -k or –convert-links option to convert the links in the HTML pages to point to the local files, rather than the original URLs. This is useful if you want to be able to browse the mirrored website offline.
For example, the following command will create a mirror of the website and convert the links to point to the local files:
wget --mirror -p -k https://example.comIt’s important to note that creating a mirror of a website can take a significant amount of time, especially for large websites. Additionally, some websites may block or limit the number of requests made by Wget, so it’s best to use this feature with caution and respect the terms of service of the website.
Skipping Certificate Check with Wget Command In Linux
By default, when Wget retrieves files over HTTPS, it will check the authenticity of the SSL/TLS certificate presented by the server. This is to ensure that the files are coming from a trusted source and are not being intercepted by a malicious party.
However, in some cases, you may need to download files from a server that has an invalid or self-signed certificate. In this situation, you can use the –no-check-certificate option to skip the certificate check and download the files without validation.
For example, to download a file called “file.zip” from a server with an invalid certificate, you can use the following command:
wget --no-check-certificate https://example.com/file.zipIt’s important to note that skipping the certificate check can be a security risk, as it allows any SSL/TLS certificate to be accepted, regardless of whether it is valid or not. This means that the files may be coming from an untrusted source and may be compromised. Therefore, it’s best to use this option only when necessary and with caution.
Additionally, this option is not recommended for regular usage or for situations where security is a concern. You should only use it if you are sure that the certificate is invalid, and not because the certificate is from an untrusted CA or self-signed.
Downloading to the Standard Output with Wget
By default, when you use Wget to download a file, it will save the file to your local machine. However, in some cases, you may want to download the file and send it directly to the standard output, rather than saving it to a file.
This can be useful for piping the output of Wget to another command or for displaying the file on the screen without saving it to disk.
To download a file and send it to the standard output, you can use the –output-document=- option along with the -O or –output-file option. The –output-document=- option tells Wget to send the output to the standard output, and the -O or –output-file option tells Wget to save the file to a specific file name.
For example, to download the file “file.zip” and send it to the standard output, you can use the following command:
wget --output-document=- https://example.com/file.zipYou can also use the -O – option as an alternative way to send the output to the standard output
wget -O - https://example.com/file.zipIt’s important to note that when you download a file to the standard output, you will not be able to save the file to your local machine. Therefore, if you need to save the file to disk, you will need to redirect the output to a file. For example, if you want to save the file as “file.zip” you can use the following command:
wget --output-document=- https://example.com/file.zip > file.zipConclusion:
In this article, we have discussed the Wget command and its various options in detail. By understanding these options, you can customize the behavior of the Wget command to suit your specific needs. With Wget, you can easily download files from the internet, making it a valuable tool for any Linux user.
If our tutorials helped you, please consider buying us a coffee. We appreciate your support!
Thank you for your support.
