In Linux, we can execute processes in parallel. This is very crucial because we have powerful processors and we want to take advantage of those fully. By leveraging the capabilities of your system’s multi-core processors, you can effectively reduce execution times and accomplish tasks faster than ever before. In this tutorial, we will delve into various techniques for running processes in parallel on the Linux terminal.
Overview of Parallel Execution
Traditionally, when executing commands or scripts on the command line, each process runs sequentially, meaning that one process is completed before the next one starts. However, with parallel execution, multiple processes are launched simultaneously and run concurrently. This approach allows us to take full advantage of the available system resources and achieve substantial performance improvements.
How to run processes in parallel?
This section shows three methods to run processes in parallel in Linux.
1. Backgrounding Processes:
One simple technique to run processes in parallel involves utilizing backgrounding. By appending an ampersand (&) at the end of a command or script, it can be executed as a background process while allowing you to continue working in the same terminal session.
command1 &
command2 &
Bash provides built-in job control mechanisms that allow you to manage multiple tasks efficiently. The wait command can be used to pause the parent shell until all background processes complete.
command1 &
command2 &
wait # Wait for both commands to finish
3. xargs Command:
The xargs command allows for efficient execution of commands in parallel by taking input from standard input (stdin) and running them with multiple instances concurrently.
echo "command1" | xargs -P 0 -I {} sh -c "{} &"
# OR
cat list.txt | xargs -P 4 -I {} sh -c "{} &"
Note: -P specifies the maximum number of concurrent processes.
3. GNU Parallel:
GNU Parallel is a versatile tool that allows for easy parallelization of shell commands and scripts. It provides a simple interface and supports several methods of distributing tasks across multiple cores or even different machines over a network.
To install GNU Parallel on Debian-based systems, use apt.
sudo apt-get install parallel
To execute commands in parallel using GNU Parallel, you can leverage the parallel command or use the --jobs option to specify the maximum number of simultaneous processes.
parallel ::: command1 command2 command3
parallel --jobs 4 ::: command1 command2 command3
Running processes in parallel is a powerful technique to optimize your workflow and save time on various Linux tasks. Whether it’s backgrounding, using GNU Parallel, or employing the xargs command, Linux offers several methods to accomplish this goal.
Comments