[Linux](EN) Use multi processes(or threads) in shell script

Make multi processes environment by parallelly running command in background.

Environment and Prerequisite

• Linux base system
• Bash shell(/bin/bash)

Multi Process? Parallel Process? Multi Thread? Background Process?

• In this post, I will run command or shell script in background by using & operator.
• By searching, found that by using & operator it will make new child process(in new sub-shell) and run command or shell script in that process(in sub-shell)
• Specifically, it is ambiguous that call above method as multi process or multi thread… However some of english materials said that & operator makes new child process so I call it as multi process in this post’s title. If there is anyone who know well about this, it is pleasure to give a comment.

Linux &(Ampersand) Opearator

What is & operator?

• In linux, this operator make shell to run the command in the background, that is, it is forked and run in a separate sub-shell, as a job, asynchronously
• Running command or script using &, that process will be run in background so that it is possible to keep doing other things in current shell.
• Examples are below. Just add & to end of command.

& operator examples

• (Example) Run sleep in background

$ sleep 10 &
[1] 79287
$ # Type enter to make below result
[1]+  Done                    sleep 10

• (Example) Run shell script in background
• Like below you can make stdout to log file by using command > filename &.

$ tee test_shell.sh << EOF
> #!/bin/bash
>
> echo 1
> sleep 1
> echo 2
> sleep 1
> echo 3
> EOF
echo 1
sleep 1
echo 2
sleep 1
echo 3
$ ./test_shell.sh > tmp_test_shell.log & # You can redirect stdout to specific file.

(Example) Run shell script function in background parallelly

Explanation

• Make function and run that function with argument in background.
• Below example script shows that main script pass array’s element to function and sleeps depends on random value . If there is no & operator then it will be run sequentially but if there is & operator then each process will be run in background separately.
• Add log file to check each child process is run well.

Code

test_multi_process.sh

#!/bin/bash

function back_ground_process () {
	# Random number between 10 and 15
	sleep_time=$(($RANDOM*12345%6 + 10))

	echo "${1} will sleep for ${sleep_time} seconds"

	sleep ${sleep_time}

	echo "${1} sleep done!"
}

# array in shell script
arr=("a_worker" "b_worker" "c_worker")

# @ means all elements in array
for i in ${arr[@]}; do
    # run back_ground_process function in background
    # pass element of array as argument
    # make log file
    back_ground_process $i > ~/log_${i}.txt &
done

• wait command wait until all child processes(which are made in current script) are done.

test_multi_process.sh(continue)

...
# @ means all elements in array
for i in ${arr[@]}; do
    # run back_ground_process function in background
    # pass element of array as argument
    # make log file
    back_ground_process $i > ~/log_${i}.txt &
done

# wait until all child processes are done
wait

echo "All background processes are done!"

Result

• Check logs immediately after running script
• We can notice that script is finished just after running the script

$ ./test_multi_process.sh
$ cat log_a_worker.txt && cat log_b_worker.txt && cat log_c_worker.txt
a_worker will sleep for 13 seconds
b_worker will sleep for 13 seconds
c_worker will sleep for 10 seconds

• After 15 seconds, we can notice that all child processes are done well.

$ cat log_a_worker.txt && cat log_b_worker.txt && cat log_c_worker.txt
a_worker will sleep for 13 seconds
a_worker sleep done!
b_worker will sleep for 13 seconds
b_worker sleep done!
c_worker will sleep for 10 seconds
c_worker sleep done!

Applications

• Use & to run process or daemon in background.
• For example, you can minimize time for installing some files to multi server computers by running in background simultaneously and it is useful when keep using shell while running other programs in same shell.
• Additionally, you can use it for multi processing or threading in shell script.

References

• https://stackoverflow.com/questions/3004811/how-do-you-run-multiple-programs-in-parallel-from-a-bash-script/3004814
• https://wiki.kldp.org/HOWTO/html/Adv-Bash-Scr-HOWTO/x5472.html
• https://unix.stackexchange.com/questions/86247/what-does-ampersand-mean-at-the-end-of-a-shell-script-line
• https://bashitout.com/2013/05/18/Ampersands-on-the-command-line.html