(One)-minute geek news

Group talks, Laboratoire de Chimie, ENS de Lyon

Background jobs


A single shell session can handle multiple running processes at the same time, so let us learn how to recycle your idle terminals!

Processes and jobs

In the Linux jargon, a process is any "currently running" program. You can list all processes currently active on your machine with the ps axu command. Processes are a vast subject that won't be discussed here in details, but let us mention a few simplified properties:

  • Each process is given a unique identifier called PID (Process IDentifier)
  • A process can have different states: running, stopped, ... or even zombie!
  • A process can receive some specific signals to trigger actions or state changes (e.g. the kill PID_of_process command sends a specific signal (SIGTERM) to the process whose PID is PID_of_process (see man signal for other possible signals))
  • Each process has a parent, and can have multiple children
  • A process can have multiple threads

Jobs are processes whose parent is a shell session (e.g. the ipython3 command on a Bash session will create a process running the ipython3 program whose parent is the bash session from which the command was executed).

A Bash session is associated with a process like any other (whose parent is likely the process running the associated terminal emulator), whose PID can be obtained with the special parameter $$ (e.g. echo $$ will print your current Bash session associated PID).

Foreground and background jobs

In your typical Bash session, you run one job at a time: you enter a command and then wait until the end of the command (either normally or because lost patience and pressed Ctrl-C to kill the job, sending a SIGINT signal behind the scene). Only then you are prompted with a bash prompt ready to receive new commands. Such job being executed "in the foreground" is naturally called a foreground job.

A single Bash session can have only one foreground job at a time, and the bash prompt is not available during its execution. In other words, a foreground job is blocking your shell during its execution.

This behavior is particularly frustrating for long commands or GUI-based programs (e.g. if you execute molden on your terminal, you will be prevented to write new commands in the same terminal/bash session, forcing you to open a new terminal), and is encouraging users to have multiple terminal windows, which can quickly become a hassle.

Fortunately, a command can be run in the background by prepending an & at the end of your command. Such job being executed "in the background" is naturally called a background job (e.g. molden & will display a 90's flavored Molden window while allowing you to run other comands!)

A single Bash session can have multiple background jobs, so use as much as you like! You can see currently running background jobs (with their PID) with the jobs command. But the real power of foreground and background jobs is that you can easily switch between them! Let us now see some commands or keystrokes of interest for manipulating jobs:

  • You can execute a command in the background by appending an & to it
  • You can bring a background job to the foreground with the fg command
  • You can stop a foreground job with the Ctrl-Z keystroke...
  • ... then use the bg to run it in the background!

The possibilities are endless! For example: you can view a file with a basic less, then stop it (with Ctrl-Z), check something with your same terminal and switch back to your less in the same exact state as before you stopped it (no need to re-scroll)! Or you can simply use it to reduce your number of open terminal windows...

Bonus: Quiet background execution

A background job has its standard output and error (see previous redirection geek news) still redirected to your current terminal. In other words, a running background job can display messages in your Bash session, which can make it hardly usable... Fortunately, you can silence background jobs through many ways:

  • Use nohup to redirect the output (e.g. nohup echo test & will not display "test", but will write "test" to the file nohup.out)
  • Use a redirection (see previous redirection geek news), for example: echo test >& /dev/null & will execute echo test in the background and throw both the standard output and error to the linux equivalent of a black hole: /dev/null
  • Use the tsp utility, a simple unix batch system.

Bonus: Immediate report upon termination

By default, when a background job finishes, you get notified only when you get a new prompt (in other words, the next time you will execute a command), which can become confusing. This behavior can be changed by adding in your ~/.bashrc the command set -b

Bonus: Jobs specifications

You can quickly refer to a specific background job with a job specification. Using these job specification, the last background job is refered with %%, and the i-th job can be refered with %i. Job specifications can be used as arguments for fg, bg, kill, ...

Bonus: Jobs pooling made easy

You can easily treat a list of arguments with a pool of commands running in parallel with the xargs utility. Example: ls ./tests/*.py | xargs -n 1 -P 4 python3 will maintain a pool of 4 (-P option) Python3 processes executing each argument (i.e. python test files, here) one by one (-n option).

Bonus: Detaching a job

If you have background jobs running and you close your terminal, those jobs might be killed (i.e. they seem to remain only if you terminated your bash session with a Ctrl-D, logout or exit and the huponexit shell option has not been set up with the shopt -s huponexit command (default behavior)).

Note that background commands executed with the nohup utility are preserved upon any kind of terminal closing. You can also make sure that a background job will survive closing your terminal or closing a SSH connection by using the disown -h command, that mimicks the nohup command on an existing background process.

Unfortunately, once a session is closed, the background jobs are turned into deamons (their parent is no longer a Bash session, but a process called init) and can no longer be brought back as foreground jobs. However, you can use terminal multiplexers like screen or tmux that allow you to detach and reattach entire Bash sessions, meaning that you can connect to a distant machine through SSH, work on a virtual terminal, detach it, close your SSH connection, change your computer, reconnect via SSH and retrieve your previous working terminal fully operational! Check out those awesome utilities!

References

man bash
man tsp
help set
man xargs
https://unix.stackexchange.com/questions/4214/what-is-the-difference-between-a-job-and-a-process