Basics of using bash, and shell tools for covering several of the most common tasks
Introduction
Most shells have their own scripting language with variables, control flow and its own syntax.
Shell scripting optimized for performing shell-related tasks.
- Creating command pipelines
- Saving results into files
- Reading from standard input are primitives in shell scripting
Catalog
- Basics of using bash, and shell tools for covering several of the most common tasks
- Introduction
- Catalog
- Assign variables in bash
- SDifferences between
' and" - Use functions in bash
- Write the scripts
- bash manner
- if
- case
- select
- for loop
- while loop
- until loop
- Stream
- Return code or exit status
- Operator
- Get the output of a command as a variable
- Substitutions
- No need to give arguments one by one
- The usage of shebang
- Differences between shell functions and scripts
- Shell Tools
- Finding how to use commands
- Finding files
- Finding code(the content of file)
- Finding shell commands(already used)
- Directory Navigation
- Shortcut key
- Exercises
- Refferences
Assign variables in bash
Syntax:
- Assign variables:
variableName=value - Access variables' value:
$variableName
Note: variableName = value will not work if it is interpreted as calling the variableName program with arguments = and value
Added: the space character will perform argument splitting in shell.
SDifferences between ' and "
Strings in bash can be defined with
' and" delimiters, they are not equivalent.
-
' : Only the literal strings -
" : Will substitude variable values that strings contains
Use functions in bash
mcd () {mkdir -p "$1"cd "$1" }
-
$0 - Name of the script -
$1 to$9 - Arguments to the script.$1 is the first argument and so on. -
$@ - All the arguments -
$# - Number of arguments -
$? - Return code of the previous command -
$$ - Process identification number (PID) for the current script -
!! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doingsudo !! -
$_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typingEsc followed by. orAlt+.
Write the scripts
We can type it directly in our shell, but we always prefer to write those commands into a file and execute it.
After writting it into a script(e.g. mcd.sh), we could use source to execute and load it.
The next, just give the arguments to the script (e.g. mcd test), it will works.
bash manner
if
if [ condition ]
thencommands
elif [ condition ]
thencommands
elsecommands
fi
case
case variable inpattern1)commands;;pattern2)commands;;*)commands;;
esac
select
select variable in list
docommandsbreak
done
for loop
for variable in list
docommands
done
while loop
while condition
docommands
done
until loop
until condition
docommands
done
Stream
- output:
STDOUT> - errors :
STDERR2>
Return code or exit status
- value of
0 -> OK; - anything different from
0 -> error occurred
Operator
- and operator:
&& - or operator:
|| - seperate commands:
; -
true: 0 return code -
false: 1 return code -
-ne : not equal
Get the output of a command as a variable
Substitutions
command substitution: $(CMD)
process substitution: <(CMD)
Execute CMD and place the output in a temporary file and substitute the <() with that file’s name.
/dev/null : You can write sth. to it, but all the written things will be discarded.
No need to give arguments one by one
Sometimes, you don't need to give arguments one by one, with the usage of globbing(e.g. *, ?, {}) .
In general, you can just use it by the way of using small regex.
Also, you can use other language(e.g. Python) to interact with the Shell.
The usage of shebang
We can use the shebang (e.g. #!/usr/local/bin/python) to let the Shell know how to run this program.
But, python may not be installed at this path, and you may can't know where they actually lives in your machine, we can use env to use the PATH environment variable to resolve such a kind of question.(e.g. #!/usr/bin/env python)
As writing bash scripts can be tricky and unintuitive, we can use tools like shellcheck to help us to find errors in our sh/bash scripts.
Differences between shell functions and scripts
-
Language
- shell functions: The same language as the shell
- scripts: Any language(So it's important to include a shebang for scripts)
-
Load
- shell functions: Loaded once (Slightly fast to load, but whenever you change them you will have to reload their definition)
- scripts: Loaded whenever they are executed
-
execute
-
shell functions: In the current shell environment(Can modify environment variables)
-
scripts: In their own process(Can't modify environment variables)
Scripts will be passed by value environment variables that have been exported using
export
-
-
As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code
-
Often shell scripts will include their own function definitions.
Shell Tools
Finding how to use commands
-
man provides a manual page (called manpage) for a command you specify. -
-h or--help flags -
:help or?(For interactive tools such as the ones based on ncurses) -
tldr focuses on giving example use cases of a command so you can quickly figure out which options to use.
Finding files
-
find-
-name -
-path pattern -
-type-
d : directory -
f : file
-
-
-mtime -
-exec
-
-
fd
A simple, fast, and user-friendly alternative tofind. -
locate
locate uses a database that is updated usingupdatedb- In most systems,
updatedb is updated daily viacron. - A more in-depth comparison between
find andlocate can be found here.
- In most systems,
Finding code(the content of file)
-
grep
grep is an incredibly valuable shell tool that we will cover in greater detail during the data wrangling lecture.-
-R: go through the entire directory-
Many
grep alternatives have been developed, including ack, ag and rg. -
riggrep : more colorful-
-u: not ignore hidden files -
-t: type -
-C: show lines around the matched -
--files-without-match -
--stats
-
-
-
Finding shell commands(already used)
-
up -
history-
history | grep find
-
-
fzf bindings.
Directory Navigation
tree-
broot -
nnn
Shortcut key
- Ctl-A
Moves cursor to beginning of line of text (on the command-line). - Ctl-B
Backspace (nondestructive). - Ctl-C
Break. Terminate a foreground job. - Ctl-D
Log out from a shell (similar to exit).
EOF (end-of-file). This also terminates input from stdin.
When typing text on the console or in anxterm window, Ctl-D erases the character under the cursor. When there are no characters present, Ctl-D logs out of the session, as expected. In an xterm window, this has the effect of closing the window. - Ctl-E
Moves cursor to end of line of text (on the command-line). - Ctl-F
Moves cursor forward one character position (on the command-line). - Ctl-G
BEL. On some old-time teletype terminals, this would actually ring a bell. In an xterm it might beep. - Ctl-H
Rubout (destructive backspace). Erases characters the cursor backs over while backspacing. - Ctl-I
Horizontal tab. - Ctl-J
Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'. - Ctl-K
Vertical tab.
When typing text on the console or in anxterm window, Ctl-K erases from the character under the cursor to end of line. Within a script, Ctl-K may behave differently, as in Lee Lee Maschmeyer's example, below. - Ctl-L
Formfeed (clear the terminal screen). In a terminal, this has the same effect as the clear command. When sent to a printer, a Ctl-L causes an advance to end of the paper sheet. - Ctl-M
Carriage return. - Ctl-N
Erases a line of text recalled from history buffer [8] (on the command-line). - Ctl-O
Issues a newline (on the command-line). - Ctl-P
Recalls last command from history buffer (on the command-line). - Ctl-Q
Resume (XON).
This resumes stdin in a terminal. - Ctl-R
Backwards search for text in history buffer (on the command-line). - Ctl-S
Suspend (XOFF).
This freezes stdin in a terminal. (Use Ctl-Q to restore input.) - Ctl-T
Reverses the position of the character the cursor is on with the previous character (on the command-line). - Ctl-U
Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position. - Ctl-V
When inputting text, Ctl-V permits inserting control characters. - Ctl-W
When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace. In some settings, Ctl-W erases backwards to first non-alphanumeric character. - Ctl-X
In certain word processing programs, Cuts highlighted text and copies to clipboard. - Ctl-Y
Pastes back text previously erased (with Ctl-U or Ctl-W). - Ctl-Z
Pauses a foreground job.
Substitute operation in certain word processing applications.
EOF (end-of-file) character in the MSDOS filesystem.
Exercises
-
ls -laht --color-
ls -a -
ls -h -
ls -lh -
ls -lt -
ls --color
-
-
bash functions
marco andpolo-
marco.sh#!/bin/bash marco(){path_dir=$(pwd)export path_direcho "set $path_dir" } -
polo.sh#!/bin/bash polo(){cd "$path_dir"echo -n 'cd to ' && echo "$path_dir" }
-
-
runUFail-
fail.sh#!/usr/bin/env bashn=$(( RANDOM % 100 ))if [[ n -eq 42 ]]; thenecho "Something went wrong">&2 echo "The error was using magic numbers"exit 1 fiecho "Everything went according to plan" -
runUFail#!/bin/bashrunUFail(){./fail.sh > output.txt 2> error.txtcode=$?fail_times=0while [ $code -ne 1 ]dofail_times=$((fail_times+1))./fail.sh >> output.txt 2>> error.txtcode=$?doneecho "fail times:$fail_times"echo "output.txt:" && cat output.txtecho "error.txt:" && cat error.txt }
-
-
xargs command which will execute a command using STDIN as arguments.
touch ex{0,1,2,3,4,5,asb,'a b','bb'}.html
find -path "*html" -type f -print0 | ls | xargs -I {} zip output.zip {} -
find . -type f -mtime -1 | ls -lt
Refferences
- Advanced Bash-Scripting Guide: https://tldp.org/LDP/abs/html/special-chars.html
- The Missing Semester of Your CS Education: https://missing.csail.mit.edu/2020/shell-tools/
