Shell Script
from Scratch

How to Write a Complex Shell Script

Jak psát složitější skript v shellu

Lukáš Bařinka

If you're doing something a second time,
you're probably not doing it last time

Pokud něco děláte podruhé,
pravděpodobně to neděláte naposled.

TOC

Obsah

1. Problem
2. Skeleton
3. Prototype
4. Polishing

1. Zadání
2. Kostra řešení
3. Prototyp
4. Vylepšování

ProblemZadání

MotivationMotivace

							[23/11/2005-08:01:20] core: state process
							[23/11/2005-08:01:20] core: I:  0.025 [0.173438] {L:S}
							[23/11/2005-08:01:20] core: U:  0.559 [0.290247-0.173492]
							[23/11/2005-08:01:20] core: T: 50.09 [1.201045]
							[23/11/2005-08:01:20] core: L: 1 [3.870664]
							[23/11/2005-08:01:20] core: P:  6.7 [0.008472]
							[23/11/2005-08:01:21] core: M: 1 [0.016943]
							[23/11/2005-08:01:21] core: heat off (tempH)
							[23/11/2005-08:01:21] core: O:  0.386 <0.1.0.0.0.0.1.0>
							[23/11/2005-08:01:21] core: state process
							[23/11/2005-08:01:21] core: I:  0.025 [0.173551] {L:S}
							[23/11/2005-08:01:21] core: U:  0.562 [0.292037-0.173707]
							[23/11/2005-08:01:21] core: T: 50.09 [1.201033]
							[23/11/2005-08:01:22] core: L: 1 [3.845922]
							[23/11/2005-08:01:22] core: heat off (tempH)
							[23/11/2005-08:01:22] core: O:  0.386 <0.1.0.0.0.0.0.1>
							[23/11/2005-08:01:22] core: state process
							[23/11/2005-08:01:22] core: I:  0.025 [0.173553] {L:S}
							[23/11/2005-08:01:22] core: U:  0.559 [0.290166-0.173604]
							[23/11/2005-08:01:22] core: T: 50.08 [1.201016]
							[23/11/2005-08:01:22] core: L: 1 [3.865338]
							[23/11/2005-08:01:22] core: heat off (tempH)
							[23/11/2005-08:01:22] core: O:  0.386 <0.1.0.0.0.0.1.0>
							[23/11/2005-08:01:22] core: state process
							[23/11/2005-08:01:23] core: I:  0.025 [0.173529] {L:S}
							[23/11/2005-08:01:23] core: U:  0.562 [0.292059-0.173551]
							[23/11/2005-08:01:23] core: T: 50.08 [1.201018]
							[23/11/2005-08:01:23] core: L: 1 [3.851480]
							[23/11/2005-08:01:23] core: heat off (tempH)
							[23/11/2005-08:01:23] core: O:  0.386 <0.1.0.0.0.0.0.1>
							[23/11/2005-08:01:23] core: state process
							[23/11/2005-08:01:24] core: I:  0.025 [0.173516] {L:S}
							[23/11/2005-08:01:24] core: U:  0.558 [0.290081-0.173680]
							[23/11/2005-08:01:24] core: T: 50.09 [1.201045]
							[23/11/2005-08:01:24] core: L: 1 [3.867546]
							[23/11/2005-08:01:24] core: heat off (tempH)
							[23/11/2005-08:01:24] core: O:  0.386 <0.1.0.0.0.0.1.0>
							[23/11/2005-08:01:24] core: state process
							[23/11/2005-08:01:24] core: I:  0.025 [0.173526] {L:S}
							[23/11/2005-08:01:25] core: U:  0.562 [0.292005-0.173802]
							[23/11/2005-08:01:25] core: T: 50.08 [1.200999]
							[23/11/2005-08:01:25] core: L: 1 [3.848423]
							[23/11/2005-08:01:25] core: heat off (tempH)
							[23/11/2005-08:01:25] core: O:  0.386 <0.1.0.0.0.0.0.1>
							[23/11/2005-08:01:25] core: state process
							[23/11/2005-08:01:25] core: I:  0.025 [0.173534] {L:S}
							[23/11/2005-08:01:25] core: U:  0.559 [0.290176-0.172840]
							[23/11/2005-08:01:26] core: T: 50.08 [1.201016]
							[23/11/2005-08:01:26] core: L: 1 [3.867275]
							[23/11/2005-08:01:26] core: heat off (tempH)
							[23/11/2005-08:01:26] core: O:  0.386 <0.1.0.0.0.0.1.0>
							[23/11/2005-08:01:26] core: state process
							[23/11/2005-08:01:26] core: I:  0.025 [0.173419] {L:S}
							[23/11/2005-08:01:26] core: U:  0.563 [0.292161-0.173685]
							[23/11/2005-08:01:26] core: T: 50.08 [1.200982]
							[23/11/2005-08:01:27] core: L: 1 [3.853658]
							[23/11/2005-08:01:27] core: heat off (tempH)
							[23/11/2005-08:01:27] core: O:  0.386 <0.1.0.0.0.0.0.1>
							[23/11/2005-08:01:27] core: state process
							[23/11/2005-08:01:27] core: I:  0.025 [0.173394] {L:S}
							[23/11/2005-08:01:27] core: U:  0.559 [0.290337-0.173639]
							[23/11/2005-08:01:27] core: T: 50.08 [1.200987]
							[23/11/2005-08:01:28] core: L: 1 [3.870080]
							[23/11/2005-08:01:28] core: heat off (tempH)
							[23/11/2005-08:01:28] core: O:  0.386 <0.1.0.0.0.0.1.0>
						

The taskÚkol

To make a graph animation of input data.

Animace dat vynesených do grafu.

We have

Data (values) from measurements

We need

Value progress in time (an animation)

Máme

Data (hodnoty) z měření

Potřebujeme

Průběh hodnot v čase (animaci)

Example of outputUkázka výstupu

Do we have necessary tools?Máme potřebné nástroje?

Animation (video)

Animace (video)

ffmpeg - ffmpeg video converter Converts images into video file

ffmpeg - ffmpeg video converter Převede obrázky do video souboru

Plots (images)

Grafy (obrázky)

gnuplot - an interactive plotting program Plots data (values) into image

gnuplot - an interactive plotting program Vykreslí data (hodnoty) do obrázků

Data (values)

Data (hodnoty)

awk - data generator Or real data + filters

awk - data generator Nebo skutečná data + filtry

Testing data generatorGenerátor testovacích dat

							#!/usr/bin/awk -f
							BEGIN {
								  a = 100
								  srand()
								  for ( i = 1; i < 1000; i++ ) {
								    a += rand() - 0.5
								    print a
								  }
							}
						

gnuplot

							set terminal png
							set output "out.png"
							set xrange [0:1000]
							set yrange [95:105]
							plot "-" with lines t ""
						

SkeletonKostra řešení

Skeleton 1Kostra 1

							#!/bin/bash
							# Graph animation of input data
							# Usage:  $0 data

							# Prepare separate slides
							#FIXME

							# Join slides into animation
							#FIXME
						

Skeleton 2Kostra 2

							#!/bin/bash
							# Graph animation of input data
							# Usage:  $0 data

							# Prepare separate frames
							# For each frame
							  # Prepare data
							  #FIXME
							  # Run gnuplot to make single frame
							  #FIXME
							  # Rename frame according to frame #
							  #FIXME

							# Join frames into animation
							#FIXME

							# Cleanup
							#FIXME
						

PrototypePrototyp

Prototype 1Prototyp 1

							# Prepare separate frames
							# For each frame
							for (( i=1; i<=1000; i++ )); do
							  # Run gnuplot to make single frame
							  (
							    # Prepare data
							    cat script.gp
							    head -n "$i" data
							  ) | gnuplot
							  # Rename frame according to frame #
							  mv out.png $( printf "%03d" "$i" ).png
							done
						

Prototype 2Prototyp 2

							# Prepare separate frames
							# For each frame
							for (( i=1; i<=900; i++ )); do
							  # Run gnuplot to make single frame (use part of data)
							  echo "set terminal png;
							        set output '$( printf %03d $i ).png';
							        set format x '';
							        plot [0:100][95:105] \
							             '<sed -n $i,$((100+i))p data' \
							             with boxes t'$i-$((100+i))';
							  " | gnuplot
							done
						

Video

							# Join frames into animation
							ffmpeg -i %03d.png anim.mp4
						

CleanupÚklid

							#Cleanup
							rm *.png
							find . -maxdepth 1 -name '*.png' -delete
						

PolishingVylepšování

Google Style Guide

• Comments
• Formating
• Variables expansion
• Quoting
• Tests
• Commands
• Functions

Comments

• Start each file with a description of its contents.
• Any function that is not both obvious and short must be commented. Any function in a library must be commented regardless of length or complexity.
  • Description of the function.
  • Globals: List of global variables used and modified.
  • Arguments: Arguments taken.
  • Outputs: Output to stdout or stderr.
  • Returns: Returned values other than the default exit status of the last command run.
• Use TODO comments for code that is temporary, a short-term solution, or good-enough but not perfect.

Formating

• Indent 2 spaces. No tabs.
• Use blank lines between blocks to improve readability.
• Maximum line length is 80 characters. Use Here Document
• Pipelines should be split one per line
  if they don’t all fit on one line.
• Put ; do and ; then on the same line as
  the while, for or if.
• Indent alternatives in case by 2 spaces.

Variables expansion

• Quote variables
• Prefer "${var}" over "$var".
• Stay consistent with what you find for existing code
• Don’t brace-delimit single character shell specials / positional parameters, unless strictly necessary or avoiding deep confusion.
• Use readonly or declare -r for constants. Constants and anything exported to the env. should be capitalized.

Quoting

• Quote strings containing variables, command substitutions, spaces or shell meta characters.
• Use arrays for safe quoting of lists of elements.
• Optionally quote shell-internal, readonly special variables that are defined to be integers: $?, $#, $$, $! Prefer quoting of “named” internal integer variables (PPID) for consistency.
• Prefer quoting strings that are “words”.
• Never quote literal integers.
• Be aware of the quoting rules for pattern matches in [[ … ]].
• Use "$@" unless you have a specific reason to use $* such as simply appending the arguments to a string in a message or log.

Tests

• [[ … ]] is preferred over [ … ], test and /usr/bin/[.
• To avoid confusion about what you’re testing for, explicitly use -z or -n.
• For clarity, use == for equality rather than =.
• Be careful when using < and > in [[ … ]] which performs a lexicographical comparison. Use (( … )) or -lt and -gt for numerical comparison.
• Always use (( … )) or $(( … )) rather than let or $[ … ] or expr.
• It is recommended to avoid using (( … )) as a standalone statement, and otherwise be wary of its expression evaluating to zero.

Commands

• Use $(command) instead of `command`.
• As filenames can begin with a -, it’s a lot safer to expand wildcards with ./* instead of *.
• eval should be avoided.
• All error messages should go to stderr.
• Always check return values and give informative return values.

Functions

• Declare function-specific variables with local. Declaration and assignment should be on different lines.
• Put all functions together in the file just below constants.
• Don’t hide executable code between functions.
• A function called main is required for scripts long enough to contain at least one other function. The last non-comment line in the file should be main "$@"

Generator with timestampsGenerátor s časovými značkami

							#!/usr/bin/awk -f
							BEGIN {
							  v = 100
							  t = systime()
							  srand()
							  for ( i = 1; i < 1000; i++ ) {
							    v += rand() - 0.5
							    T = strftime( "[%Y-%m-%d %T]", t+i )
							    print T, v
							  }
							}
						

Gnuplot with timestampsGnuplot s časovými značkami

							set terminal png
							set output "out.png"
							set xdata time
							set timefmt "[%Y-%d-%m %H:%M:%S]"
							set xtics format "%H:%M"
							set xrange ["[2021-03-29 07:23:37]":"[2021-03-29 07:40:15]"]
							set yrange [95:105]
							plot "-" using 1:3 with lines t ""
						

See AlsoDalší zdroje

• Shell Style Guide Google
• Advanced Bash-Scripting Guide
• The Art of Unix Programming Eric Steven Raymond

Other lectures (in Czech)Další přednášky

• Jak si přizpůsobit bashInstallfest 2017
• bash a regulární výrazy vs shellovské vzory Installfest 2019 Prezentace
• bash a zpracování textů Installfest 2016Prezentace
• bash: jemný úvod do magie * (1/2) Installfest 2015Prezentace
• bash: jemný úvod do magie * (2/2) Installfest 2015Prezentace
• bash & Mario Bros. (UNIX pipes in bash) LinuxDays 2017Prezentace

• ! (1/2) Installfest 2014Prezentace
• ! (2/2) SUT SH 2014Prezentace
• script.sh –options -and arguments LinuxDays 2015Prezentace
• g)awk in a nutshell LinuxDays 2018Prezentace
• Bash a strukturované programování InstallFest 2018Prezentace
• Shell FAQ LinuxDays 2016Prezentace
• Bash script from scratch Installfest 2020