Shell Scripting
Advanced Shell Scripting
Once you are comfortable with the basics, these are the tools that separate someone who writes quick throwaway scripts from someone who builds reliable, maintainable automation for production embedded systems.
Arrays
Indexed Arrays
bash
#!/bin/bash
# Declare an array
CHIPS=("esp32" "esp32s2" "esp32s3" "esp32c3" "esp32h2")
# Access elements (zero-indexed)
echo "${CHIPS[0]}" # esp32
echo "${CHIPS[2]}" # esp32s3
# All elements
echo "${CHIPS[@]}" # esp32 esp32s2 esp32s3 esp32c3 esp32h2
# Number of elements
echo "${#CHIPS[@]}" # 5
# Slice: 2 elements starting at index 1
echo "${CHIPS[@]:1:2}" # esp32s2 esp32s3
# Append to array
CHIPS+=("esp32p4")
# Iterate
for CHIP in "${CHIPS[@]}"; do
echo "Building: $CHIP"
done
# Iterate with index
for i in "${!CHIPS[@]}"; do
echo "$i: ${CHIPS[$i]}"
done
Associative Arrays (Dictionaries)
bash
#!/bin/bash
declare -A DEVICE_PORTS
DEVICE_PORTS["esp32"]="/dev/ttyUSB0"
DEVICE_PORTS["stm32"]="/dev/ttyACM0"
DEVICE_PORTS["pi"]="/dev/ttyAMA0"
# Access a value by key
echo "ESP32 is on: ${DEVICE_PORTS['esp32']}"
# Iterate over keys and values
for DEVICE in "${!DEVICE_PORTS[@]}"; do
echo "$DEVICE → ${DEVICE_PORTS[$DEVICE]}"
done
# Check if a key exists
if [[ -v DEVICE_PORTS["esp32"] ]]; then
echo "ESP32 port is configured"
fi
String Manipulation
Bash has powerful built-in string operations that avoid calling external tools like sed or awk.
bash
#!/bin/bash
FILENAME="esp32_firmware_v1.2.3.bin"
# String length
echo "${#FILENAME}" # 26
# Substring extraction
echo "${FILENAME:0:5}" # esp32 (offset 0, length 5)
echo "${FILENAME: -3}" # bin (last 3 characters)
# Remove prefix (shortest match)
echo "${FILENAME#esp32_}" # firmware_v1.2.3.bin
# Remove prefix (longest match)
echo "${FILENAME##*/}" # Just the filename from a path
# Remove suffix (shortest match)
echo "${FILENAME%.bin}" # esp32_firmware_v1.2.3
# Remove suffix (longest match)
echo "${FILENAME%%_*}" # esp32
# Replace first occurrence
echo "${FILENAME/firmware/app}" # esp32_app_v1.2.3.bin
# Replace all occurrences
echo "${FILENAME//_/-}" # esp32-firmware-v1.2.3.bin
# Convert to uppercase
echo "${FILENAME^^}" # ESP32_FIRMWARE_V1.2.3.BIN
# Convert to lowercase
echo "${FILENAME,,}" # esp32_firmware_v1.2.3.bin
# Default value if variable is empty
CHIP="${CHIP:-esp32}" # Use "esp32" if CHIP is unset or empty
# Set variable AND use default
BUILD_DIR="${BUILD_DIR:=./build}" # Assign default if not set
Text Processing: grep, sed, awk
These three tools form the core of text processing on Linux. Every embedded engineer needs them for log parsing, config file manipulation, and output filtering.
grep — Pattern Matching
bash
# Find all error lines in a build log
grep -i "error" build.log
# Show 5 lines after each match (context)
grep -A 5 "Linking" build.log
# Show line numbers and filename
grep -n "CONFIG_" sdkconfig
# Recursive search through source files
grep -r "gpio_set_level" ./main/
# Only print the matched part (not the whole line)
grep -o "0x[0-9a-fA-F]*" linker_map.txt # Find all hex addresses
# Invert match — show lines that do NOT contain a pattern
grep -v "^#" sdkconfig | grep -v "^$" # Strip comments and blank lines
# Extended regex
grep -E "UART[0-9]|SPI[0-9]" sdkconfig
# Count occurrences
grep -c "WARNING" build.log
sed — Stream Editor (Find and Replace)
bash
# Replace first occurrence on each line
sed 's/old/new/' file.txt
# Replace all occurrences (global flag)
sed 's/old/new/g' file.txt
# In-place edit (modify the file directly)
sed -i 's/baud_rate=9600/baud_rate=115200/g' config.ini
# Delete lines matching a pattern
sed '/^#/d' sdkconfig # Delete comment lines
sed '/^$/d' file.txt # Delete blank lines
# Print only matching lines (like grep)
sed -n '/error/p' build.log
# Print a range of lines (lines 10 to 20)
sed -n '10,20p' build.log
# Insert a line before a match
sed '/CONFIG_FREERTOS/i # FreeRTOS Settings' sdkconfig
# Append a line after a match
sed '/CONFIG_FREERTOS/a CONFIG_FREERTOS_HZ=1000' sdkconfig
# Delete the last line
sed '$d' file.txt
awk — Structured Text Processing
awk treats every line as a record and every whitespace-separated word as a field ($1, $2, ..., $NF = last field).
bash
# Print specific columns from ls -l output
ls -lh | awk '{print $5, $9}' # Print size and name
# Print only lines where column 3 is greater than 100
awk '$3 > 100' data.csv
# Sum a column
awk '{sum += $2} END {print "Total:", sum}' readings.txt
# Print header and filtered rows from CSV
awk -F',' 'NR==1 || $3 > 37' sensor_data.csv # Header + rows where temp > 37
# Calculate average of a column
awk -F',' '{sum+=$2; count++} END {print "Avg:", sum/count}' data.csv
# Print unique values in column 1
awk '{print $1}' log.txt | sort | uniq
# Process /proc/cpuinfo
awk -F': ' '/model name/ {print $2; exit}' /proc/cpuinfo
# Custom output formatting
ls -lh *.bin | awk '{printf "%-30s %s\n", $9, $5}'
Process Management
bash
# Run a command in the background
idf.py monitor &
BG_PID=$! # PID of the last background job
# List background jobs
jobs
# Bring background job to foreground
fg %1
# Kill a background job by job number
kill %1
# Kill by PID
kill $BG_PID
kill -SIGTERM $BG_PID # Graceful stop
kill -SIGKILL $BG_PID # Force stop
# Wait for a background process to finish
wait $BG_PID
echo "Background task finished with exit code: $?"
# Run a command that survives after terminal is closed
nohup python3 ml_server.py > server.log 2>&1 &
echo "Server PID: $!"
systemd — Managing Services
For scripts that should run as background services:
bash
# Create a systemd service file
sudo nano /etc/systemd/system/my_app.service
ini
1[Unit]2Description=My ESP32 ML Gateway3After=network.target45[Service]6Type=simple7User=rajath8WorkingDirectory=/home/rajath/gateway9ExecStart=/home/rajath/gateway/.venv/bin/python3 app.py10Restart=always11RestartSec=512Environment=PYTHONUNBUFFERED=11314[Install]15WantedBy=multi-user.targetbash
# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable my_app
sudo systemctl start my_app
sudo systemctl status my_app
sudo journalctl -u my_app -f # Follow live logs
cron — Scheduled Tasks
cron runs commands on a defined schedule — automated log backups, scheduled data pulls, periodic sensor readings.
bash
crontab -e # Edit current user's cron table
crontab -l # List current cron jobs
sudo crontab -e # Edit root's cron table
Cron syntax:
text
MIN HOUR DAY MONTH WEEKDAY COMMAND
* * * * * command
# Field ranges:
# MIN: 0-59
# HOUR: 0-23
# DAY: 1-31
# MONTH: 1-12
# WEEKDAY: 0-6 (0=Sunday)
Examples:
bash
# Every minute
* * * * * /home/rajath/scripts/read_sensor.sh >> /var/log/sensor.log 2>&1
# Every day at 2:30 AM
30 2 * * * /home/rajath/scripts/backup.sh
# Every Monday at 9 AM
0 9 * * 1 /home/rajath/scripts/weekly_report.sh
# Every 15 minutes
*/15 * * * * /home/rajath/scripts/check_health.sh
# First day of every month at midnight
0 0 1 * * /home/rajath/scripts/monthly_cleanup.sh
Signal Handling (trap)
When a user presses Ctrl+C or your script is killed, it receives a signal. trap lets you catch signals and run cleanup code.
bash
#!/bin/bash
LOCK_FILE="/tmp/my_script.lock"
cleanup() {
echo ""
echo "Caught interrupt. Cleaning up..."
rm -f "$LOCK_FILE"
# Kill any child processes
kill $(jobs -p) 2>/dev/null
echo "Done. Exiting."
exit 0
}
# Register the cleanup function for these signals
trap cleanup SIGINT SIGTERM EXIT
# Create a lock file to prevent duplicate runs
if [ -f "$LOCK_FILE" ]; then
echo "ERROR: Script is already running (PID: $(cat $LOCK_FILE))"
exit 1
fi
echo $$ > "$LOCK_FILE"
echo "Running... Press Ctrl+C to stop."
while true; do
echo "Working at $(date)..."
sleep 5
done
Script Debugging
bash
# Run with debug output (trace every command)
bash -x script.sh
# Add debug mode inside the script
set -x # Enable tracing from this point
# ... commands ...
set +x # Disable tracing
# Dry run — print what would be done without executing
set -n # Parse but don't execute (syntax check)
# Check syntax without running
bash -n script.sh
Practical: Build, Flash & Monitor Script
bash
#!/bin/bash
set -e
set -o pipefail
# Configuration
PORT="${1:-/dev/ttyUSB0}"
TARGET="${2:-esp32}"
BAUD="${3:-460800}"
BOLD="\033[1m"
GREEN="\033[0;32m"
RED="\033[0;31m"
RESET="\033[0m"
log_info() { echo -e "${GREEN}[INFO]${RESET} $*"; }
log_error() { echo -e "${RED}[ERROR]${RESET} $*" >&2; }
# Check prerequisites
check_prereqs() {
local missing=()
for cmd in idf.py python3; do
command -v "$cmd" &>/dev/null || missing+=("$cmd")
done
if [ ${#missing[@]} -ne 0 ]; then
log_error "Missing commands: ${missing[*]}"
exit 1
fi
}
# Check serial port
check_port() {
if [ ! -e "$PORT" ]; then
log_error "Port $PORT not found. Connect your device and try again."
log_error "Available ports: $(ls /dev/ttyUSB* /dev/ttyACM* 2>/dev/null || echo none)"
exit 1
fi
}
build_project() {
log_info "Setting target: $TARGET"
idf.py set-target "$TARGET"
log_info "Building..."
idf.py build
log_info "Build complete."
}
flash_project() {
log_info "Flashing to $PORT at ${BAUD} baud..."
idf.py -p "$PORT" -b "$BAUD" flash
log_info "Flash complete."
}
main() {
echo -e "${BOLD}ESP32 Build, Flash & Monitor${RESET}"
echo "Port: $PORT"
echo "Target: $TARGET"
echo ""
check_prereqs
check_port
build_project
flash_project
log_info "Starting monitor (Ctrl+] to exit)..."
idf.py -p "$PORT" monitor
}
main "$@"
Quick Reference
| Concept | Syntax |
|---|---|
| Array declaration | ARR=("a" "b" "c") |
| Array element | ${ARR[0]} |
| All array elements | ${ARR[@]} |
| Array length | ${#ARR[@]} |
| Assoc array | declare -A MAP; MAP[key]=value |
| String length | ${#VAR} |
| Remove prefix | ${VAR#prefix} |
| Remove suffix | ${VAR%.ext} |
| Replace | ${VAR/old/new} |
| Default value | ${VAR:-default} |
| Run in background | cmd & |
| Get background PID | $! |
| Wait for PID | wait $PID |
| Trap signal | trap cleanup SIGINT |
| Debug mode | bash -x script.sh |
| Grep recursively | grep -r "pattern" ./ |
| Sed replace | sed 's/old/new/g' file |
| Awk column | awk '{print $2}' file |
| Schedule cron | crontab -e |
| Run at boot | systemctl enable myservice |