NUS Hackers Wiki
NUS Hackers Wiki
  • NUS Hackers Wiki
  • Hackerschool
    • Virtual Machines and Linux
    • Beginners' Guide to the Terminal
      • Introduction to the Terminal
      • Modern Shell Tools
      • Shell Scripting
      • Real World Scripting
      • Resources
    • Self-Hosting: Three Easy Pieces
      • 1. Setting up your server
      • 2. Running Services
      • 3. Monitoring your server
    • Vim
    • Introduction to Zig
      • Language Basics
      • Error Handling
      • Memory Management
      • Working with C
      • Exploring comptime
    • CI/CD with Github Actions
      • Background
      • Basics of Github Actions
        • Target workflow
        • Running unit tests
        • Linting code
        • Deploying to Github Pages
      • Advanced use cases
        • Pollers
        • Github script
        • Executing third-party scripts
        • Reusable workflows
      • Cookbook
    • Lightning Git
      • Git Concepts
      • Getting Started with Git
      • Making your first commit
      • Branching
      • Merge Conflicts
      • Integrating remote repositories
      • Collaborative Workflows
      • Commit Manipulation and Reflog
      • Interactive rebasing
      • filter-repo
  • Orbital
    • JavaScript
      • Browser Developer Tools
      • Getting Started
      • Datatypes
      • Operators and Operations
      • Loops and Conditions
      • Functions
      • Strings
      • Arrays
      • HTML
        • Getting Started
        • Tag Attributes
        • HTML Forms
        • Browser Inspector
      • CSS
        • Selectors
        • Colors in CSS
        • Measurements in CSS
        • The Box Model
        • Adding Styles - Part 1
        • Adding Styles - Part 2
      • Working with the DOM
        • Querying the DOM - Selectors
        • Querying the DOM - Element Attributes
        • Querying the DOM - Element Styles
        • Events with JS and HTML
        • Exercise: Click Counter
        • Editing the DOM
        • Fetch Requests
        • Exercise: The NUSMods API
    • React
      • Setup
      • State
    • React Native
      • Setup
      • Intro to JSX
      • Basic Syntax
      • Handling UI
      • Props
      • State Management
    • Git
      • Setup
      • Command Glossary
      • Fundamental Concepts
        • Getting Started
        • Integrating Remote Repositories
        • Branching
        • Merge Conflicts
      • Collaborative Workflows
        • Fork and PR Workflow
        • Branch and PR Workflow
      • Advanced Concepts
        • Ignoring Files
        • Commit Message Conventions
        • Github Collaborators
        • CI/CD with Github Actions
        • Advanced Git Commands
      • FAQ
    • Telegram Bot
      • Creating a TeleBot
      • API Calls
      • Telebot Basics
      • Integrating API's
    • Relational Database
      • Database Overview
      • Database Design
      • Entity Relationship Diagram
      • SQL Basics & PostgreSQL
    • TypeScript
      • Types and Interfaces
      • Utility Types
      • Typing Component Props, Events, and Hooks
      • Why You Should Avoid Using any (and What to Do Instead)
      • TypeScript Tricks You’ll Use All the Time in React
Powered by GitBook
On this page
  • cURL/wget
  • Parsing large config (json) files
  • Getting the latest bus stop timings
  • Getting updates on the weather
  • Getting the weather forecast
  • Scheduling jobs with cron
Edit on GitHub
Export as PDF
  1. Hackerschool
  2. Beginners' Guide to the Terminal

Real World Scripting

PreviousShell ScriptingNextResources

Last updated 8 months ago

So that was a lot to learn! Now, what can we do with it? We haven't really gone through the useful stuff we can do with scripting, so this section will focus more on some cool things we can do with scripting.

To do this, we'll need to learn commands

cURL/wget

curl/wget are two commands that basically do the same thing with slight variations: they both allow us to send and receive data through common networking protocols. For our usecase, the most important thing is that it allows us to talk to an Application Programming Interface (API). Without going into too much detail, APIs are just a magical gateway for us to talk to programs other people have made.

Parsing large config (json) files

JSON is a common format for receiving and transporting data you will eventually have to work with, and it can sometimes be a pain to work with. Luckily, we have a nice program that can help us:

sudo apt-get install jq
brew install jq

Getting the latest bus stop timings

Here's a really nice API:

curl https://arrivelah2.busrouter.sg/?id=18331

It gives us the bus arrival timings at public bus stops! Let's say I want to write a script that gives me the time till the next bus, let's say bus 95:

#!/bin/bash

# Replace this with your actual curl command that fetches the JSON response
response=$(curl -s "https://arrivelah2.busrouter.sg/?id=18331")

# Extract the 'time' field of the next bus using jq
next_bus_time=$(echo "$response" | jq -r '.services[0].next.time')

# Convert current time and the bus time to epoch for comparison
# TODO: Implement this (hint: figure out how to use date +%s)
current_time=
bus_arrival_time=

# Calculate time difference in seconds
time_diff=

# Convert time difference to minutes and seconds
minutes=$((time_diff / 60))
seconds=$((time_diff % 60))

# Display the result
if [ "$time_diff" -gt 0 ]; then
    echo "Bus number 95 will arrive in $minutes minutes and $seconds seconds."
else
    echo "Bus number 95 has already arrived or will arrive shortly."
fi
Solution
#!/bin/bash

# Replace this with your actual curl command that fetches the JSON response
response=$(curl -s "https://arrivelah2.busrouter.sg/?id=18331")

# Extract the 'time' field of the next bus using jq
next_bus_time=$(echo "$response" | jq -r '.services[0].next.time')

# Convert current time and the bus time to epoch for comparison
current_time=$(date +%s)
bus_arrival_time=$(date -d "$next_bus_time" +%s)

# Calculate time difference in seconds
time_diff=$((bus_arrival_time - current_time))

# Convert time difference to minutes and seconds
minutes=$((time_diff / 60))
seconds=$((time_diff % 60))

# Display the result
if [ "$time_diff" -gt 0 ]; then
    echo "Bus number 95 will arrive in $minutes minutes and $seconds seconds."
else
    echo "Bus number 95 has already arrived or will arrive shortly."
fi

Cool, but we don't really want to have to find the directory where our script is every time we want to run it! We can put it in special folders that allow us to run it from anywhere. To find this directory, we can do:

echo $PATH

This checks the environment variable PATH, which what the shell looks at when looking for your normal commands like ls and cd. We can then add our program to the PATH locations, most likely something like /usr/local/bin.

cp bus.sh /usr/local/bin/bus

Now we can just run the bus command from anywhere like a normal shell command!

Getting updates on the weather

Let's do something else that way more complex! We have an API that gives us the 24 hour weather forecast in Singapore. We want to:

  • Check this forecast every morning

  • If it's about to rain, send us an alert. For simplicity, lets send a telegram message!

Getting the weather forecast

curl https://api.data.gov.sg/v1/environment/24-hour-weather-forecast | jq

Here's the rough script we want to send:

#!/bin/bash

# Replace these with your actual Telegram bot token and chat ID
TELEGRAM_BOT_TOKEN="YOUR_BOT_TOKEN"
CHAT_ID="YOUR_CHAT_ID"
TELEGRAM_API_URL="https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage"

# Replace this with your actual curl command that fetches the JSON weather data
response=$(curl -s "https://api.data.gov.sg/v1/environment/24-hour-weather-forecast
")

# Define an array of rain-related keywords
rain_keywords=("Rain" "Showers" "Thundery Showers" "Heavy Thundery Showers")

# Extract the general weather forecast using jq
general_forecast=$(echo "$response" | jq -r '.items[0].general.forecast')

# Initialize a flag to check if rain is found
rain_found=0

# Check for rain-related keywords in the general forecast
# TODO: Try implementing this! Your code should follow this general idea:
# for each keyword in rain_keywords check if the general forecast matches it. If
# so, set rain_found to 1

# If rain is detected, send an alert
if [ "$rain_found" -eq 1 ]; then
    message="Weather Alert: Rain expected! General forecast is '$general_forecast'. Stay prepared!"
    
    # Send the message to the Telegram bot
    curl -s -X POST $TELEGRAM_API_URL \
        -d chat_id=$CHAT_ID \
        -d text="$message" > /dev/null

    echo "Alert sent: $message"
else
    echo "No rain expected. General forecast is '$general_forecast'."
fi
  • To get a telegram bot id, just go to @BotFather, and create a new bot, enter the token we receive inside

  • To get your chat id, just go to @getmyid_bot and copy your chat id there

  • You'll need to start a chat with your bot first, go to your bot and do '/start' before you try running the script

Solution
#!/bin/bash

# Replace these with your actual Telegram bot token and chat ID
TELEGRAM_BOT_TOKEN="YOUR_BOT_TOKEN"
CHAT_ID="YOUR_CHAT_ID"
TELEGRAM_API_URL="https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage"

# Replace this with your actual curl command that fetches the JSON weather data
response=$(curl -s "https://api.data.gov.sg/v1/environment/24-hour-weather-forecast
")

# Define an array of rain-related keywords
rain_keywords=("Rain" "Showers" "Thundery Showers" "Heavy Thundery Showers")

# Extract the general weather forecast using jq
general_forecast=$(echo "$response" | jq -r '.items[0].general.forecast')

# Initialize a flag to check if rain is found
rain_found=0

# Check for rain-related keywords in the general forecast
for keyword in "${rain_keywords[@]}"; do
    if [[ "$general_forecast" == *"$keyword"* ]]; then
        rain_found=1
        break
    fi
done

# If rain is detected, send an alert
if [ "$rain_found" -eq 1 ]; then
    message="Weather Alert: Rain expected! General forecast is '$general_forecast'. Stay prepared!"
    
    # Send the message to the Telegram bot
    curl -s -X POST $TELEGRAM_API_URL \
        -d chat_id=$CHAT_ID \
        -d text="$message" > /dev/null

    echo "Alert sent: $message"
else
    echo "No rain expected. General forecast is '$general_forecast'."
fi

Great! We have a working weather alert! But we don't really want to have to manually run this every morning, it would defeat the purpose of the script! Assuming your laptop runs 24/7, we can make this script run on a schedule using cronjobs!

Obviously, your laptop running 24/7 is unrealistic. However, this method of scheduling scripts to be run is similar to how you would run really important scripts on a schedule: you put a cronjob on an always available server, whether it's in the cloud or somewher running in your basement

Scheduling jobs with cron

  • Cron is a scheduling daemon that executes tasks at specified intervals.

  • These tasks are called cron jobs and are mostly used to automate system maintenance or administration.

Essentially, we can schedule jobs on our machine to run at a specific time by putting in some magic string + the command we want to run. So what is this magic string?

Without going into too much detail about how cronjobs work, we can grab our magic string from the site below

and pick out the crontab we want, in this case:

So our expression should follow the format:

<magic string> <command>

In our case:

0 9 * * * /path/to/script/weather.sh

It is important we use absolute paths here.

And that's it! As long as our laptop or machine is running at 9am in the morning, this script should run automagically!

jq
Logo
data.gov.sgdata.gov.sg
Cron Expression Examples - Crontab.guru
Logo
Logo
Crontab.guru - The cron schedule expression generator
Logo
https://github.com/cheeaun/arrivelah