Stage 2 · Tools

Text Processing

jq — JSON Processing

Parse, filter, transform, and generate JSON from the command line.

7 min readShell Scripting & AutomationTools

jq Basics

jq is a command-line JSON processor — sed for JSON. It takes JSON input, applies filters, and produces formatted output. Essential for working with APIs, configuration files, and any JSON data in shell scripts.

Basic jq usage


# Pretty-print JSON
echo '{"name":"Alice","age":30}' | jq .

# Extract a field
echo '{"name":"Alice","age":30}' | jq '.name'

# Extract multiple fields
echo '{"name":"Alice","age":30}' | jq '{name, age}'

# Process a JSON file
jq '.users[]' data.json

# Compact output (no pretty-printing)
jq -c '.' data.json

jq filters are applied to JSON input. The . operator selects the root. .name selects the name field. .[] iterates over arrays. -c produces compact output for piping.

Use jq for API responses

When working with REST APIs, pipe responses through jq: curl -s https://api.example.com/data | jq '.results[] | {name, id}'. This extracts and formats the data you need.

Filtering and Selecting

jq provides powerful filters for selecting specific data from complex JSON structures. These filters can be combined to extract exactly what you need.

Filter patterns


# Array indexing
echo '[1,2,3,4,5]' | jq '.[0]'       # 1
echo '[1,2,3,4,5]' | jq '.[-1]'      # 5
echo '[1,2,3,4,5]' | jq '.[2:4]'     # [3,4]

# Object field access
echo '{"user":{"name":"Alice","role":"admin"}}' | jq '.user.name'

# Iterate array
echo '[{"name":"a"},{"name":"b"}]' | jq '.[].name'

# Select with condition
echo '[{"name":"a","active":true},{"name":"b","active":false}]' |
  jq '.[] | select(.active == true)'

# Recursive descent
echo '{"a":{"b":{"c":42}}}' | jq '.. | .c? // empty'

jq filters can chain: .users[] | select(.age > 25) | .name iterates users, filters by age, and extracts names. The // operator provides defaults for missing values.

Transforming Data

jq can reshape JSON structures — creating new objects, arrays, and computed fields. This is where jq becomes truly powerful for data transformation.

Transform examples


# Create new object
echo '{"first":"Alice","last":"Smith"}' |
  jq '{fullName: (.first + " " + .last)}'

# Map over array
echo '[1,2,3]' | jq 'map(. * 2)'    # [2,4,6]

# String interpolation
echo '{"name":"Alice","age":30}' |
  jq '"(.name) is (.age) years old"'

# Merge objects
echo '{"a":1}' | jq '. + {"b":2}'   # {"a":1,"b":2}

# Reduce array
echo '[1,2,3,4,5]' | jq 'reduce .[] as $x (0; . + $x)'  # 15

jq's transformation capabilities make it a full data processing language. Use map() for array transformations, string interpolation for formatting, and reduce() for aggregations.

Arrays and Objects

jq has first-class support for building and manipulating arrays and objects. This is essential for reshaping API responses and building JSON payloads.

Building arrays and objects


# Collect into array
echo '1\n2\n3' | jq -s '.'   # [1,2,3]

# Group by field
echo '[{"t":"a","v":1},{"t":"a","v":2},{"t":"b","v":3}]' |
  jq 'group_by(.t) | map({key: .[0].t, values: map(.v)})'

# Sort by field
echo '[{"n":"c"},{"n":"a"},{"n":"b"}]' | jq 'sort_by(.n)'

# Flatten nested arrays
echo '[[1,2],[3,4],[5]]' | jq 'flatten'   # [1,2,3,4,5]

# Zip arrays
echo '[1,2,3]' | jq '[., [10,20,30]] | transpose'

Use -s (slurp) to read multiple JSON inputs into an array. group_by() groups objects by a field. sort_by() sorts by a field. These are essential for data analysis.

Conditional Logic

jq supports if/then/else, try/catch, and error handling. This allows complex transformations and data validation.

Conditional patterns


# if/then/else
echo '{"role":"admin"}' |
  jq 'if .role == "admin" then "privileged" else "normal" end'

# try/catch
echo '{"name":null}' |
  jq '.name // "default"'

# Error handling
echo 'not json' | jq '.' 2>/dev/null || echo "Invalid JSON"

# Map with conditions
echo '[1,2,3,4,5]' |
  jq 'map(if . > 3 then "big" else "small" end)'

The // operator provides a default value when the left side is null or false. try/catch handles errors in filters. if/then/else allows conditional transformations.

jq in shell scripts

In shell scripts, use jq -r for raw strings (no quotes), jq -e to set exit code based on result, and jq -c for compact output. Always check jq's exit code: data=$(curl ... | jq '.field') || handle_error.

Practical Examples

Real-world jq patterns


# Extract Docker container IPs
docker inspect $(docker ps -q) | jq '.[] | {name: .Name, ip: .NetworkSettings.IPAddress}'

# Process GitHub API response
curl -s "https://api.github.com/repos owner/repo/releases" |
  jq '.[0] | {tag: .tag_name, date: .published_at, assets: [.assets[].name]}'

# Merge multiple JSON files
jq -s '.[0] * .[1]' defaults.json overrides.json

# Validate JSON structure
echo '{"name":"test","items":[1,2,3]}' |
  jq 'has("name") and (.items | type == "array")'

# Extract nested values
echo '{"data":{"users":[{"profile":{"email":"a@b.com"}}]}}' |
  jq '.data.users[].profile.email'

These patterns cover common tasks: API response processing, configuration merging, data extraction, and validation. jq is indispensable for any script that works with JSON.

Use jq -r for shell integration

Without -r, jq wraps strings in quotes: "Alice". With -r, you get raw output: Alice. Use -r when the output feeds into shell variables or other commands.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.