15 min to read

Understanding jq - The Powerful JSON Processor for Command Line

Learn how to use jq command effectively for JSON processing

Featured image


What is jq?

jq is a lightweight and flexible command-line JSON processor that functions like sed for JSON data.

It allows you to slice, filter, map, and transform structured data with the same ease that sed, awk, and grep let you play with text.

Why Use jq?

✅ Powerful: Process complex JSON structures with concise expressions
✅ Fast: Written in C, handles large JSON files efficiently
✅ Flexible: Can extract, modify, and restructure JSON data
✅ Scriptable: Perfect for shell scripts and automation
✅ Portable: Available on Linux, macOS, Windows and most Unix-like systems
✅ No Dependencies: Standalone tool with zero dependencies


Installation Methods

Standard Package Managers

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install jq

# Red Hat/CentOS/Fedora
sudo yum install jq  # or sudo dnf install jq

# macOS with Homebrew
brew install jq

# Windows with Chocolatey
choco install jq

Installing Specific Versions

# apt (specific version)
sudo apt-get install jq=1.5*

# yum (specific version)
sudo yum install jq-1.5

From Source Code

For the latest features or if packages aren’t available:

git clone https://github.com/stedolan/jq.git
cd jq
autoreconf -i
./configure
make
sudo make install

Verifying Installation

jq --version


Basic jq Syntax

The basic syntax of jq follows this pattern:

jq [options] 'filter' [JSON_FILE]

If no JSON_FILE is provided, jq reads from stdin.

Filters and Expressions

jq uses a domain-specific language for its filters. Here are the basic components:

Symbol Description Example
. Identity operator jq '.' (outputs the entire JSON input)
.property Object property access jq '.name' (outputs the value of "name")
[] Array iterator jq '.[]' (outputs each element of an array)
| Pipe operator jq '.[] | .name' (outputs "name" from each array element)
select() Filter items jq '.[] | select(.age > 30)' (outputs items where age > 30)
{} Object constructor jq '.[] | {name: .name}' (creates new objects with only name field)


Basic Usage

JSON Pretty Printing

The most basic usage is to format JSON for readability:

echo '{"name":"Somaz","age":31,"skills":["Linux","Docker","AWS"]}' | jq '.'

Output:

{
  "name": "Somaz",
  "age": 31,
  "skills": [
    "Linux",
    "Docker",
    "AWS"
  ]
}

Simple Element Extraction

Accessing Object Properties

# Output single element
echo '{"name": "Somaz", "age": 31}' | jq '.name'
# Output
"Somaz"

# Accessing nested properties
echo '{"person": {"name": "Somaz", "details": {"age": 31}}}' | jq '.person.details.age'
# Output
31

Working with Arrays

# Access array elements
echo '{"skills": ["Linux", "Docker", "AWS"]}' | jq '.skills[1]'
# Output
"Docker"

# Extract all array elements
echo '{"team": [{"name": "Somaz"}, {"name": "Oh9Lee"}]}' | jq '.team[].name'
# Output
"Somaz"
"Oh9Lee"

Basic Filtering

# Filter by condition
echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 28}]' | jq '.[] | select(.age < 30)'
# Output
{
  "name": "Oh9Lee",
  "age": 28
}

# Filter by pattern matching
echo '[{"name": "Oh9Lee", "face": "squid"}, {"name": "Oh9Lee", "face": "cuttlefish"}]' | jq '.[] | select(.face | test("^squid"))'
# Output
{
  "name": "Oh9Lee",
  "face": "squid"
}

# Multiple conditions (AND)
echo '[{"name": "Somaz", "age": 31, "role": "DevOps"}, {"name": "John", "age": 35, "role": "Developer"}]' | jq '.[] | select(.age > 30 and .role == "DevOps")'
# Output
{
  "name": "Somaz",
  "age": 31,
  "role": "DevOps"
}


Advanced Usage

Mapping and Transformation

Create new JSON structures from input data:

# Transform objects in an array
echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 28}]' | jq '.[] | {personName: .name, yearOfBirth: (2024 - .age + 1)}'
# Output
{
  "personName": "Somaz",
  "yearOfBirth": 1994
}
{
  "personName": "Oh9Lee",
  "yearOfBirth": 1997
}

# Transform array output back into an array
echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 28}]' | jq '[.[] | {personName: .name, yearOfBirth: (2024 - .age + 1)}]'
# Output
[
  {
    "personName": "Somaz",
    "yearOfBirth": 1994
  },
  {
    "personName": "Oh9Lee",
    "yearOfBirth": 1997
  }
]

Conditional Processing

Apply conditional logic in jq expressions:

# Using if-then-else
echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 28}]' | jq '.[] | {name: .name, status: (if .age >= 30 then "Senior" else "Junior" end)}'
# Output
{
  "name": "Somaz",
  "status": "Senior"
}
{
  "name": "Oh9Lee",
  "status": "Junior"
}

# Using map with conditions
echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 28}]' | jq 'map(if .age > 30 then .name else empty end)'
# Output
[
  "Somaz"
]

Working with Arrays and Objects

Powerful array manipulations:

# Count elements
echo '[{"name": "Somaz"}, {"name": "Oh9Lee"}, {"name": "John"}]' | jq 'length'
# Output
3

# Find minimum/maximum
echo '[{"value": 10}, {"value": 5}, {"value": 15}]' | jq 'map(.value) | min'
# Output
5

# Grouping by a field
echo '[{"name": "Somaz", "team": "DevOps"}, {"name": "John", "team": "Dev"}, {"name": "Jane", "team": "DevOps"}]' | jq 'group_by(.team)'
# Output
[
  [
    {
      "name": "John",
      "team": "Dev"
    }
  ],
  [
    {
      "name": "Somaz",
      "team": "DevOps"
    },
    {
      "name": "Jane",
      "team": "DevOps"
    }
  ]
]

String Manipulation

jq offers many string functions:

# String concatenation
echo '{"first": "Somaz", "last": "Kim"}' | jq '.first + " " + .last'
# Output
"Somaz Kim"

# String operations
echo '{"text": "hello world"}' | jq '.text | upcase'
# Output
"HELLO WORLD"

# Split and join
echo '{"csv": "a,b,c,d"}' | jq '.csv | split(",") | join("-")'
# Output
"a-b-c-d"


Useful Options and Features

Command Line Options

Common jq Options:
  • -r, --raw-output: Output strings without quotes
  • -c, --compact-output: Produce compact output (no pretty-printing)
  • -s, --slurp: Read all inputs into an array before processing
  • -f, --from-file: Read filter from file
  • --arg name value: Pass a value to the jq program as a variable
  • --argjson name value: Pass a JSON-formatted value as a variable
  • --sort-keys: Output object keys in sorted order

Raw Output (-r)

Remove quotes from the output:

echo '{"name": "Somaz", "age": 31}' | jq -r '.name'
# Output
Somaz

# Useful for scripting
NAME=$(echo '{"name": "Somaz", "age": 31}' | jq -r '.name')
echo "Hello, $NAME!"
# Output
Hello, Somaz!

Compact Output (-c)

Output in a compact format, useful for piping to other tools:

echo '[{"name": "Somaz", "age": 31}, {"name": "Oh9Lee", "age": 25}]' | jq -c '.[]'
# Output
{"name":"Somaz","age":31}
{"name":"Oh9Lee","age":25}

Using Variables with –arg

Pass arguments to jq expressions:

# Create data.json
cat <<EOF > data.json
[
  {"name": "Somaz", "age": 31},
  {"name": "Jane", "age": 25},
  {"name": "Doe", "age": 22},
  {"name": "John", "age": 45}
]
EOF

# Filter using a variable
jq --arg min_age 30 '.[] | select(.age > ($min_age | tonumber))' data.json
# Output
{
  "name": "Somaz",
  "age": 31
}
{
  "name": "John",
  "age": 45
}

Slurping Multiple Inputs (-s)

Combine multiple inputs into an array:

echo '{"name": "file1"}' > file1.json
echo '{"name": "file2"}' > file2.json
jq -s '.' file1.json file2.json
# Output
[
  {
    "name": "file1"
  },
  {
    "name": "file2"
  }
]


Real-World Examples

Parsing API Responses

Working with REST API responses:

# Fetch and process GitHub API data
curl -s "https://api.github.com/repos/stedolan/jq/issues?per_page=3" | 
  jq '.[] | {title: .title, author: .user.login, labels: [.labels[].name]}'

# Output might look like:
# {
#   "title": "Issue with parsing complex JSON",
#   "author": "someuser",
#   "labels": ["bug", "help wanted"]
# }
# ...

JSON to CSV Conversion

Transform JSON data to CSV format:

# Create sample data
cat <<EOF > users.json
[
  {"name": "Somaz", "email": "somaz@example.com", "age": 31},
  {"name": "Oh9Lee", "email": "oh9lee@example.com", "age": 28},
  {"name": "John", "email": "john@example.com", "age": 42}
]
EOF

# Convert to CSV (header row + data rows)
jq -r '(["NAME","EMAIL","AGE"] | join(",")), (.[] | [.name, .email, .age] | join(","))' users.json

# Output
NAME,EMAIL,AGE
Somaz,somaz@example.com,31
Oh9Lee,oh9lee@example.com,28
John,john@example.com,42

Processing Configuration Files

Update values in a configuration file:

# Original config file
cat <<EOF > config.json
{
  "app": {
    "name": "MyApp",
    "version": "1.0.0",
    "settings": {
      "debug": false,
      "timeout": 30,
      "maxRetries": 3
    }
  }
}
EOF

# Update specific values
jq '.app.version = "1.1.0" | .app.settings.timeout = 60' config.json > config.new.json

Log Analysis

Parse and analyze JSON logs:

# Sample log file with JSON entries
cat <<EOF > app.log
{"timestamp": "2023-01-01T12:00:00Z", "level": "INFO", "message": "Application started", "user": "admin"}
{"timestamp": "2023-01-01T12:01:30Z", "level": "ERROR", "message": "Database connection failed", "user": "system"}
{"timestamp": "2023-01-01T12:02:15Z", "level": "INFO", "message": "User logged in", "user": "somaz"}
{"timestamp": "2023-01-01T12:05:00Z", "level": "ERROR", "message": "Invalid request", "user": "somaz"}
EOF

# Count log entries by level
jq -r '.level' app.log | sort | uniq -c

# Output
      2 ERROR
      2 INFO

# Filter error logs for a specific user
jq 'select(.level == "ERROR" and .user == "somaz")' app.log

# Output
{"timestamp": "2023-01-01T12:05:00Z", "level": "ERROR", "message": "Invalid request", "user": "somaz"}


jq Functions Reference

jq comes with a rich set of built-in functions:

Category Functions Example
Array Functions length, map, select, sort, group_by, unique, flatten, range jq '[1,2,3,1,2] | unique' → [1,2,3]
Object Functions has, keys, to_entries, from_entries, with_entries jq '{a:1, b:2} | keys' → ["a", "b"]
String Functions length, split, join, ascii_downcase, ascii_upcase, startswith, endswith jq '"abc" | split("")' → ["a","b","c"]
Math Functions +, -, *, /, %, floor, ceil, round, sqrt, pow jq '10 / 3 | floor' → 3
Type Conversion tostring, tonumber, type, tojson, fromjson jq '[1,2,3] | tostring' → "[1,2,3]"


Tips and Best Practices

⚙️ jq Best Practices
  • Testing Filters
    • Start simple and build up complexity gradually
    • Test each part of a complex filter separately
    • Use online tools like jqplay.org for experimentation
  • Performance
    • Filter early to reduce the data being processed
    • Use indices when accessing large arrays
    • Avoid unnecessary array iterations
    • Consider using streaming parsers for very large files
  • Scripting
    • Use -r with variables to avoid quote problems in shell scripts
    • Store complex jq filters in separate files
    • Use --arg for passing variables into filters
    • Remember to escape quotes when embedding jq in scripts
  • Error Handling
    • Use try/catch for operations that might fail
    • Validate input with the type function
    • Provide defaults with the // operator
    • Check the exit code in scripts

Shell Script Integration

Example of integrating jq into a shell script:

#!/bin/bash

# Sample script to update a configuration file
CONFIG_FILE="config.json"
NEW_VERSION="2.0.0"
DEBUG_MODE="true"

# Ensure the config file exists
if [ ! -f "$CONFIG_FILE" ]; then
    echo "Error: Config file not found!"
    exit 1
fi

# Update configuration
jq --arg version "$NEW_VERSION" \
   --arg debug "$DEBUG_MODE" \
   '.version = $version | .settings.debug = ($debug == "true")' \
   "$CONFIG_FILE" > "${CONFIG_FILE}.tmp"

# Check if jq succeeded
if [ $? -ne 0 ]; then
    echo "Error: Failed to update configuration!"
    rm -f "${CONFIG_FILE}.tmp"
    exit 2
fi

# Replace the original file
mv "${CONFIG_FILE}.tmp" "$CONFIG_FILE"
echo "Configuration updated successfully!"


Troubleshooting

Common Issues and Solutions

Problem: "parse error: Invalid numeric literal at line X, column Y"

Solution: Check for malformed JSON. Ensure all strings are quoted, arrays/objects are properly closed, and there are no trailing commas.

Problem: Output includes backslashes or unexpected escaping

Solution: Use the -r (raw) option to avoid JSON string escaping when you need plain text output.

Problem: Cannot access nested properties that might not exist

Solution: Use the ? operator to safely navigate potentially missing properties: .person.address?.zipcode

Problem: Working with property names that contain special characters

Solution: Use bracket notation: .["property-with-dashes"] or .["@specialChars"]


Alternatives to jq

While jq is excellent, there are other tools for JSON processing:

Tool Description Best for
Python General-purpose language with json module Complex processing, integration with other Python libraries
jq alternative (jaq) Rust implementation of jq, faster for some operations Performance-critical applications
fx Command-line JSON processing tool with interactive mode Interactive JSON exploration
gron Makes JSON greppable by flattening it Searching through JSON with grep and other text tools
jsawk JavaScript-based processor similar to awk Those already familiar with JavaScript


Reference