10 min to read

Understanding and Using Curl - A Comprehensive Guide

A detailed guide to using Curl for HTTP requests and API testing

Featured image

Image Reference link


Introduction to Curl

Curl (Client URL) is a powerful command-line tool and library for transferring data with URLs using various protocols. It's an essential tool for developers, system administrators, and security professionals, enabling everything from simple web requests to complex API interactions, with support for over 25 protocols including HTTP, HTTPS, FTP, and SMTP.
Before You Begin

Before diving deep into Curl, it's recommended to understand HTTP methods, which you can find in my previous post: HTTP Methods and Status Codes.

Curl offers several key capabilities that make it indispensable for development and troubleshooting:

  • Universal Protocol Support: Works with numerous protocols beyond just HTTP
  • Cross-Platform Compatibility: Available on virtually all operating systems
  • Scriptability: Easily incorporated into shell scripts and automation
  • No GUI Dependencies: Works in headless environments and terminals
  • Extensive Documentation: Comprehensive man pages and online resources
  • Active Development: Regular updates and security patches


Curl Command Structure

Understanding the structure of curl commands is essential for effective usage. This section explains the basic syntax, common options, and how to combine them for different tasks.

Basic Syntax

Command Structure

The basic syntax for curl commands follows this pattern:

curl [options] [URL]

For example, a simple GET request would look like:

curl https://www.example.com

While a more complex POST request might be:

curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://api.example.com/endpoint

Common Curl Options

graph TD A[Curl Options] --> B[Request Methods] A --> C[Headers and Data] A --> D[Output Control] A --> E[Connection Options] A --> F[Authentication] B --> B1[-X, --request] B --> B2[-G, --get] C --> C1[-H, --header] C --> C2[-d, --data] C --> C3[--json] D --> D1[-o, --output] D --> D2[-O, --remote-name] D --> D3[-s, --silent] D --> D4[-v, --verbose] E --> E1[-L, --location] E --> E2[-k, --insecure] E --> E3[--connect-timeout] F --> F1[-u, --user] F --> F2[--oauth2-bearer] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#a5d6a7,stroke:#333,stroke-width:1px style C fill:#64b5f6,stroke:#333,stroke-width:1px style D fill:#ffcc80,stroke:#333,stroke-width:1px style E fill:#ce93d8,stroke:#333,stroke-width:1px style F fill:#ef9a9a,stroke:#333,stroke-width:1px
Category Option Description
Request Methods -X, --request Specifies the request method to use (GET, POST, PUT, DELETE, etc.)
-G, --get Forces the request to be sent as GET even with data parameters
Headers and Data -H, --header Adds a custom header to the request (can be used multiple times)
-d, --data Sends data in the request body, typically with POST, PUT (automatically sets method to POST)
--json Sets Content-Type header to application/json and data format to JSON
Output Control -o, --output Writes output to a specified file instead of stdout
-O, --remote-name Saves the output using the name of the remote file
-s, --silent Suppresses progress meters and error messages
-v, --verbose Makes curl display detailed information about the request and response
Connection Options -L, --location Follows HTTP redirects, useful for websites that redirect requests
-k, --insecure Allows insecure connections when using SSL, skipping certificate validation
--connect-timeout Sets maximum time allowed for connection in seconds
Authentication -u, --user Provides username and password for server authentication
--oauth2-bearer Sends OAuth 2.0 Bearer Token in the Authorization header


Basic Curl Usage Examples

These examples demonstrate the fundamental ways to use curl for common tasks. Each example includes the command and a brief explanation of what it does, providing a practical foundation for using curl effectively.

HTTP Request Methods

Simple GET Request

Fetch the content of a webpage:

curl https://www.example.com
Making a POST Request

Send form data to a server:

curl -X POST -d "param1=value1&param2=value2" https://www.example.com/form
Sending JSON Data

Submit JSON data to an API endpoint:



PUT Request for Updates

Update an existing resource:



DELETE Request

Remove a resource from the server:

curl -X DELETE https://api.example.com/resources/123

Handling Output

Saving Response to a File

Save the response body to a specific file:

# Save with custom name
curl -o example.html https://www.example.com

# Save with original filename
curl -O https://www.example.com/file.zip
Viewing Headers Only

Check HTTP headers without downloading the body:

curl -I https://www.example.com
Verbose Output for Debugging

Show detailed information about the request and response:

curl -v https://www.example.com
sequenceDiagram participant Client as Curl Client participant Server as Web Server Note over Client,Server: Simple GET Request Client->>Server: GET / HTTP/1.1 Server-->>Client: 200 OK (HTML Content) Note over Client,Server: POST with Form Data Client->>Server: POST /form HTTP/1.1
Content-Type: application/x-www-form-urlencoded
param1=value1&param2=value2 Server-->>Client: 200 OK (Response) Note over Client,Server: JSON API Request Client->>Server: POST /api HTTP/1.1
Content-Type: application/json
{"key":"value"} Server-->>Client: 201 Created (JSON Response)

Common Connection Options

Following Redirects

Automatically follow HTTP redirect responses:

curl -L https://github.com
Setting Timeouts

Limit the connection time to prevent hanging:

curl --connect-timeout 5 https://slow-server.example.com
Ignoring SSL Certificate Validation

Bypass SSL verification for testing or self-signed certificates (use with caution):

curl -k https://self-signed.badssl.com/


Advanced Curl Techniques

For more complex tasks, curl offers advanced features that allow for sophisticated request handling, authentication, and integration with other tools and services. These examples demonstrate how to leverage curl's powerful capabilities for professional use cases.

Working with APIs

GitHub API Example

Create a pull request using GitHub's API:



Using Query Parameters

Search for repositories on GitHub:



Authentication Methods

Basic Authentication

Access resources that require username and password:

curl -u username:password https://api.example.com/protected-resource
Bearer Token Authentication

Use OAuth 2.0 or JWT tokens for authentication:



Cookies and Sessions

Handling Cookies

Maintain session state across requests:



File Upload

Uploading Files

Send files to a server using multipart/form-data:



Custom File Uploads

Set content type and filename for uploaded files:



Practical Curl Use Cases

Beyond basic web requests, curl serves as a versatile tool for a wide range of practical applications. This section demonstrates useful real-world examples that showcase curl's flexibility and utility for everyday tasks.

Network Diagnostics

Check Public IP Address

Quickly determine your public IP address:

curl -s ipinfo.io/ip
# or
curl ifconfig.me
Testing DNS Resolution

Verify DNS resolution by forcing curl to use a specific IP:

curl -v --resolve example.com:443:93.184.216.34 https://example.com

Web Services and Information

Weather Information

Get current weather conditions in a terminal-friendly format:

curl wttr.in/Seoul



Currency Exchange Rates

Get the latest exchange rates:

curl -s https://api.exchangerate-api.com/v4/latest/USD | jq

HTTP Server Testing

Load Testing

Perform basic load testing with multiple requests:

# Make 10 concurrent requests
for i in {1..10}; do
  curl -s https://example.com > /dev/null &
done
Website Response Time

Measure how long it takes to connect to a website and receive data:



Best Practices and Tips

Following best practices ensures effective and secure usage of curl. These guidelines help avoid common pitfalls and enhance the usefulness of curl in both development and production environments.

Security Considerations

Protecting Sensitive Data
  1. Avoid Exposing Credentials: Don't include credentials directly in command lines as they may be visible in process lists or command history
  2. Use Environment Variables: Store tokens and passwords in environment variables instead of directly in scripts
  3. Clear History: Clear command history after running commands with sensitive information
  4. Prefer .netrc for Stored Credentials: Use the .netrc file with proper permissions for recurring authentication
  5. Be Cautious with -k/--insecure: Only bypass SSL verification in controlled test environments, never in production

Scripting Efficiency

Building Reliable Scripts
  1. Use Silent Mode in Scripts: Always use -s (silent) in scripts to suppress progress meters
  2. Add Error Handling: Check exit codes to handle failures gracefully
  3. Set Timeouts: Always set appropriate timeouts to prevent scripts from hanging indefinitely
  4. Parse Responses Properly: Use tools like jq for JSON or grep/sed for text parsing
  5. Follow Redirects When Needed: Include -L for sites that might redirect
Script Example with Best Practices

Here's an example of a curl script following best practices:



Key Points

💡 Curl Essentials
  • Core Capabilities
    - Supports over 25 protocols including HTTP, HTTPS, FTP, FTPS, SFTP
    - Works on virtually all operating systems and environments
    - Handles all common HTTP methods (GET, POST, PUT, DELETE, etc.)
    - Offers comprehensive header and request customization
  • Common Use Cases
    - API testing and interaction
    - Downloading and uploading files
    - Debugging network and web services
    - Automation and scripting of web requests
    - Web service health checks and monitoring
  • Best Practices
    - Use environment variables for sensitive data
    - Include proper error handling in scripts
    - Set appropriate timeouts for requests
    - Follow redirects when working with web apps
    - Use silent mode in scripts and with proper parsing


References