Deep Dive into OpenStack Keystone

Understanding OpenStack's Identity Service

Featured image

Image Reference link



Understanding OpenStack Keystone

Keystone is OpenStack’s identity service that provides centralized authentication and authorization for all OpenStack services. It manages users, projects, and their access to various OpenStack resources through a token-based authentication system.


What is Keystone?


The Identity Management Service

Keystone serves as the central identity service in OpenStack, providing essential functionality:

As the security backbone of OpenStack, Keystone ensures secure access to cloud resources while maintaining multi-tenancy and role-based access control.


Keystone Architecture Overview (Diagram Description)

graph LR A[OpenStack Keystone] A --> B[Core Services] A --> C[Identity Management] A --> D[Access Control] A --> E[Service Integration] B --> B1[Identity Backend] B --> B2[Token Backend] B --> B3[Policy Backend] B --> B4[Catalog Backend] C --> C1[Users & Groups] C --> C2[Projects] C --> C3[Domains] D --> D1[Roles] D --> D2[Permissions] D --> D3[Policies] E --> E1[Service Endpoints] E --> E2[API Integration] E --> E3[Token Validation] style A stroke:#333,stroke-width:1px,fill:#f5f5f5 style B stroke:#333,stroke-width:1px,fill:#a5d6a7 style C stroke:#333,stroke-width:1px,fill:#64b5f6 style D stroke:#333,stroke-width:1px,fill:#ffcc80 style E stroke:#333,stroke-width:1px,fill:#ce93d8


Keystone vs Other Identity Systems

Aspect OpenStack Keystone AWS IAM Azure Active Directory
Primary Focus OpenStack cloud identity management AWS service access control Enterprise directory services
Authentication Token-based, LDAP, Database Access keys, temporary credentials SAML, OAuth, OpenID Connect
Multi-tenancy Projects and domains AWS accounts Tenants and directories
Federation SAML, OIDC support Cross-account roles Native federation capabilities
Use Case Private/hybrid cloud Public cloud services Enterprise applications



Keystone Architecture and Components

Keystone employs a modular architecture with multiple backend services that work together to provide comprehensive identity management. This design enables flexible authentication methods and robust access control across the OpenStack cloud.


Core Components

Component Role Description
Identity Backend User Management
  • Manages user and group authentication
  • Supports LDAP and SQL database integration
  • Handles user credentials and profiles
  • Provides identity verification services
Token Backend Session Management
  • Issues temporary access tokens
  • Manages token lifecycle
  • Handles token validation
  • Supports token revocation
Policy Backend Access Control
  • Manages role-based access control
  • Defines access policies
  • Enforces permission rules
  • Handles authorization decisions
Catalog Backend Service Management
  • Maintains service endpoints
  • Manages service URLs
  • Handles service discovery
  • Provides endpoint information


Authentication Flow

Sequence Diagram Description

sequenceDiagram participant User as "User/Service" participant Keystone as "Keystone" participant Service as "OpenStack Service" User->>Keystone: Authentication Request Keystone->>Keystone: Validate Credentials Keystone-->>User: Issue Token User->>Service: Request with Token Service->>Keystone: Validate Token Keystone-->>Service: Token Valid Service-->>User: Service Response


Key Concepts

Keystone manages several key concepts:

These concepts work together to provide comprehensive identity and access management.



Installation and Configuration

Setting up Keystone requires careful planning and configuration. This section covers installation methods, database setup, and initial configuration for production environments.


Prerequisites and Requirements

System Requirements

Configuration File Setup

Apache HTTP Server Configuration

# /etc/apache2/sites-available/keystone.conf
Listen 5000

<VirtualHost *:5000>
    WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
    WSGIProcessGroup keystone-public
    WSGIScriptAlias / /usr/bin/keystone-wsgi-public
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    LimitRequestBody 114688
    
    ErrorLogFormat "%{cu}t %M"
    ErrorLog /var/log/apache2/keystone.log
    CustomLog /var/log/apache2/keystone_access.log combined
    
    <Directory /usr/bin>
        Require all granted
    </Directory>
</VirtualHost>

# Enable the site and restart Apache
sudo a2ensite keystone
sudo systemctl restart apache2



Advanced Configuration and Integration

Advanced Keystone deployments require integration with external systems, high availability setup, and comprehensive security configurations.


LDAP Integration

Active Directory Integration

# /etc/keystone/keystone.conf - LDAP configuration
[identity]
driver = ldap
default_domain_id = default

[ldap]
url = ldap://ad.example.com:389
user = cn=keystone,ou=service-accounts,dc=example,dc=com
password = service_account_password
suffix = dc=example,dc=com

# User configuration
user_tree_dn = ou=users,dc=example,dc=com
user_objectclass = person
user_id_attribute = sAMAccountName
user_name_attribute = sAMAccountName
user_mail_attribute = mail
user_enabled_attribute = userAccountControl
user_enabled_mask = 2
user_enabled_default = 512

# Group configuration
group_tree_dn = ou=groups,dc=example,dc=com
group_objectclass = group
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description

# Query configuration
query_scope = sub
page_size = 1000
alias_dereferencing = never
debug_level = 0

# Connection pool
use_pool = true
pool_size = 10
pool_retry_max = 3
pool_retry_delay = 0.1
pool_connection_timeout = -1
pool_connection_lifetime = 600

# TLS configuration
use_tls = true
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt
tls_req_cert = demand

Multi-Domain LDAP Setup

#!/bin/bash
# Script to configure multiple LDAP domains

# Create domain-specific configuration files
sudo mkdir -p /etc/keystone/domains

# Engineering domain
cat > /etc/keystone/domains/keystone.engineering.conf << EOF
[ldap]
url = ldap://engineering.example.com:389
user = cn=keystone,ou=service,dc=engineering,dc=example,dc=com
password = eng_service_password
suffix = dc=engineering,dc=example,dc=com
user_tree_dn = ou=users,dc=engineering,dc=example,dc=com
group_tree_dn = ou=groups,dc=engineering,dc=example,dc=com
EOF

# Sales domain
cat > /etc/keystone/domains/keystone.sales.conf << EOF
[ldap]
url = ldap://sales.example.com:389
user = cn=keystone,ou=service,dc=sales,dc=example,dc=com
password = sales_service_password
suffix = dc=sales,dc=example,dc=com
user_tree_dn = ou=users,dc=sales,dc=example,dc=com
group_tree_dn = ou=groups,dc=sales,dc=example,dc=com
EOF

# Set proper permissions
sudo chown keystone:keystone /etc/keystone/domains/*
sudo chmod 640 /etc/keystone/domains/*

# Create domains in Keystone
openstack domain create --description "Engineering Department" engineering
openstack domain create --description "Sales Department" sales

High Availability Configuration

Multi-Node Keystone Setup

#!/bin/bash
# High availability Keystone deployment script

# Configure shared database (Galera cluster)
# /etc/mysql/mariadb.conf.d/99-galera.cnf
cat > /etc/mysql/mariadb.conf.d/99-galera.cnf << EOF
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://controller1,controller2,controller3"
wsrep_cluster_name="openstack"
wsrep_node_address="$(hostname -I | awk '{print $1}')"
wsrep_node_name="$(hostname)"
wsrep_sst_method=rsync
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
EOF

# Shared Fernet keys setup
# Sync Fernet keys across all nodes
rsync -av /etc/keystone/fernet-keys/ controller2:/etc/keystone/fernet-keys/
rsync -av /etc/keystone/fernet-keys/ controller3:/etc/keystone/fernet-keys/

# Load balancer configuration (HAProxy)
cat >> /etc/haproxy/haproxy.cfg << EOF
listen keystone_admin_cluster
  bind 10.0.0.100:35357
  balance source
  option httpchk
  option httplog
  option httpclose
  server controller1 10.0.0.11:35357 check inter 2000 rise 2 fall 5
  server controller2 10.0.0.12:35357 check inter 2000 rise 2 fall 5
  server controller3 10.0.0.13:35357 check inter 2000 rise 2 fall 5

listen keystone_public_cluster
  bind 10.0.0.100:5000
  balance source
  option httpchk
  option httplog
  option httpclose
  server controller1 10.0.0.11:5000 check inter 2000 rise 2 fall 5
  server controller2 10.0.0.12:5000 check inter 2000 rise 2 fall 5
  server controller3 10.0.0.13:5000 check inter 2000 rise 2 fall 5
EOF

systemctl restart haproxy

Federation and SAML Integration

SAML2 Federation Setup

<!-- /etc/keystone/saml2_idp_metadata.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
                     entityID="https://idp.example.com/simplesaml/saml2/idp/metadata.php">
  <md:IDPSSODescriptor WantAuthnRequestsSigned="false"
                       protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
    <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
                            Location="https://idp.example.com/simplesaml/saml2/idp/SSOService.php"/>
    <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
                            Location="https://idp.example.com/simplesaml/saml2/idp/SSOService.php"/>
  </md:IDPSSODescriptor>
</md:EntityDescriptor>
# Federation configuration in keystone.conf
[federation]
driver = sql
assertion_prefix = ADFS_
remote_id_attribute = ADFS_CLAIMS

[saml]
certfile = /etc/keystone/ssl/certs/keystone.pem
keyfile = /etc/keystone/ssl/private/keystone.key
idp_entity_id = https://idp.example.com/simplesaml/saml2/idp/metadata.php
idp_sso_endpoint = https://idp.example.com/simplesaml/saml2/idp/SSOService.php
idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml
# Configure federation
openstack identity provider create --remote-id https://idp.example.com/simplesaml/saml2/idp/metadata.php saml2_idp

openstack mapping create --rules mapping_rules.json saml_mapping

openstack federation protocol create --identity-provider saml2_idp --mapping saml_mapping saml2

Access Control and Authorization

Keystone provides robust access control mechanisms through role-based access control (RBAC) and policy-based authorization. This enables fine-grained control over resource access while maintaining security and compliance.


Role-Based Access Control

Component Description Use Cases
Roles Define access permissions for users and groups
  • Admin access
  • Developer access
  • Read-only access
Projects Isolate resources and manage access
  • Resource isolation
  • Multi-tenant environments
  • Project-based access
Domains Group related resources and users
  • Organization separation
  • Department isolation
  • Resource grouping


Service Integration

Keystone integrates with various OpenStack services:

This integration ensures consistent authentication and authorization across all OpenStack services.



Practical Operations and Management

Effective Keystone management requires understanding daily operations, troubleshooting techniques, and performance optimization strategies for production environments.


User and Project Management

Command-Line Administration

User Lifecycle Management

Token Management and Security

Token Lifecycle Management

Performance Monitoring and Optimization

Performance Metrics Collection


Usage and Best Practices

Effective use of Keystone requires understanding its capabilities and implementing appropriate security practices. Here are key considerations and best practices for utilizing Keystone effectively.


Security Strategies

Strategy Description Benefits
Role Management Implement least privilege principle
  • Enhanced security
  • Reduced risk
  • Better control
Token Security Manage token lifecycle and security
  • Secure access
  • Session control
  • Access tracking
Policy Management Define clear access policies
  • Consistent access
  • Clear permissions
  • Audit compliance


Implementation Guidelines

For effective Keystone implementation:

Following these guidelines helps ensure secure and efficient identity management.



Troubleshooting and Maintenance

Effective troubleshooting requires understanding common issues, log analysis techniques, and maintenance procedures for Keystone deployments.


Common Issues and Solutions

Authentication and Token Issues

#!/bin/bash
# Keystone troubleshooting toolkit

# Diagnose authentication failures
diagnose_auth_failures() {
    local log_file="/var/log/keystone/keystone.log"
    local user_name=$1
    
    echo "Diagnosing authentication issues for user: $user_name"
    
    # Check recent authentication attempts
    echo "Recent authentication attempts:"
    grep "$user_name" $log_file | grep -E "(authentication|failed|success)" | tail -10
    
    # Check user status
    user_info=$(openstack user show $user_name --format json 2>/dev/null)
    if [[ $? -eq 0 ]]; then
        enabled=$(echo $user_info | jq -r '.enabled')
        echo "User enabled: $enabled"
        
        if [[ $enabled == "false" ]]; then
            echo "ISSUE: User account is disabled"
        fi
    else
        echo "ISSUE: User not found"
    fi
    
    # Check password policy violations
    echo "Checking password policy violations:"
    grep "$user_name" $log_file | grep -i "password" | grep -E "(expired|policy|violation)" | tail -5
    
    # Check account lockout
    lockout_count=$(grep "$user_name" $log_file | grep "lockout" | wc -l)
    if [[ $lockout_count -gt 0 ]]; then
        echo "WARNING: Account lockout detected ($lockout_count occurrences)"
    fi
}

# Fix common token issues
fix_token_issues() {
    echo "Checking and fixing token-related issues..."
    
    # Check Fernet key permissions
    echo "Checking Fernet key permissions:"
    ls -la /etc/keystone/fernet-keys/
    
    # Fix permissions if needed
    sudo chown -R keystone:keystone /etc/keystone/fernet-keys/
    sudo chmod 600 /etc/keystone/fernet-keys/*
    
    # Check key rotation status
    echo "Fernet key rotation status:"
    key_count=$(ls /etc/keystone/fernet-keys/ | wc -l)
    echo "Number of Fernet keys: $key_count"
    
    if [[ $key_count -lt 2 ]]; then
        echo "WARNING: Insufficient Fernet keys, rotating..."
        sudo keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
    fi
    
    # Clean token cache
    echo "Cleaning token cache..."
    if command -v memcached &> /dev/null; then
        echo "flush_all" | nc localhost 11211
        echo "Memcached cache cleared"
    fi
    
    # Restart Keystone services
    echo "Restarting Keystone services..."
    sudo systemctl restart apache2
    echo "Services restarted"
}

# Database connectivity check
check_database_connectivity() {
    local db_host="localhost"
    local db_user="keystone"
    local db_pass="$KEYSTONE_DB_PASS"
    local db_name="keystone"
    
    echo "Checking database connectivity..."
    
    # Test connection
    if mysql -h $db_host -u $db_user -p$db_pass -e "USE $db_name; SELECT 1;" &>/dev/null; then
        echo "Database connection: OK"
        
        # Check table integrity
        echo "Checking table integrity:"
        tables=$(mysql -h $db_host -u $db_user -p$db_pass $db_name -e "SHOW TABLES;" | grep -v Tables)
        
        for table in $tables; do
            count=$(mysql -h $db_host -u $db_user -p$db_pass $db_name -e "SELECT COUNT(*) FROM $table;" | tail -1)
            echo "Table $table: $count rows"
        done
    else
        echo "ERROR: Database connection failed"
        echo "Check database credentials and network connectivity"
    fi
}

Service Integration Problems

#!/bin/bash
# Service integration troubleshooting

# Check service catalog issues
check_service_catalog() {
    echo "Checking service catalog configuration..."
    
    # List all services
    openstack service list --format table
    
    # Check for missing endpoints
    services=$(openstack service list -f value -c ID)
    
    for service_id in $services; do
        service_name=$(openstack service show $service_id -f value -c name)
        endpoint_count=$(openstack endpoint list --service $service_id -f value | wc -l)
        
        if [[ $endpoint_count -eq 0 ]]; then
            echo "WARNING: Service '$service_name' has no endpoints"
        elif [[ $endpoint_count -lt 3 ]]; then
            echo "WARNING: Service '$service_name' missing some endpoint types"
        fi
    done
}

# Test service connectivity
test_service_connectivity() {
    local service_name=$1
    
    echo "Testing connectivity to $service_name service..."
    
    # Get service endpoints
    endpoints=$(openstack endpoint list --service $service_name -f value -c URL)
    
    for endpoint in $endpoints; do
        echo "Testing endpoint: $endpoint"
        
        # Test HTTP connectivity
        if curl -s --connect-timeout 5 "$endpoint/health" &>/dev/null; then
            echo "  Status: OK"
        elif curl -s --connect-timeout 5 "$endpoint" &>/dev/null; then
            echo "  Status: Reachable (no health endpoint)"
        else
            echo "  Status: FAILED"
        fi
    done
}

# Fix endpoint URLs
fix_endpoint_urls() {
    local old_url=$1
    local new_url=$2
    
    echo "Updating endpoint URLs from $old_url to $new_url"
    
    # Find endpoints with old URL
    endpoints=$(openstack endpoint list -f value -c ID -c URL | grep "$old_url" | cut -d' ' -f1)
    
    for endpoint_id in $endpoints; do
        current_url=$(openstack endpoint show $endpoint_id -f value -c url)
        updated_url=${current_url/$old_url/$new_url}
        
        openstack endpoint set --url "$updated_url" $endpoint_id
        echo "Updated endpoint $endpoint_id: $current_url -> $updated_url"
    done
}

Log Analysis and Monitoring

Comprehensive Log Analysis

Backup and Recovery Procedures

Database Backup and Recovery


Key Points

Keystone Essentials
  • Core Functionality
    - Centralized authentication and authorization
    - Role-based access control
    - Service endpoint management
    - Token-based security
    - Multi-domain support for organization isolation
  • Architecture
    - Modular backend design
    - Flexible authentication methods
    - Comprehensive policy engine
    - Service catalog management
    - High availability clustering support
  • Key Features
    - User and group management
    - Project and domain isolation
    - Token-based authentication
    - Service endpoint discovery
    - LDAP and federation integration
  • Production Considerations
    - Database optimization and monitoring
    - Fernet key rotation automation
    - Security compliance and auditing
    - Performance monitoring and tuning
    - Comprehensive backup and recovery procedures


Best Practices Summary

Enterprise Keystone Deployment Guidelines:

Security First: Implement comprehensive security measures including strong authentication, proper access controls, and regular security audits. Use federated identity when possible to leverage existing enterprise identity systems.

High Availability: Deploy Keystone in a clustered configuration with database replication, load balancing, and proper failover mechanisms to ensure service continuity.

Performance Optimization: Monitor and optimize database performance, implement proper caching strategies, and tune token expiration policies based on your organization's security requirements.

Operational Excellence: Establish automated backup procedures, implement comprehensive monitoring, and create detailed runbooks for common operational tasks and troubleshooting scenarios.



References

Additional Resources

Tools and Utilities

Training and Certification

Community Resources