19 min to read
Deep Dive into OpenStack Keystone
Understanding OpenStack's Identity Service

Understanding OpenStack Keystone
Keystone is OpenStack’s identity service that provides centralized authentication and authorization for all OpenStack services. It manages users, projects, and their access to various OpenStack resources through a token-based authentication system.
What is Keystone?
The Identity Management Service
Keystone serves as the central identity service in OpenStack, providing essential functionality:
- Authentication: Verifies user identity and credentials
- Authorization: Manages access control and permissions
- Service Catalog: Maintains service endpoints and URLs
- Token Management: Issues and validates access tokens
- Multi-tenancy: Enables resource isolation through projects and domains
- Federation: Supports identity federation with external systems
As the security backbone of OpenStack, Keystone ensures secure access to cloud resources while maintaining multi-tenancy and role-based access control.
Keystone Architecture Overview (Diagram Description)
- OpenStack Keystone
- Core Services: Identity Backend, Token Backend, Policy Backend, Catalog Backend
- Identity Management: Users & Groups, Projects, Domains
- Access Control: Roles, Permissions, Policies
- Service Integration: Service Endpoints, API Integration, Token Validation
Keystone vs Other Identity Systems
Aspect | OpenStack Keystone | AWS IAM | Azure Active Directory |
---|---|---|---|
Primary Focus | OpenStack cloud identity management | AWS service access control | Enterprise directory services |
Authentication | Token-based, LDAP, Database | Access keys, temporary credentials | SAML, OAuth, OpenID Connect |
Multi-tenancy | Projects and domains | AWS accounts | Tenants and directories |
Federation | SAML, OIDC support | Cross-account roles | Native federation capabilities |
Use Case | Private/hybrid cloud | Public cloud services | Enterprise applications |
Keystone Architecture and Components
Keystone employs a modular architecture with multiple backend services that work together to provide comprehensive identity management. This design enables flexible authentication methods and robust access control across the OpenStack cloud.
Core Components
Component | Role | Description |
---|---|---|
Identity Backend | User Management |
|
Token Backend | Session Management |
|
Policy Backend | Access Control |
|
Catalog Backend | Service Management |
|
Authentication Flow
Sequence Diagram Description
- User/Service sends authentication request to Keystone
- Keystone validates credentials
- Keystone issues token to User/Service
- User/Service requests OpenStack service with token
- Service validates token with Keystone
- Service responds to User/Service
Key Concepts
Keystone manages several key concepts:
- Domains: Top-level containers for users, groups, and projects
- Projects: Resource isolation units (formerly tenants)
- Users & Groups: Individual accounts and collections
- Roles: Access control definitions
- Tokens: Temporary access credentials
- Endpoints: Service access points
These concepts work together to provide comprehensive identity and access management.
Installation and Configuration
Setting up Keystone requires careful planning and configuration. This section covers installation methods, database setup, and initial configuration for production environments.
Prerequisites and Requirements
System Requirements
Configuration File Setup
Apache HTTP Server Configuration
# /etc/apache2/sites-available/keystone.conf
Listen 5000
<VirtualHost *:5000>
WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
WSGIProcessGroup keystone-public
WSGIScriptAlias / /usr/bin/keystone-wsgi-public
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
LimitRequestBody 114688
ErrorLogFormat "%{cu}t %M"
ErrorLog /var/log/apache2/keystone.log
CustomLog /var/log/apache2/keystone_access.log combined
<Directory /usr/bin>
Require all granted
</Directory>
</VirtualHost>
# Enable the site and restart Apache
sudo a2ensite keystone
sudo systemctl restart apache2
Advanced Configuration and Integration
Advanced Keystone deployments require integration with external systems, high availability setup, and comprehensive security configurations.
LDAP Integration
Active Directory Integration
# /etc/keystone/keystone.conf - LDAP configuration
[identity]
driver = ldap
default_domain_id = default
[ldap]
url = ldap://ad.example.com:389
user = cn=keystone,ou=service-accounts,dc=example,dc=com
password = service_account_password
suffix = dc=example,dc=com
# User configuration
user_tree_dn = ou=users,dc=example,dc=com
user_objectclass = person
user_id_attribute = sAMAccountName
user_name_attribute = sAMAccountName
user_mail_attribute = mail
user_enabled_attribute = userAccountControl
user_enabled_mask = 2
user_enabled_default = 512
# Group configuration
group_tree_dn = ou=groups,dc=example,dc=com
group_objectclass = group
group_id_attribute = cn
group_name_attribute = cn
group_member_attribute = member
group_desc_attribute = description
# Query configuration
query_scope = sub
page_size = 1000
alias_dereferencing = never
debug_level = 0
# Connection pool
use_pool = true
pool_size = 10
pool_retry_max = 3
pool_retry_delay = 0.1
pool_connection_timeout = -1
pool_connection_lifetime = 600
# TLS configuration
use_tls = true
tls_cacertfile = /etc/ssl/certs/ca-certificates.crt
tls_req_cert = demand
Multi-Domain LDAP Setup
#!/bin/bash
# Script to configure multiple LDAP domains
# Create domain-specific configuration files
sudo mkdir -p /etc/keystone/domains
# Engineering domain
cat > /etc/keystone/domains/keystone.engineering.conf << EOF
[ldap]
url = ldap://engineering.example.com:389
user = cn=keystone,ou=service,dc=engineering,dc=example,dc=com
password = eng_service_password
suffix = dc=engineering,dc=example,dc=com
user_tree_dn = ou=users,dc=engineering,dc=example,dc=com
group_tree_dn = ou=groups,dc=engineering,dc=example,dc=com
EOF
# Sales domain
cat > /etc/keystone/domains/keystone.sales.conf << EOF
[ldap]
url = ldap://sales.example.com:389
user = cn=keystone,ou=service,dc=sales,dc=example,dc=com
password = sales_service_password
suffix = dc=sales,dc=example,dc=com
user_tree_dn = ou=users,dc=sales,dc=example,dc=com
group_tree_dn = ou=groups,dc=sales,dc=example,dc=com
EOF
# Set proper permissions
sudo chown keystone:keystone /etc/keystone/domains/*
sudo chmod 640 /etc/keystone/domains/*
# Create domains in Keystone
openstack domain create --description "Engineering Department" engineering
openstack domain create --description "Sales Department" sales
High Availability Configuration
Multi-Node Keystone Setup
#!/bin/bash
# High availability Keystone deployment script
# Configure shared database (Galera cluster)
# /etc/mysql/mariadb.conf.d/99-galera.cnf
cat > /etc/mysql/mariadb.conf.d/99-galera.cnf << EOF
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://controller1,controller2,controller3"
wsrep_cluster_name="openstack"
wsrep_node_address="$(hostname -I | awk '{print $1}')"
wsrep_node_name="$(hostname)"
wsrep_sst_method=rsync
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
EOF
# Shared Fernet keys setup
# Sync Fernet keys across all nodes
rsync -av /etc/keystone/fernet-keys/ controller2:/etc/keystone/fernet-keys/
rsync -av /etc/keystone/fernet-keys/ controller3:/etc/keystone/fernet-keys/
# Load balancer configuration (HAProxy)
cat >> /etc/haproxy/haproxy.cfg << EOF
listen keystone_admin_cluster
bind 10.0.0.100:35357
balance source
option httpchk
option httplog
option httpclose
server controller1 10.0.0.11:35357 check inter 2000 rise 2 fall 5
server controller2 10.0.0.12:35357 check inter 2000 rise 2 fall 5
server controller3 10.0.0.13:35357 check inter 2000 rise 2 fall 5
listen keystone_public_cluster
bind 10.0.0.100:5000
balance source
option httpchk
option httplog
option httpclose
server controller1 10.0.0.11:5000 check inter 2000 rise 2 fall 5
server controller2 10.0.0.12:5000 check inter 2000 rise 2 fall 5
server controller3 10.0.0.13:5000 check inter 2000 rise 2 fall 5
EOF
systemctl restart haproxy
Federation and SAML Integration
SAML2 Federation Setup
<!-- /etc/keystone/saml2_idp_metadata.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
entityID="https://idp.example.com/simplesaml/saml2/idp/metadata.php">
<md:IDPSSODescriptor WantAuthnRequestsSigned="false"
protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
<md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
Location="https://idp.example.com/simplesaml/saml2/idp/SSOService.php"/>
<md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
Location="https://idp.example.com/simplesaml/saml2/idp/SSOService.php"/>
</md:IDPSSODescriptor>
</md:EntityDescriptor>
# Federation configuration in keystone.conf
[federation]
driver = sql
assertion_prefix = ADFS_
remote_id_attribute = ADFS_CLAIMS
[saml]
certfile = /etc/keystone/ssl/certs/keystone.pem
keyfile = /etc/keystone/ssl/private/keystone.key
idp_entity_id = https://idp.example.com/simplesaml/saml2/idp/metadata.php
idp_sso_endpoint = https://idp.example.com/simplesaml/saml2/idp/SSOService.php
idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml
# Configure federation
openstack identity provider create --remote-id https://idp.example.com/simplesaml/saml2/idp/metadata.php saml2_idp
openstack mapping create --rules mapping_rules.json saml_mapping
openstack federation protocol create --identity-provider saml2_idp --mapping saml_mapping saml2
Access Control and Authorization
Keystone provides robust access control mechanisms through role-based access control (RBAC) and policy-based authorization. This enables fine-grained control over resource access while maintaining security and compliance.
Role-Based Access Control
Component | Description | Use Cases |
---|---|---|
Roles | Define access permissions for users and groups |
|
Projects | Isolate resources and manage access |
|
Domains | Group related resources and users |
|
Service Integration
Keystone integrates with various OpenStack services:
- Nova Integration: Manages compute service access
- Neutron Integration: Controls network service access
- Glance Integration: Handles image service access
- Cinder Integration: Manages storage service access
This integration ensures consistent authentication and authorization across all OpenStack services.
Practical Operations and Management
Effective Keystone management requires understanding daily operations, troubleshooting techniques, and performance optimization strategies for production environments.
User and Project Management
Command-Line Administration
User Lifecycle Management
Token Management and Security
Token Lifecycle Management
Performance Monitoring and Optimization
Performance Metrics Collection
Usage and Best Practices
Effective use of Keystone requires understanding its capabilities and implementing appropriate security practices. Here are key considerations and best practices for utilizing Keystone effectively.
Security Strategies
Strategy | Description | Benefits |
---|---|---|
Role Management | Implement least privilege principle |
|
Token Security | Manage token lifecycle and security |
|
Policy Management | Define clear access policies |
|
Implementation Guidelines
For effective Keystone implementation:
- Security Planning: Implement comprehensive security measures
- Role Design: Create clear role hierarchies
- Token Management: Configure appropriate token lifetimes
- Monitoring: Track authentication and access patterns
Following these guidelines helps ensure secure and efficient identity management.
Troubleshooting and Maintenance
Effective troubleshooting requires understanding common issues, log analysis techniques, and maintenance procedures for Keystone deployments.
Common Issues and Solutions
Authentication and Token Issues
#!/bin/bash
# Keystone troubleshooting toolkit
# Diagnose authentication failures
diagnose_auth_failures() {
local log_file="/var/log/keystone/keystone.log"
local user_name=$1
echo "Diagnosing authentication issues for user: $user_name"
# Check recent authentication attempts
echo "Recent authentication attempts:"
grep "$user_name" $log_file | grep -E "(authentication|failed|success)" | tail -10
# Check user status
user_info=$(openstack user show $user_name --format json 2>/dev/null)
if [[ $? -eq 0 ]]; then
enabled=$(echo $user_info | jq -r '.enabled')
echo "User enabled: $enabled"
if [[ $enabled == "false" ]]; then
echo "ISSUE: User account is disabled"
fi
else
echo "ISSUE: User not found"
fi
# Check password policy violations
echo "Checking password policy violations:"
grep "$user_name" $log_file | grep -i "password" | grep -E "(expired|policy|violation)" | tail -5
# Check account lockout
lockout_count=$(grep "$user_name" $log_file | grep "lockout" | wc -l)
if [[ $lockout_count -gt 0 ]]; then
echo "WARNING: Account lockout detected ($lockout_count occurrences)"
fi
}
# Fix common token issues
fix_token_issues() {
echo "Checking and fixing token-related issues..."
# Check Fernet key permissions
echo "Checking Fernet key permissions:"
ls -la /etc/keystone/fernet-keys/
# Fix permissions if needed
sudo chown -R keystone:keystone /etc/keystone/fernet-keys/
sudo chmod 600 /etc/keystone/fernet-keys/*
# Check key rotation status
echo "Fernet key rotation status:"
key_count=$(ls /etc/keystone/fernet-keys/ | wc -l)
echo "Number of Fernet keys: $key_count"
if [[ $key_count -lt 2 ]]; then
echo "WARNING: Insufficient Fernet keys, rotating..."
sudo keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
fi
# Clean token cache
echo "Cleaning token cache..."
if command -v memcached &> /dev/null; then
echo "flush_all" | nc localhost 11211
echo "Memcached cache cleared"
fi
# Restart Keystone services
echo "Restarting Keystone services..."
sudo systemctl restart apache2
echo "Services restarted"
}
# Database connectivity check
check_database_connectivity() {
local db_host="localhost"
local db_user="keystone"
local db_pass="$KEYSTONE_DB_PASS"
local db_name="keystone"
echo "Checking database connectivity..."
# Test connection
if mysql -h $db_host -u $db_user -p$db_pass -e "USE $db_name; SELECT 1;" &>/dev/null; then
echo "Database connection: OK"
# Check table integrity
echo "Checking table integrity:"
tables=$(mysql -h $db_host -u $db_user -p$db_pass $db_name -e "SHOW TABLES;" | grep -v Tables)
for table in $tables; do
count=$(mysql -h $db_host -u $db_user -p$db_pass $db_name -e "SELECT COUNT(*) FROM $table;" | tail -1)
echo "Table $table: $count rows"
done
else
echo "ERROR: Database connection failed"
echo "Check database credentials and network connectivity"
fi
}
Service Integration Problems
#!/bin/bash
# Service integration troubleshooting
# Check service catalog issues
check_service_catalog() {
echo "Checking service catalog configuration..."
# List all services
openstack service list --format table
# Check for missing endpoints
services=$(openstack service list -f value -c ID)
for service_id in $services; do
service_name=$(openstack service show $service_id -f value -c name)
endpoint_count=$(openstack endpoint list --service $service_id -f value | wc -l)
if [[ $endpoint_count -eq 0 ]]; then
echo "WARNING: Service '$service_name' has no endpoints"
elif [[ $endpoint_count -lt 3 ]]; then
echo "WARNING: Service '$service_name' missing some endpoint types"
fi
done
}
# Test service connectivity
test_service_connectivity() {
local service_name=$1
echo "Testing connectivity to $service_name service..."
# Get service endpoints
endpoints=$(openstack endpoint list --service $service_name -f value -c URL)
for endpoint in $endpoints; do
echo "Testing endpoint: $endpoint"
# Test HTTP connectivity
if curl -s --connect-timeout 5 "$endpoint/health" &>/dev/null; then
echo " Status: OK"
elif curl -s --connect-timeout 5 "$endpoint" &>/dev/null; then
echo " Status: Reachable (no health endpoint)"
else
echo " Status: FAILED"
fi
done
}
# Fix endpoint URLs
fix_endpoint_urls() {
local old_url=$1
local new_url=$2
echo "Updating endpoint URLs from $old_url to $new_url"
# Find endpoints with old URL
endpoints=$(openstack endpoint list -f value -c ID -c URL | grep "$old_url" | cut -d' ' -f1)
for endpoint_id in $endpoints; do
current_url=$(openstack endpoint show $endpoint_id -f value -c url)
updated_url=${current_url/$old_url/$new_url}
openstack endpoint set --url "$updated_url" $endpoint_id
echo "Updated endpoint $endpoint_id: $current_url -> $updated_url"
done
}
Log Analysis and Monitoring
Comprehensive Log Analysis
Backup and Recovery Procedures
Database Backup and Recovery
Key Points
-
Core Functionality
- Centralized authentication and authorization
- Role-based access control
- Service endpoint management
- Token-based security
- Multi-domain support for organization isolation -
Architecture
- Modular backend design
- Flexible authentication methods
- Comprehensive policy engine
- Service catalog management
- High availability clustering support -
Key Features
- User and group management
- Project and domain isolation
- Token-based authentication
- Service endpoint discovery
- LDAP and federation integration -
Production Considerations
- Database optimization and monitoring
- Fernet key rotation automation
- Security compliance and auditing
- Performance monitoring and tuning
- Comprehensive backup and recovery procedures
Best Practices Summary
Security First: Implement comprehensive security measures including strong authentication, proper access controls, and regular security audits. Use federated identity when possible to leverage existing enterprise identity systems.
High Availability: Deploy Keystone in a clustered configuration with database replication, load balancing, and proper failover mechanisms to ensure service continuity.
Performance Optimization: Monitor and optimize database performance, implement proper caching strategies, and tune token expiration policies based on your organization's security requirements.
Operational Excellence: Establish automated backup procedures, implement comprehensive monitoring, and create detailed runbooks for common operational tasks and troubleshooting scenarios.
References
- OpenStack Keystone Documentation
- Red Hat OpenStack Platform Documentation
- OpenStack Wiki - Keystone
- Keystone API Reference
- OpenStack Identity Management Guide
- Keystone Configuration Reference
- OpenStack Security Guide
- Keystone Federation Documentation
- LDAP Integration Guide
- Token Management Best Practices
Additional Resources
Tools and Utilities
- OpenStack CLI: python-openstackclient
- Keystone Management: python-keystoneclient
- Federation Tools: pysaml2, mod_auth_mellon
- Monitoring: OpenStack Monitoring Guide
Training and Certification
- OpenStack Foundation Training
- Certified OpenStack Administrator (COA)
- Red Hat OpenStack Training
- SUSE OpenStack Cloud Training
Comments