OpenStack Keystone Identity Service

Keystone is the identity service for OpenStack. Every API call to every OpenStack service passes through Keystone for authentication and authorization. It is the first service deployed and the last decommissioned. If Keystone is down, the entire cloud is down.

Keystone provides five core functions: identity (users, groups), resources (projects, domains), assignment (roles mapped to users on projects), token (authentication proof with configurable lifetime), and catalog (service endpoint registry). Understanding Keystone means understanding how every OpenStack service discovers and trusts every other service.

Deploy

Kolla-Ansible Configuration

globals.yml settings:

# Required -- set strong passwords
keystone_admin_password: "{{ vault_keystone_admin_password }}"
keystone_database_password: "{{ vault_keystone_database_password }}"

# TLS (recommended for production)
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"

# Token provider (Fernet is default and recommended)
keystone_token_provider: "fernet"

# Optional tuning
keystone_token_expiration: 3600  # seconds, default 1 hour

Deployment sequence:

# 1. Bootstrap -- creates databases, service users, initial endpoints
kolla-ansible -i inventory bootstrap-servers

# 2. Deploy Keystone (runs as part of full deploy or targeted)
kolla-ansible -i inventory deploy --tags keystone

# 3. Post-deploy -- creates admin credentials file
kolla-ansible -i inventory post-deploy

Container verification:

# Verify Keystone containers are running
docker ps --filter "name=keystone" --format "table {{.Names}}\t{{.Status}}"
# Expected: keystone_api, keystone_fernet (both Up)

# Verify service responds
source /etc/kolla/admin-openrc.sh
openstack token issue  # Must return a valid token table

Service catalog registration:

Kolla-Ansible auto-registers Keystone in the service catalog. Verify:

openstack service list  # Should show "identity" service
openstack endpoint list --service keystone
# Expected: 3 endpoints (public, internal, admin) per region

Initial admin setup:

# admin-openrc.sh is generated by post-deploy
source /etc/kolla/admin-openrc.sh

# Verify admin project and user exist
openstack project show admin
openstack user show admin
openstack role assignment list --user admin --project admin
# admin should have "admin" role on "admin" project

Configure

Fernet Token Provider

Fernet tokens are cryptographic tokens validated without database lookup. They require synchronized key repositories across Keystone nodes.

Key repository:

/etc/kolla/keystone/fernet-keys/

Key types:

• Staging key (index 0): Used for decryption only; becomes primary on rotation
• Primary key (highest index): Used for encryption; tokens it created remain valid
• Secondary keys (middle indices): Decrypt only; pruned after max_active_keys exceeded

Rotation procedure:

# Rotate Fernet keys (Kolla-Ansible managed)
kolla-ansible -i inventory keystone_fernet_rotate

# Manual rotation (if needed)
docker exec keystone_api keystone-manage fernet_rotate \
  --keystone-user keystone --keystone-group keystone

Configuration:

keystone.conf [fernet_tokens]

• max_active_keys

  : Default 3. Set to (token_expiration / rotation_interval) + 2

Domain, Project, and User Hierarchy

Domain (organizational boundary)
  +-- Project (resource container, formerly "tenant")
       +-- User (identity with credentials)
       +-- Group (collection of users)
       +-- Role Assignment (user/group + role on this project)

Best practices:

• Create a domain per organization or department
• Use the "default" domain for service accounts
• Never modify the "admin" project; create separate operator projects
• Assign roles to groups, not individual users, for scalability

RBAC Policies

Policy files:

/etc/kolla/keystone/policy.yaml

# Example: restrict user creation to domain-scoped admins
"identity:create_user": "rule:admin_required and domain_id:%(target.user.domain_id)s"

# Example: allow project members to list users in their project
"identity:list_users": "role:member and project_id:%(scope.project.id)s"

Policy evaluation order:

1. Check explicit rule match in policy.yaml
2. Fall back to code default (oslo.policy)
3. Deny if no rule matches

Verify policy changes:

# Test a specific policy rule
openstack --os-auth-url http://keystone:5000/v3 \
  --os-username testuser --os-password testpass \
  --os-project-name testproject --os-user-domain-name default \
  --os-project-domain-name default user list

Service Catalog Management

# List all registered services and endpoints
openstack service list
openstack endpoint list

# Register a new service
openstack service create --name nova --description "Compute" compute
openstack endpoint create --region RegionOne \
  compute public http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute internal http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute admin http://controller:8774/v2.1

Federation (SAML/OIDC)

Federation allows external identity providers to authenticate OpenStack users.

Key concepts:

• Identity Provider (IdP): External auth source (AD, Okta, Keycloak)
• Mapping: Rules that translate IdP assertions to Keystone users/groups/projects
• Protocol: SAML 2.0 or OpenID Connect
• Service Provider (SP): Keystone acting as relying party

Configuration reference:

keystone.conf [federation]

[saml]

[openid]

Credential Encryption

# Initialize credential encryption (first time)
docker exec keystone_api keystone-manage credential_setup \
  --keystone-user keystone --keystone-group keystone

# Rotate credential encryption keys
docker exec keystone_api keystone-manage credential_rotate \
  --keystone-user keystone --keystone-group keystone

# Migrate credentials to new encryption
docker exec keystone_api keystone-manage credential_migrate \
  --keystone-user keystone --keystone-group keystone

Operate

User and Project Management

# Create domain
openstack domain create --description "Engineering" engineering

# Create project in domain
openstack project create --domain engineering --description "Dev team" dev-team

# Create user in domain
openstack user create --domain engineering --password-prompt \
  --email dev@example.com devuser

# Assign role
openstack role add --project dev-team --user devuser --user-domain engineering member

# List role assignments
openstack role assignment list --project dev-team --names

Token Debugging

# Issue a token and inspect it
openstack token issue -f json

# Check token expiration
openstack token issue -c expires -f value
# Compare with system clock -- clock skew causes premature expiration

# Revoke a specific token
openstack token revoke <token-id>

Fernet Key Rotation Procedure

Schedule: Rotate at half the token expiration interval. Default token lifetime is 1 hour, so rotate every 30 minutes in production.

# 1. Check current key count
docker exec keystone_api ls /etc/kolla/keystone/fernet-keys/ | wc -l

# 2. Rotate keys
kolla-ansible -i inventory keystone_fernet_rotate

# 3. Verify rotation
docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/
# New highest-numbered key should be primary

# 4. Verify tokens still work
openstack token issue

Backup and Restore

# Backup Keystone database
docker exec mariadb mysqldump -u root -p keystone > keystone_backup.sql

# Backup Fernet keys
docker cp keystone_api:/etc/kolla/keystone/fernet-keys/ ./fernet-keys-backup/

# Restore database
docker exec -i mariadb mysql -u root -p keystone < keystone_backup.sql

# Restore Fernet keys
docker cp ./fernet-keys-backup/ keystone_api:/etc/kolla/keystone/fernet-keys/
docker restart keystone_api

Quota Management

# Keystone itself has no quotas, but controls project-level quotas for other services
# Set Nova quotas for a project
openstack quota set --instances 20 --cores 40 --ram 81920 dev-team

# Set Cinder quotas
openstack quota set --volumes 50 --gigabytes 1000 dev-team

Troubleshoot

1. "401 Unauthorized" on All Services

Symptoms: Every

openstack

Root causes and resolution:

docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/

kolla-ansible -i inventory keystone_fernet_rotate

date

chronyc sources

chronyc tracking

cat /etc/kolla/admin-openrc.sh

kolla-ansible -i inventory post-deploy

docker ps --filter name=keystone

docker restart keystone_api keystone_fernet

docker logs keystone_api 2>&1 | grep -i "database|maria|mysql"

docker exec mariadb mysql -u root -p -e "SELECT 1"

2. Service Catalog Endpoint Mismatch

Symptoms: Services fail to find each other. Errors like "Could not find service endpoint" or wrong URLs in API responses.

Diagnosis:

openstack endpoint list --long
# Check: public/internal/admin URLs match actual service locations
# Check: region names are consistent across all services

Fix: Correct the mismatched endpoint:

openstack endpoint set --url http://correct-host:port/v3 <endpoint-id>

3. Token Validation Failures

Symptoms: Tokens issue successfully but fail validation on other services. Intermittent auth failures.

Root causes:

kolla-ansible -i inventory keystone_fernet_rotate

docker logs keystone_api 2>&1 | grep memcache

docker exec memcached memcstat --servers=localhost

[keystone_authtoken]

auth_url

memcached_servers

project_name

4. Database Connection Failures

Symptoms: Keystone API returns 503. Container logs show database errors.

Diagnosis:

docker logs keystone_api 2>&1 | tail -50
# Look for: "OperationalError", "Can't connect to MySQL", "Access denied"

# Test MariaDB directly
docker exec mariadb mysql -u keystone -p -e "USE keystone; SELECT COUNT(*) FROM project;"

Fix:

• Connection refused:

  docker restart mariadb

  , verify MariaDB is listening on expected port
• Access denied: Check credentials in

  /etc/kolla/keystone/keystone.conf

  [database]

  section
• Database missing:

  kolla-ansible -i inventory deploy --tags mariadb,keystone

5. Federation/SSO Issues

Symptoms: External IdP users cannot authenticate. SAML assertions rejected.

Diagnosis:

docker logs keystone_api 2>&1 | grep -i "federation\|saml\|oidc\|mapping"
# Check: certificate trust chain, metadata freshness, mapping rules

Common fixes:

• Update IdP metadata: re-download and reimport
• Fix certificate chain: ensure CA cert is in Keystone trust store
• Debug mapping rules: use

  keystone-manage mapping_engine

  to test rule evaluation

6. RBAC Policy Denials

Symptoms: Users get 403 Forbidden despite having seemingly correct roles.

Diagnosis:

# Enable debug logging temporarily
docker exec keystone_api sed -i 's/^#debug = .*/debug = true/' /etc/keystone/keystone.conf
docker restart keystone_api

# Check policy evaluation
docker logs keystone_api 2>&1 | grep "Policy check"
# Look for the specific rule that denied the request

Common fixes:

• Policy syntax error: validate YAML with

  python3 -c "import yaml; yaml.safe_load(open('policy.yaml'))"

• Rule evaluation order: explicit rules override code defaults; check for conflicting rules
• Scope mismatch: request is project-scoped but rule requires domain-scope, or vice versa

Integration Points

Every OpenStack service authenticates through Keystone. The integration pattern is consistent:

keystonemiddleware configuration (in each service's config file):

[keystone_authtoken]
www_authenticate_uri = http://keystone:5000
auth_url = http://keystone:5000
memcached_servers = memcached:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova          # service-specific
password = nova_pass     # service-specific

Service user pattern: Each service has a dedicated user in the

service

admin

Endpoint discovery: Services find each other through the Keystone service catalog. When Nova needs to fetch an image from Glance, it queries the catalog for the

image

Token flow:

1. User authenticates to Keystone, receives scoped token
2. User sends request to Nova with token in

   X-Auth-Token

   header
3. Nova's keystonemiddleware validates token against Keystone (or memcached cache)
4. Nova uses its own service user token to call Glance, Neutron, Cinder as needed

NASA SE Cross-References

openstack token issue

openstack endpoint list

openstack user list