Deploying OpenStack with Ceph Tentacle: A 3-Node Converged Setup Guide

In this guide, I’ll walk you through deploying a fully functional OpenStack cloud platform integrated with Ceph Tentacle as the distributed storage backend — all on a 3-node converged cluster running Ubuntu Noble 24.04.

This setup uses Kolla-Ansible for deploying OpenStack services as Docker containers, and cephadm for bootstrapping the Ceph cluster. By the end of this guide, you’ll have a working private cloud with:

☁️ OpenStack — Keystone, Glance, Nova, Neutron, Cinder, Horizon
💾 Ceph — Distributed storage for images, volumes, VMs, and backups
🌐 Networking — Provider + tenant networks with floating IPs
🖥️ VM Instances — Running CirrOS test instances with SSH access

Table of Contents#

Topology & Specifications
Node Preparation
Deploy Ceph Tentacle with cephadm
Ceph Pool & Keyring Setup for OpenStack
Install Kolla-Ansible
Kolla-Ansible Configuration
External Ceph Integration with Kolla-Ansible
Deploy OpenStack
OpenStack Network Setup
Create VM Instance
Verify VM Connectivity

1. Topology & Specifications#

Network Topology#

1
                    ┌──────────────────┐
2
                    │   Proxmox Host   │
3
                    │  vmbr0 (eno1)    │
4
                    │ 192.168.205.253  │
5
                    └────────┬─────────┘
6
           ┌─────────────────┼─────────────────┐
7
           │                 │                 │
8
    ens18 (Mgmt)       ens18 (Mgmt)       ens18 (Mgmt)
9
  192.168.205.101    192.168.205.102    192.168.205.103
10
    ens19 (Provider)   ens19 (Provider)   ens19 (Provider)
11
    (no IP)            (no IP)            (no IP)
12
           │                 │                 │
13
  ┌─────────────────┐ ┌──────────────┐ ┌──────────────┐
14
  │  ops-ceph-ctrl  │ │ops-ceph-comp1│ │ops-ceph-comp2│
15
  │  (controller)   │ │  (compute)   │ │  (compute)   │
16
  │                 │ │              │ │              │
17
  │ /dev/sda (OS)   │ │ /dev/sda(OS) │ │ /dev/sda(OS) │
18
  │ /dev/sdb OSD.0  │ │ /dev/sdb OSD │ │ /dev/sdb OSD │
19
  │ /dev/sdc OSD.1  │ │ /dev/sdc OSD │ │ /dev/sdc OSD │
20
  │ /dev/sdd OSD.2  │ │ /dev/sdd OSD │ │ /dev/sdd OSD │
21
  └─────────────────┘ └──────────────┘ └──────────────┘

Node Specifications#

Parameter	ops-ceph-ctrl	ops-ceph-comp1	ops-ceph-comp2
CPU	8 vCPU	6 vCPU	6 vCPU
RAM	16 GB	12 GB	12 GB
Disk OS	50 GB `/dev/sda`	50 GB `/dev/sda`	50 GB `/dev/sda`
Disk OSD	3x 25 GB (`/dev/sdb`,`sdc`,`sdd`)	3x 25 GB	3x 25 GB
NIC 1	`ens18` → 192.168.205.101/24	`ens18` → 192.168.205.102/24	`ens18` → 192.168.205.103/24
NIC 2	`ens19` → no IP	`ens19` → no IP	`ens19` → no IP
OS	Ubuntu Noble 24.04	Ubuntu Noble 24.04	Ubuntu Noble 24.04
Ceph Role	MON, MGR, OSD	OSD	OSD
OpenStack Role	Controller, Network	Compute	Compute

2. Node Preparation#

Run the following steps on all nodes unless stated otherwise.

2.1 Set Hostname#

1
# On ops-ceph-ctrl
2
hostnamectl set-hostname ops-ceph-ctrl
3

4
# On ops-ceph-comp1
5
hostnamectl set-hostname ops-ceph-comp1
6

7
# On ops-ceph-comp2
8
hostnamectl set-hostname ops-ceph-comp2

2.2 Configure /etc/hosts#

1
sudo nano /etc/hosts

Add on all nodes:

1
192.168.205.101  ops-ceph-ctrl
2
192.168.205.102  ops-ceph-comp1
3
192.168.205.103  ops-ceph-comp2

2.3 Configure Netplan#

1
sudo nano /etc/netplan/50-cloud-init.yaml

Example for ops-ceph-ctrl:

1
network:
2
  version: 2
3
  renderer: networkd
4
  ethernets:
5
    ens18:
6
      addresses:
7
        - 192.168.205.101/24
8
      routes:
9
        - to: default
10
          via: 192.168.205.1
11
      nameservers:
12
        addresses: [8.8.8.8, 1.1.1.1]
13
    ens19:
14
      dhcp4: false

Adjust IP for comp1 (192.168.205.102) and comp2 (192.168.205.103). ens19 only needs dhcp4: false without an IP — Neutron will use it as the provider interface.

1
sudo netplan apply

2.4 Install Dependencies#

1
sudo apt update
2
sudo apt install -y git curl wget python3-dev python3-venv \
3
  libffi-dev gcc libssl-dev libdbus-glib-1-dev chrony sshpass nano

2.5 Configure Passwordless SSH (from ops-ceph-ctrl)#

1
# Generate SSH key (ed25519)
2
ssh-keygen -t ed25519 -C "ops-ceph-ctrl"
3

4
# Distribute public key to all nodes including self
5
ssh-copy-id opsce@ops-ceph-ctrl
6
ssh-copy-id opsce@ops-ceph-comp1
7
ssh-copy-id opsce@ops-ceph-comp2

2.6 Set NOPASSWD sudo (all nodes)#

1
echo "opsce ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/opsce

2.7 Disable Swap & Time Sync#

1
sudo swapoff -a
2
sudo sed -i '/swap/d' /etc/fstab
3

4
sudo timedatectl set-timezone Asia/Jakarta
5
sudo systemctl enable --now chrony
6
chronyc tracking

2.8 Sysctl Configuration#

1
sudo nano /etc/sysctl.conf

Add:

1
net.ipv4.ip_forward = 1
2
net.bridge.bridge-nf-call-iptables = 1
3
net.bridge.bridge-nf-call-ip6tables = 1

1
sudo sysctl -p

2.9 Install Docker#

1
curl -sL https://get.docker.com | bash
2
sudo systemctl enable --now docker

3. Deploy Ceph Tentacle with cephadm#

All steps are executed from ops-ceph-ctrl unless stated otherwise.

3.1 Install cephadm#

1
curl --silent --remote-name --location \
2
  https://download.ceph.com/rpm-tentacle/el9/noarch/cephadm
3

4
chmod +x cephadm
5
sudo install -m 0755 cephadm /usr/local/sbin/cephadm
6

7
sudo cephadm add-repo --release tentacle
8
sudo apt update
9
sudo apt install -y cephadm

3.2 Bootstrap Ceph Cluster#

1
sudo cephadm bootstrap \
2
  --mon-ip <your-controller-ip> \
3
  --cluster-network <your-subnet>/24 \
4
  --initial-dashboard-user <dashboard-user> \
5
  --initial-dashboard-password <dashboard-password> \
6
  --allow-overwrite

Verify:

1
sudo cephadm shell -- ceph -s

3.3 Install ceph-common#

1
# On ctrl
2
sudo cephadm install ceph-common
3

4
# On comp1 and comp2
5
ssh -t ops-ceph-comp1 "sudo cephadm add-repo --release tentacle && sudo apt update && sudo apt install -y ceph-common"
6
ssh -t ops-ceph-comp2 "sudo cephadm add-repo --release tentacle && sudo apt update && sudo apt install -y ceph-common"

3.4 Copy cephadm SSH Key to All Nodes#

1
sudo cephadm shell -- ceph cephadm get-pub-key | \
2
  sudo tee /etc/ceph/ceph.pub
3

4
# Must run as root
5
sudo su
6
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ops-ceph-comp1
7
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ops-ceph-comp2
8
exit

3.5 Add Nodes to Cluster#

1
sudo cephadm shell -- ceph orch host add ops-ceph-comp1 192.168.205.102
2
sudo cephadm shell -- ceph orch host add ops-ceph-comp2 192.168.205.103
3

4
# Verify
5
sudo cephadm shell -- ceph orch host ls

3.6 Deploy OSD#

1
# ops-ceph-ctrl
2
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdb
3
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdc
4
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdd
5

6
# ops-ceph-comp1
7
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdb
8
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdc
9
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdd
10

11
# ops-ceph-comp2
12
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdb
13
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdc
14
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdd

Tip: Use hostname:/dev/sdX format — not user@hostname:/dev/sdX. The latter causes a “host not found” error. Also, warnings like Not using image... not in the list of non-dangling images are cosmetic and safe to ignore.

3.7 Verify Cluster#

1
sudo cephadm shell -- ceph -s
2
sudo cephadm shell -- ceph osd tree
3
sudo cephadm shell -- ceph df

Make sure all 9 OSDs (3 nodes × 3 disks) are up and in.

Here’s what the Ceph Block Images dashboard looks like after a successful deployment:

And the Grafana monitoring dashboard showing cluster health, OSD performance, and resource utilization:

4. Ceph Pool & Keyring Setup for OpenStack#

4.1 Create Pools#

1
sudo cephadm shell -- bash -c "
2
  ceph osd pool create volumes
3
  ceph osd pool create images
4
  ceph osd pool create backups
5
  ceph osd pool create vms
6

7
  rbd pool init volumes
8
  rbd pool init images
9
  rbd pool init backups
10
  rbd pool init vms
11
"

4.2 Create Keyrings#

1
sudo cephadm shell -- bash -c "
2
  ceph auth get-or-create client.glance \
3
    mon 'allow r' \
4
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' \
5
    -o /etc/ceph/ceph.client.glance.keyring
6

7
  ceph auth get-or-create client.cinder \
8
    mon 'allow r' \
9
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=images' \
10
    -o /etc/ceph/ceph.client.cinder.keyring
11

12
  ceph auth get-or-create client.nova \
13
    mon 'allow r' \
14
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' \
15
    -o /etc/ceph/ceph.client.nova.keyring
16

17
  ceph auth get-or-create client.cinder-backup \
18
    mon 'allow r' \
19
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups' \
20
    -o /etc/ceph/ceph.client.cinder-backup.keyring
21
"

4.3 Export Keyrings to Host#

1
sudo cephadm shell -- ceph auth get client.glance | sudo tee /etc/ceph/ceph.client.glance.keyring
2
sudo cephadm shell -- ceph auth get client.cinder | sudo tee /etc/ceph/ceph.client.cinder.keyring
3
sudo cephadm shell -- ceph auth get client.nova | sudo tee /etc/ceph/ceph.client.nova.keyring
4
sudo cephadm shell -- ceph auth get client.cinder-backup | sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
5

6
sudo chmod 640 /etc/ceph/ceph.client.*.keyring

4.4 Generate ceph.conf#

1
sudo cephadm shell -- ceph config generate-minimal-conf | sudo tee /etc/ceph/ceph.conf
2

3
# Remove leading tabs to prevent kolla-ansible issues
4
sudo sed -i 's/^\t//' /etc/ceph/ceph.conf
5

6
cat /etc/ceph/ceph.conf

5. Install Kolla-Ansible#

Run from ops-ceph-ctrl as user opsce.

5.1 Create Virtual Environment#

1
python3 -m venv ~/kolla-venv
2
source ~/kolla-venv/bin/activate
3

4
# Auto-activate on login
5
echo 'source ~/kolla-venv/bin/activate' >> ~/.bashrc

5.2 Install Kolla-Ansible#

1
pip install -U pip
2
pip install 'ansible-core>=2.16,<2.17'
3
pip install docker dbus-python
4

5
# Install kolla-ansible from master
6
pip install git+https://opendev.org/openstack/kolla-ansible@master

5.3 Setup Kolla Directories#

1
sudo mkdir -p /etc/kolla
2
sudo chown $USER:$USER /etc/kolla
3

4
cp -r ~/kolla-venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/
5
cp ~/kolla-venv/share/kolla-ansible/ansible/inventory/multinode ~/multinode

5.4 Install Ansible Galaxy Dependencies#

1
kolla-ansible install-deps

6. Kolla-Ansible Configuration#

6.1 Ansible Config#

1
sudo mkdir -p /etc/ansible
2
sudo tee /etc/ansible/ansible.cfg <<EOF
3
[defaults]
4
host_key_checking = False
5
pipelining = True
6
forks = 100
7
interpreter_python = /usr/bin/python3
8
EOF

6.2 Multinode Inventory#

1
nano ~/multinode

Update the top section:

1
[control]
2
ops-ceph-ctrl
3

4
[network]
5
ops-ceph-ctrl
6

7
[compute]
8
ops-ceph-comp1
9
ops-ceph-comp2
10

11
[monitoring]
12
ops-ceph-ctrl
13

14
[storage]
15
ops-ceph-ctrl
16
ops-ceph-comp1
17
ops-ceph-comp2
18

19
[deployment]
20
localhost       ansible_connection=local

Leave all [xxx:children] groups as default from the template.

Verify:

1
ansible -i ~/multinode all -m ping

6.3 Generate Passwords#

1
kolla-genpwd
2

3
# Set admin password (replace with your own secure password)
4
sed -i 's/^keystone_admin_password:.*/keystone_admin_password: <your-password>/' /etc/kolla/passwords.yml

6.4 Configure globals.yml#

1
nano /etc/kolla/globals.yml

1
# =====================
2
# BASE IMAGE
3
# =====================
4
kolla_base_distro: "ubuntu"
5

6
# =====================
7
# NETWORKING
8
# =====================
9
network_interface: "ens18"
10
neutron_external_interface: "ens19"
11
kolla_internal_vip_address: "192.168.205.100"
12

13
# =====================
14
# ENABLE SERVICES
15
# =====================
16
enable_openstack_core: "yes"
17
enable_cinder: "yes"
18
enable_cinder_backup: "yes"
19

20
# =====================
21
# CEPH INTEGRATION
22
# =====================
23
glance_backend_ceph: "yes"
24
cinder_backend_ceph: "yes"
25
nova_backend_ceph: "yes"
26

27
ceph_glance_user: "glance"
28
ceph_glance_pool_name: "images"
29
ceph_cinder_user: "cinder"
30
ceph_cinder_pool_name: "volumes"
31
ceph_cinder_backup_user: "cinder-backup"
32
ceph_cinder_backup_pool_name: "backups"
33
ceph_nova_user: "nova"
34
ceph_nova_pool_name: "vms"

kolla_internal_vip_address must be an unused IP in the ens18 subnet. This IP will automatically appear on the ens18 interface of ops-ceph-ctrl after deployment as the HAProxy VIP.

7. External Ceph Integration with Kolla-Ansible#

7.1 Create Directories#

1
mkdir -p /etc/kolla/config/glance
2
mkdir -p /etc/kolla/config/nova
3
mkdir -p /etc/kolla/config/cinder/cinder-volume
4
mkdir -p /etc/kolla/config/cinder/cinder-backup

7.2 Copy ceph.conf#

1
cp /etc/ceph/ceph.conf /etc/kolla/config/glance/ceph.conf
2
cp /etc/ceph/ceph.conf /etc/kolla/config/nova/ceph.conf
3
cp /etc/ceph/ceph.conf /etc/kolla/config/cinder/ceph.conf

7.3 Copy Keyrings#

1
# Glance
2
cp /etc/ceph/ceph.client.glance.keyring \
3
   /etc/kolla/config/glance/ceph.client.glance.keyring
4

5
# Nova (needs both nova AND cinder keyrings)
6
sudo cp /etc/ceph/ceph.client.nova.keyring \
7
   /etc/kolla/config/nova/ceph.client.nova.keyring
8
sudo cp /etc/ceph/ceph.client.cinder.keyring \
9
   /etc/kolla/config/nova/ceph.client.cinder.keyring
10

11
# Cinder Volume
12
sudo cp /etc/ceph/ceph.client.cinder.keyring \
13
   /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
14

15
# Cinder Backup
16
sudo cp /etc/ceph/ceph.client.cinder.keyring \
17
   /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
18
sudo cp /etc/ceph/ceph.client.cinder-backup.keyring \
19
   /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring

7.4 Fix Ownership & Permissions#

1
sudo chown -R opsce:opsce /etc/kolla/config
2
sudo chmod 640 /etc/kolla/config/cinder/cinder-backup/*.keyring
3
sudo chmod 640 /etc/kolla/config/cinder/cinder-volume/*.keyring
4
sudo chmod 640 /etc/kolla/config/nova/*.keyring
5
sudo chmod 640 /etc/kolla/config/glance/*.keyring

7.5 Verify File Structure#

1
find /etc/kolla/config -type f -printf "%u %p\n" | sort

Expected output (all owned by opsce):

1
opsce /etc/kolla/config/cinder/ceph.conf
2
opsce /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring
3
opsce /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
4
opsce /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
5
opsce /etc/kolla/config/glance/ceph.conf
6
opsce /etc/kolla/config/glance/ceph.client.glance.keyring
7
opsce /etc/kolla/config/nova/ceph.conf
8
opsce /etc/kolla/config/nova/ceph.client.cinder.keyring
9
opsce /etc/kolla/config/nova/ceph.client.nova.keyring

8. Deploy OpenStack#

8.1 Bootstrap Servers#

1
kolla-ansible bootstrap-servers -i ~/multinode

Heads up: It’s common for Ceph MON/MGR services to go inactive dead after this step, since kolla-ansible reconfigures Docker daemon settings. If this happens, a simple sudo reboot on affected nodes is the cleanest fix. After reboot, verify Ceph health with sudo ceph -s before proceeding.

8.2 Pre-checks#

1
kolla-ansible prechecks -i ~/multinode

8.3 Deploy#

1
kolla-ansible deploy -i ~/multinode

Expect this to take 20-60 minutes depending on your hardware and network speed. Occasional single-task failures on compute nodes are typically timing-related and generally resolve themselves — verify with openstack compute service list after deployment.

8.4 Post-Deploy#

1
kolla-ansible post-deploy -i ~/multinode

8.5 Install OpenStack Client & Load Credentials#

1
pip install python-openstackclient \
2
  -c https://releases.openstack.org/constraints/upper/2025.2
3

4
source /etc/kolla/admin-openrc.sh

8.6 Verify Services#

1
openstack service list
2
openstack compute service list
3
openstack network agent list
4
openstack volume service list

All services should be up. Note that cinder-volume backend shows rbd-1, confirming successful Ceph integration.

Here’s the actual output from a working deployment — all services, network agents, compute services, and volume services are confirmed up:

9. OpenStack Network Setup#

9.1 Create Provider Network (External)#

1
openstack network create \
2
  --share \
3
  --external \
4
  --provider-physical-network physnet1 \
5
  --provider-network-type flat \
6
  public-net
7

8
openstack subnet create \
9
  --network public-net \
10
  --subnet-range 17.19.21.0/24 \
11
  --gateway 17.19.21.254 \
12
  --allocation-pool start=17.19.21.100,end=17.19.21.200 \
13
  --dns-nameserver 8.8.8.8 \
14
  public-subnet

9.2 Create Private/Tenant Network#

1
openstack network create private-net
2

3
openstack subnet create \
4
  --network private-net \
5
  --subnet-range 10.0.0.0/24 \
6
  --gateway 10.0.0.1 \
7
  --dns-nameserver 8.8.8.8 \
8
  private-subnet

9.3 Create Router#

1
openstack router create main-router
2
openstack router set main-router --external-gateway public-net
3
openstack router add subnet main-router private-subnet
4

5
# Verify
6
openstack router show main-router
7
openstack port list --router main-router

Once the router is created, you can visualize the network topology in Horizon:

9.4 Create Security Group#

1
openstack security group create security-group-net \
2
  --description 'Allow SSH and ICMP'
3

4
openstack security group rule create \
5
  --protocol icmp security-group-net
6

7
openstack security group rule create \
8
  --protocol tcp --ingress --dst-port 22 security-group-net
9

10
openstack security group rule list security-group-net

9.5 Create Floating IP#

1
openstack floating ip create public-net
2
openstack floating ip list

Note: Floating IPs are only externally reachable if the provider network interface is bridged to your physical network. In isolated lab environments, you can still access VMs via their private IPs using the Neutron router namespace (see Section 11).

10. Create VM Instance#

10.1 Upload Image#

1
wget http://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img
2

3
openstack image create \
4
  --disk-format qcow2 \
5
  --container-format bare \
6
  --public \
7
  --file ./cirros-0.6.2-x86_64-disk.img \
8
  cirros-0.6.2
9

10
openstack image list

10.2 Create Keypair#

1
openstack keypair create \
2
  --public-key ~/.ssh/id_ed25519.pub \
3
  controller-key
4

5
openstack keypair list

10.3 Create Flavor#

1
openstack flavor create \
2
  --ram 512 \
3
  --disk 8 \
4
  --vcpus 1 \
5
  --public \
6
  c1-small
7

8
openstack flavor list

10.4 Create Instance#

1
# First instance
2
openstack server create \
3
  --flavor c1-small \
4
  --image cirros-0.6.2 \
5
  --key-name controller-key \
6
  --security-group security-group-net \
7
  --network private-net \
8
  cirros-instance-test
9

10
# Second instance (for inter-VM connectivity test)
11
openstack server create \
12
  --flavor c1-small \
13
  --image cirros-0.6.2 \
14
  --key-name controller-key \
15
  --security-group security-group-net \
16
  --network private-net \
17
  cirros-instance-test-2
18

19
# Check status
20
openstack server list

Wait until status is ACTIVE.

10.5 Assign Floating IP#

1
# List available floating IPs
2
openstack floating ip list
3

4
# Assign to each instance
5
openstack server add floating ip cirros-instance-test <floating-ip-1>
6
openstack server add floating ip cirros-instance-test-2 <floating-ip-2>
7

8
# Verify — instances should have 2 IPs (private + floating)
9
openstack server list

Here’s what it looks like in Horizon — both instances are Active and Running with their private and floating IPs assigned:

And the detailed server list from the CLI, confirming Ceph-backed storage and network connectivity:

11. Verify VM Connectivity#

11.1 Check Router Namespace#

1
# List all namespaces
2
sudo ip netns list
3
# Output: qrouter-xxx (id: N), qdhcp-xxx (id: N)

11.2 Ping VM from Namespace#

1
# Replace <router-id> with the ID from ip netns list output
2
sudo ip netns exec qrouter-<router-id> ping -c 4 10.0.0.71
3
sudo ip netns exec qrouter-<router-id> ping -c 4 10.0.0.104

11.3 SSH to VM from Namespace#

1
# SSH using key (recommended)
2
sudo ip netns exec qrouter-<router-id> ssh -i ~/.ssh/id_ed25519 cirros@10.0.0.71
3

4
# Or SSH with password (cirros password: gocubsgo)
5
sudo ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71

11.4 Test Inter-VM Connectivity#

From inside cirros-instance-test, ping the other VM:

1
# Enter the first VM
2
sudo ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71
3

4
# From inside the VM, ping the second VM
5
ping 10.0.0.104

Note: You cannot ping VMs directly from the controller host — there’s no route to the tenant or provider networks from the host’s default namespace. Always use sudo ip netns exec qrouter-<id> to access VM networks from the controller.

11.5 Verify OVS Bridge#

1
sudo docker exec openvswitch_vswitchd ovs-vsctl show

Verify:

br-ex has port ens19
br-int has qr-xxx (router port) and tap-xxx (VM port)
All bridge controllers show is_connected: true

Troubleshooting#

MON/MGR dies after bootstrap-servers#

kolla-ansible reconfigures Docker settings which can prevent Ceph containers from restarting. A sudo reboot is the cleanest fix. Alternatively, manually start the services:

1
sudo systemctl start ceph-<fsid>@mon.<hostname>.service
2
sudo systemctl start ceph-<fsid>@mgr.<hostname>.<id>.service

Error “host not found” when adding OSD#

Use hostname:/dev/sdX format — not user@hostname:/dev/sdX:

1
# CORRECT
2
sudo cephadm shell -- ceph orch daemon add osd <hostname>:/dev/sdb
3
# WRONG
4
sudo cephadm shell -- ceph orch daemon add osd user@<hostname>:/dev/sdb

Warning “Not using image… non-dangling images”#

Cosmetic warning, not an error. Safe to ignore.

kolla-ansible: command not found#

Ensure the Python virtual environment is activated:

1
source ~/kolla-venv/bin/activate
2
which kolla-ansible

Missing sudo password during bootstrap-servers#

Ensure passwordless sudo is configured for your deploy user:

1
echo "<user> ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/<user>

ceph.conf with leading tabs#

Kolla-ansible may fail to parse ceph.conf if it contains tabs. Strip them:

1
sudo sed -i 's/^\t//' /etc/kolla/config/*/ceph.conf

Cannot ping VM directly from controller#

This is expected — the host has no route to tenant/provider networks. Use the Neutron router namespace:

1
sudo ip netns list
2
sudo ip netns exec qrouter-<id> ping -c 4 <vm-ip>

References#

This guide is based on a real-world deployment. Adjust IPs, hostnames, credentials, and disk paths to match your own environment.