A Private Cloud Inside One Google Cloud VM

The assignment was to run OpenStack and administer one tenant end to end: its project and user, quotas, network and instances, and an application on them. I ran DevStack on a single Google Cloud VM.

That puts two hypervisors in one stack. The VM is the first layer; the instances Nova starts are virtual machines inside it. I created it with nested virtualisation enabled.

DevStack installed OpenStack on that one host, its services running as 24 systemd units. Inside it I built a single tenant:

Keystone

a project, a user with the member role and an openrc file for that user

Glance

an Ubuntu 22.04 Minimal image for both instances

Nova

custom flavors and project quotas

Neutron

a private network and subnet, a router to the public network, a security group and floating IPs

Keeping the tenant apart from admin caused the most trouble. Of the 18 problems I logged, 8 came from creating or looking up a resource with the admin credentials: a security group “not found”, an “invalid” key pair, an instance with no private address. A resource belongs to the project that created it, so the credentials in use decide what exists.

An instance’s traffic crosses two networks. Inside OpenStack, the Neutron router gives it an address from the floating range and sends it out through br-ex, the host’s Open vSwitch bridge for the external network. Outside, Google Cloud knows one address for the whole machine: the VM’s internal address on ens4, its only network interface.

Symptom

In the first instance, ping got no reply and apt update hung at “Connecting to archive.ubuntu.com”. No error, anywhere.

Trace

tcpdump on each interface along the path: the packets showed up on the instance’s tap interface and on br-ex, never on ens4.

Cause

The host was not forwarding between br-ex and ens4, and anything it did forward would leave with a floating-range source address, which Google Cloud drops without a reply.

Fix

IP forwarding on; FORWARD rules out from br-ex and back for established connections only; MASQUERADE on ens4; all saved with iptables-persistent.

That held until the VM was stopped and started again. The iptables rules came back, but br-ex came up without its address and route, and the floating IPs were unreachable again. I kept a checklist of commands to rerun, then replaced it with brex-setup.service, a oneshot systemd unit ordered after the network and DevStack that restores the bridge at boot.

The lessons I kept: the fault lies one step past the last interface where the packet is still visible, and a fix that a restart undoes is not finished until it is part of the boot.

To test the tenant end to end, I deployed a small two-tier application. The lab exercises put PostgreSQL and Flask on one VM; I split them across two instances, so that the API had to reach its database over OpenStack’s private network.

vm-db runs PostgreSQL 14, set to accept remote connections. vm-app runs a Flask API under Gunicorn as a systemd service, behind NGINX on port 80, and reaches vm-db by its private address. The final check went through an SSH tunnel via the host: /api/products returned four products as JSON, read from the other instance.

It worked, and it was built for a lab. Both instances share one security group, so PostgreSQL’s port is open on the application VM and port 80 on the database; I would give each tier its own group, with 5432 open only to vm-app. The database password sits in the application code, and PostgreSQL listens on every interface. And the host’s networking belongs in how the VM is provisioned, not in fixes added along the way.