I was working on a cinder_manage module for Ansible to invoke the equivalent of “cinder-manage db sync” inside of Python code. This is used to initialize the OpenStack Block Storage database. I had already written one for glance-manage and keystone-manage, but each of the OpenStack projects uses a slightly different internal API for database initialization. This blog post documents my thought process trying to go through the Python code to figure out how it works.
To implement this, I need to invoke the equivalent of “cinder-manage db sync” inside of Python code. I could do this with the shell, but I prefer to dig into the guts of the cinder internals and do this in pure Python.
(Technically, I do call “cinder-manage db sync” from the shell, but I implement check mode using Python code).
cinder/db/migration.py has db_sync and db_version methods, but they don’t provide a way to pass in the path to the cinder.conf file that has the database connection info, so I can’t call them directly.
They both defer to methods with the same name in cinder/db/sqlalchemy/migration.py
Looking at cinder/db/sqlalchemy/migration.py:db_version, we see this:
repository = _find_migrate_repo() try: return versioning_api.db_version(get_engine(), repository)
The _find_migration_repo function is looking for the sqlalchemy migration scripts, that’s going to look relative to the current directory of the Python script, no need to mess with that. The connection string is going to be needed by that get_engine() method:from
cinder.db.sqlalchemy.session import get_engine
OK, let’s look at cinder/db/sqlalchemy/session.py:get_engine
def get_engine(): """Return a SQLAlchemy engine.""" global _ENGINE if _ENGINE is None: connection_dict = sqlalchemy.engine.url.make_url(FLAGS.sql_connection)
A-ha, it’s FLAGS.sql_connection. What’s flags?
FLAGS = flags.FLAGS
import cinder.flags as flags
There we go, it’s cinder.flags.FLAGS
cinder/flags.py: from cinder.openstack.common import cfg FLAGS = cfg.CONF
All right, so flags come from cinder/openstack/common/cfg.py
CONF = CommonConfigOpts()
Hmm, CommonConfigOpts doesn’t take any arguments. Let’s look back at cinder/flags.py
def parse_args(argv, default_config_files=None): FLAGS.disable_interspersed_args() return argv[:1] + FLAGS(argv[1:], project='cinder', default_config_files=default_config_files)
That’s interesting, it’s actually calling FLAGS and adding to it. That’s what we want. Except we don’t really want to call parse_args, because we don’t have an argv. I think we just want to call FLAGS with our arguments.
But, is default_config_files going to be set for us already? And what’s that first argument? Recall that Flags are of type CommonConfigOpts. Is that callable? Let’s take a look.
Its parent, ConfigOpts, is callable:
def __call__(self, args=None, project=None, prog=None, version=None, usage=None, default_config_files=None)
Let’s see if we can test things out. We want to do something like
CONF.(args=, project=’cinder’, default_config_files=[‘/etc/cinder/cinder.conf’])
One way to test this is to check if the value changes from a default.
>>> from cinder.flags import FLAGS >>> FLAGS.verbose False >>> FLAGS(args=, project='cinder', default_config_files=['/etc/cinder/cinder.conf'])  >>> FLAGS.verbose True
Here’s another test
>>> from cinder.flags import FLAGS >>> FLAGS.sql_connection 'sqlite:////usr/lib/python2.7/dist-packages/cinder.sqlite' >>> FLAGS(args=, project='cinder', default_config_files=['/etc/cinder/cinder.conf'])  >>> FLAGS.sql_connection 'sqlite:////var/lib/cinder/cinder.sqlite'
OK, so we should be able to write a method in cinder_manage to load the config file
def load_config_file(conf): flags.FLAGS(args=, project='cinder', default_config_files=['/etc/cinder/cinder.conf'])
Now we need to figure out the current version and the repo version. Current version is easy:
from cinder.db import migration current_version = migration.db_version()
How about the repo version? Let’s look back at how it was done in cinder code.
The db_sync method in cinder/db/sqlalchemy/migration.py isn’t too helpful here:
def db_sync(version=None): if version is not None: try: version = int(version) except ValueError: raise exception.Error(_("version should be an integer")) current_version = db_version() repository = _find_migrate_repo() if version is None or version > current_version: return versioning_api.upgrade(get_engine(), repository, version) else: return versioning_api.downgrade(get_engine(), repository, version)
It tells us how to find the sqlalchemy repository:
repository = _find_migrate_repo()
But it doesn’t actually retrieve the repo version.
In keystone_manage, we did this:
repo_path = migration._find_migrate_repo() repo_version = versioning_api.repository.Repository(repo_path).latest
Will that still work? Let’s check on the command-line. We’ll need to do this:
import cinder.db.sqlalchemy.migration repo_path = cinder.db.sqlalchemy.migration._find_migrate_repo()
It turns out that this returns a repository, not a path
In : cinder.db.sqlalchemy.migration._find_migrate_repo() Out: <migrate.versioning.repository.Repository at 0x37fd250>
We need to change code a little, we can just do this:
from cinder.db import migration repository = migration._find_migrate_repo() repo_version = repository.latest
Of course, this uses an internal API, which means its likely to change in the next release, but we can just update the ansible module when that happens.
Do you mind if I quote a couple of your posts as long as
I provide credit and sources back to your website? My website is in
the very same niche as yours and my visitors would genuinely benefit from a lot
of the information you provide here. Please let me
know if this ok with you. Cheers!
Sure, that’s fine.