Grokking OpenStack

OpenStack - little pieces

Testing

When I first started developing with my Devstack installation I was told to use ./run_tests.sh to run the test suite and that was good enough for me. Then I began to see some other commands to run tests, tox -epy27, tox -epep8 and I ran the commands but didn’t fully understand why some commands are better than others.

Then I had to run some tests in /opt/stack/nova and every command I ran failed. Why? I didn’t know. But now I do.

Recently there has been a move in the repos to testr from nosetests. I got caught in this and have learned a bit about how to dig myself out of this hole.

First of all I am running Ubuntu 12.04 and when I first installed the OS I also installed: apt-get install git libxml2-dev libxslt1-dev and after installing devstack I installed tox with pip install tox. I also had python 2.7 on my machine and wanted to install 2.6 for testing which I did with apt-get install python-software-properties, add-apt-repository ppa:j5-dev/python2.6, apt-get update, apt-get install python2.6 python2.6-dev. I also learned I needed the mysql development libraries which I got with apt-get install mysql-client libmysqlclient-dev.

So I followed my current workflow upon cd’ing into a new directory: git status, git pull, rm -rf .venv, sudo python setup.py install. Then I tried both ./run_tests.sh which failed and tox which failed.

I found out that the “Error: pg_config executable not found.” in my output was due to missing the libpq-dev in Ubuntu (rhel is postgresql-devel).

I also learned that this error output “Please install a more recent version first, using ‘easy_install -U distribute’.” suggesting that running easy_install -U distribute might solve the problem didn’t for me. Distribute is bundled by virtualenv which is used by tox. The solution? Upgrade tox with pip install tox --upgrade.

Also when running tox if I got an error “invalid command ‘testr’” I learned that tox -r will create an updated test environment using testr and the tests passed for me. So if I run tox -r I don’t need to remove the virtual environment with rm -rf .venv since the -r flag to tox will rebuild venv from stratch.

So right now my workflow is to run tests with tox which runs tests against python 2.6 and 2.7 (since I have both installed) and pep8 or tox -epy26 for just testing against python 2.6, tox -epy27 for testing just against python 2.7 and tox -epep8 for just running the pep8 tests. If you are inclined, taking a peek into tox.ini will show you the configuration for the different environments. Some tox environments are not run by default but can be helpful for debugging and cleanup, an example would be pyflakes.

I hope this information is helpful to you.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Current Status

Yesterday marked the end of my first 4 weeks as an intern working on OpenStack. I have merged code for 1 feature and 2 bug fixes. I also wrote code that I abandoned which was necessary to close a bug. It may not sound like much but I am pretty happy with what I have accomplished.

I would like to thank my mentor, Iccha Sethi, for helping me so much by requiring my status updates, helping me set deadlines, encouraging me to reach out to others and challenging me to learn new skills. Thank you, Iccha. I also want to thank the other interns Laura Alves and Victoria Martínez de la Cruz for being so wonderful to work with and so supportive as I learn these new skills. Laura’s mentor Anne Gentle and Vicky’s mentor Julie Pichon are wonderfully helpful to all of us, thank you to them. We also have some mentors-at-large in the form of Mark McLoughlin, Russell Bryant and Brian Waldon, thank you to you. We all learn so much every time you explain the nuance of a situation, which we couldn’t get from a manual.

If you, as reader of this blog post, feel like joining in supporting our learning, you too are welcome to join us in #openstack-opw on the freenode server.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Logging & Debugging

I addressed my first foray into debugging, setting it up using the python console, in this post.

For some reason, when I turned to debugging one day I couldn’t get it to work and I couldn’t figure out why (I will explain the solution at the conclusion of this post). Fortunately I had a great guide in the form of russellb who likes to use logging and just happened to be willing to make a bit of time for me. What follows is what I learned about logging.

Logging

Logging is useful to see the status of something either as it is running or after it has run. If you are using Devstack (as I am) you already have access to continual logging of processes via the screen tool. When Devstack installs it sets up a window for each process, so executing a variation of screen -r stack gets you in to view the process logs as each process runs and updates the log. (Remember to exit screen with Ctrl+a d to detach the shell from screen while leaving screen running.) Now I am learning to parse these logs slowly but the point is they exist, which I needed pointed out to me so I am glad I know now.

Next is setting up logging to provide output for a local process I am working on understanding and addressing. The simplest example involves adding import logging to the top of the file, creating an instance of logging with LOG = logging.getLogger(__name__) outside of any of the classes and then an individual log statement inside the class and function you wish to explore LOG.debug('class.function was called").

Examples are gold to me, so lets work with this as a series of incremental examples.

First the bare minimum to get some logging working:

If this were the contents of a file entitled logging_example_01.py, you could expect the following output:

Let’s see what other kind of logging statements are available.

Here is the expected output:

As you can see, critical, warning and error logging messages were output and debug and info messages were skipped.

How do we get the debug and info messages output? We have to change the logging level.

By passing level=logging.INFO as an argument to basicConfig() we set the logging level to INFO.

Alas, our debug message still is not output. We change the logging level once again, this time to DEBUG.

Now we see all of our logging messages:

We are ready to put this knowledge to use in an OpenStack project file.

I was working with glanceclient Image.delete so let’s look at putting logging in that file. With the imported files add a line with import logging. Below the constants SORT_DIR_VALUES and SORT_KEY_VALUES add LOG = logging.getLogger(__name__). We don’t need the logging.basicConfig() line of code since this is taken care of for us elsewhere. This is the set up, finally the execution with the logging statement. Within the Image class inside the delete() function add LOG.debug('ImageManager.delete was called').

After I make these edits and save the file I have to reinstall the client with sudo python setup.py install otherwise my edits will not take effect. Then I can run glance --debug image-delete <image_id> and my log message will be output, amongst the rest of the debugging information.

This is what I understand about logging so far, and it is very helpful to have this as a tool.

Debugging

The only code I need for debugging is import pdb set at the top of the file with the other imports (alphabetically) and pdb.set_trace() wherever I want the trace to start. But here is the key, which I didn’t know until I was learning logging with russellb, after I edit the file to include the import and set_trace() I have to reinstall the client with sudo python setup.py install otherwise no debugging happens. Also after I remove the debugging code I must again reinstall the client to have my working code reflect the code in the files.

If you reached the end, I thank you. Here is a cat picture for you to enjoy.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Reboot the Shell

Why would I need to reboot the shell when working on code in any of the projects? Well, if I edit any of the shell scripts (shell.py) the changes won’t be picked up by my shell unless I reboot it somehow.

My first inclination was to find a command for OpenStack along the lines of source ~/.bashrc but there was no command I could find.

It wasn’t until I pulled from a git repo that had moved to testtools that I recognized that sudo python setup.py install, executed within the root directory of the project, rebooted the shell to update the version of shell.py that it was sourcing.

I don’t know if this is the only way to reboot the shell when working on changes to shell.py, but so far it seems to be an effective way.

Since I got caught this week moving back to the master branch with a clean git status but with stale responses to shell help, I now incorporate sudo python setup.py install as part of my workflow. I already executed git status and ./run_tests.sh everywhere I went and I just throw in sudo python setup.py install if ever I have less than full confidence that the shell responses are reflective of the shell.py code in the repo in which I am located.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

How I Am Visualizing Python-novaclient

Why am I visualizing novaclient this way? Well since I am just getting my bearings regarding the code the first thing I need to do is look for landmarks. It was really confusing seeing python-novaclient/novaclient in the path at first and seeing nova/nova in the nova project really threw me at first but I am starting to understand the purpose behind the file structure.

My big landmarks are the directories: python-novalclient, novaclient and openstack or v1. The openstack directory is at the same level as v1 (directly underneath novaclient) and the common directory is underneath the openstack directory, so why have I visualized it this way? Well for the sake of simplicity I like to visualize one path of flow and functionally v1 is imported by files within common and openstack so I consider openstack-common to be one entity and for it to be above the code flow of v1. The v1 stands for version 1 and in the projects with a second version of code to interact with the API, the directory is called v2. When novaclient has a v2 directory it will be at the same level in my mental model as v1, so a fork.

I cherry picked file names to place in this graphic. Why, because there are so many of them. At a certain point it becomes too much data to see the pattern and I mostly wanted to illustrate the mental pattern I am working with right now. I picked the files I have spent the most time in so far.

This graphic might seem simplistic or silly but it took me about 2 weeks to adopt this mental picture and another week to be able to communicate it. Not directly contributing to code but knowing where to look is a very valuable skill. I feel in some regards I am learning how to narrow down my search when looking for the location of a specific function.

Thank you for supporting this GNOME OPW intern,
Anita Kuno.

Stepping Through Authenticate()

Armed with the investigative abilities of the debugger, I now can do a little spelunking.

While playing with the debugger set-up in my previous post, I saw that it took me through authenticate() in a rather tidy fashion so I thought I would spend a little time here.

I won’t walk through the entire function, just the first part. We will be looking at the consequences of stepping through parts of the code below. This is from novaclient/client.py:

With the help of the debugger I will be looking at the values of the various arguments and variables both before and after they are defined, and in the case of key and index as their value changes.

So the debugger heads into authenticate(). Before the first line (#301) is executed, I check the values of self and self.authenticate().

Then I check the value of has_keyring: before line #302 runs.

Lines #303 - #305 take a list of values and assigns them to the variable keys. I check the value of keys, which I learn is undefined before the code runs (makes sense) and also the individual values of all the attributes in the list.

After lines #303 - #305 have run, I check the value of keys.

Next we move into lines #306 and #307 which run in a loop 4 times before breaking out to execute line #308. I check the value of key and index during the process. These variables begin as undefined and then their value changes as the code runs through the for loop.

Then back into lines #306 and #307 3 more times before breaking out to execute line #308.

Then we loop through lines #306 - #308 one last time.

We go back up to line #306 but we have enumerated over every element in the list, so then we execute lines #309 and #310, checking the values of the variables as we go.

We have come to the last line of code in this examination, line #325. We check the values before it is executed and after execution it takes us into the function urlsplit() in the python library urlparse.py.

I hope that stepping through the authenticate function with me has been useful for you. I learned a lot creating this post. For learning a code base, I feel that using a debugger can be a very powerful tool.

We didn’t go through the entire authenticate function, but the part that we did takes the various attributes of the HTTPClient object and creates a url to be parsed. I found the investigation rather fun.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Who Are You and What Are You Going to Do?

This is the post in which I discuss how I set up a debugger to work for me with OpenStack, at least in the development environment. I like to be able to ask the code what object it is using, what the type of object is and what are the attributes. I always feel better when I can dialog directly with the code, no one knows better what is happening, and I dislike guessing since I am usually wrong. Better to ask, I find.

The first thing I had to do was select a debugger. I went with pdb, the python debugger that comes with python, simply because it was the easiest. I knew I needed to get the debugger working with OpenStack and I felt that the integration step would be my trickiest part so I elected for the simplest version of a debugger I could find and it appears that my decision to go with pdb was the right decision for me.

Then I had to learn how to use the debugger in a vanilla set-up. Since I didn’t know how the python debugger worked before I began, I spent some time with a few tutorial blog posts and am glad that I did. First I went here to go through the debugging A B C’s. I found it to be an easy tutorial and a great introduction. This tutorial showed me how to drop into the debugger by calling both the debugger and a file name from the command line.

I knew I wanted to access the debugger from the console so this was a great next tutorial. It creates a class in a file and then calls an instance of the class from the console with pdb.run(). It also calls the debugger after an exception with pdb.pm(). I liked the look of pdb.pm() since it could be invoked after an error or exception but every time I tried it, regardless of the pdb command I used, the debugger exited rather than stepping through the code. Initially I thought it was due to a mistake I made, but after further investigation I now believe that this seems to be the intended action of pdb.pm() though I have no idea why.

So while the second tutorial was helpful to show me that using the debugger from the console/interpreter was possible, I didn’t have the exact syntax I needed for my situation. Next I worked with this tutorial, rather long but for my purposes it gave me what I needed. The first part was the most helpful for me, I found the code for the last exercise - the advanced portion - didn’t match the output the tutorial displayed. I tried to locate the author but was unsuccessful.

So I decided to go with pdb import at the top of the file I wanted to start to step through and pdb.set_trace() at the location in the code where I wanted to drop into the debugger. Look at line #7 and line #43 in the code below. Line #44 was the beginning of the code I edited, which I wanted to evaluate, so I placed pdb.set_trace() on the line prior.

Next I needed to create an instance of novaclient in the python console. So I opened a console in my OpenStack development environment, imported the class for the client and created an instance.
python
import novaclient.v1_1.client
client = novaclient.v1_1.client.Client('admin', '<admin_password>', 'admin', 'http://50.56.25.223:5000/v2.0') Your ip will be different than mine but :5000/v2.0 should work for you, that is until it changes. So check the date on this blog post.

Then I requested the client fetch me an image by passing in the id of an existing image, which I had gotten with nova image-list earlier.
client.images.get('9e44f0ee-083d-40a4-804c-534d66fb15ac')

Now without pdb.set_trace() in the images.py code, the output to client.images.get(image id) is something like this:
<Image: cirros-0.3.0-x86_64-uec>

But with pdb.set_trace() in the images.py file, I got:

… and I’m in the debugger. Which was my goal.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Status Updates

One of the things that I am doing that seems to be working very well for me, is constantly posting my current status: to my mentor, in my notes, for my reference, just to clear my thoughts.

I find that posting my status is my way of inviting comment without depending upon it. If my mentor has questions about what I am doing or how I am progressing, hopefully my status updates can help her keep tabs on me. If she has any thing to offer, a new direction, a suggestion, some support, she can post in response to an update. By continually posting my status without being asked, I allow her to check in as she has time, without having to wait for her to ask. It gives me the responsibility of scheduling my time productively and selecting tasks I can do on my own while incorporating feedback as it arrives.

It is serving a good secondary purpose. It is helping me to track when I am stuck and when I am not. If I am able to articulate my status, I probably am making productive use of my time. If I can’t answer the question “What are you doing?” with anything other than the response “foundering” it is probably time to seek some guidance.

The one caveat that I would add is the 3 day rule, in any new undertaking or situation I give myself 3 days to just be in the environment, with permission to not know anything and just listen and read and search and look and get my bearings. After 3 days, something just happens and some things which were completely unstructured just fall into place.

So I find that volunteering and posting my status, continually throughout the day for my mentor, contributes to a good work flow between us.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Setting Up Devstack

With any kind of development, the first step is creating your environment.

The Devstack project sets up an OpenStack development environment rather quickly (the downloads take a while) with one script. Devstack.

Devstack has been tested on both Ubuntu Server 12.04 and Fedora 17. I have tried it on both and am most familiar with Ubuntu Server so that is the environment I will address going forward. The only thing I have noticed with Fedora is the necessity to disable selinux after the OS is installed.

So install Ubuntu Server 12.04 as an openssh server. Once installation is finished and you have rebooted and signed in, I like to execute the following: sudo apt-get install git libxml2-dev libxslt1-dev eatmydata lynx irssi vim-gtk. The two libs are necessary to run some of the OpenStack module tests and aren’t installed with Devstack. Git is required for Devstack to install properly, eatmydata is useful when running the OpenStack tests. I like to have lynx and irssi around in case I need them. Vim-gtk has a few more features than the default Ubuntu vim so I grab that too.

After apt-get is finished, I generate some keys and then configure my git: git config -- global user.editor "Vim", and also user.name "My Name", and user.email "Must Match Email in Gerrit".

Then I get Devstack: git clone git://github.com/openstack-dev/devstack.git and then cd devstack; cp samples/localrc .; vi localrc and add the following lines to localrc:
FLOATING_RANGE=192.168.1.224/27
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth0
then close localrc and run ./stack.sh. Full Guide Here

Usually Devstack installs successfully but the last couple of installations have exited with non-specific failure messages although everything seems to be running.

After Devstack is finished, execute source ~/devstack/openrc.

I test my installation by executing nova image-list, if I get a list of images I am usually in good shape.

The other thing I do is execute screen -r stack and see if all 17 processes have windows. Less then 17 processes and I may have to stop devstack with ./unstack.sh and restart with ./stack.sh. Detach screen with Ctrl-a d since you want screen to continue running.

This is basically the format I follow to install Devstack on Ubuntu Server 12.04.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Taking Note

One of the biggest challenges that I am experiencing right now is understanding how to describe what I am experiencing. Writing down notes and developing my own workflow is feeling like a process that needs to evolve as my internship proceeds.

At the moment, I am working with IRC as my main form of communication and I have a private dialog window open with my mentor during my workdays. At the conclusion of every workday I copy the text of our interactions and save it to a Tomboy note. Tomboy notes are very flexible and I really like that I can search all of my notes from the main Tomboy page. Iccha, my mentor, often tells me commands or file paths in IRC that are helpful. Being able to search my notes for that command again or access a conversation or url from the day previous is really quite easy. Using a private dialog in IRC and and saving the text to Tomboy notes is working well for me.

I have a google document shared with my mentor and myself and I try to keep it updated as often as I can. I try for once or twice a day. It is a good place for summaries, ideas that carry over from day to day and urls for extra reading. It is nice because the viewers of the document can be administered and I can access the document easily from my browser.

I have been using a blank text file as a whiteboard in my local environment as a place to write pseudo code and collect my thoughts. Yesterday I expanded that idea to this gist. I like that the gist is private within github but I can also share the url with whomever I would like to access the information, readers of this blog for instance. I am still getting the hang of using comments on bug reports to communicate so I created this gist to explain my work and my thought processes to the creator of the bug and get his input going forward. Since it gives an up-to-date status of my understanding, after creating it, my mental energy was free to move onto other things, like blogging while waiting to communicate with the originator of the bug report. I like to have my mental work in tidy packages so that I can view a new and separate task with my full attention. Creating this gist as a whiteboard really served that purpose for me.

Taking notes will evolve as I go along and I am glad to learn new methods to make the best use of how I like to organize my mental space.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.