I addressed my first foray into debugging, setting it up using the python console, in this post.

For some reason, when I turned to debugging one day I couldn’t get it to work and I couldn’t figure out why (I will explain the solution at the conclusion of this post). Fortunately I had a great guide in the form of russellb who likes to use logging and just happened to be willing to make a bit of time for me. What follows is what I learned about logging.

Logging

Logging is useful to see the status of something either as it is running or after it has run. If you are using Devstack (as I am) you already have access to continual logging of processes via the screen tool. When Devstack installs it sets up a window for each process, so executing a variation of screen -r stack gets you in to view the process logs as each process runs and updates the log. (Remember to exit screen with Ctrl+a d to detach the shell from screen while leaving screen running.) Now I am learning to parse these logs slowly but the point is they exist, which I needed pointed out to me so I am glad I know now.

Next is setting up logging to provide output for a local process I am working on understanding and addressing. The simplest example involves adding import logging to the top of the file, creating an instance of logging with LOG = logging.getLogger(__name__) outside of any of the classes and then an individual log statement inside the class and function you wish to explore LOG.debug('class.function was called").

Examples are gold to me, so lets work with this as a series of incremental examples.

First the bare minimum to get some logging working:

If this were the contents of a file entitled logging_example_01.py, you could expect the following output:

Let’s see what other kind of logging statements are available.

Here is the expected output:

As you can see, critical, warning and error logging messages were output and debug and info messages were skipped.

How do we get the debug and info messages output? We have to change the logging level.

By passing level=logging.INFO as an argument to basicConfig() we set the logging level to INFO.

Alas, our debug message still is not output. We change the logging level once again, this time to DEBUG.

Now we see all of our logging messages:

We are ready to put this knowledge to use in an OpenStack project file.

I was working with glanceclient Image.delete so let’s look at putting logging in that file. With the imported files add a line with import logging. Below the constants SORT_DIR_VALUES and SORT_KEY_VALUES add LOG = logging.getLogger(__name__). We don’t need the logging.basicConfig() line of code since this is taken care of for us elsewhere. This is the set up, finally the execution with the logging statement. Within the Image class inside the delete() function add LOG.debug('ImageManager.delete was called').

After I make these edits and save the file I have to reinstall the client with sudo python setup.py install otherwise my edits will not take effect. Then I can run glance --debug image-delete <image_id> and my log message will be output, amongst the rest of the debugging information.

This is what I understand about logging so far, and it is very helpful to have this as a tool.

Debugging

The only code I need for debugging is import pdb set at the top of the file with the other imports (alphabetically) and pdb.set_trace() wherever I want the trace to start. But here is the key, which I didn’t know until I was learning logging with russellb, after I edit the file to include the import and set_trace() I have to reinstall the client with sudo python setup.py install otherwise no debugging happens. Also after I remove the debugging code I must again reinstall the client to have my working code reflect the code in the files.

If you reached the end, I thank you. Here is a cat picture for you to enjoy.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Why would I need to reboot the shell when working on code in any of the projects? Well, if I edit any of the shell scripts (shell.py) the changes won’t be picked up by my shell unless I reboot it somehow.

My first inclination was to find a command for OpenStack along the lines of source ~/.bashrc but there was no command I could find.

It wasn’t until I pulled from a git repo that had moved to testtools that I recognized that sudo python setup.py install, executed within the root directory of the project, rebooted the shell to update the version of shell.py that it was sourcing.

I don’t know if this is the only way to reboot the shell when working on changes to shell.py, but so far it seems to be an effective way.

Since I got caught this week moving back to the master branch with a clean git status but with stale responses to shell help, I now incorporate sudo python setup.py install as part of my workflow. I already executed git status and ./run_tests.sh everywhere I went and I just throw in sudo python setup.py install if ever I have less than full confidence that the shell responses are reflective of the shell.py code in the repo in which I am located.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

Why am I visualizing novaclient this way? Well since I am just getting my bearings regarding the code the first thing I need to do is look for landmarks. It was really confusing seeing python-novaclient/novaclient in the path at first and seeing nova/nova in the nova project really threw me at first but I am starting to understand the purpose behind the file structure.

My big landmarks are the directories: python-novalclient, novaclient and openstack or v1. The openstack directory is at the same level as v1 (directly underneath novaclient) and the common directory is underneath the openstack directory, so why have I visualized it this way? Well for the sake of simplicity I like to visualize one path of flow and functionally v1 is imported by files within common and openstack so I consider openstack-common to be one entity and for it to be above the code flow of v1. The v1 stands for version 1 and in the projects with a second version of code to interact with the API, the directory is called v2. When novaclient has a v2 directory it will be at the same level in my mental model as v1, so a fork.

I cherry picked file names to place in this graphic. Why, because there are so many of them. At a certain point it becomes too much data to see the pattern and I mostly wanted to illustrate the mental pattern I am working with right now. I picked the files I have spent the most time in so far.

This graphic might seem simplistic or silly but it took me about 2 weeks to adopt this mental picture and another week to be able to communicate it. Not directly contributing to code but knowing where to look is a very valuable skill. I feel in some regards I am learning how to narrow down my search when looking for the location of a specific function.

Thank you for supporting this GNOME OPW intern,
Anita Kuno.

Armed with the investigative abilities of the debugger, I now can do a little spelunking.

While playing with the debugger set-up in my previous post, I saw that it took me through authenticate() in a rather tidy fashion so I thought I would spend a little time here.

I won’t walk through the entire function, just the first part. We will be looking at the consequences of stepping through parts of the code below. This is from novaclient/client.py:

With the help of the debugger I will be looking at the values of the various arguments and variables both before and after they are defined, and in the case of key and index as their value changes.

So the debugger heads into authenticate(). Before the first line (#301) is executed, I check the values of self and self.authenticate().

Then I check the value of has_keyring: before line #302 runs.

Lines #303 - #305 take a list of values and assigns them to the variable keys. I check the value of keys, which I learn is undefined before the code runs (makes sense) and also the individual values of all the attributes in the list.

After lines #303 - #305 have run, I check the value of keys.

Next we move into lines #306 and #307 which run in a loop 4 times before breaking out to execute line #308. I check the value of key and index during the process. These variables begin as undefined and then their value changes as the code runs through the for loop.

Then back into lines #306 and #307 3 more times before breaking out to execute line #308.

Then we loop through lines #306 - #308 one last time.

We go back up to line #306 but we have enumerated over every element in the list, so then we execute lines #309 and #310, checking the values of the variables as we go.

We have come to the last line of code in this examination, line #325. We check the values before it is executed and after execution it takes us into the function urlsplit() in the python library urlparse.py.

I hope that stepping through the authenticate function with me has been useful for you. I learned a lot creating this post. For learning a code base, I feel that using a debugger can be a very powerful tool.

We didn’t go through the entire authenticate function, but the part that we did takes the various attributes of the HTTPClient object and creates a url to be parsed. I found the investigation rather fun.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

This is the post in which I discuss how I set up a debugger to work for me with OpenStack, at least in the development environment. I like to be able to ask the code what object it is using, what the type of object is and what are the attributes. I always feel better when I can dialog directly with the code, no one knows better what is happening, and I dislike guessing since I am usually wrong. Better to ask, I find.

The first thing I had to do was select a debugger. I went with pdb, the python debugger that comes with python, simply because it was the easiest. I knew I needed to get the debugger working with OpenStack and I felt that the integration step would be my trickiest part so I elected for the simplest version of a debugger I could find and it appears that my decision to go with pdb was the right decision for me.

Then I had to learn how to use the debugger in a vanilla set-up. Since I didn’t know how the python debugger worked before I began, I spent some time with a few tutorial blog posts and am glad that I did. First I went here to go through the debugging A B C’s. I found it to be an easy tutorial and a great introduction. This tutorial showed me how to drop into the debugger by calling both the debugger and a file name from the command line.

I knew I wanted to access the debugger from the console so this was a great next tutorial. It creates a class in a file and then calls an instance of the class from the console with pdb.run(). It also calls the debugger after an exception with pdb.pm(). I liked the look of pdb.pm() since it could be invoked after an error or exception but every time I tried it, regardless of the pdb command I used, the debugger exited rather than stepping through the code. Initially I thought it was due to a mistake I made, but after further investigation I now believe that this seems to be the intended action of pdb.pm() though I have no idea why.

So while the second tutorial was helpful to show me that using the debugger from the console/interpreter was possible, I didn’t have the exact syntax I needed for my situation. Next I worked with this tutorial, rather long but for my purposes it gave me what I needed. The first part was the most helpful for me, I found the code for the last exercise - the advanced portion - didn’t match the output the tutorial displayed. I tried to locate the author but was unsuccessful.

So I decided to go with pdb import at the top of the file I wanted to start to step through and pdb.set_trace() at the location in the code where I wanted to drop into the debugger. Look at line #7 and line #43 in the code below. Line #44 was the beginning of the code I edited, which I wanted to evaluate, so I placed pdb.set_trace() on the line prior.

Next I needed to create an instance of novaclient in the python console. So I opened a console in my OpenStack development environment, imported the class for the client and created an instance.
python
import novaclient.v1_1.client
client = novaclient.v1_1.client.Client('admin', '<admin_password>', 'admin', 'http://50.56.25.223:5000/v2.0') Your ip will be different than mine but :5000/v2.0 should work for you, that is until it changes. So check the date on this blog post.

Then I requested the client fetch me an image by passing in the id of an existing image, which I had gotten with nova image-list earlier.
client.images.get('9e44f0ee-083d-40a4-804c-534d66fb15ac')

Now without pdb.set_trace() in the images.py code, the output to client.images.get(image id) is something like this:
<Image: cirros-0.3.0-x86_64-uec>

But with pdb.set_trace() in the images.py file, I got:

… and I’m in the debugger. Which was my goal.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

One of the things that I am doing that seems to be working very well for me, is constantly posting my current status: to my mentor, in my notes, for my reference, just to clear my thoughts.

I find that posting my status is my way of inviting comment without depending upon it. If my mentor has questions about what I am doing or how I am progressing, hopefully my status updates can help her keep tabs on me. If she has any thing to offer, a new direction, a suggestion, some support, she can post in response to an update. By continually posting my status without being asked, I allow her to check in as she has time, without having to wait for her to ask. It gives me the responsibility of scheduling my time productively and selecting tasks I can do on my own while incorporating feedback as it arrives.

It is serving a good secondary purpose. It is helping me to track when I am stuck and when I am not. If I am able to articulate my status, I probably am making productive use of my time. If I can’t answer the question “What are you doing?” with anything other than the response “foundering” it is probably time to seek some guidance.

The one caveat that I would add is the 3 day rule, in any new undertaking or situation I give myself 3 days to just be in the environment, with permission to not know anything and just listen and read and search and look and get my bearings. After 3 days, something just happens and some things which were completely unstructured just fall into place.

So I find that volunteering and posting my status, continually throughout the day for my mentor, contributes to a good work flow between us.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

With any kind of development, the first step is creating your environment.

The Devstack project sets up an OpenStack development environment rather quickly (the downloads take a while) with one script. Devstack.

Devstack has been tested on both Ubuntu Server 12.04 and Fedora 17. I have tried it on both and am most familiar with Ubuntu Server so that is the environment I will address going forward. The only thing I have noticed with Fedora is the necessity to disable selinux after the OS is installed.

So install Ubuntu Server 12.04 as an openssh server. Once installation is finished and you have rebooted and signed in, I like to execute the following: sudo apt-get install git libxml2-dev libxslt1-dev eatmydata lynx irssi vim-gtk. The two libs are necessary to run some of the OpenStack module tests and aren’t installed with Devstack. Git is required for Devstack to install properly, eatmydata is useful when running the OpenStack tests. I like to have lynx and irssi around in case I need them. Vim-gtk has a few more features than the default Ubuntu vim so I grab that too.

After apt-get is finished, I generate some keys and then configure my git: git config -- global user.editor "Vim", and also user.name "My Name", and user.email "Must Match Email in Gerrit".

Then I get Devstack: git clone git://github.com/openstack-dev/devstack.git and then cd devstack; cp samples/localrc .; vi localrc and add the following lines to localrc:
FLOATING_RANGE=192.168.1.224/27
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth0
then close localrc and run ./stack.sh. Full Guide Here

Usually Devstack installs successfully but the last couple of installations have exited with non-specific failure messages although everything seems to be running.

After Devstack is finished, execute source ~/devstack/openrc.

I test my installation by executing nova image-list, if I get a list of images I am usually in good shape.

The other thing I do is execute screen -r stack and see if all 17 processes have windows. Less then 17 processes and I may have to stop devstack with ./unstack.sh and restart with ./stack.sh. Detach screen with Ctrl-a d since you want screen to continue running.

This is basically the format I follow to install Devstack on Ubuntu Server 12.04.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

One of the biggest challenges that I am experiencing right now is understanding how to describe what I am experiencing. Writing down notes and developing my own workflow is feeling like a process that needs to evolve as my internship proceeds.

At the moment, I am working with IRC as my main form of communication and I have a private dialog window open with my mentor during my workdays. At the conclusion of every workday I copy the text of our interactions and save it to a Tomboy note. Tomboy notes are very flexible and I really like that I can search all of my notes from the main Tomboy page. Iccha, my mentor, often tells me commands or file paths in IRC that are helpful. Being able to search my notes for that command again or access a conversation or url from the day previous is really quite easy. Using a private dialog in IRC and and saving the text to Tomboy notes is working well for me.

I have a google document shared with my mentor and myself and I try to keep it updated as often as I can. I try for once or twice a day. It is a good place for summaries, ideas that carry over from day to day and urls for extra reading. It is nice because the viewers of the document can be administered and I can access the document easily from my browser.

I have been using a blank text file as a whiteboard in my local environment as a place to write pseudo code and collect my thoughts. Yesterday I expanded that idea to this gist. I like that the gist is private within github but I can also share the url with whomever I would like to access the information, readers of this blog for instance. I am still getting the hang of using comments on bug reports to communicate so I created this gist to explain my work and my thought processes to the creator of the bug and get his input going forward. Since it gives an up-to-date status of my understanding, after creating it, my mental energy was free to move onto other things, like blogging while waiting to communicate with the originator of the bug report. I like to have my mental work in tidy packages so that I can view a new and separate task with my full attention. Creating this gist as a whiteboard really served that purpose for me.

Taking notes will evolve as I go along and I am glad to learn new methods to make the best use of how I like to organize my mental space.

Thanks for supporting this GNOME OPW intern,
Anita Kuno.

A strong part of the open source community is working with others. I have had the good fortune to be in a position to work closely with another applicant over the last week.

Shruti (who blogs here) lives in India and is currently a student. She found out about the GNOME Outreach Program for Women through her love of OpenStack (via Facebook) and had contacted the same mentor as myself, Iccha. Iccha introduced us via email and we have been working together as a team since that introduction.

It was funny the way it worked out, we met each other the day prior to American Thanksgiving, which Iccha being in the US celebrates and neither Shruti or I do (I’m in Canada). So there we were, neither Shruti or I really knowing what we were doing, trying to get devstack working. Off to IRC we go.

I had previously met jpich on IRC and introduced her and Shruti. We had a lively interaction understanding various error messages and reading documentation trying to get operational environments on our respective systems.

Now had Shruti or I approached a different mentor or had Iccha not introduced us, I may have spent my time flailing away by myself waiting for Iccha to resurface after her holidays. By introducing us to each other and leaving us to figure it out on our own, we filled in the gaps ourselves and continued to work away, asking questions, having doubts, making mistakes, reading documentation and having success. Success is nicer when you have someone to share it with.

I’m not sure if this is quite the way the GNOME Outreach Program for Women is supposed to work, but this is how it is working for us. We have a daily email thread where Shruti and I share with Iccha our initial topic for the day with intended next steps. This allows for feedback and suggestions from the other two. When one of us is unavailable perhaps the other can offer something supportive if only an ascii smile.

We have good conversations with other OpenStack mentors and OpenStackers not involved in the program at all. Turns out they are genuinely nice even if they don’t know we are applicants.

I also think we encourage each other. Having another applicant to work with is a better gauge of accomplishment than comparing ones work with the mentor. I think it heartens me to work with another applicant and I am glad it is Shruti. She is friendly and intelligent and a hard worker.

I really like the energy we have created between and amongst us during the application process and I hope we can continue to grow and expand it during the execution of the program.

Thanks for reading,
Anita Kuno.

It may not sound like much, but having a successful first commit makes me feel very proud. The commit? It was a bug fix. The bug? It was a typo. Silly, yeah, I know but the point is less the code and more the process. It is a process which I had attempted and failed at previously.

About a year ago, I wandered into the Mozilla channel and received a lot of support when I identified myself as someone wanting to learn how to contribute. I had at least two people in the channel giving me their full attention and support. And yet I failed. I wasn’t able to navigate the process and I couldn’t put any more energy into trying. Life intervened and the necessity to feed dependants and take care of daily activities eliminated the brief burst of energy I had. I never made it back into the Mozilla channel after that first visit.

I am a big open source software consumer and I love to make useful contributions to products I believe are worthy. It wasn’t for lack of desire on my part or lack of welcome on the behalf of those in the Mozilla channel. It was all the other stuff that had to be dealt with to get to the point of committing a patch.

So what is different now? How was I able to commit a patch to OpenStack when I couldn’t figure it out with Mozilla? Well there are a couple differences, let me discuss them.

Money. Yeah, I don’t want to be crass, but money makes a difference. The GNOME Outreach Program for Women provides a stipend which when I devote my 40 hours per week to the project, I can buy groceries. I don’t have to do the work and figure out to buy groceries, I can focus on the work knowing I can also eat at the same time. Might sound not worth discussing to some, but to me it is a mainstay in any kind of learning/internship/apprenticeship program in which I have participated and I have participated in a few. So money isn’t the reason but it does make a contribution possible, for me at least.

Identified mentors available via IRC, email and Google Hangout (I think other venues are possible but these are the ones suggested and implemented thus far). Also the mentors checked in with me. Iccha knew I was considering applying and followed up with me, asking me questions about what I would like, offering me links and information, setting up times to meet with me. So for the brief time I was sitting on the fence, a mentor knew my status and helped me to make up my mind. That too counts for a lot.

The duration of the interaction also helps, I didn’t need to get everything done in one day. Many of the projects require specialized environments for executing and testing code and these environments don’t get installed by a n00b in one sitting. Having someone come to me and purposefully decide to help me as I tried and made mistakes and reinstalled and tried again made the difference for me to persist through multiple failures to ensure that I had an environment to edit code and successfully submit a patch.

Open source software is all about code but it also has a very large framework consisting of individuals who have a very specific way of working. By working together over several days, certainly at least a week if not two, I am starting to learn the people and the process. The edit itself took less than 5 seconds. Setting up the environment and learning the culture in order to enable me to edit the code and submit the patch with confidence? At least a week. Creating an environment filled with great people allowing me to take the week I needed is the difference between this successful commit and my prior failed attempt.

Thanks for reading,
Anita Kuno.