Sections

 Home
 Dev Log
 Resume
 Free Media
 Riddles
 Bob's Gallery
 Links & Resources
 Contact


Projects

 Stem
 Arm
 Kana Quizzer
 Keybinding Util
 Model Gallery
 Applets
 Nexus


Arm Development Log

This is not the page you're looking for.

Development status reports now live on http://blog.atagar.com/.


Status Report for February 2015

March 2nd, 2015

Why can't it always be winter? With murderous weather keeping you indoors life is so much more productive.



Hidden Service Descriptor Support

Stem now has the ability to read tor hidden service descriptors!

This is moot for the moment as there isn't a method to get them, but with David's work on #14847 this is becoming a thing.



Google Summer of Code

Sadly we weren't accepted into GSoC 2015. The program was trimmed back this year from 190 to 137 orgs, with several veteran orgs dropped to make room for newcomers. We're in good company. Mozilla, Linux Foundation, Blender, and others were not included this year too. Friday we're gonna get tips to improve our chances next year.

Pity, but c'est la vie. Regardless of GSoC we're always happy to welcome newcomers, so if you'd like to get involved then don't be shy!

Few other noteworthy things were...


Cheers! -Damian




Status Report for January 2015

January 31st, 2015

Greetings wonderful world. January has been delightfully focused on a couple substantial feature branches that are now merged!



Descriptor Lazy Loading

When reading descriptors without validation (which is now the default) Stem is 25-70% faster!

This is because we now parse descriptors fields on-demand. Just need a server descriptor's exit policy? That's all we'll bother reading. This is why the heavier the descriptor type the larger the benefit.



Unified Python 2/3 Codebase

Thanks to Foxboron Stem now has a unified python 2/3 codebase!

This means that Stem now runs directly with any version of python from 2.6 to 3.x. Prior to this Stem did a 2to3 export when you installed it to provide cross-version compatibility.

For users, the benefit of Foxboron's overhaul is that python3 installation and test runs are now much, much quicker.

Beside these a few other noteworthy tasks were...


Cheers! -Damian




Status Report for December 2014

December 27th, 2014

Hi all! Tis that time of year coal appears in all my sox so guess it's time for a status report. This month I set arm aside to polish improvements that accumulated in the Stem 1.3 release.

There were a lot of small tweaks, but the highlights for this month were...



Hidden Service Tutorial

By a bit of serendipity I came across a nice article by Jordan Wright for spinning up hidden services with Stem.

Federico Ceratto and Patrick O'Doherty have been improving our hidden service support, so I spruced up their contributions and wrote a tutorial of our own that demos it.



Gentoo Support

Thanks to Anthony Basile Stem is now packaged for Gentoo. Installing is as simple as...

  % emerge stem

I've been delighted at how many Gentoo people have jumped out of the woodwork to help. toralf worked with me on ticket to address testing issues, and Manuel took on Stem 1.3 packaging the same day as its release - thanks guys!


Couple other quick things of note is that I updated our OpenHub repository listing so it includes most of our newer projects, and trimmed membership of our internal lists.

Cheers! -Damian




Status Report for November 2014

November 23rd, 2014

Hi all. Next weekend I'm returning to ye olde isle of Vashon to feast with family, so figured I might as well send this out early. I'll be in a Thanksgiving food coma this time next week, after all...

My devious scheme for November was to finish rewriting arm's graph panel and, while that certainly made progress, Stem, DocTor, and internal discussions stole the spotlight this month...


DocTor


  • We now notify directory authority operators directly of issues in addition to the tor-consensus-health@ list. This should reduce duration of outages and other issues with the authorities.
  • We replaced turtles with longclaw. Sebastian deserves major kudos for orchestrating this.
  • Properly fixed DocTor's OOM issues by not shelling out to send notification emails.
  • DocTor detected a burst of relays from Google App Engine. These relays lacked any contact information so we dropped them from the network as a potential Sybil attack. If you're the operator of these then please let us know! We'd be delighted to add you back in once there's a proper family and contact information.

Stem


A couple other quick things of note is that I spruced up our volunteer page and attended this month's Seattle TA3M. The later had a nice bit of serendipity in that I met Anna, a UW PhD student who's looking into our Bandwidth Authorities. Best of luck Anna! That's certainly an area that could do with a lot of improvement. ;)

Cheers! -Damian




Status Report for October 2014

November 5th, 2014

I love the beginning of winter. Landscape a chilly tundra, no sun, yet still no snow to slip on. What I *don't* love, however, is all the sicknesses that float around. This month has been uncommonly unlucky for me with three separate bugs making the rounds. No single one was a big whoop, but still can't say I'm a fan of things making themselves at home in my warm, cozy respiratory track.

Oh well. Enough of that. This month was mostly focused on arm, specifically rewriting the graph panel. Work here is ongoing, but I hope to have this section done in November.

Other noteworthy things this month include...

  • Stem now has a set of methods to more easily work with hidden services thanks to federico3 and Patrick. New methods include...


    • get_hidden_service_conf
    • set_hidden_service_conf
    • create_hidden_service
    • remove_hidden_service
  • Addressed OOM issues with DocTor. DocTor uses quite a bit of memory, but the real trouble was that sending a notification at the end forked, doubling allocated memory. I addressed this by spawning the mail subprocess earlier, before we read in descriptor data.

  • Couple DocTor patches from Tom Ritter - thanks Tom! (1, 2)

  • Subteam discussions with Karsten and OONI folks accumulating in the VegasTeam3 wiki. I'm dubious that this will mean much (or anything) in practice, but we'll see.

  • Attended a couple local tech events including TA3M Seattle and Seagl.

  • Got promoted at work! This was five years in the making, so I'm pretty delighted. :P

Cheers! -Damian




Status Report for September 2014

September 30th, 2014

Code! Beautiful code! How I've missed thee!

As mentioned before I'm hip deep in the middle of an arm rewrite. Not only to overhaul the arm codebase, but to ensure Stem will be up to snuff for a Tor GUI or web dashboard (projects I'd love to jump into someday). I'm delighted to announce a nice bit of progress on this front.

September went toward rewriting its header panel (the top portion of arm's interface)...


As mentioned though this is really more about Stem than arm. Stem improvements made along the way this month include...


  • Far better handling for the 'private' keyword (the default prefix) and the default suffix in our ExitPolicy class. (ticket)
  • New BaseController.connection_time() method that provides when our control connection was established or detached.
  • New Controller.get_accounting_stats() method which reports accounting information for our relay.
  • Controller now fetches our own descriptor if no fingerprint is provided to get_server_descriptor(), get_network_status(), or get_microdescriptor().
  • New crop() and join() methods for the str_tools module.
  • Dropped a redundant get_* prefix from most of our util function names. Old names still work as aliases.
  • Our launch_tor_with_config() function raised an OSError if called too many times, caught thanks to RSenet. (ticket)

DocTor, the monitor for directory authority health, also got some love this month...


  • Corrected suppression behavior when we have staggered alerts. This should greatly cut down on the amount of noise on the list.
  • We now check if too many relays lack a bandwidth authority measurement... or we would if this wouldn't perpetually be in alarm. (ticket 1, 2)
  • Revised DocTor's schedule as requested by Sebastian. (ticket)
  • Expanded suppression for messages about turtles and kicked off a conversation to at long last sort this out. (ticket)

One final note is that I uploaded my Tor Ecosystem presentation to YouTube so volunteer page visitors can see a streaming presentation rather than downloading a 53 MB video.





Cheers! -Damian




Status Report for August 2014

August 24th, 2014

Hi all, getting my status report out of the way a tad early. Deadlines for my day job at Amazon required working a weekend so April was mostly spent just getting caught up on code reviewing contributions...


This concludes my backlog of things I owe other people, so next up I'll finally be getting back to arm development! -Damian




Status Report for July 2014

August 6th, 2014

For one brief, glorious moment in time I touched code. Really, it happened! House plants can vouch as my witness.

Or that is to say, I've coded a lot less than I'd like. Barring a long overdue vacation to Sol Duc this month went toward a lot of random distractions...

  • Worked with Philipp on overhauling our procedure for flagging bad relays. Philipp's taken the lead here and is doing a great job! While I find this space interesting I'm already spread too thin, so going forward I won't be a co-maintainer for it after all. :(

  • Looked over Nick Hopper's consensus validation change, adding crypto blob type validation to simplify his work. I now owe him a more thorough code review.

  • Handful of DocTor issues including OOM troubles, unpleasant amount of noise from consensus parameters, and a several minor revisions.

  • Discussed adding a new X- extrainfo descriptor field with Virgil, suggesting revisions and helping him make it conformant with the spec.

  • System I do development on died a puzzling death. Even after twenty hours investigating I'm still not quite sure what happened. Cleaning out dust, removing the RAID controller, and starting over seems to have done the trick.

  • GSoC administrative work. We're now less than two weeks from the final evaluations!

  • Continued overhauling arm but as already mentioned focused on this a lot less on this than I'd like.

So little coding makes me grumpy, so like many of you I'm taking a hard look at where my time's evaporating to. Maybe there's something I can change or extricate myself from to better focus.

Cheers! -Damian




Status Report for June 2014

June 26th, 2014

Oooh summer. Pollen scratching my eyes and a fiery ball of death hellbent on frying me. We oughta nuke the sun! Teach that lazy, freeloading ball of plasma who's boss. Just think of all the savings on air conditioning!

In those rare moments not staving off heat stroke I spent much of this month visiting family, and just a tad on code...

Recently I shifted my focus back to arm. I've been truly impressed at the level of interest in it on IRC! New arm users are appearing almost daily, and I owe them a shiny new release!




Status Report for May 2014

June 1st, 2014

Hi all, this month was focused on work leading up to today's Stem 1.2.0 release. The control interpreter in particular turned out extremely well, so if you haven't tried it yet then please check it out!

Besides implementing and writing a tutorial for the interpreter this involved addressing a host of other issues that had accumulated in trac. By far the most important was an issue where Stem spawned Tor instances would halt due to unread stdout content.

Last, I looked over the code of several new Stem users this month and added them to our examples page...


Cheers! -Damian




Status Report for April 2014

May 1st, 2014

Hi all. This April's status report is gonna be short and sweet. The bulk of my time went toward Google Summer of Code student selection. I'm delighted to announce that we have thirteen great students working with us this summer (twelve with Tor and one with the EFF)! This is twice as many as last year, so it will be interesting to see if we can manage it.

This concludes the bulk of my work as the org admin, so this next month I should be able to focus back on writing sweet, sweet code.

On April 26th I presented at LinuxFest Northwest. For those who haven't seen it, this was a repeat of my TA3M 'Tor Ecosystem' presentation.

And last but not least, bit by bit I've been picking away at a feature branch for Stem. It's not quite ready yet, but expect an announement and a new Stem release sometime in May. One improvement that *is* now available though is the following:


  • The connection module's connect_port() and connect_socket_file() functions have been deprecated in favor of a new, slicker connect().

This new function is quite a bit smarter about the error responses it provides, and is more intuitive to use.

Cheers! -Damian




Status Report for March 2014

March 31st, 2014

Hi all. GSoC coordination and a couple particularly tenacious illnesses pretty well knocked me out of commission this March. The neatest changes I managed to slip in concerned Stem's site...

  • FAQ entry for how to interact with Tor's controller interface directly. This comes up reasonably frequently on Stack Overflow. In the answer we discuss each connection and authentication scenario with examples.

  • A few weeks ago Coderman made the great suggestion for a Stem Cookbook. I've put together a few scripts that demonstrate neat things. If you have any suggestions for other things to exemplify then please let me know!

  • Code reviewed Martin's munin-tor and Lunar's exit-funding scripts. Both are now listed on our examples page, and I also made Exit Map more prominent (thanks Philipp for mentioning Stem in Spoiled Onions!).

  • Handful of other minor improvements such as making FAQ entries more concise, syntax highlighting torrc snippets, and better module docs.

Stem now supports HS_DESC events, and fixed an issue with ExitPolicy pickle-ability thanks to Nick Hopper.

Final note for March is that as anybody who knows me can attest I make a lot of typos. I spend enough time on irc that I finally invested some effort into getting an Irssi spellchecker working.

I love Irssi to death but user friendly it is not. If you would like a similar setup then here's my notes.

Cheers! -Damian




Status Report for February 2014

March 1st, 2014

It was great meeting everyone in Reykjavík! February largely went toward GSoC prep. Thanks to everybody that sent me project ideas!

    Blog Post

This left a lot less time than I'd like for coding. The only noteworthy changes this month were...

  • Added a new 'stem.util.test_tools' module for easily running Pyflakes and PEP8 checks.

    Stem, ExitMap, and arm all do these checks as part of their tests so this let them shed quite a bit of boilerplate.

    Want these checks for your own project? It's easy, so give it a try!

  • Stem tutorial example for checking the Tor exits you're using for your connections.

  • Couple new Controller methods for checking where Tor is listening for connections of a given type...

    >>> for listener_type in stem.control.Listener:
    ...   ports = map(str, controller.get_ports(listener_type))
    ...   print "%s ports: %s" % (listener_type, ', '.join(ports))
    ...
    OR ports: 
    DIR ports: 
    SOCKS ports: 9050
    TRANS ports: 
    NATD ports: 
    DNS ports: 
    CONTROL ports: 2779
                  
  • Gave Sphinx's aafig module a try to see if it could improve our API overviews...


    In the end I decided against it since this doesn't add much, and it causes us to render the overview as a svg (so users can't copy/paste function names).

  • Sent Philipp some improvements for ExitMap.

Cheers! -Damian




Status Report for January 2014

January 30th, 2014

Hi all. This month I decided to switch from a depth-first to breadth-first approach for overhauling arm. Changes included...

  • Made arm PEP8 compliant. This took around a week during which sed and I were close friends. Like Stem, both pyflakes and pep8 checks are now included in arm's test runs.

  • Completely dropped tor_tools from the codebase. This is a neat milestone as tor_tools was a 2,700 line wrapper around TorCtl providing a friendlier API, thread safety, caching, etc. Stem provides all of these capabilities.

    Initially migration was simply switching tor_tools from TorCtl to Stem, but now arm works directly with a Stem controller allowing us to drop the tor_tools module completely.

  • Finished arm's new tracker module which provides tor resource usage, connections, and application names. It's not just much nicer code, but now has unit tests!

Why am I sinking all this effort into arm? After all, it presently does pretty much what we want, right?

Well, yes. And I suspect user's won't see many obvious changes in the next arm release. Rather, my present focus on arm is partly for code quality but more importantly to make sure Stem is ready for prime time.

Arm and Vidalia are our two most complicated and demanding controller applications, and making a Stem-based arm release is a surprisingly good way of making sure Stem has all the pieces we need for tasks like a next-gen Tor GUI or a localhost status panel for relay operators.

As I've rewritten arm Stem has improved as well. Stem changes this month included...

  • Added a port_usage() function for getting a description of a port's common usage. This nicely complements our other connection utils...

    >>> from stem.util.connection import port_usage
    
    >>> port_usage(80)
    'HTTP'
    
    >>> port_usage(22)
    'SSH'
    
    >>> port_usage(9418)
    'Git'
    
  • Couple new Controller methods to help with NEWNYM handling: is_newnym_available() and get_newnym_wait().

  • Pyflakes and pep8 are python modules so why shell out to them? We now use their APIs instead reduced the runtime of Stem's static checks by 60%.

  • Greatly reduced the verbosity of stem's test output. You can still get the previous, more detailed results with the '--verbose' argument.

Last but not least, non-development tasks from this month included...


  • Finally moved doctor to its new home on cappadocicum. (ticket)
  • Submitted my Tor Ecosystem talk to LinuxFest Northwest. (presentation)
  • Tickets for Iceland. Looking forward to seeing everyone there!

Cheers! -Damian




Status Report for December 2013

January 1st, 2014

Hi all, hope you've enjoyed the holidays (and for a few of you CCC)! This December I finally swapped back to focusing on arm in earnest. Good news is that the codebase is rapidly improving, the focus for this month being...


  • Finished overhauling the starter module.
  • Expanded test coverage as more of arm gets rewritten.
  • Integrated pyflakes and pep8 with our tests.
  • Moved user facing strings to the internal configuration. I did this to improve our code, but it could also allow us to add internationalization support to arm in the future if folks want it.

Bad news is unfortunately that none of these improvements are user facing, and even by the rosiest estimates arm's next release is still many months off.

This December I also finished with my efforts to revise our internal lists. None of the more substantial changes got much support, but this still resulted in some worthwhile improvements...


  • A wiki with our lists, maintainers, and a brief summary of each: link
  • We now have a public, documented policy for adding folks to our internal lists.
  • The tor-assistants list is now a proper subset of tor-internal.
  • Identified some unused lists we can drop. (ticket)

Cheers! -Damian




Status Report for November 2013

December 8th, 2013

Hi all. Work and life is conspiring to keep me busier than usual. What time was left this November mostly went toward non-development activities. I miss coding...



Tor Ecosystem Presentation


On 11/18 I gave a presentation at Seattle's TA3M overviewing projects in the Tor ecosystem, with an eye toward getting more developers involved...


Preparing for this gobbled up a surprising amount of time (I don't do much public speaking), but I'm really happy with how it turned out. I'll likely give this presentation again at other local venues, such as LinuxFest Northwest in the spring.



Email List Management


Much of this last month went toward managing our lists and leading a discussion about what we'd like to do with our non-public lists. Work on this is ongoing, but at the very least we'll finally get a public, documented process for adding folks.

I also sunk several hours into a frustrating GMail issue where it bounces Mailman's moderator queue as spam. At one point this seems to have been responsible for unsubscribing everyone using GMail from tor-assistants@. I've changed Mailman's configuration so this hopefully won't happen again, but I'm out of ideas for the GMail bounces.



Other news this month includes...


  • Fixed an important Stem ImportError under Windows. This resulted in a hotfix release of Stem (version 1.1.1). (ticket)
  • Added Stem support for the new event types from proposal 218. (change)
  • Code reviewed Philipp's ExitMap project.
  • Read over tbbscraper and added to Stem's examples page.
  • Helped provide stats for the impact of changing tor's default exit policy. (ticket)

Cheers! -Damian




Status Report for October 2013

October 27th, 2013

Hi all. Here's a quick update of what I was up to this October...



Stem Release 1.1.0


After seven months of work Stem version 1.1.0 is finally out!

This month was primarily focused on pre-release performance and compatibility improvements in preparation for this...




GSoC Mentor Summit


Nick, Moritz and I attended the GSoC Mentor Summit at Google's main campus. By and large it was very similar to previous years, with many of the same faces and discussion topics. Unlike the prior couple years however I did not focus on keeping notes which, on reflection, was a mistake. Here's the small bit I remember...


  • Had a couple fun language discussions with developers from SciRuby and Scala. The former mostly concerned the advantages and disadvantages of Ruby with respect to Python (I use the former at work, so I've had some time to form an opinion on this). The later concerned Scala support for issuing database queries via list comprehensions.
  • Couple discussions with Wikimedia developers regarding Tor abuse. Unfortunately this is a perennial discussion that never seems to go anywhere despite having plenty of technical solutions.
  • Nick gave me a tour of Chutney's codebase and tried to persuade me to take on development of it. Nearly worked too. But while Chutney, SoaT, TorBEL, and a dozen other projects would be a lot of fun to work on I only have so much I can squeeze in alongside a full-time job.
  • A hackathon organizer with Random Hacks of Kindness discussed what they've found most successfully attracts contributors. The two most memorable tips were momentum and recognition. Regularly organizing events is necessary to maintain a core contributor group, and engaging local media doesn't hurt as it provides recognition for the contributors.
  • Shadowed Nick during discussions with a Mozilla developer concerning the upcoming Tor IM bundle. They certainly seem excited for OTR support...
  • Visited with Terry and Arc from the PSF quite a bit. Still no ETA on when distributions will switch to Python 3, and Mailman 3 is still a ways off. Momentum on lots of projects though. Evidently they acted as an umbrella org for roughly sixty students. I would weep bitter, bitter tears if I had to be the admin for that...


Other news this month includes...


  • Karsten, Andrew and I are now all moderators for the tor email lists. I'm taking point in this area, making daily sweeps through the moderator queue and tweaking Mailman to reduce friction.
  • Handful of DocTor fixes and improvements. Karsten shut down the Java DocTor early in the month so our shiny, new Python DocTor is now flying solo!
  • Attended events in the Seattle area including 'Go for Python Developers' at Google Fremont and the second TA3M meetup.
  • Worked with authority operators to resolve brief outages with dizum and urras.

Cheers! -Damian




Status Report for September 2013

September 29th, 2013

Hi all. Present status: gobbling birthday carrot cake. Before that, though, did a few things worth noting in September...



Stem: Connection Resolution Support


Stem can now provide you with the active connections belonging to a process. This is similar to arm, capable of doing lookups via seven *nix and FreeBSD resolution methods.

Unlike arm, however, this is a far cleaner implementation including both unit and integ tests. Unfortunately this doesn't yet work with Tor's DisableDebuggerAttachment feature, but I plan to look into workarounds for that soon.

'How do I use this', you ask? It's easy!



Arm: Rewrote Starter Module


Arm is presently undergoing an overhaul, and this month I focused on its starter module. It's now shorter, has unit tests, and is just generally damn better (before and after).



Other news this month includes...


  • Google Summer of Code finals were last week. All the remaining students passed, having done great work - fingers crossed they stick around!
  • DocTor's python replacement is now happily chugging along on yatei and provides #tor-bots notifications.
  • Attended a local TA3M meetup hosted by Lee Fisher. I'll be presenting a 'tor ecosystem' talk at it in November.
  • Notified outdates relays that will soon be dropped from the network if they don't upgrade. (ticket)
  • Investigated tor regression reported by stem's jenkins tests. (ticket)
  • Code reviewed Aaron's TorPS.
  • Fixed a stem caching bug regarding hidden service config options caught by wayzard.(ticket)
  • My site (https://www.atagar.com, which hosts arm) finally has SSL support.

Cheers! -Damian




Status Report for August 2013

September 7th, 2013

Hi all, after a month of work I'm pleased to announce that the Stem based replacement for Doctor is now live!


For those who aren't familiar with it, Doctor is a service that pulls hourly consensus information and checks it for a host of issues (directory authority outages, expiring certificates, etc). In the case of a problem it notifies tor-consensus-health@, and we in turn give the authority operator a heads up.

The new version of Doctor replaces Karsten's java based counterpart and is part of our ongoing scheme to eventually deprecate Metrics-lib in favor of Stem.



Other news this month includes...


Cheers! -Damian




Status Report for July 2013

August 3rd, 2013

Hi all. This month was mostly spent on non-tor work including a server migration, bad service outage at work, and a full week of cleaning my apartment. Still, plenty of spiffy news in stem land...



Remote Descriptor Fetching


Major feature for this month was the addition of a module to remotely fetch tor descriptors...


This works much like tor itself does, downloading descriptor content from directory authorities and mirrors. With it we can now easily script against the present state of the tor network without piggybacking on a live tor instance.

Curious what you can use it for? See our present monitors for some ideas.

This also included a little work with Nick on the spec and tor side...


  • Dropped the unimplemented microdescriptor query from the spec. (ticket)
  • Noting the max queryable fingerprints/hashes in spec. (ticket)
  • We're getting a high failure rate on the downloads we make. A little more investigation is needed on my part to help narrow this down. (ticket)


Other news includes...


  • Revised the appearance of stem's frontpage. The blue buttons were pretty jarring, so switched to something that matches the rest of the page. (before, after)
  • Added Slackware to our download page. Many thanks to Markus for adding us to SlackBuilds!
  • Worked with Sreenatha to port Tor Weather to stem. Unfortunately Weather does not presently have an active maintainer so I'm not sure how we will proceed on this front. (thread, ticket)
  • The Munich dev meeting has attracted quite a few potential volunteers. After discussing prospective projects with seros I tidied up our volunteer page. Changes included...
  • Our automated Jenkins test runs ran into another regression in tor that caused it to segfault. (ticket)
  • STREAM events mishandled IPv6 addresses. (caught and patched by soult, ticket)
  • Thread with Pierre about TorPylle which turned into a discussion with Nick regarding future language direction for the tor codebase. I'm looking forward to seeing where it goes!
  • We just finished with midterms for Google Summer of Code. Chang unfortunately did not pass, but the other projects are going well.
  • Sorted out travel arrangements for the GSoC mentor summit. Nick and I will be going, and Moritz is presently on the waitlist.
  • Code reviewed ra's rttprober. He provided some great feedback for which I still owe him a reply.

Cheers! -Damian




Status Report for June 2013

June 27th, 2013

Hi all. Without much ado here's my status report for June...



Stem: Migrated to the Mock module


Our homemade mocking framework has served us well, but over time it taught me one very important lesson: writing a mocking framework is hard. On the surface it seems pretty simple: apply and revert a set of monkey patches. But how do you monkey patch class methods? What about alias imports like the os module? And god forbid you want to mock python's open() function.

I'm finally taking a lesson from one of my coworkers and using a library for this. Python has several options but the most common is PyPI's mock module, which became part of the standard library in Python 3.3. (change)



Arm: Pruning the torTools utility


Arm's torTools module was a wrapper around TorCtl that provided caching, thread safety, and a better API. Now that we're using stem it's obsolete, and my aim is to completely eliminate it from the codebase. This is easier said than done however. This month I pruned vast swaths of the module, reducing it from 2768 lines to 1020. (before, after)

Doing this required some expansions on stem's part. Added functionality includes...


  • get_pid() and get_user() methods in the Controller
  • system functions for getting a process' start time and FreeBSD jail path
  • the system module's call() wasn't respecting exit codes


Stem: Remote descriptor fetching


Throughout the month Karsten and I have been bouncing ideas back and fourth concerning a stem API for remote descriptor fetching. I have some s3krit ideas for improving this that will address both our use cases even better (spoiler: future style instances). We're coming up on a four day weekend so hopefully I'll be able to start implementing this soon!



Stem: Expanded FAQ with answers to common Stack Overflow questions


I took a pass over the tor related questions on Stack Overflow, answering fifteen that concerned controller scripting. The vast majority of those unfortunately were some variation of 'how do I programmatically change my IP?' which I answered with a Stem FAQ entry.

The only question I thought was especially interesting went along the lines of 'How do I check the IPs of the exits I'm presently using?' (link). I answered this with a FAQ entry too.



... and a handful of smaller tasks


  • Investigated the descriptor version provided via tor's 'GETINFO ns/*' options. Contrary to to the spec it turns out these have been v3 all along, and stem now parses them as such. (#7953)
  • Our automated Jenkins test runs caught their first instance of tor regression. This concerned LOADCONF's behavior after merging a branch for ticket #6752. (#9122)
  • Handful of GSoC tasks (welcome/sorry emails after acceptance was announced, and org admin prep for the program's start)
  • Added msg_type argument to ControlMessage.from_str() (request by meejah, #8605)
  • Investigated cache consistency issue (thanks to Ravi, #7713)
  • Fixes for ONLINE target (patch by Jeremy, #8692)
  • Minor revisions to how consensus bandwidth-weights are validated (#6872)
  • Addressed an arm issue with certain TERMs (#9064)
  • Answered some questions from Sreenatha that came up while migrating Tor Weather to Stem.

Cheers! -Damian




Status Report for May 2013

May 26th, 2013

Hi all. This is a little early but considering the time sink GSoC has been I doubt I'll get much more done, so here's my status report for May. The few stem tasks I've snuck in included...


  • Added Ubuntu and Fedora to our download page, Fedora package is thanks to Juan Orti
  • Several testing fixes so our Jenkins tests runs now pass
  • The Controller mangled non-unicode descriptor content when using python 3.x (caught thanks to aj00200, #8755)
  • Expanded our client usage tutorial to use SocksiPy and include an example for polling Twitter (thanks to Ashish)
  • Integ tests now assert ownership on the tor process (patch by Jeremy Kushner, #8634)
  • The DescriptorReader mishandled relative paths (patch from Kostas, #8815)
  • Fix so the Controller cache is thread safe (patch by Akshit, #8607)
  • Running our integ tests littered the /tmp directory (patch by Ashish, #8622)
  • Improvements so we use the staticmethod decorator and new style exception catching (patches from Sean, #8824 and #8823)

Cheers! -Damian




Status Report for April 2013

May 6th, 2013

Hi all. Though April had a bit of stem work the month was mostly gobbled up by the chaos that is GSoC. I never cease to be amazed at how much time orchestrating it takes, but enough of that. Other tasks from last month included...


Improved Stem's Site



General Stem Improvements


  • Overhaul for a large part of our testing framework, run_tests.py in particular was in need of a rewrite
  • Fix for broken process renaming (patch from ragwater, ticket)
  • We now have a custom :trac: and :spec: role for Sphinx (patch from ragwater, ticket)
  • ATTACHSTREAM provided an unexpected 555 response (caught thanks to a cypherpunks, ticket)
  • Added support for the ADDRMAP event's new CACHED attribute
  • Looked into anomalous bridge-ip-transports lines in the consensus, turned out to be from an unmerged tor patch running on George's relay

Cheers! -Damian




Status Report for March 2013

March 31st, 2013

Hi all. Even without counting the Boston dev meeting March was a highly productive month. Noteworthy things include...


Stem Tutorials

Stem's tutorials got an overhaul, including:


  • A much friendlier layout. No more intimidating wall of text - the tutorials have been rewritten and broken into subsections.
  • "To Russia With Love" tutorial, exemplifying client usage and programmatically managing a tor process.
  • "Tortoise and the Hare" tutorial, demoing tor event handling through a curses bandwidth graph.
  • "Double Double Toil and Trouble", which ends the tutorials with a page of scripts and applications that use stem.

Feedback welcome! The shiny new tutorials are available at...

https://stem.torproject.org/tutorials.html


Stem Packaging

Thanks to a half dozen package maintainers stem is now available on several platforms, with more in the works. The most recent was the Python Package Index (PyPI), which can make stem installation as simple as 'pip install stem'.


Google Summer of Code

The timing of this year's winter developer meeting couldn't have been better. During it I begged, bribed, and poked people with pointy sticks until they 'volunteered' to mentor something in this year's GSoC. Thanks to them our project ideas page is no longer overly sparse. Google will be announcing the selected orgs on April 8th.


Stem Release 1.0

Last, but certainly not least I wrote numerous finishing touches for stem and made its long overdue initial release!

https://blog.torproject.org/blog/stem-release-10

Cheers! -Damian




Status Report for February 2013

March 2nd, 2013

Hi all. Between being a short month and oncall for work I didn't get as much done as I'd like. What time I did have for tor mostly went into stem's descriptor functionality. In particular...


Microdescriptor Support

Stem can now read and parse microdescritpors, with controller methods coming later this weekend.

I understand the desire for lightweight descriptors but they're a step backward for controllers. Their lack of fingerprints make them clunky to use, and tor lacks the usual methods for retrieving them (ticket) and v3 network status information (ticket). Controllers will often need to read descriptor content from the data directory until those are fixed.


General Descriptor Improvements

  • Invalid descriptor content within archives caused the reader to stop processing content from the archive. This bug is simple in retrospect but cost me around a week of hair pulling frustration to sort out. Thanks to Karsten for catching it! (ticket)

  • Descriptor readers can now optionally provide network status documents rather than the entries they contain. Feature request by Karsten. (ticket)

  • Calling str() on Descriptors choked if it contained unicode content. Caught by Sathyanarayanan. (ticket)

  • Thanks to Karsten Descriptors now provide hex digests. (ticket)

  • Discussed the new 'flag-thresholds' attribute and added support for it to stem. (ticket)

  • Descriptor parsers that used readline() could choke if derived from a descriptor archive. Caught by Karsten.


Other Tasks...

  • Discussed proposal 218 with Karsten on tor-dev@. (thread)

  • Changed our consensus-tracker script to alarm for non-exits (we had an undetected sybil attack early in the month). Also fixed an issue where the script gobbled up way too much memory. (ticket)

  • Made a few backward incompatible changes to improve stem's usability in anticipation of our March API freeze...


    • Version comparison is now done through normal comparison operators rather than a meets_requirements() method.
    • Renamed the keyword arguments for Controller.from_port() and others to be less verbose (for instance 'control_port' to just 'port').
    • Dropped the 'path' arg from parse_file(), it was never intended for external callers.
  • Variety of bug fixes...


    • We didn't recognize a 'NEVER' date in ADDRMAP events. Caught by Desoxy. (ticket)
    • Patch from Abhishek so our tests avoid static /tmp usage. (ticket)
    • Added copyright notices throughout most of our codebase. Suggested by Juan. (ticket)
    • Addressed issues with get_process_name() on OSX. Caught by Sathyanarayanan. (ticket)

Cheers! -Damian




Status Report for January 2013

February 2nd, 2013

Hi all. This January I've been integrating feedback from first time stem users and implementing their feature requests. Projects included...


Python 3.x Support

Stem now works under both the python 2.x and 3.x series. To use stem with python 3 simply use it when you install...

python3 setup.py install

This turned out to be a larger project than I had anticipated, taking almost half the month. But it was well worth the effort.


PEP8 Compliance

Ditched most of my odd coding preferences in favor of the standard python style guide. I've integrated both pep8 and pyflakes with our tests to prevent regression. Hopefully this'll make it easier for others to contribute to stem.


Arm Codebase Refactoring

Prior to being shanghaied into the projects above I sunk quite a bit of time into overhauling arm. Thus far I've dropped around a third of the codebase in favor of similar (but tested!) capabilities in stem.


Descriptor Improvements

Got lots of input concerning stem's descriptor module (thanks Karsten!), leading to quite a few improvements...


  • used feedback from Aaron Johnson to make the descriptor API less confusing
  • we now support bridge network status documents (ticket)
  • added support for '-legacy' authorities (ticket)
  • parsing error if network status documents lacked a 'directory-footer' line (ticket)
  • empty 'bridge-ip-versions' lines caused problems (ticket)
  • we didn't recognize the @type annotation for key certificates (ticket)
  • we weren't parsing the new ntor-onion-key lines (caught by sonu, ticket)
  • error when ran with pypy (caught by peer)

Sean helped quite a bit at the start of the month, but has since been busy with other things.


  • added a Controller.get_streams() method (ticket)
  • tests for the close_stream() method (ticket)
  • expanded the Controller's unit tests (ticket)
  • fixed type checks (ticket)
  • fixed test_reattaching_listeners (ticket)

Cheers! -Damian




Status Report for December 2012

December 31st, 2012

Hi all. Between the holidays and being oncall for work I didn't have very high hopes for December. However, by the time the dust settled it turned out to be a surprisingly productive month. Projects included...

  • Finishing stem support for event handling. This was the last major feature we were missing before having feature parity with TorCtl.

  • Ported arm and the consensus-tracker to stem. The arm migration went surprisingly smoothly, but there's still a lot of cleanup work left to do here. Ideally arm will be a far simpler codebase now that it doesn't need a wrapper module around the controller.

  • Moved stem's site to "https://stem.torproject.org/". (ticket)

  • Smaller things include...


    • finally fixed the periodic freezes in arm (ticket)
    • uniform support for a default response in Controller getters (ticket)
    • vastly improved performance and memory usage for the ExitPolicy class
    • expansions for descriptor handling (ticket 1, 2, 3)
    • extend_circuit(), attach_stream(), and get_circuits() support (patches by Ravi, ticket 1, 2)
    • TAKEOWNERSHIP support (thanks to Lunar^ for the initial patch, ticket)
    • fixed a bug where circuit/stream ids were sometimes ints (caught by Lunar^, change)
    • added a post-authentication hook so event listeners can be reattached to Tor
    • several OPW discussions with Marina (we didn't get any substantial applications)
    • added flash proxy and txtorcon to the volunteer page, and made lots of general revisions
    • discussed TorCtl deprecation with Mike and made the announcement
  • Besides this, Sean Robinson has been submitting an absurd number of fixes, improvements, and code reviews of his own. Many thanks!


    • version pre-requirement checks for events and tests (ticket)
    • testing expansion for malformed events (ticket)
    • close_stream() method (ticket)
    • STREAM_BW event handling (ticket)
    • testing util expansion to make it easier to test client use cases (ticket)
    • get_socks_listeners() method and related mocking changes (ticket)
    • ... and many, many more (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)

Cheers! -Damian




Status Report for November 2012

December 1st, 2012

November was a month that included eating far too much pie. There was some stem work too on the side, though delicious pumpkin pies are always how that month should be remembered.

My primary focus for November was tor event handling, which is the last major feature we need before having parity with TorCtl. We presently support nine of the nineteen event types, including the most commonly used ones (logging, BW, CIRC, STREAM, etc). I'll be spending a good chunk of December finishing this up.

Besides this I've been really thrilled at how contributors are coming out of the woodwork to help...


Ravi

  • Ravi has volunteered to take a lead on moving stem onto tor's site. Unfortunately this is presently blocked on getting a subdomain.

  • Provided a patch to move stem's controller exceptions to the top level namespace. (ticket)

  • Fix for the repurpose_circuit() integ test (ticket) and discovered an issue with the stem.process test. (ticket)

Eoin

  • Submitted a great patch overhauling and expanding verification of server descriptor content. (ticket)

  • Caught a possible tor bug related to 'GETINFO orconn-status' queries when disconnected. (ticket)

  • Numerous spelling fixes (change) and caught an issue with respect to how the descriptor reader handles archives. (change)

Sean

  • Reviewed my event parsing branch, offering feedback (ticket) and catching a bug where STREAM events could have a zero port. (ticket)

  • Submitted patches to add close_circuit() to the Controller (ticket) and a setup.py (ticket). The former led to a discussion about stem's licensing and copyright for patches.

  • Helped resolve an issue with EXTENDCIRCUIT where we weren't taking into account when the path was optional or not. (ticket)

Other things I did this month includes...

  • Preparation for the 2013 Outreach Program for Women, the application deadline for which is now only two days away. This mostly involved helping others add their project ideas to the volunteer page and adding one of my own.

  • Made a landing page for stem's bug tracking and linked to it from stem's site.

  • Revamped stem's enum documentation to be both more readable and support interlinking. (change)

  • Provided a code review for Karsten's pygeodate.py. (ticket)

  • Answered a handful of controller inquiries on our lists. Stem's now at a point where I don't mind suggesting it to developers. If you're scripting or writing an application around tor then please give stem a try! I'd love to get more feedback on where its rough edges are before we make an initial release. (1, 2, 3, 4)

Cheers! -Damian




Status Report for October 2012

November 2nd, 2012

Ahhh, the beginning of winter. That very special time of the year when Seattle skies are perpetual overcast and gloomy. When crowds flock to the coffee shops and the best thing to do with your time is hook up to a caffeine IV and code.

... on a side note it has been a really productive month.


  • Stem Website

    Stem now has a far more developer friendly website!

    https://stem.readthedocs.org/en/latest/index.html

    Barring the obvious new sections I revised vast swaths of stem's API documentation.

    Feedback welcome! The tutorial section is just starting, but I have some ideas for expanding it.


  • Network Status Document Parsing

    After a couple months of work by Ravi and me the network status document handling is finally done. It turned out to be a... very big feature branch.


    With this done we're very nearly at feature parity with metrics-lib (and I suspect a bit past it in terms of testing). In finishing this up I also spotted an undocumented oddity with microdescriptor 'directory-signature' lines.


  • GSoC Mentor Summit

    Met with the developers of numerous other open source projects. I was really impressed at how many worthwhile conversations were crammed into such a short conference. Highlights include...


    • Arc from Python had distutils suggestions, thanks to which stem and arm will soon have both Python 2.x and 3.x support.
    • OSU's Open Source Labs are entertaining the thought of running Tor relays.
    • Discussed TorBirdy and other Tor projects with Sukhbir. He mentioned an issue with trac permissions that is now fixed.
    • Talked with Terri from Python about Mailman 3, which would be a great answer for the requests we get to have a Tor forum.
    • Attended a talk led by Marina from Gnome about their outreach program for women. We'll likely be taking part in it this year.

Other changes include...


  • Cleaning up orphaned pyc files as a part of running our tests (feature suggested by Ravi, ticket)
  • Ported arm's str_tools module to stem and added unit tests (ticket, docs)
  • Code reviewed Ravi's attach_stream addition, waiting on the revisions (ticket)
  • Arm troubleshooting on irc with zack and ultramage. Also fixed a broken link on arm's site spotted by wh6iQ.
  • A new contributor, Eoin, spotted some great issues and contributed patches. Tickets include...


    • python 2.5 compatibility bugs (ticket)
    • count of the number of skipped tests (ticket)
    • error with the whitespace checker (ticket)
    • spelling corrections (ticket)
  • Sathyanarayanan spotted some nice bugs including...


    • on python 2.5-2.6 a missing microdescriptor consensus wasn't causing the related test to be skipped (ticket)
    • integ tests for the process module would fail if tor was already running (ticket)

Next up: Ravi and I are working on tor event handling, the last major feature stem is missing as a controller library. After that we plan to port arm over, then tidy up loose ends in preparation for stem's initial release.

Cheers! -Damian




Google Summer of Code Mentor Summit 2012

October 21st, 2012

As per conferencing tradition Friday was spent on travel and meeting the other attendees. Some of the highlights for me were...


  • David from KDE

    Besides demoing some KDE eye candy we discussed their project infrastructure. KDE is a federation of smaller projects and had over sixty students this year (ten times the number mentored by Tor).

    Their project's scale has led to some unusual infrastructure decisions. For instance, they have a partly decentralized git infrastructure where pushes go to a single master host and pulls are from any of several mirrors. The config they use to do this leads to some... odd behavior. For instance a 'git pull' updates your tracking branch but not the origin branch reference. The result is that to do a pull for realz you need to call *both* pull and fetch. No doubt they also get fun behavior from mirroring delays...

    We also talked a bit about post-review and defaults they could set to better support their setup. KDE has the largest public ReviewBoard instance, but the above git setup makes it a bit confusing to use.


  • Sukhbir from Debian

    In 2011 Sukhbir applied to us for GSoC to work on TorBirdy. We loved his proposal, but due to prior commitments he ended up working with Debian instead. Since then he has become a GSoC mentor for Debian and involved with the Tor by implementing his earlier proposal for TorBirdy.

    Sukhbir's interested in getting even more involved with Tor so we discussed other projects that might interest him, and ways that we could better publicize TorBirdy on our site.


  • Arc from Python

    When I found out that Python had a mentor at the summit I made a mental note to hunt him down and ask about packaging best practices. After an unexpected discussion about rugby I found out that it's actually easy to support both python's 2.x and 3.x series by including a 2to3 conversion at build time. This can be done via either distutils or distribute.

    I also asked him to look into his crystal ball for when python 3 would take over the world and he said 'Next year. Ubuntu and Fedora are ready and willing to make the switch. The last main holdout is Gnome. They tried to migrate but work there isn't finished yet.'


Saturday was the first day of the unconference. After an amusingly confused attempt to have each of the couple hundred attendees shake each other's hand there were sessions. Some were a little interesting, but I spent more time on the hallway track since that's the real benefit of the summit. The only useful tidbits I got from the talks were...


  • Do outreach early. The successful GSoC students who stick around tend to be the ones that get involved before the application phase. We should try harder to recruit college students to hack on tor, with the carrot that this'll give them a leg up when applying for the program. OpenHatch might be something to look into for this. This would be a nice task for a community manager if we get one...

  • Google Code In is a program somewhat similar to GSoC where highschool students become involved with open source. Last year they had 18 organizations and this year they're narrowing it down to 10. I was already highly tentative about having us apply and now that I've heard more I'm sure we don't have enough bandwidth for the hand-holding this would require.

As for the hallway track...


  • Adriano and Luis from Umit

    Last year Adriano showed me Open Monitor, a censorship detector written in python. Sounds familiar? I thought so too, and tried a few times to get them to talk with Ooni Probe and vice versa without success. My impression is that they're UI developers (a skillset we sorely lack in the tor project) with a rather unscalable backend, while Ooni Probe's backend is far more mature but lacks any sort of UI for rendering real time censorship information.

    I made another stab at getting the two projects to talk, after which the meeting took a weird turn with Adriano arguing that 'some censorship is good'. Evidently they decided that Open Monitor won't look for censorship concerning 'porn or terrorism'. I argued that this was a slippery slope and that censorship monitoring shouldn't try to pass a moral judgment on the content being censored, but after a time it was clear that we were talking past each other.

    I still think that we should leverage their UI expertise, but that's up to the Ooni Probe devs.


  • Open Source Lab

    Met with a couple administrators from the OSU's Open Source Labs. They provide hosting for several of the largest open source projects including Apache and the Linux Foundation. Mostly we talked about amusing legal threats they get for hosting the phpBB project. Evidently lawyers are quite skilled at clicking the 'this is a phpBB forum' link followed by 'hosted by the OSL' before sending their angry emails. We also talked a bit about setting up non-exit relays. They might be pretty receptive to this if we want to follow up.


  • Sumana and Rob from Wikimedia

    Unsurprisingly Wikipedia occasionally has issues with spammers using Tor. We talked about some possible options, such as requiring accounts for Tor users to edit with a sort of proof of work in account creation to make ban evasion more of a pita.


  • Terri from Python

    Mailman 3 is coming, and with it an interface that *doesn't* look like it came from the 1980s! Most importantly for us, the new version of Mailman provides a forum interface, letting email and forum users communicate by whichever method they prefer. This would be a good answer to our forums ticket. She estimates that it'll be ready in six months or so.


Flying out on Sunday cut my day short, but there was one session that I thought was interesting. Gnome and Wikimedia are launching a program similar to GSoC to encourage more women to get involved with open source. It runs later this winter. One gotcha is that Google's not involved so mentoring orgs need to cover the $5k stipend.

I like the idea. Is this something we want to take part in? If so then I'd be happy to administer the non-financial parts of it.

Cheers! -Damian




Status Report for September 2012

October 4th, 2012

Hi all. As is often the case work and such meant not too much time for tor. For September all I have to report is...


  • Network Status Document Parsing

    This has been my main focus for September and it's still not finished... but it's close! Version 3 document parsing just has a couple days of work left, then abstracting it to cover v2 documents and microdescriptors should be relatively easy-ish. I'm really looking forward to merging this feature branch. It has grown quite monstrous...


  • Stem Documentation Hosting

    For a while now I've had a TODO item for making a nightly cron that built and hosted Stem's new sphinx documentation. I was about to do this when I recalled that meejah once recommended ReadTheDocs, a service that does... well, exactly that. After a few minor bumps (1, 2, 3) it's now live...

    https://stem.readthedocs.org/

    We definitely need to put effort into making them more reader friendly. At present it's just a dump of all the pydocs which, while informative, is actually a bit overwhelming for new users. Module summary pages would greatly help.


  • MAPADDRESS Support

    Ravi submitted a patch for adding MAPADDRESS support to stem's controller. It's a nice addition, especially the integ test.


  • Arm Issues

    Looked into a couple arm issues...


    • Tor's start time didn't show up if the system has proc contents but we fail to parse it (ticket).
    • Can't connect when using a control socket with password auth (ticket).

  • Updated Dev Wiki

    In response to a potential volunteer I wrote a summary of several development tasks. Updated stem's wiki with what was in that email.


Cheers! -Damian




Status Report for August 2012

September 3rd, 2012

Hi all. I spent most of August, like the prior month, traveling. This time I attended Toorcamp and went on vacation. Both were pleasant and relaxing, but not terribly conducive to coding so I don't have much to report...


  • Descriptor CSV Export Functionality

    Naif proposed export functionality for Tor server descriptors a while back, which Eric and Megan took the first stab at. I ended up revising this quite a bit, but it turned out nicely.


  • Caching Expansion and Test Prompt

    Added caching for GETCONF and static GETINFO queries. I also added a handy little script for a debugging interpretor. It kicks off Tor, then provides an interactive python prompt with a controller instance to it (optionally shutting Tor down afterward). Ravi code reviewed these changes and volunteered to do code reviews of my future work as well.


  • Consensus Parsing

    Over the last few weeks Ravi's been working on a patch to parse network status documents. It's functional, but missing unit tests and deviates from the parsing style and strict validation done for the other descriptor types so I'm taking a turn with the code. Thus far I'm done revising and adding tests for router status entries, and now working on the document. Changes are available in the 'document-parsing' branch of my repo.

    Parsing these documents is a far larger task than I thought (especially if you include v2 documents and microdescriptors), so working on this branch will probably keep me occupied for much of September.


  • Controller Expansion

    Ravi has gone on a hacking binge, adding support for USEFEATURE, SIGNAL, EXTENDCIRCUIT, and SETCIRCUITPURPOSE.


Cheers! -Damian




Status Report for July 2012

August 5th, 2012

Hi all. My July was mostly spent traveling, both for Defcon and a couple visits with my family on Vashon (for a funeral and Strawberry Festival). August will be much the same. I'll be gone...


  • 8/8 - 8/12 for Toorcamp
  • 8/17 - 9/3 for a family trip

Besides that and cat wrangling for the GSoC midterms, here's the things I did this month...

Cheers! -Damian




Trip Report for Defcon 2012

August 1st, 2012

Hi all. When tor developers go on trips we commonly write a trip report afterward. My recent trip to Defcon didn't contain anything sekrit so sending the notes I took here. If any of these presentations sound interesting to you then videos should be available later on the Defcon site.

Cheers! -Damian

PS. I was new to many of the things being presented so my notes quite likely have some inaccuracies. Don't quote me on anything - watch it for yourself if interested.

tl;dr
Only presentations related to the privacy field were...


  • Can You Track Me Now?

    Panel discussion on mobile GPS tracking with lots of interesting stats.


  • Crypto and the Cops

    EFF presentation on the 5th amendment protections and how it applies to compulsory decryption.


Day 1

Las Vegas is... an unusual city. Like most people I knew I wasn't in Kansas any more when I saw slot machines in the airport. By day the city's not much to look at. Mostly just a bunch of dusty buildings baking in the 110 degree desert heat. But by night the whole city illuminates in neon lights making it quite a spectacle.

My first night I explored the strip and several of the larger casinos including Bally's, Bellagio, Caesar's Palace, and the Rio where the conference was taking place. They were... enormous. Caesar's in particular took me almost a half hour simply to walk the length of it. Probably my favorite sight was the Bellagio fountains which had a magnificent water show choreographed to 'Singing in the Rain'. It was very classy.


Day 2

The second day was the first of the conference and mostly consisted of two things: standing in line a couple hours for registration and introductory talks. The talks were just a single track and included...


  • Wireless Security: Breaking Wireless Encryption Keys

    Handshake process for WPA, WEP, and WEP2 among other things. I was still sorting out the material that I got during registration so I didn't pay very close attention which was a pity since I could use a refresher.


  • Intro to Forensics: Tools & Tactics

    A very brief introduction to nmap, tcpdump, netcat, ntop, and Metasploit. Nothing new to me here though it was a nice reminder that I know less about them than I'd like. I haven't played with them since grad school and should try them again when I get home...


  • Cerebral Source Code

    Presentation on social engineering. Sadly missed a large part of this one since I still hadn't had breakfast. The QA portion that I saw was fun, mostly questions about how he'd extricate himself from various sticky situations. His answers were clever, as well as entertaining. My favorite...

    Q: What would it take to have more trust in this community?
    A: LSD?


  • DC101: Defcon Survival

    This was a panel discussion with general advice for enjoying Defcon. Mostly it boiled down to 'you get what you put in' which caught me a little off guard. I came here expecting to watch some talks, see Vegas, and schmooze with other conference attendees. Defcon, however, aims to be a lot more interactive than that and the organizers put a lot of effort into events that weave throughout the conference.


  • HF Skiddies suck, don't be one. Learn some basic Python.

    This one made me twitch. The speaker both gave a very bad tutorial for getting started with python (who the hell fires up *Eclipse* every time they want to write python?!?) and seemed to be incoherently drunk for most of it. Mostly I spent this time reading the program.


  • Hacking the Hackers: How Firm is Your Foundation

    This was a long checklist of various topics the speaker thought the Defcon audience should know about, largely focused on hardware hacking since that was his specialty. Interestingly 'TOR' was the first thing he listed. I always wondered how that spelling persists despite our efforts to stamp it out. Considering that this conference had roughly 20,000 attendees, each of whom will pass it along I guess now I know...


  • Fun with the Lockpicking Village

    A few years back I read the MIT guide to lockpicking and spent a couple days practicing against my door. Since then I've fallen out of practice and this was a nice reminder about both the basics and a few things that I didn't know (how bump keys work, defeating deadlatches, how to use a can of compressed air to defeat magnetic locks that allow egress traffic, etc).


The hallway track wasn't anything too notable. Met three developers with Metasploit and someone from the security group I attended at WSU. But that was about it.


Day 3

Second day of the conference I ran into someone from Amazon Infosec. A talk I wanted to attend concerning the attack surface of near field communication (NFC) devices was full, but several of the others were interesting...


  • Can You Track Me Now?

    This was my favorite presentation of the whole conference. A panel of four presenters, including Chris Soghoian, discussed geolocational privacy of mobile devices including local threats, historical information, and requests for real time tracking.

    Why does this matter, you ask? Because when it takes a team of agents to track your every step surveillance is costly. But when a single agent can track you and your every acquaintance from the comfort of his desk we get a whole lot closer to a 1984 style surveillance state real fast.

    Sprint now has a department of 200 people to service wiretapping requests. They also provide a self-service portal to law enforcement that has been used six to seven hundred thousand times a year, for 8 million GPS coordinates. This includes your every move going back 6 months to 2 years depending on your provider. 7 years in the case of AT&T.

    And no, none of this requires probable cause or a warrant. In other words, beware if your ex is a cop.

    Another eye opening bit of the talk was phone encryption. Apple has a master skeleton key for all iPhone devices so, when requested, they can send law enforcement a DVD with all the contents of your phone. Android is a little different in that they remotely reset your password to something that they can give the law enforcement agent.

    Beside three letter agencies, ad networks and apps are also a concern. 47 of the top 100 mobile applications send locational information to 3rd parties. For instance, 'Best Alarm Clock' sends locational information in the clear to 3rd parties whenever the app is used (somehow I doubt they need to do that for the alarm clock to function...).

    On a policy front one slight glimmer of hope comes from a recent case which ruled that placing a GPS tracking device on your car *does* constitute a search and hence requires a warrant (which means they kinda sorta need evidence of wrongdoing, *gasp*!). Don't read too much into this, though. The majority opinion ruled this way because it technically constituted trespassing on your property.

    Just because you're paranoid doesn't mean they're *not* out to get you...


  • Crypto and the Cops

    Marcia Hofmann from the EFF discussed 5th amendment protections and how they apply to being compelled to reveal your passphrase for an encrypted device. There have been five cases in this area so far, some that ruled each way in terms of compulsory decryption. Though not intuitive, the main determining factor in those cases constituted a gap in knowledge. If law enforcement knows exactly what they're looking for and where it is then there's less chance of the amendment applying than if the search is a fishing expedition. Moral of the presentation: never volunteer information, even if you're innocent. Ask for a lawyer first.


  • Making Sense of Static

    Walkthrough of GPS signal acquisition, mostly focusing on how receivers account for Doppler shift and determine the code phase. The speakers also presented libswiftnav, an open source implementation of a GPS receiver system.


  • Life Inside the Skinner Box

    Cautionary presentation about the automation of law enforcement and where it might someday lead. This was pretty high level, discussing the societal implications.


  • Anti-Forensics and Anti-Anti-Forensics

    Discussion by a forensic examiner of several tricks that can draw out the length of an investigation (and by extension cost, increasing the chance of settlement). He also covered how they can be mitigated. Some of them included...

    • using a non-standard RAID configuration
    • randomizing file crated/modified timestamps
    • altering file header types
    • using restricted windows filenames (CON, PRN, AUX, etc)
    • owning lots 'o media
    • modifying files provided with your system so the checksums no longer match what a default install would include (and hence adds to the haystack of things to sift through)

Before heading back for dinner I also caught part of a Skytalk. I didn't find the presentation to be too interesting (brainstorming about mass data aggregation), but I got a chuckle from the sign on the door: "Absolutely no cameras allowed. Violaters will be violated violently." :P


Day 4

I had less luck on the third day of the conference, hopping between several talks that weren't particularly interesting. The best I found were...


  • World War 3.0 - Chaos, Control, the Battle of the Net

    Policy talk about Dubai's upcoming WCIT negotiations (aka, the thing that's gonna politicize the Internet). Most of the presentation was surprisingly fluffy, the only interesting bit coming from Dan Kaminsky who made the point that there's generally four interests involved...

    • privacy
    • security
    • anti-piracy
    • sovereignty

    ... with 'reliability' being the golden goose that no one wants to cook. His point was simply that these four camps war and forge alliances, privacy and security for instance sometimes at odds over anonymity but thick as thieves on other obvious fronts. He also said something like "We have made a system optimized for moving pictures of cats. And it's *really* good at it. Though we'd like it to do more." (likely horribly misquoted)


  • Exploit Archeology: Raiders of the Lost Payphone

    Step by step process of reverse engineering and reprogramming an Exetel payphone. Twenty year old software from a company that imploded without a trace long, long ago. Ye gods each step turned out to be a trek to Mt. Doom and back...


  • Off Grid Communication with Android

    Mesh network by introducing a transparent proxy after going through the Java networking interface. He wrote something based on the 'Wi-Fi Tether for Root Users' app to put his phones into ad-hoc mode then tried both the OLSR and BATMAN proactive routing protocols. He also tried a reactive protocol that broadcasts messages, then left routing decisions to the receiver which can trivially pick the shortest path though this had issues scaling above 200 devices or so.


  • Back Ops

    My favorite talk of the day, given by Dan Kaminsky. It was originally slated for two hours, but got crunched at the last minute into a one hour slot so he went pretty fast over five different topics including...


    • Timing Attacks

      Attacks can distinguish 15-100 ms deltas over the Internet and 100 nanoseconds over a LAN. Making everything take an identical amount of time is generally unrealistic since it forces everything to worst case performance. His view was not to let the perfect be the enemy of the good, and simply introduce random delay into requests.


    • Bad Random Number Generation

      First a rant that for twenty years security engineers have worried about entropy sources, yet we still haven't gotten it right. The common sources of entropy are: hardware RNG, keyboards, mouse, and disk rotation. However... most servers don't have any of these, especially with the move to virtual storage devices. Roughly 1/200 keys on the Internet are poorly generated, and he proposed using idiosyncrasies in clock timing as an alternative source of randomness.


    • Security Aspects of Languages

      Starting with the question 'why is PHP so damn popular?', he discussed why secure coding practices like prepared statements often aren't followed. His conclusion: SDEs are in charge and they just want their service to work. The more of a pita security makes itself the more it'll be viewed as an obstruction and hence circumvented. He then proposed more usable security patterns, for instance making the languages smart enough to translate queries like "SELECT name FROM users WHERE username=$id" into the proper prepared statement counterpart.


    • Censorship Detection

      After a brief mention of OONI-Probe he discussed a censorship detection method of his via the use of favicons. I saw this in an earlier presentation of his and he mostly skimmed through it.


    • Vulnerability Scanning

      Trying to answer the question of 'how much of the Internet is vulnerable to X', he used *stateless* TCP connections to scan hosts far quicker than he usually could. The trick was to skip retaining any information about outbound connections, making fire-and-forget TCP handshakes and leaving it to the other end to remember the state of the connection.


Day 5

I didn't expect much from the lineup on the last day, but some of them were surprisingly good. I also ran into a couple more people from Amazon and one from Tor.


  • SIGINT and Traffic Analysis for the Rest of Us

    Very good talk on APCO Project 25 (P25), two way radio communication used by police, fire departments, federal enforcement, the secret service, and the DoD. A drop-in replacement for analog FM, this was first introduced in the early 1990s. It can optionally be encrypted, with federal/DoD users commonly enabling it and police/fire leaving it off. If you see an earpiece or bulky walki-talky then this is what it's probably using.

    The speaker discussed several vulnerabilities in P25. Firstly, it uses a forward correcting protocol that is easy to jam. Usually a jammer needs to be stronger than the signal it's jamming, but with P25 it's sufficient to selectively block a message's header to cause the rest of the message to be ignored. How hard is this, you ask? To cheaply jam a P25 signal you just need a $15 GirlTech IMME toy (it cost less to buy the toy than the transmitter that comes in it). As an added bonus it comes in pink or purple and comes with ponies on the box!

    Second, P25 has no notion of authentication. Anyone can transmit and user ids can be trivially spoofed. The user id of transmitters are sent in the clear, even when encryption is enabled.

    Third, any P25 device that receives a malformed signal will reply with the user id (again, in cleartext). This is invisible to the user and means that you can ping all P25 devices and triangulate their positions. Think of it as the "Marauder's Map" for the secret service.

    Fourth, P25 is a narrow band radio broadcast (12.5 Khz) which is fairly easy to intercept via a scanner (such as an Icom R-2500). This makes encryption a must for private communications. However, the encryption of P25 devices has several usability issues...


    • Encryption is enabled or disabled via a switch on the handset. The only indication of if encryption is enabled or not is a "1" or "(/)" symbol on the handset. Which means 'encrypted' and which means 'cleartext' is evidently a common point of confusion.

    • The beep that the device makes to indicate that it's in encrypted mode is the same that's used to indicate a low battery among other things.

    • Encryption is transparent to the receiver, so if an individual in a group is broadcasting in cleartext then it's impossible for anyone else in the group to know and correct them.

    • Rekeyed P25 devices become incompatible while being rekeyed, forcing users to drop to cleartext if any of the devices haven't yet been updated.

    As a result, about 30 minutes per day of federal communications are sent in the clear. In a multi-city study on what this included researchers were able to gather the names of informants, license places of vehicles being tailed, etc. These snippets were also enough to reasonably figure out what investigations were going on at a given time.

    When brought up with the federal field offices encryption rates improved for a week or so, then dropped back to their prior ratio of cleartext. The researchers also discussed these with the manufacturers, who insisted that it wasn't their fault if their users couldn't properly use their devices.

    Btw, the only federal service to have a spotless record in terms of encrypting their communications? The US Postal Service.


  • SCADA Strangelove or: How I Learned to Start Worrying and Love the Nuclear Plants

    This was a talk on the security of SCADA HMI software (or lack thereof), which is used throughout several minor things like nuclear power plants. Originally to prevent plant operators from installing pong on the kiosks, HMI applications hasn't evolved much since those humble origins. This talk was funny as hell, going over issues with several widely deployed HMI systems...


    • Microsoft Bob: Early attempt at a friendly, 'helpful' interface. If you made three incorrect password attempts you got a dialog saying "I see you forgot your password. Please enter a new one here...". It also stores passwords in plaintext.

    • InvisiLink: Stored passwords are an xor with a static key. What is that key, you ask? "0123456789"

    • KingView: Passwords simply have their last byte subtracted from 0xff.

    • Iconies Genesis32 9.2.2: The login dialog includes a 'challenge code' that can be used as an alternative method of getting local admin access. To use this code the user is supposed to call customer service, who then provides the password corresponding with the code. Alternatively, you can take the first eight characters of the challenge code's md4 hash to do the same.

There were also a couple talks on 'firesheep inspired' frontends for various exploits. The first did NTLM relaying against Windows corporate networks to allow for a variety of not-so-nice things like reading through another person's Exchange inbox.

The second was a UI to automatically ARP poison a network then use sslstrip to snag credentials. I felt a little embarrassed for the speakers in this talk since they first tried to sensationalize what sslstrip could do. After that they claimed to have a stealthy MITM, then described the noisiest attack I can think of. Not only did they ARP poison the network and do SSL downgrading, but they did an nmap scan of every host on the network and, just for added stealthyness, blocked all tunneling protocols (ssh/vpn). The later was on the theory that "they'll just use the local network instead". Uggg.

So more annoying push-button solutions for script kiddies. Yay.


Ending Conclusions

Fun and interesting conference, but *damn* vegas is pricey. The trip was 1.5k and only $200 of that was the conference registration. I enjoyed it and it was a good thing to do at least once, but way too costly to do again.




Status Report for June 2012

July 3rd, 2012

Hi all. This June I exchanged my developer hat to be a mentor instead, spending more time reviewing code than writing it myself. Fingers crossed that at least one or two of them stick around after the summer ends!

As such, this status report is more about other people than me. Apologies if I miss anything.


  • Ravi (GSoC Student)

    This month Ravi discovered a bug with Tor's GETCONF method and wrote numerous features including SAFECOOKIE support and a GETCONF method for the controller. The GETCONF handling is complicated by the accursed HiddenServiceOptions so it has needed several iterations, but I plan to merge it this week. After that Ravi already has two more feature branches lined up for me to review (*sob*)...


  • Beck (Volunteer)

    Somehow I've never been able to bring myself to do development on Windows (if you haven't seen Neal Stephenson's Hole Hawg article then I recommend it). Fortunately Beck does, and has done a fantastic job of fixing stem and its tests to work there (1, 2). He also added get_version(), authenticate(), and protocolinfo() methods to the controller.


  • Erik and Megan (Wesleyan Students)

    Erik and Megan have been focusing on stem's tests, first submitting a couple fixes for the mocking module (1, 2) then writing unit and integration tests for the proc utilities. Next they plan to implement the CSV export functionality suggested by naif.


  • Sathyanarayanan (Volunteer)

    Though work has kept him pretty occupied, Sathyanarayanan has been working on an ExitPolicy class and is presently at the dev meeting making plans to implement a python based Onionoo.


  • Karsten

    Though he isn't hacking on stem itself, Karsten has been helping by reviewing its descriptor handling and suggesting improvements (1, 2).


Besides those, I implemented a few stem improvements too this month...


  • Sphinx Documentation

    At the start of June I rewrote stem's documentation into reStructuredText so it could be compiled by Sphinx. The results are very pretty...


  • Python 2.5 Compatibility

    Stem aims to support python versions 2.5 and above (in the 2.x series). However, most of our development has been on 2.7, letting backward incompatible changes slip in inadvertently. This ended up being a two week bug fixing odyssey, but now that it's done stem and its tests should now work if users want an interpretor that harks back to 2006...


  • Test Freezing Issue on Mac OSX

    Both Sathyanarayanan and Karsten reported that stem's integ tests freeze on Mac OSX. After a dozen hours of hair pulling I've narrowed it down to an issue where control sockets are left in a CLOSE_WAIT state when closed, and eventually we lose the ability to make new control sockets...
    https://trac.torproject.org/6235

    From what I can tell this is either an issue with Tor, Python, or Mac OSX (my money's on the last). Help welcome if anyone has ideas.


Other random things from this month include going to the Fremont Fair, attending a SEAPIG meeting (local python developer group) and making travel arrangements to attend Defcon later in July.

Cheers! -Damian




Status Report for May 2012

June 1st, 2012

Hi all. Spring is in the air, and with it many lovely things including the UW Street Fair, Folklife, and an iSec Open Forum. It also included a week of oncall but that doesn't count in the 'lovely things' column. On the upside though, this time it was pleasantly light (first time I've gone a whole week without being woken up by a pager at 3am!).

As a GSoC admin I don't have much to report besides a blog posting at the start of the month. However, as a mentor some neat things are in the works...


  • Ravi is adding SafeCookie support in stem. I've code reviewed the first couple iterations and it's looking great. Given some tests and revising to fit with recent refactoring it should soon be ready to merge.

    After that he'll start working on the general controller. This last weekend I refactored how response classes are organized and implemented GETINFO as an example, so this will hopefully be an easy project for him to start hacking on.


  • Beck added descriptor validation to check that the signing key's hash matches its fingerprint. This is the first part of descriptor integrity validation that Karsten suggested, however this project has gotten stuck so he's moving on to other stem tasks.

  • Investigated a couple issues brought up by Sathyanarayanan. (1, 2)

  • Two Wesleyan students will start working on stem soon!

Needless to say helping these projects has occupies much of my time, and will take even more once the Wesleyan students get started. But after years of trying in vain to attract developers to my projects I wouldn't have it any other way. :)

Other stem tasks I finished this month includes...


  • ExtraInfo descriptor parsing. This took me a couple weeks since it contains so many attributes, but I'm glad it's finished. Now all that remains before we can port Onionoo are the consensus' network status entries. That's now at the top of my todo list for descriptor work, but I'm setting it aside for now in favor of Sphinx and helping Ravi and Beck with controller work.

  • Improved the launch_tor() function, making the test instance easily configurable via a temporary torrc (similar to what Vidalia does) and adding integ tests. I also added a workaround so it'll work on Windows. (1, 2)

  • Updated the stem development wiki so it'll be easier for new volunteers to find tasks that interest them.

  • Discussed descriptor type annotations with Karsten and implemented stem's side of it. We also discussed some changes to bridge descriptor sanatization which led to some minor stem changes too.

Cheers! -Damian




Status Report for April 2012

May 2nd, 2012

Ducks are awesome. Especially cute Indian Runners who waddle about upright like penguins. Besides this startling discovery, in April I wrote some code, released some code, and did a bunch of GSoC mentor/administration work.

Stem had a reasonably good month. Work included...


  • Finished and merged all of the outstanding todo items from my discussions with Karsten (merge diff). This included additional testing, support for bridge descriptors, re-discovering a tor bug that caused negative uptimes, and a variety of other things. I'm keeping an eye on metrics-lib tickets as they roll in so stem can improve from the issues that Karsten discovers (example), and discussion is ongoing about the addition of a descriptor header field.

  • Discussed stem's copyright with Wendy and others. We now have a plan for contributors that'll allow us to reuse stem code under other licenses when needed.

  • Ongoing discussions with Beck about potential stem projects. He sounds eager to work with Ravi and me on the general controller.

  • The whitespace conventions of my projects drive you guys nuts just as badly as yours annoy me. This was gonna be a continued pain point for accepting contributions from others so I've added a whitespace checker to stem's tests that'll yell at people if they start doing something funky.

  • Stole a trick from git and dropped the '--no-color' argument from the test runner in favor of autodetecting if stdout is a tty terminal or not. This means that test output to your console will have pretty colors, but the ANSI escape sequences will be omitted if you're piping the output to something else (less, a file, etc).

  • Merged a chroot testing target so we can ensure that stem plays nicely with those environments. This is a use case that traditionally causes problems for our controllers since they don't account for a path prefix in cookie authentication, determining the data directory, etc.

  • More intermittent concurrency woes. Hopefully I fixed it for realz this time!

Arm also got some love, including a couple important fixes which were released in version 1.4.5...


  • Fix for unrecognized authentication methods. I also filed a ticket with the fix for TorCtl which got a less-than-heartwarming thanks.

  • Added a notice when ptrace is disabled, which by extension causes some proc contents to only be readable by root (breaking arm's connection panel). Only Jake has expressed an opinion that this is a good feature to have, but others don't seem interested in discussing it so guess it's something that I'll just need to work around. The message tells users how to disable the feature and cites the ticket if they want to know more.

  • Helped arm users including Eric, LoneWolf, and MoPac.

Finally, I spent a good chunk of this month cat herding for GSoC. We survived the student selection process (yay!) and for the moment at least things are proceeding smoothly. Thanks to Sebastian for leading the student selection meeting and covering for the #gsoc deduplication discussions.

Cheers! -Damian

PS. The aforementioned ducks are in reference to an email thread dispensing free flightless avian waterfowl to the masses. Alas though, I couldn't get any since my small apartment isn't especially duck-friendly. *sob*




Status Report for March 2012

April 5th, 2012

I never cease to be amazed at how fast a month sweeps by. This March I fell in love, fell out of love, got ill with a horrible stomach bug, and wrote a bunch of python code. My favorite was the last one and this is just about that.

My time developing stem this month was almost entirely dedicated to writing a python counterpart for metrics-lib. Most of the effort here went into reader concurrency, server descriptor validation, and lots of testing. For my part this project has the following goals...


  • provide the server descriptor, network status, and microdescriptor parsing needed by the controller

  • validate that new tor versions comply with the spec and don't break our parsing

  • replace the java metrics-lib so we have a single codebase with multiple maintainers (in other words, persuade Karsten to hack on stem)

  • allow applications that just need descriptor data (such as the consensus tracker script) to use cached descriptor data so they don't require an open control port

At present stem's implementation just handles server descriptors. A lot more work will be needed to cover the rest of what metrics-lib does.


In other stem news Ravi Padmala, a contributor to several of our projects and a GSoC applicant, made multiple fixes to stem's version parsing. I never cease to be amazed at how error prone something that sounds as simple as 'parse the tor version' can be. Guess that's why we're writing a library...

Sathyanarayanan also took the first stab at porting arm's ExitPolicy class to stem, though more work is still needed there.

Besides stem, roughly an equal amount of my time has gone into this year's GSoC (for anyone living under a rock we were accepted, yay!). With my org admin hat on I revised our application, made lots 'o revisions to our volunteer page, and helped to respond to general GSoC inquiries.

With my mentor hat on I reviewed Ravi's proposal and decided afterward that I dislike the fire-and-forget approach that we usually take with GSoC projects. We give students isolated projects where they can work independently because it is less work for us. On occasion we end the summer with a new core contributor or something we can use, but in general it fails on both counts. I want to try something a little different this year and actually work on the tasks with my applicant. Maybe it'll work, maybe it'll drive them mad. We'll see...

Other random things that I did this month included...


  • Attending local presentations by Bruce Schneier and Dan Kaminsky.

  • Looking over meejah's txtorcon, a python controller lib using twisted. It's impressive that he got this up so quickly and it's neat to see what a twisted implementation looks like. However, it is missing large and very basic controller functionality (such as parsing controller replies), and what parsing it does do is hacky (if "COOKIE" in protocolinfo_reply: do cookie auth which will obviously fail with replies like 'SAFECOOKIE'). With work though this could be a nice alternative implementation. Meejah is obviously very capable and it'll be interesting to see where he goes with it. (Correction: mistake on my part, txtorcon actually does have parsing for GETINFO and GETCONF responses)

  • Discussed with Norman and Karsten the possibility of Weslayan students working with us again this year on either Stem or Onionoo. It will be a smaller scope than last year's project if it materializes (just a couple students) which in my opinion is a good thing. Large groups are hard to manage.

Cheers! -Damian




Status Report for February 2012

March 2nd, 2012

Oh, how todo lists never seem to get any shorter.

My first half of February was focused on stem development, most importantly the implementation and testing of the BaseController class. This is the foundation on which useful controller activity can be based, providing a parallel to TorCtl's asynchronous controller communication (event handling) and sendAndRecv function. Good news is that the BaseController is also designed to be thread safe. Bad news is that getting the deadlocks worked out was a pain in the ass and consumed well over a week. Sometimes concurrency is hard. >:(

Other stem development included...


  • Simulated chroot setups for integration testing (ticket). This hasn't yet been merged because I haven't added a method for users to provide their chroot prefixes (and hence these integ tests for things like cookie authentication rightfully fail). Not hard, just haven't gotten to it yet.

  • Gave some input on Robert's Safe Cookie proposal and filed a ticket for supporting it in stem. Sathyanarayanan has offered to take the first pass at implementing it.

  • Discussions with people helping to make stem better. Sathyanarayanan put the finishing touches on configuration saving and Neena fixed an integration testing bug. Many thanks!

Later in the month I began making the arduous trek (half hour walk) to the Tor developer meeting. For everyone who could attend it was great to see you again! Highlights for me included...


  • Demoed stem to several people and schemed about its future plans.

  • Discussed a python metrics-lib with Karsten. Making a skeleton for that now holds the top slot on my dance card when I finally get some time to do development again.

  • Talked with potential mentors about ideas for GSoC. There was quite a bit of interest but not any concrete plans at the time.

  • Discussed the burden of proof needed for badexiting and resolved a ticket we had for an automated exit setup we've been seeing.

  • Brainstormed alternative names for the third incarnation of TorStatus with Karsten and Arturo. In the end we went with "Atlas". I later filed tickets (1, 2, 3) to move it and Onionoo to tor's infrastructure (tpo vm, git repos, trac, etc).

  • Talked with Runa and Karsten about the monitoring infrastructure project. It won't be a GSoC project, but rather something that Runa plans to hack on later.

  • Organized for us to go on the Underground Tour. Note to future self: leaving with twice as much transit time as you need doesn't work. Quadruple it.

Sadly as the month went on I've shifted more and more from development to helping others. Of late the little time I have has gone toward GSoC preparation...


  • Revised our GSoC landing page, rewriting a few of the sections.

  • Nagged lots of people for project ideas and added them to the ideas page.

  • Added a project idea for stem.

  • Dug up our application from last year. With only a few minor tweaks it should still be fine.

  • Discussions about if we'll be filing a joint application with the EFF again or not. Conclusion was that it probably isn't as vital to our acceptance as we once believed, but we're still gonna do it because we like the EFF and they've pinky promised to communicate better this year.

... and then of course there were other things...


  • Code reviewed Karsten's script for gathering obfsproxy statistics (ticket).

  • Several volunteer page changes, like adding Obfsproxy, Ooniprobe, and Shadow.

  • Discussions about trac with proper. On one hand I'm glad that he's trying to help, but on the other I feel like there's a growing need for us to include a banner on our wiki warning that it's community maintained. With our logo and the official domain there's a sizable risk that visitors won't realize that we don't review several of the pages at all.

  • Thanks to Sebastian for fixing an arm bug where reading tor logs from February 29th on leap years would crash arm. This problem was reported by dozens of people, which is actually really heart warming. :)

While I'd like to get back to my own coding, I doubt that these distractions will subside much any time soon. C'est la vie, I shouldn't complain - it's all good stuff.

Cheers! -Damian




Status Report for January 2012

February 2nd, 2012

Hi all. Performance reviews and oncall kept me occupied for much of January, and Megan had dibs on most of what remained. I'm still hacking on stem but progress isn't as fast as I'd like. C'est la vie.

Stem's development in January mostly focused on...


  • Writing a proper mocking module and refactoring the tests to use it. This will greatly improve the maintainability and ease of writing new tests going forward. Originally this began with the humble goal of 'remove a built-in mocking hack from the system module', then went down the rabbit hole of larger scale testing improvements. Still, I'm happy with the results.

  • Sathyanarayanan took on development tasks including integration tests for chroot setups, saving configurations, and troubleshooting test failures on OSX. Design discussions and code reviews take a fair bit of time but I'm thrilled to finally have someone to hack on the codebase with me. A couple other potential volunteers (piffey and blackpaw) showed interest but have since disappeared.

  • A large part of my discussions with Sathyanarayanan centered around making stem more developer friendly, both in terms of its utility APIs and easier collaboration. As it turns out keeping stem's todo list in a text file on my netbook is not the most optimal location for other people. I've since moved it to a development wiki.

  • Expansion of the configuration utility. The most notable changes include multi-line configuration options and moving to a listener architecture. The former lets us move user facing strings out of the source (good if we ever translate) and the later greatly simplifies usage of this utility. It'll also allow for runtime configuration editability later.

  • Additional options for running stem's tests...
    • '--tor' - Runs integration tests against a given tor binary (obviously needed to test during tor development).
    • '--no-color' - Removes ANSI escape sequence formatting which is preferable when piping test output.
    • '--log' - Makes stem provide its logging output with the test results. Hopefully by making log messages more visible during development we'll get better, more user friendly logging for stem's users. Actually, I've already rewritten most of stem's log messages because of this option...

Non-development things I did include...


  • Sent tor posters to international people. The pile of customs slips was pesky, but worse was twiddling my fingers at the post office as they typed each form in one by one. Hunt and peck is not the fastest method for data entry...

  • The consensus tracker script had a couple interesting finds this month. The first was an oddly configured exit from the University of Waterloo and the second was 41 exits with what looks to be an auto-generated configuration.

  • Realized that my git-fu wasn't up to par for some of the things we're doing at work, so I read 'Git from the Bottom Up'. If you've ever been curious about git's internal data model then this is the article for you. It's short and gives a very well written overview starting with git's most basic components (blobs) and building up from that. I've heard that Pro Git is also good so I might skim some of that next.

Looking forward to seeing most of you at the development meeting! -Damian




Status Report for December 2011

January 2nd, 2012

Hi all. Between being ill, oncall for work, visiting with family over the holidays, and finally meeting a brilliant, wonderful girl named Megan I didn't accomplish too much in December. This isn't likely to change any time soon so the projects I maintain may get a little less attention - such a pity. ;)

As normal I've mostly been split between developing stem and maintaining arm. Ideally I'd like to sink all my time into the former but arm had several issues that demanded attention this month...


  • The ptrace change from ticket 3313 caused tor's file descriptors to only be readable by root, breaking lsof, netstat, sockstat, ss, and some of arm's proc based utilities. This does not break connection resolution itself, but rather the file descriptor to inode mappings used to associate connections to processes. Thanks to Jake we have a partial workaround which is to filter netstat results by the owner's uid instead which, at least for the tor deb, should give the same results. For non-deb users I'll need to just give a notice about why it's broken or, in the case of Ubuntu users, suggest that they turn off 'DisableDebuggerAttachment'.

  • The latest arm release sometimes exhibits strange terminal glitches, caused by an interaction between the readline and curses modules. Thanks to Stephan Seitz for providing a reliable method for reproducing the issue.

  • Kamran submitted a patch for UPnP support in arm which I spent a couple days reviewing. It's a nice addition, though it's gonna need some work before it's merged.

  • Tor Cloud use cases revealed a couple bugs with arm's torrc validation including case sensitivity and an unexpected logging default for Debian. Thanks to koolfy for reporting the issues and Runa for the test system.

  • Saving a snapshot of the log in arm had a couple issues, as reported by Jeff Bonner on the Debian bug tracker.

  • Non-proc based connection resolution could fail due to terminal localization and an issue with the getProcessName() function. Issues caught thanks to Stephan Seitz.

The little time I've had for stem has mostly gone into improving and testing connection and authentication to the tor process. This module took a lot longer than I'd intended to finish but I'm really happy with the result. Also, both Sathyanarayanan and boerni have taken an interest in stem, making me a little hopeful that developing this library won't be as lonely as arm was.

Cheers! -Damian




Status Report for November 2011

December 5th, 2011

Hi all. I spent November chiefly focusing on stem, shoring up its testing and handling the connection/authentication handshake. The scope of the library is expanding very slowly but I'd argue that this is a good thing. Stem has almost compete code coverage with its tests exercising most use cases and edge situations I can think of. For instance, connection and authentication are run against configurations with...

  • an open control port
  • password authentication
  • cookie authentication
  • both password and cookie authentication
  • control socket
  • control socket with cookie authentication
  • no method for controllers to connect

and soon I'll be adding tests for chroot setups (a use case where our projects traditionally have a lot of issues). If you're making Tor changes that touch on PROTOCOLINFO or AUTHENTICATE then please run stem's integ suite. It's quick, it's easy, and it'll give your change a very, very good workout. To run it simply...

git clone git://git.torproject.org/stem.git
./stem/run_tests.py --integ --target CONN_ALL

Besides stem I've been involved with a smattering of other issues...


  • Reviewed Filiol's slides which, between the FUD, had a few reasonable concerns. This mostly concerned better monitoring so I filed a ticket which probably won't get much traction until the next GSoC.

  • Discussed a library for fetching consensus information directly from a variety of sources like cached descriptors and directory authorities/mirrors (ticket). I'll be implementing this functionality in stem and Karsten plans to do the same with a java library.

  • Discussed a Tor forum (ticket). Imho it will be doomed without Tor dev involvement and, since realistically we'll give up on clicking through clunky web interfaces, we should have an email frontend too. Andrew disagrees so I'm taking my hands off of that project.

  • Variety of small arm issues including ASC mishandling, torrc validation issues spotted by kookfy, and fixing an rpm dependency issue spotted by unspawn.

  • Packaged the Tor posters and sent the ones to people in the US. I've had my fill of gigantic post office lines so the rest (and the stack of customs slips that go with 'em) will be waiting until after the holidays.

  • Filed tickets (4629, 4630, 4627, and 3958) for TorCtl issues I've spotted while developing stem. I probably won't keep doing this - it's time consuming and pointless if stem replaces TorCtl. The last issue might also exist with Vidalia though Tomás hasn't commented yet.

  • Though not really Tor related, I submitted a patch to ReviewBoard for a weird XSS issue via comment fields. This next GSoC I'd like to set up a ReviewBoard instance for us. While I realize that shiny, ajax websites aren't our style it would make code reviewing a lot nicer. code review, commit

All in all a good month. Cheers! -Damian




Status Report for October 2011

November 1st, 2011

Between the normal October hubbub of baking peanut butter cookies and Halloween (the spiffiest US holiday, imho) I've been hacking a fair bit on our python projects.

Arm is now in maintenance mode, but has been getting plenty of love...

  • Thanks to Carlo Strub arm now has a FreeBSD port!
  • Sebastian and Robert spotted a couple substantial issues ([1], [2]), now fixed
  • Jordi Espasa Clofent generously lent me an OpenBSD vm for arm testing. I fixed the issues that I could ([1], [2], [3]), but there's still a couple bad ones outstanding...
    • The control connection gets intermittent interrupt signals while arm starts. This one has me completely stumped. Wherever this fun-loving gremlin lives it's deeper than I'd care to go (maybe a vm issue, OpenBSD quirk, or it's just a conscientious objector of localhost socket connection - who knows).
    • The uptime attribute for OpenBSD's variant of ps is... er, difficult to parse. It's in local time, has am/pm rather than being 24-hour time, and the whole format changes based on if the uptime is over a day or not. This whole platform has been scientifically designed to get on my nerves...

My main focus, however, has been on Stem. I've finished the ControlMessage class, a counterpart for TorCtl's core sendAndRecv functionality which handles the base control protocol message parsing. From here it'll be easy to implement counterparts for most of TorCtl's functions (get_info, get/set_conf, etc), but that's not really a high priority. Only a small fraction of my time has been spent working on the stem library - much more has been spent on the documentation and unit/integration testing which is what'll give this library its worth. Besides being developer friendly and well tested, this will let us check when cutting new Tor releases if its changes will cause issues for stem's users or not. I've also submitted a TorCtl change to take advantage of this but it's looking kinda unlike that will happen.

At present the stem integration tests are a good basic verification test for Tor's controller functionality, and will become better as I expand stem. If we become interested in testing for Tor then this will also give a very good starting point for writing those. However, while I'm happy to help with Tor testing I'm also tired of working alone on things that only I care about. If we expand testing to focus more on Tor then someone else will need to take a lead there.

Besides development, I did a code review for Tom's torperf changes and attended the GSoC Mentor Summit where I met Mitar Milutinovic, David Fifield, the Umit developers, and took part in a counter-censorship discussion. We should follow up with Rodolfo Carvalho who's developing Open Monitor (they have a skill set we lack and vice versa), but that's up to others. From irc it sounds like we're too overloaded right now to mentor for Google Code In - pity but maybe next year.

All in all a great month. Cheers! -Damian




Status Report for September 2011

October 1st, 2011

Hi all. For my part September was spent working toward the 1.4.4 arm release which tidies up the rough edges of the prior version and adds the control port interpretor (a pretty spiffy feature, imho). This feature was made with people either learning or commonly using the control port in mind so I'd really appreciate more feedback. Unfortunately Sebastian is the only dev so far to give it a try. :(

Besides that and contrary to my usual avoidance of fellow human beings, I've worked with others on a few things...

  • Roger and others spotted an issue in Vidalia and TorCtl's handling of control port authentication when there's multiple authentication methods being used. I fixed this for TorCtl and arm.
  • Juan Alcaine is helping with the arm RPMs, providing much needed testing and splitting arm from its dependencies. Next step is to get help from Erinn for uploading the arm/torctl rpms to the deb.tpo repos.
  • Kamran has been working on a patch for exit locale selection in arm. It's functional, but not quite done yet (I'm looking forward to seeing his finished version!).
  • Met with Alasdair Young, another Seattleite who's interested in checking Pidgin for leaks and hacking on MAT. I should get drinks with him again in a few weeks...
  • Helped Rob Jansen a bit with a curses setup wizard for Shadow.

The 1.4.4 release is the finishing point for major arm development (for realz this time!), and next I'll be shifting my focus to Stem (a fork of TorCtl) and our shiny, new django TorStatus site.

Cheers! -Damian




Status Report for August 2011

August 31st, 2011

"Yup, all done hacking on arm!" I told myself. I'm such a liar. My August was mostly spent adding features that didn't make it into the 1.4.3 release. In particular...


  • a dialog with stats for exiting port usage (for exits) and client locales (for guards and bridges)
  • control socket support
  • torctl event parsing rewrite
  • descriptor dialog rewrite
  • expanded the projects listed on the tor front page

Google Summer of Code finished last week with all students passing. For me the real question of how we did will be answered in another few weeks when we discover which students stay and which evaporate. Most have expressed an interest in staying so that's a good sign.

Finally, I've spent this last week writing a control port interpretor. Its purpose is to provide raw control port access (like a telnet session with the control port) but with usability improvements. In particular...


  • auto-negotiate authentication
  • tab completion for valid controller commands (which are fetched from the attached tor instance via the 'GETINFO */names' options)
  • up/down cycles through the history and ctrl+r provides history auto-completion
  • * nice formatting for the responses (context specific color/bolding)
  • * support for mutli-line controller commands and event listening
  • irc style interpretor commands...
    • /write [PATH] - saves interpretor backlog to the given path (PATH defaults to the last used location)
    • /find PATTERN - regex search through the backlog, highlighting matches
    • /quit - I'll let you guess
    • * /help [OPTION] - provides usage information for both interpretor and tor commands
    • * /window [0-9] - switches between workspaces (like multiple telnet connections in screen sessions)
    • * /info RELAY - dumps consensus/descriptor entries for a relay by fingerprint or nickname (see the arm descriptor dialog for what this'll look like)
* these are the todo items, everything else is done - ideas welcome for other features, especially if it'll make your life easier!

This interpretor can both be a terminal prompt (by running "arm --prompt" or "arm -p"):

Interpretor Prompt

or used from the arm interface:

Interpretor Panel

They work from the same backend, but the curses/getstr vs prompt/readline frontends provide different capabilities...


  • Only the prompt provides line wrapping. I haven't decided if I'll do this in the panel or not since it's a pita to code (many gory details due to scrolling) and not desirable for all commands...
  • Only the prompt provides suggested tab completion results or ctrl+r history search.
  • Only the panel can provide input syntax highlighting and nice scrolling keybindings.
  • Only the panel will be able to have a /window option.

Most of this next month will be spent polishing this new addition, then making the 1.4.4 arm release.

Cheers! -Damian




Status Report for July 2011

August 1st, 2011

Hi all. For most of July I've been traveling. First along the ocean, visiting Ashland's theaters and the Strawberry Festival, then ending with PETS in Waterloo. It was fun and great to see everyone, though I'm glad to finally have some time back at home.

During those trips I finished arm's relay setup wizard and released version 1.4.3. At the dev meeting I also worked with Nick on refactoring TorCtl's event parsing and Jake on a safe method for customizing Debian's system wide torrc.

This release marks the end of my plans for major feature expansion of arm's terminal interface. From here I'll be shifting my focus to either Kamran's gtk interface or the new TorStatus site (probably picking based on if Kamran wants to stay after GSoC or not).

Cheers! -Damian




Status Report for June 2011

July 3rd, 2011

June could have started a little better, beginning with a nasty flu bug that had me bedridden for the better part of a week. But once that was over with arm got several new features and is now tantalizingly close to its 1.4.3 release. Improvements include...


  • Menu interface (thanks to Kamran for implementing its first version)
  • TorCtl fixes for 2412, 2812, 2065, 1329, 2580, 3406, and 3409
  • Newnym option
  • Dependency auto-fetching via mirrors with signature checks (issue spotted by Sebastian and Robert)
  • Relay setup wizard. This is still in the works and about a week away from completion, but it's turning out very nicely.

Kamran has made some progress with the arm gui, porting the bandwidth graphs and nearly finishing the log panel. This has slipped quite a bit due to illness and family issues, though the parts that are done look great. For a description and screenshot of his work see his blog posting.

Finally, I dug into arm's resource consumption and performance. I was able to reduce its memory usage by 12% and the shutdown time's now instantaneous. However, besides this arm's about as lean as I can reasonably make it...


  • Memory:
    • 17.9 MB total memory usage
    • 3.0 MB (16.8%) is from the idle python interpretor
    • 7.5 MB (41.9%) is from importing the codebase
    • 7.4 MB (41.3%) is consumed at runtime, contribution from individual panels being negligible

  • Startup time is 0.142 seconds. 0.123 is the baseline startup, with graphing contributing an extra 0.02 seconds (probably from reading the state file for bandwidth prepopulation). On the first startup there's around an extra second, probably for importing the libraries.

  • As for cpu usage, there's spikes from connection and resource usage fetches but otherwise it's flat (very little curses or controller activity due to caching and being smart with redraws). Individual panels don't contribute noticeably to the baseline.

Cheers! -Damian




Status Report for May 2011

May 31st, 2011

May was a beautiful, chaotic haze that began with the GSoC acceptance fallout...

  • blog posting
  • acceptance introductions
  • last minute coordination to get a student for the EFF
  • adding proposals, mirrors, and minor template additions for next year's GSoC
  • git repository and ldap discussions

... and somehow ended with me as the mentor (or co-mentor) for five students. Karsten, Norman, and I will be mentoring three students from Wesleyan college to work on a Django rewrite of the TorStatus site. I only have a little prior experience with Django so a good portion of this month was spent reading the Django book to get up to speed (only got to chapter ten - I'm a horribly slow reader).

Meanwhile Kamran Khan has been hacking on the arm codebase, finishing his first week with a functional menus prototype. The implementation details turned out to be trickier than I'd anticipated and will need more polish, but my hat's off to him for being able to dive into a completely new codebase and develop such a substantial contribution in so little time (thanks, Kamran!). For more on this see Kamran's blog.

The fifth student is 'identity' from irc. He's doing his thesis on arm, writing both a paper on its future plans and implementing a minor feature (NEWNYM functionality). He, Kamran, and I will be having periodic sync-ups via Mumble to discuss their projects and issues they run into. If this is successful I'll suggest it to the Wesleyan students too.

Besides all of this, I'm also keeping an interested eye on Julien's Metadata Anonymisation Toolkit. I did a code review for an initial bit he implemented and might do more if I both have the time and he wants the help.

In terms of arm development, I finished a complete rewrite of the codebase that's been almost a year in the works. Besides being a far saner implementation, this paves the way for the further performance enhancements and features discussed on its wiki. /me is still not quite done doing his happy dance about this

Other interesting developments include...

Cheers! -Damian




Status Report for April 2011

April 29th, 2011

Once again my month's been split between GSoC and arm. The former's been a quite juggling act (especially the admin role), but well worth it. When the dust settled arm got a fantastic student. Kamran Khan will be hacking on arm this summer, working on several enhancement including manual path selection, UPnP support, a newnym option, and best of all a GTK front end.

In other news, this month started with the 1.4.2 arm release followed by hotfixes and improvements to its deb. Thanks to Dererk, TorCtl has been properly packaged for Debian as python-torctl and arm uses this (rather than a bundled copy) for its debs. This release also had metadata fixes suggested by intrigeri (debian bugs 623311 and 623312).

Thanks both to Sebastian and my new team of git-fanatics at Amazon, arm has finally migrated to git. Besides some workarounds for svn:externals and 'svn export' this has been a painless transition and I'm definitely glad I made the move.

As for arm development, this month has included several notable fixes, performance improvements, and cleaning of the codebase...

  • Improved arm's startup time by 83% (from 0.84 seconds to 0.14).
  • Thanks to Erinn and Andrew I finally have access to a Mac. PID resolution and several important issues for arm on that platform have been fixed.
  • The deprecated connection panel and file descriptor popup have been dropped from the codebase (together over 1500 lines).
  • Fixed a critical parsing error for circuit paths in older Tor versions thanks to asn.
  • Investigated the work needed for Windows compatibility and process renaming. Unfortunately neither are likely to happen any time soon.
  • ... and many more (file descriptor warnings, using new 'traffic/*' getinfo options, etc).

Cheers! -Damian




Status Report for March 2011

April 6th, 2011

Ok, my project's released, applicants have responses, and at long last I stand triumphant over my inbox nemesis (for the moment, anyway - he'll be back). No more excuses so here's my status report for March.

This last month was spent juggling a few things. First and foremost I've been swapping between my mentor and admin hats for GSoC. By day I've been that annoying, nagging guy asking devs to talk to all these young upstarts that won't get off their lawn. But by night I transform into a hideous slave driver, bent on demanding more and more from the poor applicants to my projects. I'm actually not sure which group is more likely to plot my assassination...

Time permitting I've continued to hack on arm, adding some sweet new features like expanded circuit paths and application identification to the connection panel. The 1.4.2 release (which has been in the works for the last three months) is finally done, for more on that see its blog post.

And finally, I've survived my last oncall with RCX Checkout (only got paged thirteen times!) and transferred to the Source team of Builder Tools. Yesterday was my first day and so far I'm loving it, though I'm suspecting my old team is trying to hint something since they left me in their oncall rotation... :P

Cheers! -Damian




Status Report for February 2011

March 2nd, 2011

I'm usually weary of randomization but this last month had too many interesting things to keep from branching out a bit. GSoC is coming up and Andrew has been drumming up interest among students from Wesleyan. In preparation for both I've been sprucing up the Tor volunteer page, prepping our GSoC application, and contacting new potential mentors like Tomás, Robert Ransom, Robert Hogan, and the T(A)ILS community for project ideas.

I've also been preparing my own project for possible contributors, moving arm's development notes and revised project ideas to the Tor wiki. This will be the canonical place for arm development information and upcoming plans (I get enough of scrums and burn-down charts at work, and this should act nicely as an alternative for keeping people informed).

Arm development has stayed relatively on track, with the revised connection panel very nearly achieving parity with its predecessor (and in most respects surpassing it). Most of what remains are refinements and tasty new features. Arm has also been added to Debian (Sid) and Ubuntu (Natty) with backports pending. Many thanks to Peter for his help.

As with last month, I have another round of being oncall for work which will sap a chunk of my time (and leave me grumpy). That aside, the 1.4.2 arm release should be done by the end of the month and I'll be keeping an eye on the channels for the first round of inquisitive students. Fingers crossed that we find some good ones that stick around this year.

Cheers! -Damian




Status Report for January 2011

February 2nd, 2011

This last month began with the release of arm 1.4.1, last minute features including better TBB compatibility and summarization/filtering of the options presented in the configuration panel. This was followed by three hotfixes for platform specific bugs in error handling and an improvement to help with Gentoo ebuilds (thanks to Fabian, Trystero, and Anthony).

Since then I've been diving into the connection panel source, splitting out and improving its functionality for mapping IP/ORPorts to fingerprints and identifying exit connections (this determines what information is publicly displayable or not). The pesky syshook concurrency error and among others have also been fixed.

I've been working weekends to make deadlines for Amazon and had a week long brush with the flu which ate a good portion of this month. I'm also scheduled for our oncall rotation next week which won't help either. However, I'm hoping to have the connection panel rewrite finished this month and begin working on some tasty new features for it soon afterward.

Cheers! -Damian




Status Report for December 2010

January 4th, 2011 (updated 1/9/11)

Hi all. In the month of December arm saw some *sweet* improvements. Here's what I've been up to:

  • Vastly Better Resolver Performance

    By far the most expensive thing that arm does is ps and netstat/lsof/etc lookups. While wandering around development forums I discovered psutil, an awesome library for cross platform resolution of system and process information. For OSX and BSD they're using ps and lsof lookups just like arm. However, for Linux they had a very different approach, querying proc contents directly. I adapted the functions for arm and it cut the runtime for resource and connection resolution by 90%. Many thanks to the authors of psutil (Jay Loden, Dave Daeschler, and Giampaolo Rodola')!

  • BSD Compatibility

    For a long time FreeBSD has been arm's nemesis. Its variant of netstat can't get connection pids, the ss resolving utility belongs to a spreadsheet program instead, and even pid resolution failed (breaking resource stats and numerous other things). However, thanks to patches and testing by Fabian Keil and Hans Schnehl arm now has BSD counterparts for all of these, plus autodetection for BSD Jails.

  • Expanded Distribution

    Peter and I have finished revisions for the arm deb and it's now pending feedback from the Debian FTP admins. Arm is also now available on ArchLinux thanks to Spider.007 and Fabian mentioned that he might be interested in doing a FreeBSD port.

  • Volunteer Recruiting

    Being the lone developer of arm is kinda lonely. I'd love to find other people interested in hacking on the code with me. To this end, and in anticipation of GSOC 2011, I've added a project to Tor's volunteer page ("Client Mode Use Cases for Arm"). I also found other island resident interested in learning Python and possibly working on both arm and OpenWrt (keeping my fingers crossed that he'll get involved!).

Plus numerous other fixes and improvements. The next release is currently planned for next week, and for January my goal is to get arm's connection panel rewritten, which is important to the project for a couple reasons:

  • The first couple iterations of arm were prototypes, figuring out what worked and what didn't. As such, the maintainability and performance of the code kinda sucked. My overarching goal this year has been to rewrite the codebase to be both elegant and have the smallest resource usage possible. This rewrite is about 90% done, with the last remnants being the connection panel and controller.

  • Client side use cases and several other potential GSOC projects involve the connection panel. Hence getting this code in shape is an important prerequisite for others to be able to hack on it.

The connection panel has several potential features queued up (identification of other attached controllers, country statistics for bridge operators, etc) so in February I'll be implementing some of those. However, there's looming deadlines for my work with Amazon so I might need to throw some of my weekends at that instead (causing this schedule to slip). -Damian




Status Report for November 2010

December 4th, 2010

Hi. Like the rest of the Tor developers I've been plenty busy throughout November. Arm advancements have included:

  • Finishing its configuration editor which lets users easily customize Tor's configuration with reference information scraped from the man page.
  • Made a new release (1.4.0) which had numerous changes including: additional torrc validation, config panel rewrite, improved text input methods, and numerous fixes.
  • Worked with Peter on getting the arm deb in shape for Debian and the Tor repositories. Looks like we're getting pretty close!

    Arm is also now in Gentoo's portage repositories and, if I'm lucky, I might also find a volunteer to maintain it for BSD. Oddly, no one's shown interest in having it on Redhat distros (despite an experimental RPM being available). Maybe no one uses Redhat anymore... :P
  • Finally got over my phobia of having a userbase and stopped hiding the project. In particular I've added it to:
This next month's plans include:
  • working more with Peter to finish the deb
  • rewrite the connection panel (this and the controller are the last parts of the old codebase to be revised)
  • begin experimenting with a panel for showing details on client connections (for more information see the "client mode use cases" in arm's TODO)
  • providing short summaries on the config panel rather than the full man descriptions
  • incorporate feedback from new users, such as to have a warning when arm cpu resource usage is high that the -b flag would be suggested (thanks to Clete)

Cheers! -Damian