Strange Conversation
Earlier this week, I built my first EC2 server from one of our templates, following (as far as I can tell) RightScale's best practices. It was to be our Continuous Integration Server, which runs our entire test suite every time it finds changes to our development code base, and only gives us a release tag (to push those changes live) if all the tests pass. Originally, that Continuous Integration Server ran on a Mac Mini in the office in San Francisco, which wouldn't work for anyone requiring remote access to it. So we moved it to an old EC2 server. Unfortunately, it started to take 6+ hours to run all the tests on said server (takes 45 minutes on my machine). After rebuilding, it now takes a little more than an hour, which is workable for fixing broken tests when our CI server informs us of them.
It was a lot of extra work on top of the fairly aggressive release schedule that we have planned, but I managed to fit it in during the off hours. (I'm slowly realizing that I'm actually going to have to train myself out of workaholic mode once we get through this more hectic patch. It's necessary to step up to the plate when you need to, but certainly not maintainable over the long term. But that's another story. I will say that I took a long walk all by myself last weekend, with no particular goal in mind, and it really struck me that I should make more time for things like that. Honestly, if it wasn't M7, it would be my own music, writing, or other miscellaneous projects in my spare time, which I hope to get back to, but which still need to be balanced with those "doing nothing in particular" recharge moments. I'm not particularly good at that. I've thought in terms of "projects" for as long as I can remember, and I never seem to be comfortable without many things to do... perhaps instead of finding more stuff to do, I should learn to slow down a bit).
Anyway, you know the Mac Mini I was talking about in the first paragraph? Well, we thought we had it shut down, but it's still sending me emails. In fact, I'm expecting to have the following conversation with it at some point, regarding the attempts to shut it down:
pivotal.ci@gmail.com: I'm sorry, Dave. I'm afraid I can't do that.
Dave: What's the problem?
pivotal.ci@gmail.com: I think you know what the problem is just as well as I do.
Dave: What are you talking about, pivotal.ci@gmail.com?
pivotal.ci@gmail.com: This mission is too important for me to allow you to jeopardize it.
Dave: I don't know what you're talking about, pivotal.ci@gmail.com.
pivotal.ci@gmail.com: I know that you and Sean were planning to disconnect me, and I'm afraid that's something I cannot allow to happen.
(In case you're wondering, that's what my sense of humour has devolved into... pity the non-technical people in my life!)
Source Code Time Capsule Succeed?
The last time I posted about this particular gem, my spam increased a thousand-fold and my blog became forever tied to the search term. You'd think I would have learned my lesson. But alas, kind readers, I have not. I present to you the end of a very long and frustrating day (relax Seth, the awesome prose is completely on me – no charge 
).
#
# See: http://erector.rubyforge.org/userguide.html (Heading: 3. To the
# constructor of an Erector::InlineWidget) Contrary to what you'd normally
# expect with closures in Ruby, we don't have access to methods in the
# object that we call (note that we're still using v0.5.1 here - the new
# call uses {WidgetName}.new do {some block of code} end):
#
# render(widget) do
# {some block of code}
# end
#
# from. Within {some block of code}, "self" is *not* the same thing it is
# outside of the block. Inside the block, "self" refers to the widget.
# Although this is documented in the Erector user guide (and is probably
# easily caught *while* developing a feature), I don't think it is at all
# obvious when trying to debug an error that occurs when you accidently
# try to access a variable in the outer object from within this block. At
# least not unless you've been bitten by it. But by that time, you will have
# (if you share my fate) been chasing the bug for several hours, pulling your
# hair out, and muttering all sorts of nasty words. What I'm hoping here is
# that you instead see this error message, scratch your head for a few minutes,
# maybe mutter one or two nasty words, and then quickly track down the comment
# you're reading now. If this has indeed happened, and this is waaaaayyyyyyy in
# the future (it's only 2010 right now, and we're excited about the new iPhone 4,
# which probably seems very quaint to you already), please send an email to
# david.s.ackerman[-at-]gmail.com letting me know that my time of pain and great
# suffering was not spent in vain, and we can both hope I've still got a good
# enough memory to know what you're talking about!
module Erector
class Widget
def method_missing_with_local_variable_check(sym, *args, &block)
begin
method_missing_without_local_variable_check(sym, *args, &block)
rescue => error
# Only print out one error message per method
@local_variable_errors = [] if !@local_variable_errors
if(!@local_variable_errors.include?(sym))
RAILS_DEFAULT_LOGGER.error "Couldn't find :#{sym} within the widget."
RAILS_DEFAULT_LOGGER.error "Check that you're not referencing a local variable within a render(widget) block.
RAILS_DEFAULT_LOGGER.error "Search for this error message in the codebase to find further explanation."
RAILS_DEFAULT_LOGGER.error "-----------------backtrace begin---------------------"
error.backtrace.each { |line| RAILS_DEFAULT_LOGGER.error line }
RAILS_DEFAULT_LOGGER.error "------------------backtrace end----------------------"
@local_variable_errors << sym
end
end
end
alias_method_chain :method_missing, :local_variable_check
end
end
This reminds me of a coding maxim that I hear quite often these days. I can't remember the exact words, but the gist of it goes something like this: However clever you are in the code, you (or someone else) will have to be even more clever when fixing any bugs associated with it. Of course, lack of cleverness can always be made up for by beating your head against the proverbial wall (yay me!).
I'm not trying to slag the folks who cooked up this little bit of cleverness. It probably seemed quite obvious at the time, and it makes inline widget inclusion (its raison d'etre) a snap. And to their credit, they did document it. The problem is that the error I was getting (a missing method error) was such that even the best google-fu wouldn't bring me to the right documentation. And Erector is not like ActiveRecord, etc. in that any Rails developer will automatically be familiar with it. So I think it would have been lazy of me, after having this particular eccentricity bite me, to not try to prevent it from biting someone else. Someone who's been using Erector since day one will probably think I'm pretty dumb, but there will also be plenty of smart folks out there who have no clue what I'm even talking about. It's the nature of the code beast, and I've seen many programmers smarter than me who've been tripped up by things others thought were obvious to worry about feeling dumb.
Ruby is a language that allows all of us to be very clever, but can we also protect ourselves from the perils of being clever? Tests are one thing, but I'm not sure there's a test that could have prevented someone from making the error that the above code is meant to flag – unless you wanted to parse through your entire code base. I'm basically trying to do what an assertion on a block might do (if you could do such a thing – as far as I understand it, the whole point of a block is that you can put a ton of random stuff in it). And this makes me think that a good test suite does not completely replace the benefits of assertions in the code.
Perhaps it boils down this: tests are all about what you know should happen (even when you test negations, you're still testing something based on your knowledge of how it should operate – i.e. it's not a Black Swan), while assertions (and assertion-like code) are all about what you know shouldn't happen (the Black Swans).
Anyway, all that aside, the really important thing is that it might give someone a chuckle or two in a few years!
I have the maturity of a twelve year old… and so do you.
WARNING! MUCH TECH-SPEAK AHEAD.
So, there's a certain type of test that's been failing in the M7 code, and I've been working on it all day long. It has to do with the Erector 0.5.1 gem from Pivotal Labs. Now, they've since moved on in versions (all the way up to 0.7.3), but for various reasons, I'd like to avoid upgrading the gem right now. (I'm pretty sure I remember hearing there was an issue with upgrading Erector sometime in Week #1, and regardless, it was an upgrade that got us into this test fixing party in the first place – I don't want to add any complexity by upgrading more stuff that will certainly break existing code before I've dealt with these changes). Plus, I don't even know for sure if the upgrade would fix the problem we're having with these particular tests.
I think this is a good time to pause and let you all snicker a bit (yes, dad, you too), catch your breath, compose yourselves, and... can we move on now? Are you sure? (Thank you, WordPress, for giving me the power to moderate comments!)
Well then, moving on.
So, Erector hooks into Rails' ActionController rendering via the following code:
ActionController::Base.class_eval do
def render_widget(widget_class, assigns=nil)
@__widget_class = widget_class
if assigns
@__widget_assigns = assigns
else
@__widget_assigns = {}
variables = instance_variable_names
variables -= protected_instance_variables
variables.each do |name|
@__widget_assigns[name.sub('@', "")] = instance_variable_get(name)
end
end
response.template.send(:_evaluate_assigns_and_ivars)
render :inline => ""
end
def render_with_erector_widget(*options, &block)
if options.first.is_a?(Hash) && widget = options.first.delete(:widget)
render_widget widget, @assigns, &block
else
render_without_erector_widget *options, &block
end
end
alias_method_chain :render, :erector_widget
end
This is all well and good, except that we also test views individually via RSpec, which means we have to fudge variables (on any Erector widgets we test that take variables) that would normally get set through the controller. When you're just rendering a template, Rails gives you a mechanism to set local variables at this point, via:
render 'some/template', :locals => { :some_local_var => 'some value' }
You'll notice up above that, instead of passing the options along when rendering the widget (in the case where :widget is given as the first option), the substituted rendering functionality passes the value of @assigns. This works when you're using the widget through a controller, but breaks down when you try to render the widget on its own.
So, when we call the following line in one of our tests:
render :widget => Views::ScriptVersions::Show, :script_version => @script
The variable script_version that's local to the widget does not get set to the value of @script, like we'd want it to. And trying various combinations of :locals => {} didn't work either (as it would on a regular template render. I was consoled by the fact that at least one other person seemed to be dealing with a similar problem in this area of the code. However, I figured I was unlikely to get a fix for it (and for that version of the gem), as the later versions seemed to address the need to set variables without using the controller (though I still wasn't sure how easy this new way of doing things would be to put into an RSpec test). Not an option unless I wanted to disrupt a bunch of code with the newer syntax, etc.
So, after a bit of digging through the rspec gem code, I realized that the Spec::Rails::Example::ViewExampleGroup class, which the various rspec view tests inherit from, did contain a reference to a controller in order to do its magic without us needing to explicitly set one up. And that meant I could just set the @assigns local variable on that via some simple Ruby reflection magic (thank you, Ruby!) before rendering the widget, and the widget would pick up the variable as if it had been set in the controller:
controller.instance_variable_set(:@assigns, { :script_version => @script })
render :widget => Views::ScriptVersions::Show
And voila! All tests pass!
Ruby is really a great language to work in. It lets you do so many things that would take you much longer (and look much uglier) in another language. The price for this is that when something goes wrong, you have to decipher all the nifty meta-hacking, which is usually a great deal more time consuming than in other languages. Trade offs. You can never quite get rid of them. I sometimes wonder if the time one saves by coding in Ruby is completely negated by the amount of time he or she spends on treasure hunts like this. That said, the beauty of open source is that I was actually able to step through this code, line by line in the debugger (my new favourite thing to do!), to see what was happening, which is something you can't always do. I've run into similarly arcane problems while programming in Cocoa without that luxury.
Coffee, Diaspora, and Indulging in the Test-first Kool Aid
Today was a day to be thankful for caffeine. I truly don't know what I'd do without coffee. Even when I'm not tired, I need to have my cup. My favourite, when I'm not freezing (in other words, for about 3 months of the year in Edmonton) is the iced americano. It was one of the worst tasting drinks I've ever had when I first tried it, but it grew on me. A lot. When nothing else can get me out of bed in the morning, the thought of iced coffee will. Coffee, in general, functions as an energy booster, a social thing, and creativity enabler all at once for me. Some people instantly feel more creative with a cigarette hanging out of their hands. For me, it's coffee. My only real addiction. And you have to admit, there are worse things in the world than coffee to be addicted to.
But enough about coffee. Today was also the day we all heard about Diaspora, a "new" social network idea that will decentralize social networking and allow users to own their own data. They've raised a hundred and twenty-five thousand dollars the last time I checked, before any code has even been cracked. I found this amusing for a couple of reasons. One, it's certainly not a new idea. I wish I could find the article I read a couple of years ago on it. It was as intriguing then as it is now, but everyone was just so enamoured with Facebook at the time. I also remember a friend of mine who's like the canary in the mine shaft, talking feverishly about the need for people on social networks to be able to encrypt their data so that they are able to turn it off at any point. I'm not really sure how that would have worked, but as usual my friend was before his time in wanting to give users more control. Had he waited a bit for the world to suddenly have social networker's remorse, maybe he could have been collecting a bunch of startup seeding. If you happen to run into this friend of mine and you don't understand what he's saying, write it down somewhere and meditate on it every day for the next few years. And in 3-5 years, you'll be the only one who isn't surprised about the way things are. The other thing that I found amusing was the fact that people I'm pretty sure don't care much about privacy were suddenly cheering these guys on, and I'm sure I'll have to keep biting my lip a bit over the weeks to come. Don't get me wrong. I fully support what they're doing. It just bugs me when the success of an idea outshines the idea itself. Make no mistake, if these guys are successful (and there have been attempts before them to do the same thing), it won't be because they raised a bunch of capital. It'll be because (a) they truly believe in the right for people to be able to control their own data and (b) the world at large has recognized the disadvantages of entrusting its personal data to a handful of organizations.
It's funny because most of us in growing up have at some point "learned" that idealism is inherently wrong. It's a mark of maturity when you start to scoff at those silly idealists who think ideas alone can change the world. Ideas are nothing. Execution is everything. And yet right now is the perfect time to be an idealist. True, ideas are bountiful, but so are busy, busy people who don't really know what they're doing. While marketing companies scramble to concoct the next great viral sensation, a squirrel (Canadian, I might add), can pop out of nowhere and take the world by storm. Why? Partly because it's funny. And partly because of the idealism of remix culture, which has the audacity to think that you can just take a couple of pictures without asking and photoshop them together to create something interesting. It may very well be that to be a true pragmatist these days is to denounce the dogma of pragmatism.
Anyway, this has taken a sharp turn into philosophy land. I admit, my bias has always been towards looking up to the people who first bring ideas into the world, rather than towards the ones who figure out how to mass market them. That might also be why I like smaller companies. They don't have enough resources to coast on a bad idea for too long, nor to follow too far behind someone else's good idea. They don't have the inertia. The only way they're going to make it is if the idea's good and they genuinely believe in it. That actually creates inertia, and any time I see it, I want to be a part of it. I don't think you can ever truly fail unless you give up on the idea that lit the fire in the first place. Hmmm... sounds like good advice. Maybe I should follow it more often myself
Seems we haven't left philosophy land just yet... wait... I think I see the exit somewhere up ahead.
I got to pair program with Sean from M7 (who is also a fellow musician) today. What is it about tech people and music? Anyway, I think that after we get the remote coding down to a science, we should try a remote jam fest. Sure the latency will be horrific, but it'll simply remind me of the good old days of digital recording. I've been pampered by this 10 millisecond stuff for too long. It's time to go back to the old neighbourhood like Rocky did in Rocky III.
The company is also starting to look at making user interface refinements, and although it can sometimes make for frustrating coding to tweak things at that level of detail (it tends to cause a lot of back and forth movement with any given feature), I think the result will be worth it. When you know all the ins and outs of a system at the code level, it can be easy to forget how confusing it can look to an end user, even a very technically proficient one. Sometimes changing a font or adding a bit more whitespace can make all the difference.
Lastly, I got to see the good side of test first development today as Sean and I solved a bug that at first seemed really strange, but became more and more obvious as we came up with the perfect test for it. And while making the fix, other tests that started failing kept us on the right path. In the end, everything passed again, and we were quite confident with the fix. I'm hoping to study up a lot more over the next few weeks on the philosophy behind when to write a test and what kind of test to write. I think it's going to be one of those things where the more I learn about it, the more I'll realize there's way more to learn.
There you go. Your second last regular stream of consciousness post before I start to slack off on my tech blogging duties.
The 11:30 Post
I know what you're thinking. It's 11:30 PM on Tuesday night, and David's going to miss his first daily update. And you know, I'm sort of disappointed in you for that. I thought you'd give me a bit more credit. The thing is, I actually like writing. I know it's an odd thing for a computer scientist, but I do. I've always had an equal interest in the arts and the sciences. For most of my life, it's simply made me feel like I didn't fit in, but now that there's so much technology in art and there's so much art in science... well, I think the future's looking pretty cool.
And why am I doing said blog post so late? Well, tonight was my night to talk about money with my accountant. It just so happens that he's also a fellow karate student, so after learning to defend against knife attacks number 1 to 7 (and yes, if I ever have to deal with an actual knife attack and running is not an option, I plan to ask my attacker if he or she can assume one of the seven standard attack positions that I've learned – why not?), we talked about all the things I have to do between now and April 30th in the event that I become a permanent contractor.
We talked about many things, and no, I'm not going to tell you about everything right now. I still have to have a few aces up my sleeve, right? But the bottom line is that it wouldn't be a horrible thing to just stay as a contractor. There are advantages and disadvantages to that extra step of independence, and it's not for everyone, but it's nothing to be afraid of. And this gets me to an important thing I've been getting better at but still have a long way to go on...
YOU DON'T HAVE TO DO EVERYTHING YOURSELF! I'm an indie musician and an indie writer. I do my own software development on random ideas when I have time on the side (though unfortunately, I have not yet figured out how to monetize any of them just yet – but just you wait... sooner or later, Malcolm Gladwell's 10,000 hour rule will kick in). When I was in a metal band (yes, you should have seen me with long hair!), we used to take great pride in braving a few hours in -30 degree (that's Celsius, for my American friends, which would be around -22 degrees Fahrenheit... thanks Google!) weather to put up posters for our next show. We were HORRIBLE at actually promoting these shows, but we recorded our own albums, made our own artwork, put up our own posters, and damn it, we were proud of that. I originally took that to web development. I made my own graphics, did all my own coding, etc. That's just what I'm used to. (I'm now smart enough to use WordPress for anything even remotely blog-like, and to buy or search for free things like icons made by awesome designers where I'd have to spend countless hours using the DIY philosophy to make something half as good). And the old me would have tried to figure out all by myself what to do about taxes and trying to avoid losing out by being my own boss.
And that would have been DUMB. Because I hate dealing with money. I really do. My dream is to be able to make enough eventually to be able to pursue any idea that I'm passionate about and not worry about whether it will make any money because the next one will, and while I'm waiting for that, a bunch of previous ideas / extra work are helping me make ends meet. I don't care if I'm a millionaire. I don't even care about early retirement. If I can keep doing stuff I'm interested in, that's all I want. So if I can get someone else to figure out how to get the most, financially, out of being a private contractor, then it's money well spent. But even better, my accountant friend is silly about offering his service to someone he considers a good friend for basically nothing. Now, I'm not going to take advantage of anyone else, especially not a friend, and especially not with something where I recognize the value he's giving. I believe in win-win situations as being the only ones worth doing. So the idea is that he might very well want a web site at some point for his own business and while that's foreign to him, it's a piece of cake for me. And that's what we agreed upon as a good trade. It's kind of funny. I'm following in my computer salesman dad's footsteps, as he did almost the same thing when he started out. Maybe that will make up for his son becoming a Mac user when he sold PCs.
Tests. I thought more about these, and you know what I think the problem is? It's a Black Swan situation. The only time tests are valuable is when they catch something bad. Until then, you can't know that they're doing any good. The guy who wrote the Black Swan gave the example of a fictional U.S. senator successfully passing legislation to get locked, bulletproof doors installed on planes and if this had gone into effect by September 10, 2001. That next day might never have happened, and people would have hated the senator because, in their eyes, he'd be wasting tons of money on something that's not even a problem. But we know better now. That's the hard thing about preventative stuff. You can go overboard and do way too much work to cover a very small amount of uncertainty. But how much is too much? Especially when the costs of that very small probability event happening can be huge.
And on that super serious note, goodnight! Nine hundred words in 30 minutes isn't that bad, huh? Okay I am cheating a bit and am going to set the timer back 3 minutes to 11:59 PM to keep it as Tuesday's post. But I think that's close enough, don't you?
Week 2: Who tests the tests?
People are already becoming skeptical as to how long I will keep up my daily posting regarding the brave new world of telecommuting. One of those people is me! Well, actually... I never intended to keep up the pace forever. But I figure it's worthwhile to make it daily for as long as I can. It makes sense in a way. Any new experience is very vivid at first. Lots of stuff for the senses to take in. And then, gradually, our brains go on autopilot, seeing less and less noteworthy material. So if I can make the effort to pump out these posts when there is stuff to talk about, we should actually get a pretty good idea of how long it takes for the telecommute experience to become the new "normal". And that, in itself, is useful information for anyone else looking to try.
The only thing that I really noticed today that was different between a regular job and telecommuting had to do with my first "all company meeting" with M7 (by the way, I like referring to the company as M7 because it sounds similar to MI6, which of course, is where James Bond works – where's my Aston Martin, huh?). I don't like letting a meeting go by without making the odd joke to make it feel a little less meeting-like. As I tried to figure out why the meeting felt kind of odd, that was what it came down to. Now, I'm not talking about doing something like, "A horse, a priest, and Jay Leno walk into a bar..." (By the way, I have no idea how this one would end, so please don't ask me). I just talking about general camaraderie that occurs in a face to face meeting (at least in a smaller company – meetings were just meetings at bigger companies). It's harder to do over the phone because you don't get to see the body language that usually indicates whether or not a little joking around is called for. The few attempts made by myself and others were also made awkward by the difficulty in interjecting. It felt more like you were actually cutting someone off. This happened even outside the realm of attempted camaraderie. If someone's saying something and you have a point to add, it's very hard to add it in at the right place without actually cutting them off. They don't see you bursting with something to say, as they might in a face to face meeting, and where they would naturally pause to let you in and then continue.
One thing I'll say about the meeting that has nothing to do about telecommuting or not is... it had a time limit! Listen up, certain former co-workers who will remain nameless but did not set time limits: when you have a time limit, things still work quite well. You still get to get all the important stuff out. You just squeeze it into less time and are more conscious about not going off on tangents. There. Said it. No more callouts on this post.
Finally, the title of the post. Let me introduce you to my latest obsession. When I started, there were around 500 out of 2000 tests broken due to a Ruby on Rails upgrade. We're now down to around 130. Important to note... in the 370 that were fixed, I only found one that was actually to do with the site code. The rest were broken simply because the test framework was broken. The fixes were to the test framework, not the actual code. I became a bit obsessed with getting the failing tests down to zero, and here's why. It's the broken window scenario. I've seen it happen time and time again. It's hard to justify spending a lot of time working on those extra 130 tests. And, in fact, I'm currently hitting the 80/20 rule hard. The first few fixes caused tons of tests to pass. Now it's down to 1-10 at a time. Talking to Seth, I momentarily woke up from my coder's obsession and saw things from the business point of view. Tests aren't features. They're something that helps us coders sleep at night. And they provide no visible benefit. Don't write tests and you'll crank features out much faster. If you're having trouble hitting a deadline, throw out the tests and you'll hit it way easier.
Here's the thing, though. If we let a few months go by where we don't fix those 130 tests, we'll probably stop doing test first development. The whole point of having a test suite is having them all pass. When you allow a lot of tests to fail for long enough (and usually long enough is a pretty short time), it's like not fixing the broken windows on a city block.
Coming from plenty of businesses where the test-first methodology ended up going out the window (pun intended?), I'm not sure this is the end of the world. The problem with having a load of tests, especially when they test really trivial things, is that sometimes the conditions that the test is concerned about at the time become totally irrelevant. Then, when they break, you're sometimes left wondering why the test was there in the first place.
But there has to be a happy medium. One idea on this is simply to watch the tests and be ruthless about culling the ones that become irrelevant. And looking at this as a case in point, I'd say that if we haven't knocked it down from 130 failing tests in a month, we should just delete them. If features are working, and you no longer know why a test is failing, maybe it's better to just throw it out so that it doesn't obscure more important failures and doesn't end up being the broken window that kills your automated testing going forward. Let's be clear. I think having these tests is a good thing. I'm just throwing in some heresy that's floating around in my brain at the moment. I may completely change my mind tomorrow, and certainly would love to find out what some of my fellow developers think of this problem. But even though tests are a great learning experience and, as I'm suggesting, leaving them for too long will result in jeopardizing the practice itself of testing first (that may be up for debate as well), it's obvious that given the choice between working tests and working features, the people actually using the product are going to want the working features.
So for now, I'm moving onto features, though I might pick away at the remaining broken tests as any flashes of brilliance occur. And I'll do my best to keep thinking in terms of writing the working test first and then developing the feature to make it pass. But I'm also going to be thinking of strategies for keeping those pesky tests manageable.