Luke Melia

I had a great time presenting this talk at GoRuCo 2010. If you watch it, please don’t be shy with your feedback, tips, or criticism.

Redis is one of the most interesting of the NOSQL solutions. It goes beyond a simple key-value store in that keys’ values can be simple strings, but can also be data structures. Redis currently supports lists, sets and sorted sets. This post provides an example of using Redis’ Set data type in a recent feature I implemented for Weplay, our social youth sports site.

The End Result

Weplay members were told us they wanted to be able to see which of their friends were online, so we decided to add the feature. Let’s look at how Redis was instrumental in this functionality.

Properties of a Set

First a quick overview of Redis’s “set” data structure. A set in Redis has the same properties as the abstract data structure of the same name:

* 0 to N elements
* Unordered
* No repeated members

In practice, this is nice because adding a value to a set does not require you to check if the value exists in the set first.

Redis Set Operations

You can see a full list of the set operations Redis supports. here are the ones were going to use:

The Approach

The idea is to have one active set per minute. During each request that comes in from a logged-in user, we’ll add a user ID to the active set. When we want to know which user IDs are online, we can union the last 5 sets to get a collection of user IDs who have made a request in the last 5 minutes.

Now, if we have a set of the user’s friend’s IDs, we can intersect that with the online users and we’ve got our online friend IDs.

The Code

Here’s the relevant code. (Note: we’re using the redis-rb gem, by Ezra Zygmuntowicz (aka @ezmobius), as our Redis client.)

I love that the code is almost as succinct as the diagrams.

With this approach, you need to empty or delete the older sets before the next hour comes around. We use a cron job for that purpose. I’m not going to show that here, but feel free to get in touch if you have any questions about it.

Hope this gives you some ideas of the Redis’ elegant simplicity and power. This code simpler than it would be if we had used an RDBMS or memcached, or both.

If you’re new to Redis, I’d recommend following it’s creator and maintainer, Salvatore Sanfilippo (@antirez). He’s a very thoughtful and transparent open source leader and is rapidly and continually improving Redis.

At Weplay, we recently switched our Ruby interpreter from MRI 1.8.6 to REE 1.8.6, and have seen dramatically improved response times (reduced by ~40%).

As of now (October, 2009), EngineYard doesn’t support Ruby Enterprise Edition, so we had to go it alone. This blog post documents our configuration.

We run in EngineYard’s cluster environment with nginx -> HAProxy -> mongrel load-balanced across a few app server slices with a utility slice running delayed_job workers. Our ruby processes are managed by monit.

Here’s how we went about things:

Install REE

I installed REE at /opt/ruby-enterprise-1.8.6-20090610. One of my goals was a side-by-side installation with MRI so that I could switch back quickly if something went wrong. (As it’s turned out, REE has been rock-solid for us.)

Note the –auto flag which provides for a non-interactive installation.

Update mongrel_rails script

Our EngineYard slices have MRI installed at /usr/bin/ruby18 with a symlink at /usr/bin/ruby. The shebang line in /usr/bin/mongrel_rails was:

#!/usr/bin/ruby18

Change it to:

#!/usr/bin/ruby

This may not be absolutely necessary based on how your environment starts mongrel, but I’d
recommend doing it anyway.

Prepare a script to update symlinks

So now you’ll have the following setup:

/opt/ruby-enterprise-1.8.6-20090610/bin/ruby

/usr/bin/ruby18

/usr/bin/ruby -> ruby18

To switch to REE, we’re going to switch that symlink to point at the REE ruby binary. You’ll also want to do the same for other binaries like rake, gem, etc. Here’s a script, eyruby_switch.rb, I wrote to switch them all at once:

You will call the script like this: ./eyruby_switch.rb ree or ./eyruby_switch.rb mri. Don’t execute it just yet.

Re-install gems

Before the switch, I wanted to get all our non-vendored gems installed in REE. The REE install has its own gems directory, and you can’t share gems with native extensions between REE and MRI anyway, so you need to reinstall. Here’s an example of installing a gem for REE without needing to have the symlinks pointed to REE:

sudo /opt/ruby-enterprise-1.8.6-20090610/bin/ruby /opt/ruby-enterprise-1.8.6-20090610/bin/gem install --no-rdoc --no-ri nokogiri

Setup GC-tuning wrapper script

For us, the big performance win for REE was the tuneable GC parameters. I’m not going to try to explain the params here. If you made it this far, you’ve probably already ready Evan Weaver’s blog posts on this topic.

Our goal was to apply the GC params to all rails processes, but not necessarily to every ruby process running on the system. So we created a wrapper script called ruby_with_env:

Put it somewhere on your system (ours is at /data/weplay/shared/bin/ruby_with_env) and make sure it’s executable.

Update monit scripts

Examine all the scripts specified by the .monitrc files in /etc/monit.d/. In any script that starts a rails process, replace the call to /usr/bin/ruby with the ruby_with_env wrapper script. This is safe to do even if MRI is still the active ruby on the system, because the environment variables will simply have no effect with MRI — everything else should run properly.

Flip the switch, hero

Update the symlinks:

./eyruby_switch.rb ree

Verify the script worked with ruby -v

Then tell monit to restart your processes and your app is now running on REE.

Feedback

I would love to see EngineYard support REE so I can use their otherwise excellent team to do setup and maintain this stuff for me. Until then, maybe we can help each other out. If you have questions or ideas to improve this configuration, let me know in the comments or on twitter.

One more thing

If you’re in the early stages of exploring REE and you just want to try it out locally, I highly recommend Wayne Seguin’s awesome rvm – Ruby Version Manager. It makes it easy to install REE and switch back in forth in a development environment.

I thoroughly enjoyed Gotham Ruby Conference (GoRuCo) this past Saturday. Excellent mix of talks, smooth flow of the day, and a intelligent, friendly crowd. Congrats to the organizers for pulling off a great conference a third year running!

One thing I’ve learned about myself is that I get more out of a conference if I a) am well rested, and b) take notes. With 7.5 hours of sleep on Friday night (a lot for me), I fired up TextMate during the opening talk and started the note-taking. I made an offhand comment in the #nyc.rb IRC room that collaborative note taking would be cool. Ari Brown suggested EtherPad, which turned out to be a great tool for the job.

EtherPad is pretty impressive. It allows multiple authors to simultaneously edit a document. You can see your own and other people’s edits and contributions highlighted in a color associated with each contributor. It supports simple-formatting, a little chat window, saved revisions, and nice import/export options. And it does this all in a web browser without plugins, logins, or any friction whatsoever. Big props to the development team behind that thing.

The primary note takers were Ari and me, but one particularly cool moment in the collaborative experience was when Gregory Brown, the lead-off speaker, came on and did a light edit of the notes from his own talk.

Link: Notes from GoRuCo

At my first conference since last year’s GoRuCo. I have a second daughter, Jemma, since then, and no blog posts. Hoping I can use this occasion to engage more!

Credit to @sd for the photo.

Gotham Ruby Conference was yesterday at Pace University. It was a great day – kudos to the organizers. If you missed it, check out the video from Confreaks when it is posted.

UPDATE: A few more conference pics, courtesy of Sebastian, including some of the afterparty/hackfest, where Jeanhee & Chiara came out to meet the ruby geeks!

After listening to Ryan Davis’ inspiring talk, “Hurting Code for Fun & Profit” at GoRuCo today, I wrote this rake task to figure out what code to hurt. It looks though your git log, and takes the 15 .rb files in your Rails’ app/ directory and runs them through flog, so you can see which are the most complicated.

[ruby]
namespace :analyze do
namespace :commits do
desc ‘Flog the most commonly revised files in the git history’
task :most_changed_files do
counts = Hash.new(0)
IO.popen(“PAGER=cat git log –name-only –pretty=oneline 2>&1″) do |pipe|
while (!pipe.eof) do
line = pipe.readline
next unless line =~ /^(app|lib).*\.rb$/
counts[line.chomp] += 1
end
end
counts.sort_by{|item| item.last}.reverse.first(15).each do |item|
flog_score = `flog -s #{item.first}`.to_f.round
puts “#{item.first} (in #{item.last} commits) (Flog: #{flog_score})”
end
end
end
end
[/ruby]

Install it with sake from here.

UPDATE: If you’re interested in watching Ryan’s stellar talk, it will be up on Confreaks soon, or you can try the RubyConf edition of the talk.

lmelia$ history 1000 | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
184 git
122 script/story
64 rake
26 sudo
12 dig
11 cap
9 script/rstakeout
9 sc
8 ./script/story
5 history

The startup I work for, weplay, opened our beta to the public on Tuesday night, just a few hours ahead of a New York Times article about us. It’s been an insane couple of weeks leading up to this launch, and I’m happy it’s behind me, though it was fun and I really enjoyed working so intensely with my team.

We got covered on ABC News today, and have spawned a bunch of blog posts and conversations. Our “official” launch is in a few weeks.

If you have any connection to youth sports, I hope you’ll consider using the site to help organize your team and activities!