Today's Email Incident

Recognizing that it is not politically correct in these parts to pose this question, I sometimes wonder if the Github team (which seems like a highly-skilled, competent team) regrets building on Rails.<p>It seems they have been able to use their high skill and competence to laboriously steer a bulky and (dare I say it?) heavily-engineered framework to serve Github as the modestly fast and fairly reliable site that it is.<p>But would not they be happier had the efforts exhausted on scale, reliability, and repair been available to spend building features had a faster, less confusing platform and framework been selected?<p>I know, it's antithetical or heretical to imply that Rails is anything but a model of efficiency.

I previously assumed the cause was some marketing intern pasting email addresses into the wrong form. Better training (and a slap on the wrist) would prevent it from happening again.<p>But it's somewhat scarier knowing this was due to a bug triggered by a security patch. Who knows what yet-undiscovered bugs are still in there.<p>That said, there's no such thing as bug-free software and I still find GitHub to be trustworthy.

Very concerning - they accidentally distributed their entire customer list and now I have to handle the internal PR fall out re: my email address and that those of non-technical co-workers (board, accounts, finance, engineering managers) have been distributed globally and then subsequently pastebined. Times like this I'm glad that our code is hosted internally and access to it is controlled internally.<p><a href="https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601.png" rel="nofollow">https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601....</a><p><a href="https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4hq.png" rel="nofollow">https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4h...</a>

This is bad - real bad. Goes to show you if the 'golden boys' can make mistakes such as these, what about the other 96% of companies handling your info?

A modern mail server should have features to prevent too many users from being present in the To: field. This will also prevent a lot of the Reply All problems we see regularly. This can be implemented as part of a Quota management system.

This is potentially a pretty big security hole for apps, right? For example:<p><pre><code> posts = current_user ? current_user.posts : Post.public posts = posts.where(:user_id =&#62; params[:user_id]) if params[:user_id] </code></pre> In the old Rails, that code is guaranteed not to show any private posts. But with the new Rails it'll show you everything written by a user.<p>Granted, it's pretty awkward code, but I can imagine situations where people might be inclined to write something like that.

In case anyone's interested, I think this is a Github Issue on the same bug:<p><a href="https://github.com/rails/rails/issues/9813" rel="nofollow">https://github.com/rails/rails/issues/9813</a>

I'm impressed. I thought it much more likely that somebody carelessly pasted a bunch of addresses into a "To" box instead of "Bcc".

Would it make sense to just always put the recipients in the BCC field to ensure you don't accidentally leak email addresses if/when something like this were to happen again?

Wait, so what exactly is the situation that's at issue here? We upgraded yesterday, too, and have both scopes and class methods that we append onto has_many/belongs_to relations, but haven't experienced anything out of the ordinary. Now I'm terrified we have some unexpected behavior lurking somewhere that's not tested or something.<p>&#62; <i>In this case, when using the scope method to define an Arel scope on Organization, the where clause of the scope is overriding the condition imposed by the Organization#teams association. The part of the WHERE clause meant to restrict the query to Team records related to the Acme organization was dropped.</i><p>This doesn't happen in all cases -- all the cases where I've done this appears to be fine. And a scope just blowing away an association in all cases would be too obvious to not be caught.<p>Is it because both the scope and the relation select by the same attribute? Does that attribute have to be `id` for this bug to occur?<p>Or does it have to do with the fact that they're nesting a query within the where clause to determine the IDs? (I don't see why this would matter).<p>I guess I'm off to a REPL unless anyone here knows more concretely what the issue is.<p>EDIT TO ADD:<p>I can't reproduce this behavior. I just spun up a new Rails 3.2.13 app, and created those two models. Going through what they do in the console, my results differ on the potentially dangerous part:<p><pre><code> 1.9.3-p0 :015 &#62; teams = acme.teams.using_octocats_scope Team Load (0.3ms) SELECT "teams".* FROM "teams" WHERE "teams"."organization_id" = 2 AND "teams"."organization_id" IN (SELECT id FROM "organizations" WHERE "organizations"."has_octocats" = 't') =&#62; [] </code></pre> I do get an empty array, as is supposed to be the case. I notice that my nested select remains a SQL statement, rather than being evaluated to '(1)' as in their example.<p>So... I don't know if I'm doing something wrong, or if GitHub has other code / configuration settings interacting strangely here.<p>Can anyone else reproduce?<p>EDIT AGAIN:<p>Here are the four files you need to reproduce yourself. <a href="https://gist.github.com/losvedir/5202121" rel="nofollow">https://gist.github.com/losvedir/5202121</a><p>ONE MORE EDIT:<p><a href="https://github.com/losvedir/test_github_thing" rel="nofollow">https://github.com/losvedir/test_github_thing</a><p>There's a repo with my history in the console. Dunno why I'm seeing the behavior I am.

Bad incident, great transparency and handling. Just don't do it again please... :)

Sending emails always requires more attention than expected. Everyone, even Apple in 2007, made some severe mistakes in delivering emails. By using a professional SMTP relay (disclaimer: we provide SMTP+) this wouldn't happen since, for example, every A:/CC:/CCN: recipient is splitted from the original message by default.

<a href="http://pastebin.com/JX1PdiMB" rel="nofollow">http://pastebin.com/JX1PdiMB</a>

How would one unit-test an app like this, where you have to talk to an external database and over SMTP at a specific time of day? Seems like in a case like this thorough unit testing needs to be very extensive to catch a bug like this, yet it would have been worthwhile.

And this is why you shouldn't use ORMs. Magic blackboxes always bite you in the ass eventually.

The excuses are on an on. I admire Github guys. They give a super cool impression to the other outside world and they're doing a great job. But the recent outages and the problems are giving high attention to Github. It's giving negative sign for the people who are interested go for Github way of hosting code. Either their own premise or on github. giving detailed analysis is a good tech thing... it fires developers heart and we are still going admire what they did to resolve this issue but definitely this isn't a good sign for github.

I whipped up a quick example Rails project to demonstrate the issue. It's a lot like the Github post, but a little simpler: for instance, there's no real reason to talk about class methods:<p><a href="https://github.com/pjungwir/scope-error" rel="nofollow">https://github.com/pjungwir/scope-error</a><p>I'm pretty concerned about this bug, because scopes are one of my favorite Rails features.

For quite some time, I have been using Sequel (<a href="http://sequel.rubyforge.org" rel="nofollow">http://sequel.rubyforge.org</a>) as an ORM with Postgres instead of ActiveRecord, and have never been happier. Overall, Rails has been a great web-framework, but ActiveRecord has been it's sore spot, at least for me.<p>Also, using the sequel_pg (<a href="https://github.com/jeremyevans/sequel_pg" rel="nofollow">https://github.com/jeremyevans/sequel_pg</a>) on Postgres ensures that even raw SQL queries are type-casted properly, plus it supports streaming and cursors, while also being very very quick.<p>If you want to stick with Ruby/Rails and relational databases, but think ActiveRecord might not be your bag, give Sequel a shot. It also fits nicely into lighter frameworks such as Sinatra.

Non-technical companies which do this tend not to apologise - it draws attention to the mistake which most of their (non-technical) customers never noticed in the first place.<p>I guess github doesn't qualify for the non-technical crowd :)

I'm trying to figure out how likely my apps are to be effected by this same bug/behavior change as Github's, and if I should delay installing the security update without more investigation.<p>Not a great situation to be in with a security update.

Seing as how this list of customers are about to run out of their trial for Github Enterprise, it'd be funny if atlassian or any other hosted Git providers were to make use these emails for their own gain.

a little more on the incident <a href="http://chadscira.com/post/5148b1f0f93e934b550112be/Github-leaks-1500-emails" rel="nofollow">http://chadscira.com/post/5148b1f0f93e934b550112be/Github-le...</a>

Frameworks are innocent.

Rails security point release breaks apps by changing the ORM.<p>News at 11.