How is team-member-1 doing?

I knew a kid once. He was a &#x27;junior operator&#x27; in a computer room .. you know, in the good ol&#x27; days, where the computers lived. (Before they escaped and attached themselves to your wrists.)<p>He thought he was smart. And sometimes, he was.<p>One day, he overheard a team leader talk to his programmers about the newly-minted database, sitting there in front of them on the table, on a brand new .. amazing .. 640Meg hard drive.<p>This database had consumed the disk. It had cost the company a cool million dollars to create. It was vital that we backed it up.<p>So, the new 640 Meg disk was on its way, onto which we&#x27;d back the database up. The first thing we&#x27;ll do, the leader said, is copy the database, sector for sector.<p>&quot;And only then, will we re-index the database!&quot;, he claimed. &quot;Until then, the indexes will remain un-sorted!&quot;<p>Well, the kid overheard all of this, but only heard &quot;the indexes will remain un-sorted!&quot;.<p>Later that night, this kid thought he&#x27;d prove himself.<p>He re-indexed the database.<p>He didn&#x27;t tell anyone.<p>The next day, a not-so-junior programmer came in, saw the database disk attached to the operator machine, and thought that the backup had been done. For reasons we shall not explain, he disconnected the disk from the operator machine.<p>The index had not been done.<p>The database was gone.<p>The new disk arrived, but nobody could mount the old database disk. Much panic ensued!<p>Operator logs were consulted. The computer room security cam tapes were spooled.<p>Oh shit!<p>Epilogue: I made a lot of money from those kids, writing a tool to recover a corrupted database, whose power had been removed mid re-indexing ..

What I think is particularly noteworthy is not that Gitlab recognized that anybody could have made that mistake, but rather how supportive Gitlab was about the whole thing.<p>When you make a big mistake, it is easy to place yourself in a mindset where you feel like a disaster even though everyone is accepting. I call it the &quot;disappointing your parents&quot; mindset, because it can feel a lot like people are just being supportive because they love you, and what you did was indeed inexcusable to a certain degree.<p>The feeling is made somewhat worse when you are an employee, because your livelihood and your future are dependent on how other people perceive you. To that point, I&#x27;m really impressed by the fact that Gitlab addressed the fact that this employee was still being promoted, and that the mistake hadn&#x27;t affected that. In my mind that is as at least as important as all of the rah-rah stuff.

I really have to just start rolling my eyes at this point. I&#x27;m just waiting for the official meme, and the cycle will be complete. Get to work making your infrastructure resilient to a simple accidental deletion, and restoring some faith in your product.

Several years I was a junior dev at a 3PL (3rd Party Logistics) company troubleshooting an issue with some overnight data imports.<p>I had it all loaded up on the test environment, then I&#x27;d delete the import, make some changes, and re-run it. Wash, rinse, repeat, trying to track down the issue. As I&#x27;m sure you can guess, at one point I executed my scirpt in the wrong window and deleted last nights import from Production.<p>I immediately told my boss, who was very understanding with a &quot;everyone does this kind of thing at some point&quot; kind of shrug and went over to our DBA&#x27;s office to ask him to re-load the last snapshot. But it turned out the snapshots had been broken for two weeks and no one had noticed.<p>And it wasn&#x27;t a simple issue of re-running the import. After all the orders had been imported, humans had manually assigned orders to trucks and dispatched them in the wee hours of the morning. Now there was no way to know what packages were on what trucks.<p>I think the DBA ended up buying ApexSQL Log out of pocket to roll back the deletion with the transaction log.<p>The result was that for several hours the delivery drivers for a national office supply company in a certain state were completely unable to use their handhelds or access their truck&#x27;s inventory. That was my team-member-1 moment.

&quot;Bob Hoover, a famous test pilot and frequent performer at air shows, was returning to his home in Los Angeles from an air show in San Diego. As described in the magazine Flight Operations, at three hundred feet in the air, both engines suddenly stopped. By deft maneuvering he managed to land the plane, but it was badly damaged although nobody was hurt. Hoover’s first act after the emergency landing was to inspect the airplane’s fuel. Just as he suspected, the World War II propeller plane he had been flying had been fueled with jet fuel rather than gasoline. Upon returning to the airport, he asked to see the mechanic who had serviced his airplane. The young man was sick with the agony of his mistake. Tears streamed down his face as Hoover approached. He had just caused the loss of a very expensive plane and could have caused the loss of three lives as well. You can imagine Hoover’s anger. One could anticipate the tongue-lashing that this proud and precise pilot would unleash for that carelessness. But Hoover didn’t scold the mechanic; he didn’t even criticize him. Instead, he put his big arm around the man’s shoulder and said, “To show you I’m sure that you’ll never do this again, I want you to service my F-51 tomorrow.”

I&#x27;m old enough to have made some serious mistakes on the job over the years. Thankfully, the worst ones were when I was much younger, but I know that I could still make a worse one before I retire.<p>Here&#x27;s what impresses me about Gitlab: They not only say they&#x27;re committed to honesty and transparency, they actually practice it.<p>It&#x27;s easy to see this as some cynical PR move, but to me it&#x27;s refreshing that they have addressed specifically what happened to the employee who made the error. It makes me believe that they are working very hard on fixing their practices to ensure this kind of failure won&#x27;t happen again, and I trust they will share (as @syste said in this thread) what those changes are once they have it sorted out.<p>&quot;Oh, how naive you are!&quot; some may say, to which I respond, &quot;Oh, how cynical and inexperienced you are!&quot; Human failure is inevitable. Designing systems (whether in code or in management practices) that tolerate this inevitable failure is very difficult.<p>This sort of event can be the catalyst for tearing out what didn&#x27;t work and creating a much stronger foundation for the future, but only if blame is set aside and honesty is allowed to prevail in the &quot;after action&quot; analysis. Call it PR if you like, but I see a healthy desire to deal with what actually happened and fix it rather than falling into the trap of pointlessly assigning blame.<p>Consider that it took congressional hearings and someone with the chutzpah of Richard Feynman for NASA to own up to the shuttle explosion. A far worse event, with far worse consequences, but the aftermath of those events and NASA&#x27;s complete unwillingness to hold itself accountable and deal with reality cost it a lot of credibility.<p>Good on you, Gitlab.

They could support their team members by not continuing to make a circus out of them

The first part of the post were interesting and I guess funny, but creating and selling a T-Shirt with the accident and stuff? IMO this would have been a fun joke inside the company, but to outsiders, eh. I don&#x27;t want to sound grumpy though :)

I generally like the openness, but, Gitlab marketing team, if you&#x27;re listening: stop spamming social media content on your blog, it seems cheesy and lazy.<p>A few tweets or comments are more than enough to prove your point.<p>They did something similar with the storage post, which was full of Hackernews opinions.<p>So, this, or at least post my comment on your blog, too :D

When I had my first IT job in ~1991 I caused millions of pounds worth of loss to my employer, a well known retailer in the UK, due to a bug I made.<p>My boss covered my arse. I love that man, and i&#x27;ve never made a serious mistake since, as it&#x27;s made me risk averse.<p>Gitlab did the right thing here by owning the situation and making it public.

The Gitlab PR team is certainly doing a much better job than their engineering team did.<p>I actually appreciate their attitude towards errors by employees.<p>Unfortunately, the appearance this spectacle creates is that the same sort of attitude should apply to them as a company, i. e. &quot;Don&#x27;t fire Gitlab! You&#x27;ve just invested 200MB of data into their education&quot;<p>It&#x27;s a very smart method to protect not just team-member-1, but also employee-1.

His page says: &quot;Database (removal) Specialist at GitLab&quot;<p>Love it.

I agree with the mentality that this is a team effort, and when it fails, a team failure. And that when something goes wrong what&#x27;s important is to understand it, and put in place a way for it to not happen again. Kudos to GitLab for their forward thinking way of working.

So who&#x27;s team-member-1 for the Amazon s3 outage a few weeks back? I&#x27;m sure they feel the same way and we&#x27;d love to send them gifts to support them just like the community supported GitLab.

Yet another gitlab post on how open and transparent they are and how they are being praised for that. It&#x27;s now just looking more and more like gitlab being known for being a transparent company than being recognized for their product or technical competency.<p>Like I said in another post a while back, it&#x27;s fine being transparent but gitlab just have taken this to an extreme. It&#x27;s important to be private about certain details and just get real work done and be known for that.

Perfectly happy with Github, but seriously, can I hire you guys as a PR agency? :)

Pedantic note: I&#x27;m sure under 1. Technical Skills, the &quot;I think this is out of the question here&quot; should read &quot;I think there&#x27;s no question about this&quot; or something along those lines.

Probably could&#x27;ve designed a good backup and restore strategy with the time that was invested in this piece. A combo of full backups with append-only storage of changes going a certain amount of time into the past. Worked for me for long, long time despite my many screwups. Even when I lost all my stuff to triple, storage failure I still recovered a tiny bit stored on my cheap, write-once solution: DVD-R&#x27;s. There was some bit rot but better than bit loss. I imagine their solution would be better done with a filesystem or backup software.<p>Note: It was neat that much of the community was supportive. I see the article as really a thank you to them.

been there. crashed a client&#x27;s website on a friday by updating the wrong plugins for a Joomla site.<p>given a sufficiently complex dependency chain for presented problems, anybody can be a &#x27;team-member-1&#x27;.<p>mistakes happen at all levels.