Twilio is erroneously over-billing and suspending accounts

Rob from Twilio here. We are still actively working on this incident and aware that erroneous auto-recharge billing and account suspensions have occurred for a number of customers.<p>If this has happened to your account, please send an email to help@twilio.com where we have our support team working on making this right for everyone affected. We are spinning up additional resources to make sure every customer issue gets resolved as quickly as we can.<p>Link for this article goes to our status board which serves as the authoritative source of information on this incident. We will be delivering updates every half hour or as new information becomes available.<p>We are very sorry. This is far short of our commitment to you. We&#x27;ll have more on the incident and what we will do to make this right later today.

For people commenting about overdraft charges, Dreamhost went through this a while back. I was in college at the time, and didn&#x27;t realize they&#x27;d drafted a second charge to my account. A series of small transactions later, and I was $300 in the red with very little income to fix it. They offered to reimburse overdraft fees for those affected, but I never was reimbursed.<p>This is also great as a &quot;how not to tell customers that you just double billed them.&quot; At least Twilio is getting this right.<p><a href="http://dreamhost.com/dreamscape/2008/01/15/um-whoops/" rel="nofollow">http:&#x2F;&#x2F;dreamhost.com&#x2F;dreamscape&#x2F;2008&#x2F;01&#x2F;15&#x2F;um-whoops&#x2F;</a><p>Step 1: Don&#x27;t complain to the customer how inconvenient your massive mistake was for <i>you.</i>

This sort of thing has happened before to another company, but that was a case of human error compounded by the fact that there were no software safeguards in place.<p><a href="http://dreamhost.com/dreamscape/2008/01/15/um-whoops/" rel="nofollow">http:&#x2F;&#x2F;dreamhost.com&#x2F;dreamscape&#x2F;2008&#x2F;01&#x2F;15&#x2F;um-whoops&#x2F;</a><p>At the very least, it would be wise for any company to implement billing features that will under bill rather than overbill. Under billing at least gives you the option of notifying users later (or you just eat it). Overbilling can sour a professional relationship to the point of ending it.

We&#x27;ve been charged about 10k and going up...<p>Looks like their auto-recharge balance test keeps returning 0 or less than 0, it charges your card, and loop.<p>EDIT: Looks like they&#x27;re doing something - the UI at least now displays $500, rather than $10k before. The only real confidence I&#x27;ll get is when I can see todays account transactions from the bank.

Apologies for nit-picking at punctuation, but occasionally it matters :-)<p>&quot;Twilio is over-billing ...&quot; = Twilio is charging more than they should.<p>&quot;Twillio is over billing ...&quot; = Twilio has given up on billing. It&#x27;s <i>so</i> over.<p>Just my personal bugbear - but sometimes it&#x27;s worth getting this sort of thing correct. Apologies if this is just a US&#x2F;UK difference, also (I&#x27;m in the UK).

I see a lot negative comments here. Frankly, I understand people&#x27;s frustration, but I think Twilio are dealing with this in the right way.<p>I was one of the customers affected. Luckily for me, my credit card automatically blocked the transaction after the 10th time I was auto-recharged, meaning I had a cap on how much I paid. I can totally understand the frustration of people discovering that they suddenly have 100&#x27;s or even 1000&#x27;s of dollars worth of charges to their credit card.<p>On the other hand, by the time I noticed that something strange was going on, Twilio&#x27;s status page already said they were looking into it. I contacted their support right away, and they answered within 20 minutes, during what must be a customer support crisis.<p>And now RobSpectre is on HN giving very helpful answers.<p>So, no, it&#x27;s not their finest work (as RobSpectre has said), but at least they&#x27;re dealing with it quickly, and hopefully professionally as well. Let&#x27;s let this play out before we jump up and down on a great company that we all love.

This is bad. It&#x27;ll badly affect people with low bank balances and high overdraft charges - it&#x27;ll be interesting to see how Twilio reimburses people for charges banks levy on customers.

I&#x27;m glad they (and most of us) are not in the business of making insulin pumps or other software where such bugs could have potentially lethal consequences.

I&#x27;ve been using Twilio for over 3 years now and haven&#x27;t experienced any significant downtime or service issues. Their support has always been amazing. Obviously this situation sucks but if there&#x27;s any company I&#x27;d trust to make it right it&#x27;s Twilio.<p>From a developer point of view, we all have bugs. I remember the first time one of my bugs impacted customers and it was awful. If there&#x27;s any community that should be understanding that things like this happen in the tech world, it should be us.

I think the fact of the matter is as a developer I want to get angry at them and then remember this is my own worst nightmare - mistakes happen, the true test will be in how Twilio reimburse people.

I work in telecom, in a slightly different vein from Twilio&#x27;s and I deeply empathize with the pain of billing.<p>Even if Twilio&#x27;s somehow able to ignore the pain of telco engine rating and least-cost-routing, there are still so many places you can make an error and cause something like this.<p>Kudos to Rob and team for owning the issue and I hope the Twilio folks get this sorted out (I&#x27;m confident they will). It&#x27;s worth mentioning that I&#x27;ve never seen another billing issue of this size from Twilio so one in 3-4 years isn&#x27;t too awful. Compared to AT&amp;T it&#x27;s probably generous! (Facetious but with a grain of truth).<p>Good luck guys and keep rocking. Here&#x27;s hoping everything works out.

Rob from Twilio again. Our CEO Jeffiel just posted an update on the situation on our blog which can be found here: <a href="http://www.twilio.com/blog/2013/07/billing-incident-update.html" rel="nofollow">http:&#x2F;&#x2F;www.twilio.com&#x2F;blog&#x2F;2013&#x2F;07&#x2F;billing-incident-update.h...</a><p>Full text also here:<p>At 3:28am PDT&#x2F;11:28am GMT, our monitoring systems reported an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. This incident affected 1.1% of customer accounts. The on call team immediately began an incident response, using the Twilio status dashboard at status.twilio.com to update customers at regular intervals. By 6:24am PDT&#x2F;2:24pm GMT all suspended accounts had had service restored. The same problem briefly re-occurred at approximately 12:30pm PDT&#x2F;8:30pm GMT, affecting 0.3% of customer accounts, which were immediately remedied.<p>At this time, the Twilio billing system is offline and account-balances are not being updated in real-time until we fully resolve the issue. We are actively processing credit card refunds, and you will see a transaction void or refund on your credit card statement shortly (most banks process these within 24-48 hours). While the billing issue is being resolved, all voice and messaging services continue to operate normally.<p>In addition to refunding erroneous credit card transactions, Twilio will also be crediting affected accounts an additional 10% of their last 30 days’ spend. We recognize that it’s not about the money, but our responsiveness to the situation that matters to you. If your account was affected, please consider this credit an acknowledgement of the inconvenience we’ve caused you. If affected customers incurred overdraft or over-limit fees due to this incident, we will also make them whole. We are in the process of contacting all affected customers via email. Additionally, we will be releasing a full postmortem on the incident once all events and root causes are known, as well as detailing the corrective steps we’ll be taking.<p>Our focus is on providing you the best quality service and experience, and we recognize that today’s disruption came up short of what you expect from Twilio. Please accept our apologies and know it is our mission to always be earning your trust and business.<p>As many have you have pointed out on Twitter, this is not a fun day for our team, and especially our engineers. We appreciate your support and your patience.<p>Sincerely, Jeff Lawson CEO &amp; Co-founder

Haha, the one time in my life having an outdated credit card on file paid off.

Goodbye, twilio...

Yeap had this ourselves. Auto recharge actually took $500 rather than $20!

This is one of the reasons I like to use my credit card over a debit card. Knowing that errors like this don&#x27;t affect my mortgage payments makes me feel safer.

This has gotta suck. Many cards have steep overdraft fees.

My account was impacted by this as well, however, it looks like my credit card company blocked the fifth recurring bill (and I woke up to a call from the Fraud Team at my credit card), so no real damage has been done. Support answered my email within 10 minutes, and promised to help me clean everything up.

The worst part is that the calls aren&#x27;t getting updated with prices right away... so if you built a system that relies on getting the price info at the end of the call, you probably aren&#x27;t pretty happy now.

Oh dear, not nice at all if you&#x27;re on a budget

Hit -- but we only see an overcharge (3x). Service seems to be back up. Do you have a support number our ops team can contact (they&#x27;re paranoid).

Another example of the amateur nature of Twilio. Good for $30 a month developers but bad for businesses that depend on their phone system.

Mistakes like this really suck, but I have no doubt that Twilio will make it right.

painful reminder that you need to test payment processing code, by hand or however, after updating it.<p>i&#x27;ve run some ecommerce sites and this is basic stuff - new payment modules, time to test card processing works properly.

Should have used &lt;= instead of &lt;<p>Best of luck

Wow. Seriously? Learn to code.