For everyone complaining about "how could they not have failover" let me ask you this: Would you take a job with Wells Fargo to fix their infrastructure?<p>And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything?<p>How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done?<p>And now you see why banks have such terrible IT.<p>(One of my mentors actually works at BofA, and says he only does it because he gets to work 6 hours a day, gets a VP title and a ton of money, but nothing ever gets done)
Apparently there was a fire in a Datacenter - theres a thread on /r/sysadmin by an insider.<p><a href="https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/" rel="nofollow">https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...</a>
for those who haven't taken the plunge, now is a good time to move to a credit union, regional bank, or online bank:<p>- for instance, in cali: <a href="https://www.golden1.com/" rel="nofollow">https://www.golden1.com/</a><p>- high interest accounts: <a href="https://www.bankrate.com/banking/checking/best-checking-account-rates/" rel="nofollow">https://www.bankrate.com/banking/checking/best-checking-acco...</a><p>- good rate online bank: <a href="https://empower.me/" rel="nofollow">https://empower.me/</a><p>get better service, lower/no fees, and good convenience without all the economy-damaging selfishness and hubris.
Regional bank checking in. We've been told to reroute certain workloads through other means until 6am central Friday. Regardless of that, the story coming out of this over the next few days is really going to help Business Continuity departments at other institutions plan for this scenario. My teams are running mock scenarios this afternoon based on the auto power shutdown causing a data center to be unable to fail over. Might end up being something else entirely, but it's still a good scenario to investigate.
For those who have worked in bank IT, I have some questions.<p>So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ.<p>Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to recover? What if a data center was nuked? What if two data centers were nuked?<p>How much redundancy is in the system? Are there third-party agencies that track private bank ledgers? How hard is it to take them out too?<p>Ever since I read "The End of Alchemy" (a great book, btw) this thought has haunted me.
You can still access Wells Fargo Online via the direct link: <a href="https://connect.secure.wellsfargo.com/auth/login/present?origin=cob&LOB=CONS" rel="nofollow">https://connect.secure.wellsfargo.com/auth/login/present?ori...</a>
From their API team:<p>Please be advised that the Wells Fargo Gateway is currently unavailable.<p>We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We're working to restore services as soon as possible.<p>We apologize for the inconvenience as we continue to work on a resolution.<p>We will update you no later than 3:00 pm, Eastern Time.
Wow, functionality is still significantly degraded over 24 hours later. I just tried to log in to check on an account balance, and the site was extremely slow. It showed my balance, but failed every time to load the transaction history with "Error in external system" or something like that. The sign in page still shows: "Alert: Some customers may be experiencing issues accessing online and mobile banking. We apologize for any inconvenience."
OK, Hanlon's razor applies here:<p><a href="https://en.wikipedia.org/wiki/Hanlon%27s_razor" rel="nofollow">https://en.wikipedia.org/wiki/Hanlon%27s_razor</a><p>recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor.<p>from the reddit:<p>"throwawayfordays75 1399 points 8 hours ago*2<p>Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. "<p>At this point im wondering what "utility work" was happening.
And I’m not surprised they don’t have a services status page either that many modern companies do offer[0][1][2].<p>[0] <a href="https://api.twitterstat.us" rel="nofollow">https://api.twitterstat.us</a>
[1] <a href="https://help.netflix.com/en/is-netflix-down" rel="nofollow">https://help.netflix.com/en/is-netflix-down</a>
[2] <a href="https://developers.facebook.com/status/dashboard/" rel="nofollow">https://developers.facebook.com/status/dashboard/</a>
I got an email today from WF titled “New year. New look. Continued commitment.” Maybe they are flipping the switch on their “New year. New look...”; which sounds like a new redesign and they are having downtime while switching over.
Well, for a second I thought they have closed business operations completely, keeping in mind how embroiled they were in graft cases in 2008 recession.<p>Pardon my immense ignorance!
Wells Fargo is nothing more than a federally regulated crime organization, supported by the federal reserve and the American tax payer, but I digress.<p>This thread has primarily focused on redundancy and software architecture. That could be the case, but there is no better way to fight proxy war via hacking of banks. It’s a domino effect... pull your money out of the bank, the bank calls loans, then there is credit crunch, investors lose confidence in the stock, value lost etc... the enemy has secretly inflicted civilian problems across the economy. Distrupting the banks and the flow of money can lead to a revolution... or take your mind off of America’s enemy or diverting energy else where.
Bullish on bitcoin b/c of its highly resilient characteristics. Might not process as many transactions per second and it might pollute more than the average bank, but sure as hell dont need to worry about a natural disaster or freak incident single handedly taking out the entire network.