Archives

Google Account Auto Suspension Takes Out Local Business

Google Account Auto Suspension Takes Out Local Business

Google Suite’s Support Channels Prove Effectively Useless

This is what happens when you trust Google to run core elements of your business and that trust is violated. It’s what happens when a company becomes so large and makes so much of its money from one service that they disregard other paid services. So much of our digital life is built around email and increasingly other associated Google products like Google Drive and Google Docs.

So what happens when locked out from all of this? Obviously it becomes very hard to do business. This is currently the case for one of our clients. A local electrician keeps his quotes, emails, and invoices all in Google Drive and Docs. He’s stuck, yesterday he reported that he couldn’t even go into work due to the issue. Problems started for him on Sunday and it is now approaching Friday: 5 days without access to his account.

What happened Here?

We received a call from our client on Monday. He had been having trouble logging in on Sunday but was hoping to resolve it himself. We are resellers so we have access to a panel that lets us reset customer passwords and usually unlock the accounts if they were suspended for too many failed logins. For whatever reason, our client’s account was automatically suspended but did not give us the option to unsuspend it. Over the course of the past week, support was only able to say it’s likely due to their password being compromised. Fine, we should be able to reset it, show them how to setup 2FA, and move on with our business.

That is not how things went.

Enter ‘Support’

Google Suite has a support button where you can be connected to a support technician by phone. This support is tied to domains by a PIN. Support makes no distinction that we are in the Partner program and have a dozen or so domains under management. So we have to provide a second PIN when we call in. Phone support couldn’t help at all, but opened an email ticket. This ticket was answered a little over 12 hours later by a support technician requesting we fill out a form to receive a new ticket number. At this point, it was after 12AM but I pushed on knowing our client was facing increasing pressure to get his account re-enabled. Eventually after a flurry of emails with support mostly confused by there being two open tickets in the system, we were asked to perform a CNAME validation. I couldn’t do this until my client was awake around 8AM the next morning.

A Entire Day Passes

Support didn’t respond to us during our regular business hours after we made the CNAME change. We poked a few times over the day via email. Over 20 hours went by until we received the next response. Here it is:

Google CNAME response

I will admit it was at this point I finally started to lose my cool. Password recovery can take up to 48 hours typically as it filters through support at large companies, but this delay is unreal. Every interaction I have with their support I get a different estimate. When I brought up our SLA agreement on the phone, they explicitly stated that it would not apply and they would not be issuing a service credit for this period of inaccessibility.

The Knowledge Test

After requesting that I add a second CNAME record two days after the first was completed, I was finally sent a list of questions to fill out:

Take a look at it. Notice the second two sections? That’s not how things work for a reseller. We are billed directly by Google for all the domains we have management of, then our clients get billed for that through us. Google doesn’t handle your payment information when you use Google Apps with us. Regardless, I answered all the questions and attached our reseller invoice as proof (at around 3AM). I then went to sleep for a few hours. This is what I woke up to:

So, we resubmitted that and are now waiting some more. I’m hoping that maybe around midnight we’ll have a resolution, but I’m not very optimistic.

To Summarize The Issue

It has truly been a difficult week. As I write this post I’m averaging just over 4 hours of sleep per night. As of right now, 26 emails have been exchanged between me and their support. What is supposed to be the correct support department only seems to respond to their emails between the hours of 12AM and 5AM EST. We’ve logged a total of 2 phone calls, performed 2 DNS CNAME verifications (support demanded we do another last night), reached out to 3 acquaintances that work at Google, and filled out a ‘knowledge test’ questionnaire twice.

All of this and it still feels like we’re at square one. We are suffering real time and monetary losses in lost opportunity (I had to postpone a client initial consult today, for instance) and our client is potentially losing thousands of dollars per day being entirely unable to work while this account is suspended. Considering this has taken up a week of my time now, I can safely estimate that our GSuite profit margin is gone for the year.

Oh. In case you think it’s just us, TechCrunch recently had an article about one of their writers getting locked out for a month.

A final thought: We’ve had sporadic support contact with Google over the years. I am unable to recall a single instance in which support was successfully able to solve an issue. There is no process for escalation. I believe Google’s ‘support’ exists for those unable to read the documentation, not for customers facing real bugs or disruptions.

Final Support Contact

Throughout the day Thursday we tried to open a new ticket to see if we could get our issue assigned to another support technician. This time I spoke to Geremy, who claimed to know Vincent personally. He told me that the case would be escalated to a case manager, but unless I wanted to start over again, I’d need to wait for Vincent to respond in approximately 4 hours. At around 11:30PM Vincent responded with a new knowledge test. We responded within a few minutes, filling out all the questions. Around a half hour later we received this email:

So, all good right? Spoiler: We never heard from Vincent again and never received a password reset link from him.

Conclusion

Support was useless. If anything it caused us far more downtime than not contacting support would have. We figured out a solution around 3PM today and I’ll document it for any Google Suite Partners stuck with the issue:

Problem: Your client’s super administrator account becomes autosuspended and there is no option to re-enable from your reseller panel.

Solution: Reassign super admin privileges to another user on the domain and set the secondary email to your reseller email. You will need to do this through your reseller panel. Go to the domain’s admin console. Select More Controls (at the bottom) > Admin Roles. Assign another user in the domain as a super administrator. If you don’t have another I guess you’ll have to find another way. Once this user is assigned, go back and select company profile. Set this user as the admin, then set your reseller account email to secondary account.

After we did this, we tried password recovery and were sent a reset link instantly that allowed us to change the password on the account without proving we owned it. It let us sign right in. The Admin Console for the domain still shows this user’s account as disabled but it’s clearly not anymore. Our final take from this is that not only is support broken, but the admin console is actually broken. There are two versions of the admin console currently live, so maybe the creation of the new one broke unlock functionality.

I don’t know how to file a bug with them, and I’m so sick of getting run around by their ‘support’ this week that I want to have zero correspondence with them. What I do know is Google doesn’t care about its customers, its partners, or the future of the Internet. Companies like Microsoft, Fastmail, and Rackspace do seem to care about supporting their email-related products. We will not recommend any of Google’s products in the future but we will assist our current clients should they want to leave using Google Suite.

Some Tips On Passwords And Digital Security

This post was originally intended for my personal blog, but I decided to move it to the company since it’s been a dreadfully long time since I worked on our content.

Background

I have around 20 years of experience actively working on researching, securing, and mitigating security issues on electronic devices. In this post I will explain some simple security concepts that any computer (or smartphone user) should be familiar with. I will describe some easy solutions to ease security fatigue. This kind of information is becoming more important as companies get compromised and personal information leaks online. This information assumes an average user. (eg. not a human rights activist operating in the UAE)

Passwords

I’m going to make a generalization: Your passwords are very weak and you reuse the same ones everywhere. I would gather >95% of the audience reading this will fit this generalization. Most have their passwords on a notepad in the drawer next to their laptop. Adding numbers and capitals don’t improve your password very much despite what your bank or government site tells you. So, how to fix?

Picking strong passwords

XKCD did a great comic on picking strong passwords that’s shared frequently:

password_strength.png

Ideal passwords are constructed using the highest entropy possible. Basically, this means we want something as random as possible. But, we want a random password that is hard for a human and a computer to guess. This doesn’t mean the password should always be difficult to type or remember. You can pick a series of words and a numeric sequence to come up with a very high entropy passphrase that’s quite easy to remember. Continue reading because it will only be important to generate a few good passwords. Proper management will allow you to not need to remember more as software can generate and store the rest.

You can calculate password entropy here: http://rumkin.com/tools/password/passchk.php

Bad password: Ubmt2017! (40.5bits)
Good password: Potato1Good2Chicken3Dumplings4 (141.9bits)

I do recommend high entropy passwords because it makes automated guessing using dictionary lists and common patterns (eg. the password mullet with letters in the front and numbers in the back) impossible. It does not hurt to go overkill if it can still be easier for you to work with.

Password Management

Organizing your passwords is important. Our memory is unreliable. I’ve also worked with multiple small business that need to share lots of passwords. Nearly all of them use some form of easily crackable mnemonic. In my bad password example above, this is the most common form of mnemonic I’ve seen: take property/client name, jam a year on, and some random tail like an exclamation point and call it a day. This will work to sometimes keep out a casual guesser. It’ll also stop reuse, so if one site gets compromised, the attacker won’t login to everything. But it won’t stop anyone who breaks into stuff for a living or some of the more . It’s common and easy to figure out. Attackers are looking for it.

Instead, consider using a password manager. Two broad categories are used, offline and online. KeePass is the most well-known offline solution. I recommend an online service for all but the most critical of applications due to the added convenience. Two great online providers are LastPass and 1Password. They are not perfect but they will be substantially safer than password reuse. These services work by giving you an encrypted database of passwords you don’t have to worry about memorizing. Only one strong password needs to be remembered to unlock your secure database of saved random passwords.

How to tell if your password should be changed

You want to change your password if it was reused on multiple sites or if the site you used it on was compromised. What usually happens here is that a hacker gains access to a copy of the compromised company’s password database. This database usually does not give them your exact password. Typically the password is run through a function called a hash function and this scrambles the password many times over and stores it in the database as a hash. When you type your password in to login to a site it runs the function and matches it with the hash that was stored. The reason this is done is so that if someone steals your password it will be difficult for the computer to guess the input that produces the same hash.

However, if you used a weak password, the computer doesn’t have to guess long and can easily access your account. So by using a strong password like I described above, you buy yourself time to change it and also makes it unlikely that a password cracker is going to spend months trying to break your password. There is a site called Have I been pwned? that will allow you to see if your email address was listed in any site compromises.

Two-Factor Security

This is becoming a default feature for banks. Basically, in addition to knowing a password, you need to provide a short number or code from a text message, USB key, or Authenticator app. Text message is not ideal for two factor because it’s not too difficult to convince a phone company to give away your phone number to an attacker. Turning on two factor authentication greatly increases security as even if an attacker has your password they may not be able to login to your account without actually having possession of your phone or keys. Turn it on where available. Google supports it and you can turn it on through the accounts setting page, https://myaccount.google.com/security. We use it inside the company for all critical accounts, and we can turn it on for client WordPress sites by request.

Device Security

Device security is tricky. The advice I give below assumes protection against remote attackers. It does not cover securing against local attackers like a stolen laptop or someone walking into your house/office and installing malware.

Mobile Security

If you’re very concerned about mobile security, buy an iPhone and keep it up to date. This is the best first-line defense I can recommend. Apple’s software has a track record of not being perfect, but since the iPhone 5S, they’ve used really clever hardware design and do things like encrypting the device by default. They also control their updates and support their devices for years longer than most other manufacturers, (the 5S came out in 2013 and gets current updates). If you must buy an Android device, the Nexus line seems to be the best supported with updates. However, these don’t have the same hardware security features of the iPhone and it’s really difficult to limit privacy tracking. Provided you use a strong PIN and/or fingerprint and have the phone signed into iCloud, there’s not much to be had with a stolen iPhone.

Computer Security

I won’t cover MacOS users here because there’s not a sizable percentage and the number of OS threats is lower. For Windows systems, I do recommend Windows 10 despite the bad press and annoying upgrade nag screens. Microsoft has spent a substantial amount of time improving the security of Windows since 7, and the UI on 10 isn’t a nightmare on desktop/laptop like 8/8.1 was. They’re also on their 3rd major revision, called the Creator’s Update, so it’s gotten quite mature.

However, it’s not without its flaws. Windows 10 has advertising and sends a massive amount of information back to Microsoft. I use O&O Shutup10 to turn most of it off, Andrew prefers Spybot Anti-Beacon. Whichever tool you use I recommend applying recommended settings, and if you want to apply extra, I do not recommend turning off features like sending virus samples back to Microsoft.

For virus scanners, most slow the system down and some actively make system security worse. I recommend just leaving Microsoft’s built-in Windows Defender (called Microsoft Security Essentials in Windows 7) as the only active scanner. If you want to run a second manual scanner Malwarebytes has a fantastic scanner. However, the best first line defense nowadays will be common sense and an ad blocker. The ad blocker to use is Ublock Origin, which works on Google Chrome, Microsoft Edge, and Mozilla Firefox. Common sense will mean being very careful of suspicious emails like earlier this year’s widespread Google phishing attack

We hope you find the information in this post useful. Should you have any questions or feel that we should include something feel free to comment. If you need help developing a more comprehensive security solution for your business, call or visit our contact page.