I’ve been a long-time user of Google Apps and have recommended it to dozens of colleagues and associates. I currently use both the Standard (free) and Enterprise (paid) version of apps for several very high profile and long-standing domains. I’ve always considered it to be a reliable and cost-effective option for providing core email, calendar, and other services. However, I’ve recently fallen victim to activity by Google that demonstrates how an enterprise is placing their operations at risk if using Google Apps for their domain.
The first indication that there was a problem was an error message on my iPhone indicating that the password was incorrect for the domain in question. Since the password hadn’t been changed recently, I decided to investigate from my desktop by trying to log into the account. I was greeted with a message that stated “This account has been disabled” and a recommendation to change my password if I was having problems.
I followed the process for changing the password successfully and attempted to log in again. “This account has been disabled.” I consulted Google and found no helpful forum or blog postings on how to resolve this issue. I searched for a way contact Google associated with this issue. No luck. There is an option to obtain dial-in support, but that requires access to your account PIN and since I couldn’t log into the Apps admin panel, I couldn’t obtain the PIN. If you call Google Apps support, there is no way to proceed without the PIN.
I decided to post a message on the support forum for Google Apps. After ten days, there were no replies from other users or “advisors” from Google despite my attempt to bump the thread.
So let’s recap the impact to the domain after more than 7 days:
No email – not only could I not send and receive email, when folks tried to email me they got a hard bounce (account disabled). It is one thing to disable access to email, but bouncing email is catastrophic as communications completely fail instead of being queued in the inbox until the issue is resolved.
No App services – Calendar, Docs, Drive, and all Google App services are completely blocked. These are not critical for this domain in particular, but would be high impact if denied for some of my other domains.
Things were starting to get desperate. What could I do to restore email? Migrate the MX entries to another provider?
Ninety-six hours later, still no resolution. I’ve deliberately delayed migrating to another service provider so that I could provide an accurate perspective and timeline for this blog post, but it would appear that Google had successfully denied service to the business with no way to seek resolution.
I’d searched my email for this domain and the back-up email associated with the domain. There had been no correspondence from Google indicating an issue with the domain or an attempt to notify me of an impending or enacted disabling of the account. Google App services on the domain were killed with no explanation and no recourse.
In desperation, I decided to try and social engineer my way to support. I dialed in and entered the PIN from another one of my domains and once I got a human on the phone explained my issue. They promised they would look into it and call me in a few hours. I gave them my home and mobile numbers and made them promise to call back before hanging up. You guessed it, no call back. During that dial-in process, I was able to get a trouble-ticket issued so after a day I emailed Google Apps support in reference to that trouble ticket. Within a few days I had elevated it to a senior support person who suddenly emailed me a resolution:
“I have reviewed you case and can see that your account was incorrectly disabled due to an internal issue which our technical team has now resolved.
Therefore I have gone ahead and re-enabled your account and you should now be able to log in to use Google Apps.
I am sorry for the inconvenience caused. I understand that you were without access to the Google Apps services and I really appreciate your patience whist we have worked through this issue.”
The problem is now fixed, but not without significant impact to the domain in question and no explanation from GOogle as to how an “internal issue” can disable a business for over a week.
If you are currently using Google Apps for your domain, or considering the service, I would recommend you be aware that you are placing your livelihood in the hands of Google, which has no incentive to provide reliable services or support. Google can disable your domain services at their sole discretion with no recourse other than to move to an alternative service provide and social engineer a resolution that took over a week to resolve.
I’ll be looking at Rackspace and Microsoft as alternative service providers and appreciate any recommendations you may have regarding your experiences with those services.
UPDATE – NOVEMBER 25 2012
There has been a lot of great discussion regarding this article here on CTOVision and at HackerNews. As promised, here are some hints and recommendations on how to resolve this issue or at least plan for minimizing the impact to operations.
Things Google should do:
1) Alert the user at their secondary address if domain services are disabled with a time-expiring dynamic link to generate a trouble-ticket if this is in error. As a provider of services (even free) Google is obligated to at least try and ensure that services are not disabled due to an error on their end. Remember, even with free services, you are paying Google with your user data!
2) Allow a secondary mechanism for obtaining the support PIN in the event the admin panel is inaccessible. Perhaps they have you generate a TXT record in your domain DNS that once verified sends and email to the back-up domain email with your customer number and support PIN.
If you are the domain administrator:
1) Store your support PIN in a secure location outside of the Google ecosystem. I’m a big fan of using 1Password for this type of content, but a piece of paper in a safe would work just as well. You need to plan for not having access to your Admin panel.
2) Ensure your DNS is hosted outside of Google. Most registrars provide free DNS services, but I’m also a huge fan of Amazon Route 53. You need to be able to change your MX record to another email provider if the outage becomes unacceptable.
3) Establish a secondary provider in advance so you can hot-swap if you need to. This step may be cost prohibitive to some, but worth determining if the several hundred dollars per year is reasonable insurance in the event your Google services are disabled. For critical accounts, forward the email to the secondary provider account so you have a back-up of the email content as well.
4) Consider purchasing Google Apps through a reseller. Resellers have experience and access to support that normal “small” enterprise users don’t have. It might be worth having that reseller relationship in the event you have an issue like this.
5) There is a way to generate a Support Ticket in a circumstance like this, it is just hidden behind several layers. I was able to discover it through my conversations with Google, so if you are having this issue here is what I recommend.
- On your Google Apps login page, select the link that says “Can’t access your account.”
- Upon solving the Captcha on the next page look for a link that says “Reset your administrator password by domain verification”
- On the next page, enter a valid email address.
- On the following page, under item 3, you’ll see a link to contact Google support. That link is
https://support.google.com/a/bin/request.py?contact_type=admin_no_access
As of the date of this update, you can navigate directly to that link to generate a support ticket.
Related Reading
