The reason I'm using GMail (as opposed to FastMail) is primarily their terrific integration with Android. The excellent GMail app, the contact and calendaring syncing.

I'd really love to use Fastmail, but I see no way I could sync my contacts in a seamless way.

I can't see how your solution would solve this, so no, I would not use.