i've used S3 on a couple of projects to offload customer storage and bandwidth costs from our network. We don't really have a clean method for billing for disk usage or bandwidth for the applications we host. Customers with large bandwidth or storage requirements for their data are now placed on S3, and we simply pass the cost onto them.
I'm a fan of it.
No experience with EC2. I read a blog about Google App Engine competing with EC2, but when I glanced at it, it's completely different(e. g. all application code must be in python...no native code, etc)