We already do that to an extent and it would work fine for the most part if the rest of the company did it. We have remote developers in the UK, Germany, and the US. There was talk of closing the doors on our TX office and having them all work remotely. The way our projects are designed, we can work from anywhere in the world.
The only disadvantage of working like that is that you lose the ease of contact and collaboration that comes with meeting in an office. So much is lost in teleconferences, but I would say it has become less of an issue over time.
I still think there is value to sitting across the table from someone during a meeting versus talking to them over the phone.