In my hobby system, I just noticed something weird about my production data… the values looked like some data had been lost.
Solution: I wiped the whole system. In seconds, it had rebuilt itself from one of its peers.
At a prior position, I spent many a night restoring data from backups, running data integrity scans, and generally trying to come up with ways to protect data. The organization spent millions of dollars on wild and complex systems. In light of all that work, I started investigating distributed systems and stumbled on CouchDB (and all its commercial variants).
As of today… I am a fan.