Kubernetes Logging with Splunk

In my work as an open source developer on the Partner Catalyst Team within Microsoft, I get a chance to work with partners and help them succeed on Azure. Recently, we hosted a hackfest with a partner to help them migrate some of their workload to Kubernetes. The problem that we helped solve with the partner is a common problem with any microservices architecture -- with your multitude of services deployed across a variety of machines, how does one centralize their logs such that they are able to debug services when things go awry?