Question
My service running in a pod output too much log and cause low ephemeral storage. As a result, the pod is evicted and other services can't deploy to k8s.
So how I can determine pod resource ephemeral storage requests and limit to avoid this situation? I can't find any best practice about ephemeral storage.
Answer
Note that by default, if you have not set any limits on ephemeral-storage
the pod has access to the entire disk of the node it is running on, so if you
are certain that the pod is being evicted because of this, then you are
certain that the pod consumed it all. You can check this from kubelet logs, as
kubelet is the guy in charge of detecting this behavior and evicting the pod.
Now, here you have two options. Either you can set an ephemeral-storage
limit, and make a controlled pod eviction, or just get an external volume, map
it into the container, and get the logs outside of the node.
You can also monitor the disk usage, as suggesting shubham_asati, but if it is eating it all, it is eating it all. You are just going to look at how it is getting filled out.
