How To Prevent Failure In Containerized Environments

How To Prevent Failure In Containerized Environments
Containers have grown in popularity due to their mobility, speed, and functionality. Since they run directly on the OS kernel, users don’t deal with many operations headaches. Unfortunately, containers come with a lot of security concerns. This can be absolved with proper management and tools. Also, users must understand new threats to maintain consistent security.

To help with managing and understanding threats, Gremlin created their failure-as-a-service software. Gremlin provides users the ability to simulate failures in real time. This builds a greater understanding of system flaws. The software also allows teams to practice for a real failure or threat.

The company recently released capabilities for containers. Gremlin’s new container attack simulation gives users the chance to experience an attack or failure in real time. This kind of experience is invaluable. I recently chatted with Gremlin’s security officer, Frederic Bull, to gain a deeper understanding of container security.

What responsibility do users have when securing containers?

Containers have all the security concerns of any other host environment, so users have the responsibility to continue to secure containers as they would their hardware hosts. But containers also add more concerns, primarily that of exposing the host upon which they reside. Care must be taken to configure host permissions as well as container configuration to ensure the host environment cannot be compromised through its resident containers.

Is there anything being overlooked in container security?

  • Containers are relatively new and do not always support the fine granularity of configuration desired.  An example is configuring resource quotas or tighter control on namespace access.  There is still a great deal of reliance on host configuration for complete security.
  • Many common security monitoring tools have immature container support, or lack support altogether.
  • Native image signing and encryption are not available or are immature.

What new threats do you expect as containers grow in popularity?

I expect to see many threats toward container images, both in exfiltrating data from images and in compromising the integrity of images.  I also expect to see novel new side-channel attacks against containers run from their hosts.

How can developers stay secure while using a public repository?

Permissions for access must be solid, and a rigorous change management process must be enforced (including mandatory security and peer code review).  How that process looks at the low level will depend somewhat on the threat model and risk profile of the application and codebase, but the process should employ technical controls to ensure compliance. Examples would be an inability to merge into critical branches without the requisite approvals and including cryptographic signing of commits to ensure proper identity-verification and non-repudiation.

How can users improve container security with Gremlin?

Practicing proper host-level security is paramount.  Gremlin has safeguards to protect itself from malicious host configuration, but no application can protect itself completely from a motivated attacker with sufficiently elevated access.  This includes things like locking env variables from unauthorized users and ensuring all files and directories have appropriate ‘least privileges’ access.

It’s important to ensure that only those resources that are necessary are shared between the host and containers.  Gremlin does not require root privileges and should never be granted them. The Gremlin Help site ( contains information and descriptions of all resource sharing configurations required, which generally involves granting some targeted Linux capabilities and sharing some Gremlin specific file locations.

A commonly overlooked feature of containers are Quotas, which allow restrictions on container resource usage, and direction on how to handle overages.  Understanding how those quotas interact with the host resources is important, and a proper configuration can improve the safety around events like DOS attacks or even misconfigured or buggy code deployments.