SaltStack: Driving SecOps Compliancy Through Event-Based Automation

SaltStack: Driving SecOps Compliancy Through Event-Based Automation

Guest Post Written by: Sean BrownArea Director, EMEA @ SaltStack

  • Automation is a necessity to deliver compliance and configuration management and for all enterprises operating at scale
  • Many existing management platforms were built for a previous generation of IT and cannot scale nor flex to cloud native operations
  • An opportunity exists to reduce cost and risk by considering bi-directional, closed loop event-based automation for configuration management and SecOps compliance

Download your own copy of the SaltStack SecOps white paper HERE.

Whether you need central heating or air conditioning, you’ll be familiar with the process of defining and setting a desired state, in this instance setting the temperature.  The sensors within the system then provide constant feedback and the desired state is maintained until the system receives a new target state.  More comprehensive systems allow for more finely grained defined states to be set, perhaps by room or by time of day. 

Platforms delivering change and configuration management in IT offer similar capabilities albeit with considerably more variables to be managed and most likely involving several people with responsibility for the service level management of the platforms in operation.  How desired state is achieved and maintained varies by platform and approach.

Some platforms can define a target state and deploy this to a system under management, it is a simplex operation – “top down”.  Once a target state is set there is no ability to receive feedback as this platform has no monitoring capability.   The target state deployment can be scheduled, to re-instantiate the target state, perhaps at an hourly, daily or weekly interval but between intervals there is no visibility into the state of the system under management. 

“75% of organizations still don’t automate server or application provisioning” [Gartner]

This is analogous to setting your room temperature at the radiator and having no knowledge of the changing weather conditions, time of day, level of occupancy in the room or knowledge of the window that someone unexpectantly left open.

Other platforms can offer more capability.  A target state can be defined, deployed and another, separate platform can be deployed to monitor state.  Many of the security compliance propositions available today work in this way, deploying agents to the systems and then reporting back when the system under management deviates from target state (or compliant state in this instance).

Average cost of a data breach in 2018 was $3.16m and the likelihood of a recurring breach over the next two years: 27.9%  Without automation, estimated cost increases to $4.43 million, a $1.55 million net cost difference. (IBM/Ponemon)

At this point two teams work together, one to report the compliance drift and the other tasked with remediating that drift back to complaint state.  It works but requires integration of both platforms and processes, resulting in additional cost, risk and operational inertia.  When the systems under management span multiple data-centres, technologies, geographies and operating domains (on premise, hosted, public cloud), more teams have to be involved resulting in more risk, inertia and ultimately cost.

 

 

Of course, the analogy of domestic heating systems and complex enterprise IT is limited. A closer comparison may be found in aviation. Modern airliners are a collection of systems and sensors, operating in concert with each other utilising real-time feedback and automation. The systems are too complicated for any individual pilot to manage and whereas initially other aircrew were added to share the workload, the last 30 years have seen the movement of many responsibilities to automated control logic capable of delivering a safe, repeatable and predictable outcome (destination reached) irrespective of the changing flight conditions (weight, balance, trim, air speed, fuel usage, wind direction, noise regulation, schedule).

IT will consume 20% of all generated electricity by 2025.  A 2016 Berkeley laboratory report for the US government estimated the country’s data centres, which held about 350m terabytes of data in 2015, could together need over 100TWh of electricity a year by 2020. This is the equivalent of about 10 large nuclear power stations.

It may be argued that IT is as complex (micro-services, containers, VM’s, public, private, hosted, Linux, Windows, legacy), equally regulated (PCI, CIS, DISA-STIG etc.) and more environmentally conscientious than ever before. 

For this reason, enterprises are looking to optimise how they manage IT and the systems they have deployed in the past for this purpose are now too brittle to successfully incorporate more modern design patterns such as cloud and containers, or too expensive to scale to the 100k plus individual management targets many mid-size and enterprise.

A solution may be more integration.  For example, the integration of a dedicated monitoring system alongside an agent-based compliance reporting system, a configuration management platform and a service level management/ticketing system that together allow the enterprise to define, target, deploy, monitor, react and report.  However, interdependent integration has associated costs and risks that many budget conscious businesses would rather avoid. 

This pattern is playing out in the public cloud today whereby enterprises are finding that to support AWS, Azure and GCP (for example) along with their associated API’s and licensing is a false economy and a more one-stop-shop approach is economic reality. 

Enterprise want to set their own direction.  Indeed, this is commercial necessity in both competitive market and in public sector where organisations are seeking any combination of differentiation, cost optimization or risk reduction. 

SaltStack addresses the customer needs stated above and more.  Designed as an extensible, massively scalable and high-performance remote execution engine, SaltStack provides event-based automation capabilities that address IT operations, DevOps and SecOps concerns.  SaltStack can help customers achieve target state using either declarative or imperative approaches and utilises an asynchronous message bus architecture to deliver change and configuration instructions at incredible speed and scale.

The SaltStack platform also includes the capability to deploy an agent-based monitoring solution (SaltStack Beacons) that utilise the same event bus to transmit messages back to the core, be they simple notifications of a successful system change/package deployment or something more time or operations risk sensitive such as a service being enabled when it should be disabled -telnet being turned on when it should be turned off for example. Being extensible means that customers can extend or build their own beacon monitors to build a unique insight, command and control capability for their extended estate. 

This capability adds orchestration to automation, resulting in a platform that can interpret environmental change (e.g. file, disk, network, service, login, CPU utilisation, system operating temperature) and can react as required, be it fully autonomously or as part of a service chain process.  When applied to the SecOps use case the ability of an enterprise to deploy a target complaint state to their estate and then have SaltStack keep that estate in target compliant state through self-healing, automated remediation.  No third-party monitoring tools.  No third-party compliance reporting tools.  No integration cost and undeniably less risk.

In March 2019 SaltStack will be launching a dedicated SecOps compliance service.  Utilising the platform described above to deploy target compliance profiles for Windows and Linux systems at launch, this will incorporate the delivery of pre-built, tested compliance profiles for CIS, DISA-STIG and NIST targets as well as dedicated SaltStack beacons for compliance reporting and alerting. 

If a system under management moves out of target compliance the CISO and ITOps team will know, in real-time, and will have the ability to remediate back to target compliant state. Assured compliance at scale.  Just another use-case for SaltStack.

To receive the SaltStack SecOps white paper, please tell us a little about yourself:


Using Salt and Vagrant for Rapid Development

Using Salt and Vagrant for Rapid Development

Written by: Peter Meulbroek, Global Head of DevOps Solutions

One of our main jobs at Risk Focus is to work closely with our clients to integrate complex tools into their environments, and we use a plethora of technologies to achieve our clients’ goals. We are constantly learning, adopting, and mastering new applications and solutions, and we find ourselves constantly creating demos and proofs of concept (PoCs) to demonstrate new configurations, methods, and tools.

We use a variety of applications in our deliverables but often rely on Salt for the foundation of our solutions. Salt is amazing: it combines a powerful declarative language that can be easily extended in Python, a set of components that supports a diverse array of use cases from configuration and orchestration to automation and healing, and a strong supportive community of practitioners and users.

In my own work I’m often off the grid, traveling to clients, at a client site, or in situations where a lack of internet access precludes me from doing work in the public cloud. In these situations, I rely on technology that allows me to experiment or demonstrate some of the key concepts in DevOps from the laptop. Towards this end, I do a lot of this work using Vagrant by Hashicorp. I love Vagrant. It’s a fantastic platform to quickly create experimental environments to test distributed applications. Vagrant’s DSL is based on Ruby, which fits in well with my past developer experience. Finally, Vagrant is easily extensible: it delegates much of the work of provisioning to external components and custom plugins and comes with Salt integration out of the box.

After working with Salt and Vagrant on a set of demos, I’ve decided to share some of the tools I put together to improve and extend this basic integration. The tools are aimed at two goals: faster development of new environments and environmental validation. In pursuit of the first goal, this post is about host provisioning and integrating Salt states. In pursuit of the second, I will post a follow-up post to describe how to generate ServerSpec tests to validate newly-created hosts. These tools offer a quick path to creating and validating virtual infrastructure using Salt.

 

Background

The ironic aspect of using Vagrant is that, although it is implemented in Ruby and the central configuration file (the Vagrantfile) is implemented in Ruby, the actual implementation of Vagrantfiles is un-Ruby-esque. The syntax can be ugly, validation of the file is difficult, and the naïve implementation is not DRY*. However, with a bit of coding, the ugliness can be overcome. I’ve put together a set of provisioning configuration helper classes written in Ruby that allow me to more succinctly define the configuration of a test cluster and share code between projects. The idea behind the classes is very simple: extract from the Vagrantfile all of the ugly and repetitive assignments so that creating a simple Salt-ified platform is trivial.

The code for this post is found at https://github.com/riskfocus/vagrant-salt. Readers are encouraged to check it out.

*Don’t Repeat Yourself 

TL;DR

The post assumes you have Vagrant installed on your local machine, along with a suitable virtualization engine (such as Oracle’s VirtualBox).

1.) In your vagrant project directory, check out the code

git clone https://github.com/riskfocus/vagrant-salt

2.) Copy the configuration file (vagrant-salt/saltconfig.yml.example) to the vagrant project directory as saltconfig.yml

3.) Copy the sample vagrant file (vagrant-salt/Vagrantfile.example) to the vagrant project directory as Vagrantfile (replacing the default), or update your Vagrantfile appropriately

4.) Set up default locations for Pillars and Salt States. From the Vagrant project directory:

./vagrant-salt/bin/bootstrap.sh

5.) Initialize the test machine(s)

vagrant up

Congratulations. You now have an example Salt cluster with one minion and one master. The bootstrap script has also created a simple Salt layout, complete with pillar and state directories and top files for both.

 

Uses

The examples directory contains a few sample configuration files that can be used to explore Salt. These are listed in vagrant-salt/examples/topology and include:

  • One master, two minions
  • One master, one syndic, two minions

To use either of these topologies, copy the example Vagrantfile and saltconfig.yml to the Vagrant project directory and follow instructions 2-5, above.

 

Deep Background: The Classes

The development necessary to get Vagrant and Salt to work together seamlessly is key management and configuration – creating and installing a unique set of keys per cluster to avoid reuse of keys or potential vulnerability.

The code consists of a factory class to create the configuration for Salt-controlled hosts and a set of classes that represent each type of Salt host (minion, syndic, master). Each class loosely follows the adapter pattern.

When put into action, the factory class takes a hash that specifies the hosts to be created. Each host created is defined by a value in this hash. It is quite convenient to use a yaml file to initialize this hash, and all examples given below list the configuration in yaml. Example yaml configurations are provided in the code distribution.

There are three classes that can be instantialized to create configuration objects for each of the Salt types. All configuration objects need two specific pieces of information: a hostname and IP. If a host is to be included in Salt topology, the configuration must include the name of its Salt master.

  • The base for “salt-ified” hosts is the minion class, which corresponds to a host running a Salt minion.
  • The master class corresponds to a host running a Salt minion and also the Salt master process.
  • The syndic class corresponds to a Salt master that also runs the syndic process.

The Configuration Structure

The configuration structure is used by the factory class to instantiate a hash of host objects. It has three sections: defaults, roles, and hosts.

The defaults section specifies project-wide defaults, such as number of virtual CPUs per host, or memory per host, as follows:

defaults:

memory: 1024

cpus: 1

Entries in this section will be overwritten by the role- and host-specific values. This section is particularly useful for bootstrapping strings.

The roles section specifies values per-role (minion, master, syndic). This gives a location to specify role-specific configuration, such as the location of configuration files.

roles:

minion:

grains:saltstack/etc/minion_grains

The hosts section specifies the hosts to create and host-specific configuration. Each host must, at minimum, contain keys for role, IP, and the name of its master (when it has one):

hosts:

minion1:

ip: 1.2.3.4

role: minion

master: master

master:

ip: 1.2.3.4

role: master

master: master

Note that per-host values (such as memory or cpu count) can be added here to overwrite defaults.:

hosts:

master:

cpus: 2

memory: 1536

ip: 1.2.3.4

role: master

master: master

 

The Vagrantfile

Incorporating the configuration classes into a Vagrantfile greatly simplifies its structure. The factory class creates a hash of configuration objects. Executing the Vagrantfile iterates through this hash, creating each VM in turn. The objects store all necessary cross-references to create the desired topology without having to write it all out. The configuration objects also hide all the messy assignments associated with the default Salt implementation in Vagrant and allow the Vagrantfile to remain clean and DRY.

The file Vagrantfile.example in the distribution shows this looping structure.

 

Integrating Salt States

The above description shows how hosts can be bootstrapped to use Salt. Of much more interest is integrating Salt with the configuration and maintenance of that host. This integration is fairly trivial. Included in the vagrant-salt/bin directory is a bash script called “bootstrap.sh” that will create a skeleton directory for Salt states and pillars. This directory structure can be used by the Salt master(s) by including the appropriate Salt master configuration. For example, with default setup, the included Salt master configuration will incorporate those directories:

hosts:

master:

role: master

ip: 10.0.44.2

memory: 1536

cpus: 2

master: master

master_config:

file_roots:

base:

– /vagrant/saltstack/salt

pillar_roots:

base:

– /vagrant/saltstack/pillar

 

Conclusion

Salt is a very powerful control system for creating and managing the health of an IT ecosystem. This post shares a foundational effort that simplifies the integration of Salt within Vagrant, allowing the user to quickly test deployment and implementation strategies both locally and within the public cloud. For developers, it gives the ability to spin up a new Salt cluster, validating configuration, states, and the more advanced capabilities of Salt such as reactors, orchestrators, mines, and security audits. The classes also provide an easy way to explore Salt Enterprise edition and the visualization capabilities it delivers.

The next post in this series will focus on validation. As a preview, at Risk Focus we strongly believe in automated infrastructure validation. In the cloud or within container management frameworks, addressable APIs for all aspects of the environment mean that unit, regression, integration, and performance testing of the infrastructure is all automatable. The framework described in this post also includes a test generation model, to quickly set up an automated test framework for the infrastructure. Such testing allows for rapid development and can be moved to external environments. In the next post, we’ll go through using automatically generated, automated tests for Salt and Vagrant.