Using Salt and Vagrant for Rapid Development

Using Salt and Vagrant for Rapid Development

Written by: Peter Meulbroek, Global Head of DevOps Solutions

One of our main jobs at Risk Focus is to work closely with our clients to integrate complex tools into their environments, and we use a plethora of technologies to achieve our clients’ goals. We are constantly learning, adopting, and mastering new applications and solutions, and we find ourselves constantly creating demos and proofs of concept (PoCs) to demonstrate new configurations, methods, and tools.

We use a variety of applications in our deliverables but often rely on Salt for the foundation of our solutions. Salt is amazing: it combines a powerful declarative language that can be easily extended in Python, a set of components that supports a diverse array of use cases from configuration and orchestration to automation and healing, and a strong supportive community of practitioners and users.

In my own work I’m often off the grid, traveling to clients, at a client site, or in situations where a lack of internet access precludes me from doing work in the public cloud. In these situations, I rely on technology that allows me to experiment or demonstrate some of the key concepts in DevOps from the laptop. Towards this end, I do a lot of this work using Vagrant by Hashicorp. I love Vagrant. It’s a fantastic platform to quickly create experimental environments to test distributed applications. Vagrant’s DSL is based on Ruby, which fits in well with my past developer experience. Finally, Vagrant is easily extensible: it delegates much of the work of provisioning to external components and custom plugins and comes with Salt integration out of the box.

After working with Salt and Vagrant on a set of demos, I’ve decided to share some of the tools I put together to improve and extend this basic integration. The tools are aimed at two goals: faster development of new environments and environmental validation. In pursuit of the first goal, this post is about host provisioning and integrating Salt states. In pursuit of the second, I will post a follow-up post to describe how to generate ServerSpec tests to validate newly-created hosts. These tools offer a quick path to creating and validating virtual infrastructure using Salt.



The ironic aspect of using Vagrant is that, although it is implemented in Ruby and the central configuration file (the Vagrantfile) is implemented in Ruby, the actual implementation of Vagrantfiles is un-Ruby-esque. The syntax can be ugly, validation of the file is difficult, and the naïve implementation is not DRY*. However, with a bit of coding, the ugliness can be overcome. I’ve put together a set of provisioning configuration helper classes written in Ruby that allow me to more succinctly define the configuration of a test cluster and share code between projects. The idea behind the classes is very simple: extract from the Vagrantfile all of the ugly and repetitive assignments so that creating a simple Salt-ified platform is trivial.

The code for this post is found at Readers are encouraged to check it out.

*Don’t Repeat Yourself 


The post assumes you have Vagrant installed on your local machine, along with a suitable virtualization engine (such as Oracle’s VirtualBox).

1.) In your vagrant project directory, check out the code

git clone

2.) Copy the configuration file (vagrant-salt/saltconfig.yml.example) to the vagrant project directory as saltconfig.yml

3.) Copy the sample vagrant file (vagrant-salt/Vagrantfile.example) to the vagrant project directory as Vagrantfile (replacing the default), or update your Vagrantfile appropriately

4.) Set up default locations for Pillars and Salt States. From the Vagrant project directory:


5.) Initialize the test machine(s)

vagrant up

Congratulations. You now have an example Salt cluster with one minion and one master. The bootstrap script has also created a simple Salt layout, complete with pillar and state directories and top files for both.



The examples directory contains a few sample configuration files that can be used to explore Salt. These are listed in vagrant-salt/examples/topology and include:

  • One master, two minions
  • One master, one syndic, two minions

To use either of these topologies, copy the example Vagrantfile and saltconfig.yml to the Vagrant project directory and follow instructions 2-5, above.


Deep Background: The Classes

The development necessary to get Vagrant and Salt to work together seamlessly is key management and configuration – creating and installing a unique set of keys per cluster to avoid reuse of keys or potential vulnerability.

The code consists of a factory class to create the configuration for Salt-controlled hosts and a set of classes that represent each type of Salt host (minion, syndic, master). Each class loosely follows the adapter pattern.

When put into action, the factory class takes a hash that specifies the hosts to be created. Each host created is defined by a value in this hash. It is quite convenient to use a yaml file to initialize this hash, and all examples given below list the configuration in yaml. Example yaml configurations are provided in the code distribution.

There are three classes that can be instantialized to create configuration objects for each of the Salt types. All configuration objects need two specific pieces of information: a hostname and IP. If a host is to be included in Salt topology, the configuration must include the name of its Salt master.

  • The base for “salt-ified” hosts is the minion class, which corresponds to a host running a Salt minion.
  • The master class corresponds to a host running a Salt minion and also the Salt master process.
  • The syndic class corresponds to a Salt master that also runs the syndic process.

The Configuration Structure

The configuration structure is used by the factory class to instantiate a hash of host objects. It has three sections: defaults, roles, and hosts.

The defaults section specifies project-wide defaults, such as number of virtual CPUs per host, or memory per host, as follows:


memory: 1024

cpus: 1

Entries in this section will be overwritten by the role- and host-specific values. This section is particularly useful for bootstrapping strings.

The roles section specifies values per-role (minion, master, syndic). This gives a location to specify role-specific configuration, such as the location of configuration files.




The hosts section specifies the hosts to create and host-specific configuration. Each host must, at minimum, contain keys for role, IP, and the name of its master (when it has one):




role: minion

master: master



role: master

master: master

Note that per-host values (such as memory or cpu count) can be added here to overwrite defaults.:



cpus: 2

memory: 1536


role: master

master: master


The Vagrantfile

Incorporating the configuration classes into a Vagrantfile greatly simplifies its structure. The factory class creates a hash of configuration objects. Executing the Vagrantfile iterates through this hash, creating each VM in turn. The objects store all necessary cross-references to create the desired topology without having to write it all out. The configuration objects also hide all the messy assignments associated with the default Salt implementation in Vagrant and allow the Vagrantfile to remain clean and DRY.

The file Vagrantfile.example in the distribution shows this looping structure.


Integrating Salt States

The above description shows how hosts can be bootstrapped to use Salt. Of much more interest is integrating Salt with the configuration and maintenance of that host. This integration is fairly trivial. Included in the vagrant-salt/bin directory is a bash script called “” that will create a skeleton directory for Salt states and pillars. This directory structure can be used by the Salt master(s) by including the appropriate Salt master configuration. For example, with default setup, the included Salt master configuration will incorporate those directories:



role: master


memory: 1536

cpus: 2

master: master




– /vagrant/saltstack/salt



– /vagrant/saltstack/pillar



Salt is a very powerful control system for creating and managing the health of an IT ecosystem. This post shares a foundational effort that simplifies the integration of Salt within Vagrant, allowing the user to quickly test deployment and implementation strategies both locally and within the public cloud. For developers, it gives the ability to spin up a new Salt cluster, validating configuration, states, and the more advanced capabilities of Salt such as reactors, orchestrators, mines, and security audits. The classes also provide an easy way to explore Salt Enterprise edition and the visualization capabilities it delivers.

The next post in this series will focus on validation. As a preview, at Risk Focus we strongly believe in automated infrastructure validation. In the cloud or within container management frameworks, addressable APIs for all aspects of the environment mean that unit, regression, integration, and performance testing of the infrastructure is all automatable. The framework described in this post also includes a test generation model, to quickly set up an automated test framework for the infrastructure. Such testing allows for rapid development and can be moved to external environments. In the next post, we’ll go through using automatically generated, automated tests for Salt and Vagrant.

Taking a Bite out of Apple – IT Projections for 2019

Taking a Bite out of Apple – IT Projections for 2019

Written by: Cary Dym

2019 is off to a rotten start for Apple.  On January 4th, Apple hit a 52-week low of $142, 39% off of its all-time high hit in October 2018.   (By some estimates, Warren Buffet has suffered a $4B paper-loss on his Apple stock).  While most of the attention to the fall of Apple has been focused on weak demand from China, another less-discussed factor is that people are keeping their phones longer.   The average hold time for iPhones has increased from 24 months to 36 months driven by higher non-subsidized costs of new phones and very few “must have” features.

So, is this recent change in consumer purchasing behavior also representative of a broader trend in corporate IT spend?   According to a recent report from Computer Economics entitled “IT Spending and Staffing Benchmarks 2018/2019”, while overall IT Budget spend in 2018 increased slightly as a percentage of revenue, capital spending as percentage of IT spending remained flat at a five-year low of 18%, off of a high of 23% in 2014.   Not surprisingly, security/privacy continues to be the major spending priority.  Interestingly, security is tied with another priority—cloud applications – and cloud infrastructure spend is a close third.   Despite the focus—and spend—on  cloud applications and infrastructure, the report goes on to state that “only 20% of companies have converted at least half of their business applications to the cloud”.   This leaves a huge growth opportunity in the Cloud. 

We saw a decrease in overall IT CapEx in 2018 offset by an increase in OpEx, primarily driven by Cloud consumption, but what about the third leg of the IT spending stool – staffing?  On average, IT staffing budgets for most organizations stayed flat in 2018.   While hiring is slowing for lower-level skills such as computer operations, scheduling, and lower-level tech support positions; higher-level skills show increasing demand.   Lower-level skills are being replaced by automation, driving increased IT organizational efficiency.   Meanwhile, demand for higher-skilled resources continues to outpace supply, driving up salaries and/or forcing IT organizations to look to outside organizations for skilled resources. Upskilling resources coupled with automation is a top priority. 

For 2019, Gartner predicts that spending on commodity items, such as communications and data center technologies like servers, is expected to be either flat or down, while spending on IT services and software is expected to go up.   They forecast an 8.3% growth for 2019 in Enterprise Software driven primarily by “Everything as a Service”.    An example for this growth is AWS Aurora – Amazon’s MySQL and PostgreSQL compatible Cloud service offer – which continues to be the fasting growing service in the history of AWS.   Meanwhile, Gartner suggests that the IT services market – the range of services that assist individuals and enterprises in implementing, managing, and operating the wide variety of processes, systems, software, equipment, and peripherals – will top $1 trillion in 2019.

I am personally quite excited about the projection for 2019 and the opportunities to help our clients with their IT priorities.     The needs of our clients reflect the priorities predicted by Gartner and outlined above: they plan to drive automation across their build, deploy and operational workstream and continue to leverage Cloud services and cloud infrastructure as they are decreasing their investment in data center technologies.   In addition, our clients look to us to augment their internal IT organization with highly skilled resources, but ultimately want coaching to help build up their expertise in house.  At Risk Focus, we see a great opportunity to help our clients form and execute their plans, while coaching their internal teams to adopt new practices and work within new processes.  To this end, we continue to build out our bench of skilled DevOps players and coaches in our centers of excellence in NY, Toronto, Pittsburgh, London, Amsterdam and Riga. 

As for the outlook of Apple, as stated in January 5th NY Times article entitled Apple’s Biggest Problem? My Mom, “If there’s an app – maybe its Fortnite 2 – that I can’t run on my existing iPhone, a new iPhone will be on every teenage boy’s shopping list.” Platforms are ultimately only as valuable as the services they enable! 

Wishing you all a Happy, Healthy and Prosperous 2019!

Risk Focus On Infrastructure As Code

Risk Focus On Infrastructure As Code

Infrastructure as Code

Written by: Cary Dym, Global Head of DevOps Business Development

Infrastructure as Code (IaC) transforms and automates the manual process of standing up datacenter environments and processes, such as hardware instantiation, networking, run books, and appliance and software configuration, into automated deployment and configuration.The IaC concept has been around for several years in both startups and many tech firms and is gaining wider traction.  TechNavio cites the increased adoption of IaC as a major trend across all industries and geographies in their Global DevOps Platform Market 2018-2022 report.

Every industry is challenged by Digital Disrupters:  firms that are competing based on enhanced capabilities and lower costs derived from digital innovation.  According to the 2018 IDC Whitepaper, Designing Tomorrow, “Over 67% of companies believe a digitally enabled competitor will gain a competitive advantage within the next five years.”  Traditional companies must be able to move faster at lower cost, and yet continue to manage risk.  Firms willing to undergo digital transformation are able to achieve this with IaC.  Infrastructure cost (CAPEX) and human cost (OPEX) can be reduced by leveraging the dynamic and self-service capabilities that IaC provides.  Increased velocity means recasting multi-step, multi-hour, manual processes—such as racking servers, loading software patches, installing services and applications, configuring networks, and enabling storage—into automated, repeatable, scalable processes that are performed in minutes.  When done properly, IaC reduces risk by addressing traditional IT problems, including configuration drift, human error, inconsistencies, and loss of context.

These additional capabilities – faster delivery of infrastructure, and consistent configuration during the software delivery cycle – allow organizations to make changes faster, with more confidence, and lower risk.

A good place to start Digital Transformation is implementing IaC to facilitate adoption of DevOps practice.  Firms starting on this journey are faced with the hard task of assessing whether the organization has skills and know-how to embark on the journey alone or requires collaboration with skilled practitioners.  Most “not-born-in-the-cloud” firms realize they need to bring in outside resources (unfortunately, sometimes after first failing internally).  Risk Focus has broad industry expertise across Finance, Healthcare and Telecom industries with deep expertise in IaC technologies.  We realize that even large journeys start with a single step and have developed a unique Player-Coach engagement model that facilitates new DevOps principles, enabling demonstration of best-practices through quick-win projects.

At Risk Focus, we are agnostic (yet opinionated) about the tools we use. Our choices are informed by a variety of factors and determined by our clients’ needs.  However, we do have our favorites.  One such tool is Terraform, which is the service provisioner and infrastructure orchestrator in the suite of offerings by HashiCorp. Terraform is cloud-agnostic and supports all major clouds, both public and private.  In hybrid environments where there are advantages to a single set of tooling, Terraform allows our practitioners to quickly develop, validate and roll out orchestration templates.

We implement CM with two tools:  Salt and Ansible.  Ansible focuses on simplicity, and getting going is quick, changes are easy to understand, and organizational adoption tends to be fast.  We recommend Saltfor organizations with greater infrastructure complexity. Salt has a completely declarative model that includes components to dynamically manage configuration and detect drift, along with the ability to layer buildouts and react to signals from the environment, changing infrastructure dynamically in response to changing conditions.  These abilities necessarily require additional complexity and result in a steeper learning curve, but clients with sufficient scale, compliance requirements, or complexity find great benefit from the additional features.

At Risk Focus, our Cloud and DevOps team support transformation initiatives and demonstrate domain expertise in the following areas:

– Infrastructure as Code Orchestration with tools like HashiCorp’s Terraform, as well as cloud-native Orchestration with CloudFormation, ARM, and HEAT.

– Configuration automation with technologies including Salt and Ansible.

– Migrating applications to public cloud, including re-architecting of applications to become more cloud-compatible or cloud-native.

– Containerization including extensive experience with Docker, Docker Swarm, OpenShift, Kubernetes, EKS, GKE, and

– Cloud migration and hybrid cloud implementation using VMWare, Openstack, AWS, GCP and Azure.

– Process and methodology improvements and CI/CD pipeline implementation leveraging tools such as Git, JIRA, Jenkins, and

– Multi-cloud Monitoring and Log aggregation via Splunk, Elastic, and InfluxDB.

“Tour of Cloud Computing” – In Depth Interview

The August 23 Jaxenter interview of Peter Meulbroek, Head of DevOps and Cloud Solutions at Risk Focus by journalist Gabriela Motroc entitled “A Tour of Cloud Computing” dives deeply into several key topics.

The interview is organized around the following themes:

Security – Discusses the new paradigm.

Benefits – Discusses key benefits like automation and the self-service nature of the cloud.

Preferred Tools and Technologies – Describes the various technologies that Risk Focus prefers for Configuration    Management, Orchestration, Packaging and Distribution, Data Masking, Containerization, and Monitoring.

The limitations of a Cloud-Neutral approach.

The article gives Meulbroek the platform to share the approach that Risk Focus brings to clients grappling with a Cloud Strategy. For instance, regarding Cloud-Neutral strategies, Meulbroek states “Cloud-neutral adds a large amount of complexity and risk to a migration, without really solving the issue”.

Regarding Security, he states “the old, obsolete paradigm for security — the perimeter defense — has gone the way of the curtain wall and needs to be replaced with defense in depth.  Nor is it enough to manage data security between applications. Data, at rest or in flight, needs to be protected at all levels within an application, and managing security for an application is largely managing access to decrypt narrowly-focused cohorts of data”

Read The Full Article Here

DevOps Culture & The Meaning of “However”

DevOps Culture & The Meaning of “However”

We’d love to embrace DevOps; however

As I prepare to embark on the next phase of a career focused on recognizing new trends and accelerating the adoption of emerging technologies, I harken back to 2013.  I had recently landed in Tel Aviv with my family for a 2 year stint and was having a discussion with an Israeli colleague about the cultural differences between the US and Israel. He was expressing his frustration from his time living in the US with the word “however”.   He explained that there is no equivalent in Hebrew for the word “however” when encountered – which happened quite frequently – as follows: “I’d really love to help you with your driver’s license application; however…” He quickly realized that no matter how he tried to negotiate, cajole or plea, in the US, “however” meant “no way”.

And this got me thinking to the challenges facing organizations going through Digital Transformation – the process of leveraging new processes, technology and data to improve productivity, increase financial performance and remain competitive. In discussions with clients,  I often hear “We’d love to embrace DevOps; however we are not properly organized or staffed to implement a DevOps Strategy we don’t know where to start our management wants to see quick results and DevOps sounds big and costly we can’t just throw everything into the cloud, we have to be mindful of compliance

 I’d argue there is a bigger HOWEVER at play which is that Digital Transformation is difficult and many initial attempts tend to fail; HOWEVER, not transforming is not an option.   According to IDC’s April 2018 report Designing Tomorrow ”73% of companies will either be out of business or marginalized if they don’t transform.”

So how to resolve these conflicting “howevers”.   The concerns expressed by organizations are valid and should not be brushed outside.   However, digital transformation can still occur while recognizing and acknowledging the challenges above.   And that is why I am so excited to join Risk Focus! Bringing a unique blend of a PLAYER/COACH model together with their years of experience in dealing with risk mitigation, Risk Focus is singularly focused on helping organizations deal with these challenges.

Our engagements begin with a listening workshop where we gather information on the client’s unique organizational challenges – structural, cultural, resources, regulatory/compliance and technology. We then align on a “quick win” project that allows our player/coaches to work directly with the  organization on a specific deliver that can be implemented using new processes and technologies including Infrastructure as Code, comprehensive CI/CD pipeline, Container solutions, and Cloud transformation. These quick win projects usually run 4-6 weeks and are intended to deliver the following objectives:

Facilitate organizational alignment around Dev/Ops

Initiate “on-the-job” hands-on training

Implement and adopt new technologies

Identify a backlog of follow-on projects and deliverables

Demonstrate a low-cost quick win as a Proof of Possible

 I am thrilled to be part of the Risk Focus team and look forward to working with you to enable your organization to address your cultural HOWEVERS!