Leading enterprises are adopting a DevOps philosophy based on better communication between developers and operations teams. These practices can enable a much faster pace of feature release, but also have the potential to introduce new software vulnerabilities. Dell Software implemented a software supply chain quality infrastructure that actually reduces risk, said David Mortman, chief security architect at Dell Software at the RSA Security Conference.
Mortman said weak software is most often attacked, yet enterprises spend the least amount of money to protect it. By some estimates, enterprises collectively annually invest $20 billion on network security, $10 billion on host security and $5 billion on data security -- but only $.5 billion on software supply chain security.
Create an Agile process
When Dell moved to Agile software development, it transformed the way software products were released. It compressed the design, development and test cycle and enabled more releases across a shorter interval. But other problems emerged as the software sat on the shelf waiting to be pushed into production. When combined with continuous integration, Dell was also able to move software into production at a faster pace.
Mortman said this was the beginning of implementing DevOps in a way that solved not only development problems, but the challenges associated with getting software into production faster. It extended the design/development/test cycle all the way to deployment. In turn, expedited production made Dell more profitable and efficient. Dell is now seeing about 2,000 deploys per day. But DevOps as a philosophy is not just about releases; it is about security, features and marketing, said Mortman.
Assembled software needs a quality supply chain
Assembling existing code is far more efficient than rewriting it from scratch, observed Mortman. The use of third party code meant that developers could spend a little bit of time reconfiguring existing libraries to do the grunt work, rather than reinventing the wheel. The downside is that this made an acceptable risk worse for Dell since much of this new code was being pushed into production without a thorough security analysis. Therefore, leading enterprises, like Kanban, Lean and now DevOps are adopting many of the same supply chain principles introduced into manufacturing by William Deming.
The pharmaceutical and auto industries have created rich supply chain audit and tracking infrastructure to help protect their products from causing harm. These systems also help to manage relationships with suppliers worthy of trust and provide traceability to allow a recall in the event something goes wrong.
This is particularly critical for software development as more enterprises move into physical Internet of Things (IoT) products that have the potential to cause harm, said Mortman. In software, developers can grab any part from a junkyard and never write down what they did, he added. When Mortman started doing software audits at Dell, he found 198 known execution flaws in the code base, which put the company at risk.
Baking security into the assembly line
Enterprises need to bake two key elements into their software supply process in order to ensure the security of their production software. First, they need to ensure that developers don't assemble applications from libraries with known security flaws. And second, they need to be able to audit production software for security defects when new vulnerabilities are discovered.
Software libraries generally consist of many versions of a given software. Developers tend to use the latest version under the presumption that it is likely to be of higher quality and have fewer bugs. But sometimes new releases open up new security vulnerabilities. For example, 81 of the 85 versions of the Spring Framework have known vulnerabilities, said Joshua Corman, CTO at Sonatype, a secure software supply chain tool vendor.
In addition, security researchers are constantly stumbling across new vulnerabilities in libraries that are widely deployed. Almost every major enterprise was affected by the Heartbleed bug found lurking in plain sight under the TLS system used for authenticating webpages. Affected enterprises rushed to update their software and also had to revoke and replace all of their security credentials. Other widespread security vulnerabilities were discovered in Bash and Apache Struts.
Dell was able to use its software supply chain tooling to identify in five minutes where Heartbleed was implemented, said Mortman. They were able to patch it in another five minutes. Meanwhile, other enterprises spent six months trying to find and patch the vulnerability. Mortman said the pain was so bad that the Cyber Supply Chain Management and Transparency Act was since introduced in Congress. This will require software vendors and SaaS providers to audit and report vulnerabilities in their software supply chain that could affect US government customers.
Standardize software libraries
Mortman said that enterprises should also consider standardizing and reducing the number of libraries in their software infrastructure. For example, he pointed out that General Electric has invested considerable resources to reduce the number of bolts used in its turbines and windmills. This makes things easier to manage, ensures better quality control and reduces the risk that a technician will inadvertently use the wrong bolt. When every server in the farm has been deployed in the same way, finding a problem on one can help to determine what is wrong with the others.
One attendee in the audience asked if a faster pace of release would make it easier for disgruntled employees to slip in backdoor malware. Automated supply chain auditing tools are only able to track public libraries that are widely followed by the security industry. Mortman countered that a good DevOps process should be implemented using automation tools that can help to track and document updates. DevOps is about change management, process flow and documenting those processes, he explained.
DevOps helps meet demands of mobility
Is DevOps best defined by what it isn't?
Is DevOps here to stay?