We were also lucky enough to acquire Third Brigade, a Canada-based security firm,  earlier this year and get our hands on their superb “Deep Security” threat protection for Virtual servers.  More than just protection ahead of the patching cycle it offers excellent resource optimisation by utilising the VMSafe APIs to do much of the work once per physical server, rather than once per virtual machine.

As one of a select group of major companies who have seen the technology evolution through DOS, Windows, Client-Server and now cloud you’d expect Trend to be working hard on what security FOR the cloud needs to look like and of course we are.  It’s easy to think that the public cloud is really just like a shared private cloud – one that you buy a little piece of when you need some computing power (and indeed it’s often marketed that way).  From a processing perspective that’s a reasonable description, but from a security perspective it’s anything but…

A private cloud is really the ultimate goal of a virtualized data centre.  First you take your physical servers and make them virtual servers to reduce your hardware costs and increase flexibility.  If you are smart you implement something like Deep Security to optimise your security posture.  Then you look towards resilience, using shared storage and dynamic migration of virtual servers from one host to another in the event of hardware failure.  Ultimately you may have burst capacity so you can spin up extra servers for particular tasks and hibernate others depending on the workload throughout the day or year (think of the holiday season rush or closing the books at the end of a fiscal quarter).  You may even go the whole way and get all of that working cross data centre to provide redundancy, scalability and performance.  We’ve been living that for a while now with the Smart Protection Network.  We’ve learnt a lot of lessons doing this for ourselves on a massive scale and we can pass those lessons onto our customers and use them to shape the products we build.  Still though, through all those stages there is a common factor – it’s just you in the private cloud.  You can still put a wall around your whole resource pool, filter everything and try to keep the bad guys out.

In the public cloud your provider runs something that looks pretty similar to the ultimate private cloud described above and they can carve you out a piece of that, charged on a per use basis, that looks pretty much like your own private cloud.  There is however a big difference.  Instead of being protected by a strong perimeter your servers are sitting alongside those of strangers, competitors and inevitably the same organized criminals that you work so hard to keep outside the perimeter of your data centre today – The only barrier to entry inside the perimeter being possessing a credit card number (stolen or otherwise)!  So how do you guard against that?

Security FOR the cloud means that the host must defend itself.  Defend itself at the front end because the firewall rules may be inadequate and because it may be attacked from within the firewall.  Defend itself, and its data, at the back end because there are a lot of strangers sharing the same storage and the “trust us our systems can’t be hacked” security model that your cloud provider offers has been proven over and over to be the worst one in town.   Can that really be done?  Can the host defend itself in a shared environment well enough to provide compliance in the cloud?  We believe it canand that we have the building blocks to augment public cloud security in the future.  Security FOR the cloud is available today (Deep Security 7.0) with further pieces under development.

In the malware realm, this is nothing new and has been referred to previously as “Malware as a Service”.  Just as legitimate companies move to the cloud for the above-mentioned benefits, cybercriminals move some of their malware onto “shared infrastructure” sites to make them harder to mitigate, block, or get taken down.  What is somewhat new is the increase of hosted malware in Google applications (e.g. Google Reader, Blogger, etc.). 

What caught my attention was that the bad guys quickly learned to leverage the PaaS cloud infrastructure for malware CnC.  It does not take a fertile imagination to see bad guys going from using PaaS to manage their malware to applying knowledge to go after IaaS applications.  The public cloud (SaaS/PaaS/IaaS) has a compelling value proposition in terms of cost, but “out of the box” IaaS only provides basic security (perimeter firewall, load balancing, etc) and applications moving into the cloud will need higher levels of security provided at the host by layers such as Trend Micro Deep Security 7.0.  Such countermeasures would mitigate the possibility that a bad guy might attack an IaaS instance or take it over for use as a botnet hub.

If someone with malicious intent buys up the IaaS instances, it’s seems to me that the Service Provider should detect and stop that as a violation of the Service Provider Service Level Agreement (SLA).  But how does a service provider assess how their IaaS/PaaS is being used without compromising the privacy of the application? If they don’t watch the usage, perhaps they have to validate the customer? And what if the service is bought with stolen personally identifiable information (PII) and credit card numbers? 

The malware threat is as old, but the cloud poses some new questions.

Do you remember when you refused to bank online because it couldn’t be safe? I do. In fact, I even remember working with one of the leading banks in Canada when the CIO declared that no employees should have access to the Internet—for any business reason, ever. He did not last long in his job.

Over the past 15 years our reliance on the Internet has steadily increased, encouraged by advancements in technology (including security), a culture of instant gratification and an obsession with efficiency. After a year of media frenzy, some of us are still skeptical on whether to adopt cloud computing for enterprise and datacenter expansion. Like the advent of email, I believe that three things are going to force your hand:

1)      Internet Pressures: cloud computing is easy and the success of public clouds like Amazon means that your internal “clients” have alternatives for computing power readily available to them

2)      Cost savings: it is cost effective and with the economy still uncertain, cost-savings are paramount

3)      Competitive advantage: it is being adopted by your competition and it will enable competitive advantage

We have heard from IDC and others that security is an overwhelming concern preventing public cloud deployments. I don’t blame you if you count yourself in this group. The lack of control over the network perimeter is just the start of the list of security challenges that should concern anyone considering cloud computing. In cloud deployments you rely on administrative connectivity to servers and applications accessible only via the internet. The potential for vulnerability exploits from co-located cloud servers and the need to ensure data protection and data integrity in these co-located cloud hosting environments is enough to keep any self-respecting CIO awake at night. And then you start to ask yourself, who owns the logs? Where is my data? How do I prove to auditors that these resources are adequately protected? These are all legitimate questions and concerns. And, as I pointed out above in item 1, these risks are being taken by your constituents, both unmanaged and unidentified.

That said, the sky is not falling from the clouds.

Gartner is predicting 10x growth in the number of virtual machines expected to be deployed over the next 3 years. You are deploying the virtualization technology underpinning cloud environments. With virtualization, you are equipped to create your own private cloud environments.

Rather than being afraid of placing your server resources in the cloud, prepare them to take flight, and choose where and how they fly. There are six areas of security that you need to identify and assess the impact, requirements and complexities of protecting workloads across the traditional-physical, virtualized-private and public cloud computing environments.

Over the next few blog entries, I will be reaching out to experts in our industry. In an interview format we will begin to contrast and compare challenges across these three environments — physical datacenter, virtual and cloud computing — to evaluate what virtualization and cloud computing is imposing on:

-          Network Security

-          Data Protection

-          Host Security

-          Identity Management

-          Security Information Management

-          Vulnerability Management

Stay tuned!

Here I will consider a range of activities as “Cloud Computing” including SaaS, PaaS and IaaS.  All three raise some concerns for companies. Companies that find the benefits of cloud computing compelling should plan and execute their cloud computing strategy in a way that avoids the risk of catastrophic failure. 

When moving systems into the cloud, a company needs to consider a range of potential catastrophic failure scenarios and consider actions they can take to mitigate those situations. Cloud computing vendors (particularly IaaS vendors) typically have tight limits on their responsibility (see Todd’s post on “Who Owns the Mess?”), so the company is unlikely to have any legal recourse there (unless the vendor was negligent). 

Companies need to consider that other customers or their provider of cloud computing facilities may be the target of attacks (such as the DDOS attack that hit Amazon EC2 customer Bitbucket). One approach to minimize the risk of this is to distribute applications across cloud computing vendors. Some applications (such as storage) may be suited to such a distributed approach, while other applications (such as using SaaS spam and virus  filtering) may be difficult to distribute across vendors. 

To mitigate against system failure, companies need to evaluate how fault tolerant the systems they intend to use really are. It is important for the company to assess their needs and the robustness of the infrastructure / systems they intend to use (for an example of a non-robust cloud infrastructure, see a discussion on Sidekick’s infrastructure in Andrew’s post on “The Sky is Falling on Cloud Computing”. 

IT staff should consider that with IaaS they are typically using a more homogenous computing environment than is typical inside a company. This monoculture has both advantages and disadvantages. The environment can be more efficient because it is better understood, security patches can be applied systematically to all instances, administration can be centralised. However, the security downside is that these very same features open the door to exploitation. Intruders potentially have the opportunity to hire the same computing environment and test it for weaknesses. Some of the potential breaches are due to the virtualization techniques used and can be quite unexpected (for example, at the recent BlackHat conference, researchers Becherer, Stamos and Wilcox considered the issue of exploiting cloud computing instances by using the lack of randomness in random number generation). 

The company deploying into the public cloud needs to consider how administrative access will be granted to cloud computing resources (see “Cloud Danger #3: Reliance on Passwords”). One approach to reduce the risk of passwords is to use two factor authentication for administrative purposes (see ). 

Another situation which comes under the heading of catastrophic failure is data theft and data loss. Given that access to the cloud computing resources will be remote, the company needs to consider measures such as encrypting data in the cloud (for example, see Amazon’s whitepaper on encrypting data in the cloud  ).  Ideally the deploying company should hold the encryption keys rather than the IaaS provider. 

The use of cloud computing does not necessarily equate with “putting all your eggs in one basket”. If due care is taken to minimize the risk of catastrophic failure then the benefits of cloud computing are available to many companies – and we will continue to see meltdowns occur where due care is not taken.

Throughout history the need for portability and interoperability has usually been dealt with through standardization. Standard railroad gauges enabled cross continental travel, just as TCP/IP unlocked worldwide communications. It’s not surprising then, that many people look at cloud computing and assume we need standards before lock-in can be avoided. But do we really need widely-adopted standards? While not ideal, interoperability can still be achieved through abstraction (or brokering) and portability through conversion in an environment with many standards.

When talking about interoperability and portability in Infrastructure-as-a-Service (IaaS) there are generally two significant issues. One is the format of the virtual machine templates (or images) which describes the disk and configuration of the virtual resources required. While this is generally dictated by the underlying virtualization solution used, some providers have created custom formats (for example, the Amazon Machine Image). The Open Virtualization Format (OVF) was designed as a single standard, but public providers may continue to push their own formats for various reasons. Without universal adoption of OVF, the next best thing is format conversion to provide practical portability. As a stop-gap, some service providers have started accepting multiple formats to avoid the conversion overhead, in the same way some devices supported HDDVD and Blu Ray until that standards battle was won.

The other challenge is the current incompatibility of the management API for uploading, downloading, inspecting, configuring, and performing actions (such as spinning up new instances). Each provider has its own API which prevents orchestration software from working with multiple service providers. There are many approaches to this issue. Some groups like the Open Grid Forum are attempting to create a standard, the Open Cloud Computing Interface (OCCI). Others, like Eucalyptus emulate the Amazon Web Services interface as a de facto standard. VMware has developed its own vCloud API which it submitted to the Distributed Management Task Force (DMTF) as an open standard. vCloud API will provide a basis for interoperability among VMware-based service providers (and perhaps other providers in the future), but almost certainly not the established players. Most providers forgo official standardization because they want (and need) to move rapidly in this evolving marketplace and standards bodies are not known for speed. But the lack of industry-wide adoption of a single API doesn’t have to prevent portability and interoperability.

Multiple APIs can be combined under a single API, even without participation of the providers. In the virtualization space, an API for the APIs already exists in the form of libvirt. For cloud computing, a group has taken on this task for with the Unified Cloud Interface Project, though the project is still in its infancy. Another initiative, cloudloop provides an API to work with multiple storage services. An API of APIs, such as these, provides a form of interoperability, where framework vendors, middleware vendors, and end users can consume a single API without worrying about service provider lock-in.

For Platform-as-a-Service (PaaS), portability and interoperability becomes much more challenging. By nature platform services can have drastically different data formats. Windows Azure, for example, provides database services and .NET application containers. Applications and data within Azure are not compatible Google AppEngine and vice versa. The only way to avoid lock-in when utilizing PaaS is to choose a framework offered by multiple providers and avoid provider specific extensions (like the Python extensions in AppEngine). We may see a similar abstraction strategy emerge where applications can be written once to run on many PaaS offerings. I expect to see a lot of development in this space as workloads shift from IaaS to PaaS.

Software-as-a-Service (SaaS) has the most challenge because of the inherent diversity of the data. One can’t expect Facebook data to be exportable and importable into another social media site (Matt Asay called this the Hotel California of Tech). Nor can we assume all software services even offer extraction of data. This is only acceptable when the service provided doesn’t have existing standards. In cases like Google Docs, it is reasonable to expect some form of conversion like the new export options Google released this week. In this environment conversion is a much more practice vehicle for portability rather than standardization.

In the rapidly evolving cloud computing marketplace, we should expect to see multiple standards emerge and as Stephen Foskett said “only useful standards will survive“. This is a healthy process in a new environment. In the mean time we can achieve portability and interoperability independent of the standards. We can break vendor lock-in and ensure the availability of services and data through conversion and abstraction.

IaaS providers are doing a decent job of baseline security (physical security, perimeter firewall, load balancing, perhaps a network IDS/IPS, etc) and have to provide a basic ante to the game.  While the occasional IaaS vendor strives to differentiate themselves with higher degrees of security, many (if not most) are focused on providing aggressive prices and flexibility that the IaaS concept promises relative to the on-premise data center.

While IaaS vendors strive for a secure environment, the security responsibility and accountability lies with the business using the service.  The Amazon Web Services Customer Agreement is quite clear in this regards:

7.2. Security. We strive to keep Your Content secure, but cannot guarantee that we will be successful at doing so, given the nature of the Internet. Accordingly, without limitation to Section 4.3 above and Section 11.5 below, you acknowledge that you bear sole responsibility for adequate security, protection and backup of Your Content and Applications.

You can visit your favorite IaaS vendors to read their Terms of Service or Service Level Agreements to see that they typically do not take responsibility for more than physical security, security personnel, and basic perimeter security of the computing environment that you might use.  This is probably because many of the IaaS providers are located in the litigious USA where people are known to sue one another(Frank Gens at IDC mentioned during a recent IDC webinar that in the near term, 75% of the cloud computing services market was in the US).  IaaS providers need to clearly limit their legal liability to stay in business, and that means the security burden falls to the cloud computing customer (e.g. the enterprise) to ensure that their data and applications are safe.

I spoke with one corporate lawyer about this (note: I am a lay person, this is not legal advice, please consult qualified legal counsel before making any decisions about anything) who made the point that if someone gets sued because of a data breach, the plaintiff will go after the party with the deepest pockets.  I tried finding a pithy quote from a lawyer to articulate this point and came up empty, but try googling “lawyer sue deep pockets” and you’ll understand what I’m talking about.  In cloud computing, the deep pockets are typically with the IaaS consumer and not the IaaS provider.

Note that we have not seen IaaS-related data breaches to date.  Distributed Denial of Service (DDoS) and lost data, but no data breaches of sensitive data.  Given that compelling cloud economics and flexibility are expected to draw in applications, and those applications will eventually include sensitive data, a data breach is only a matter of time.  When the s—t hits the fan due to a data breach, look for the lawyers to knock on the enterprise door, not the IaaS data center door.

Enterprises can offload security responsibilities to their parties by relying on the IaaS vendor security, or managed security service providers (MSSPs), but if something goes wrong, the owner of the data is the one who is accountable.

How can you mitigate the risk when deploying applications involving sensitive information into the cloud?  When considering where to deploy applications in the cloud, the “security” answer to your application developers can be “Yes, deploy into the public cloud (IaaS), as long as you take these steps…” rather than a knee-jerk “No.”  Those “yes steps” involves protecting the individual host inside the IaaS perimeter.  That protection includes host-based technologies providing functionality including deep packet inspection for IDP/IDS and firewall along with file integrity checking and log inspection (like OSSEC). And now comes my shameless sales pitch! Check out the new Trend Micro Deep Security 7.0 that addresses many cloud security and compliance issues.

What has been billed as a large scale failure of cloud computing, more specifically, cloud storage, is making headlines and generating lots of heat but little light.

• Major outage hits T-Mobile Sidekick users:  “Users of T-Mobile’s Sidekick have been suffering through a major outage over the past several days that left many without access to the Web or their address books.”

• Lawsuits filed over Sidekick outages:   “In that lawsuit, Thompson’s lawyers argue why the outage of the Sidekick was particularly devastating, noting the device’s cloud-basedarchitecture in which the primary copy of the data is stored, not on the devices, but on servers operated by Microsoft’s Danger unit. “

Reliability is the first objection to anything “cloud”-related, especially storage. The Microsoft/T-Mobile/Sidekick fiasco seems at first glance like a knife aimed at the heart of cloud storage.

However, given public reports of the nature of the Microsoft/Danger back end service architecture for the Sidekick device — Oracle RAC on top of EMC SAN — this is not an architecture that serious engineers working on fault tolerant distributed infrastructure would term cloud infrastructure. In fact this is a traditional data center architecture that failed! Oh, the irony. This is not the Google architecture, for example, or Amazon’s, nor what Trend is working on with a variety of initiatives. In these, data is truly replicated and distributed via shared nothing architectures with multiple redundancies. I think this distinction may be lost on many people however. The Sidekick business model was selling the idea that user data would be available from everywhere/anywhere, which is the definition most people have of “the cloud”. More irony: The Microsoft subsidiary involved is named of all things Danger, Inc.

Meanwhile the damage to T-Mobile can be measured in real and immediate terms, with many industry pundits mistakenly conflating the issue as a black eye for cloud storage, not merely Microsoft:

• Phone sales hit by Sidekick loss:  “The issue is seen by industry experts as the largest failing for cloud computing in recent memory…It is also being painted as a black eye for Microsoft which has pushed cloud or online services as a less expensive solution for enterprise data storage…This is the most spectacular loss of data on the web to date,’ Harry McCracken, editor of Technologizer.com told BBC News.”

This is unfortunate. Truly redundant cloud storage technology can be more reliable, scalable, and cost effective than the the traditional technologies at the root of the Microsoft/Danger problem. Meanwhile there are rough spots and areas of legitimate concern that require addressing — for example, privacy, security, and manageability of data in a fluid multitenant world — and on these terms we can and should discuss whether cloud storage is appropriate for a given customer or service, and we can and should work to further develop cloud storage solutions to address these issues.

For service providers which themselves operate on top of cloud infrastructure (SaaS, IaaS) such as Trend Micro, we can employ true cloud storage technology for fault tolerance, redundancy, and disaster mitigation and recovery. This will provide our customers highly reliable services that degrade gracefully — definitely not catastrophically — whenever the realities of fickle hardware, networks, and natural or geopolitical disasters intrude. We can employ good end to end security principles to assure data privacy and integrity. As is so painfully highlighted by the developing Sidekick fiasco, reputation will be crucial, our reputation, the reputation of our platform providers. We can gain trust with careful engineering, openness, and reputation. We can maintain trust through transparency, development as an industry of secure open data exchange solutions for cross service and cross provider data backup, and intolerance of substandard or sloppy vendors who may poison the well.

What happened to Microsoft with Danger was an IT snafu that could have happened to any data center.  While data was apparently lost, it was not compromised.  The Register article points to possible design issues in the infrastructure.  This unfortunate event was not something unique to “the cloud”, but what was different is that an apparent IT process mess-up affected lots of consumers.  When you compare the Danger episode to what happens when your internal email server dies, the difference lies in who is aware of the problem:  with your internal email server, only your company knows it is down (not the entire world).

From a security perspective, both the Microsoft Danger episode, Amazon EC’s recent DDoS, and Google’s hosted email availability challenges point to birthing pains for some Software-as-a-Service (SaaS) and Infrastructure-as-a-Service (IaaS) offerings.  Something significant to note is that while we have seen service outages, we have not yet seen security breaches that compromised sensitive data.  Such a breach might slow cloud computing’s adoption, but most enterprises are being careful in what data they move to the public cloud and how they protect it.

A data breach in the public cloud will eventually happen, but my prognostication is that the cost savings and flexibility provided by Sofware-as-a-Service/Platform-as-a-Service/Infrastructure-as-a-Service (and hosting) will drive adoption with security teams highlighting and attempting to mitigate risks.  A key challenge for security professionals is to avoid saying “no, no, no” and instead say “Yes, cloud computing is good, but you need to do X, Y and Z to secure your application and sensitive data”.

Issue #1:  The implementations are not proactive, but reactive in nature, despite better response times to new threats.

Reality:  Replacing hash signatures with intelligent static signatures can both defeat code obfuscation and detect polymorphic malware.  Furthermore, it consumes far fewer resources than emulation-based behavior detection.  Trend Micro’s intelligent pattern project, an automatic pattern generation system, shows that an intelligent static pattern can proactively detect hundreds of malware belonging to similar families at millisecond speeds without triggering false positives on more than 20 million benign samples.

Issue #2:  While detection rates are maximized (which looks good in test results), the risk of false positives is increased.

Reality:  A few years ago, Security Information Management (SIM) emerged as a solution for solving the problem of the overwhelming log volume of Intrusion Detection System (IDS). SIM includes a set of sensors which ensure that IDS events are collected, analyzed, and responded to in the shortest period of time possible. By centralizing this information, events from distributed IDS sensors can be correlated and categorized. The benefit of correlation is the sharp decrease of false positives.

The Trend Micro™ Smart Protection Network™ is similar to reputation-based SIM in that data centers aggregate URLs, emails, scripts, and files from heterogeneous data collectors. During the correlation process, Smart Protection Network measures the relationship of security events to determine the threat potential, keeping false positives at a very low and tolerable level.

Issue #3:  The results of ‘in-the-cloud’ scanning can be based on much more input data of both good and malicious files but causes an additional performance impact on the client-, network- and server-side.

Reality:  In order to maintain a balanced workload between the desktop and cloud, the agent requires a light-weight and intelligent signature database that is smaller than traditional signature databases. When a suspicious file cannot be determined, the agent can then send the file or fingerprint to the local server for the further verification, thus saving bandwidth by not sending too many packets into the cloud.  Embedding the emulator into the desktop and local server allows the agent to inspect the hidden payloads of obfuscated programs. Bandwidth will be saved because the hash value of the dumped data rather than the file itself is sent to the cloud.

Issue #4:  Due to the time required to answer a query, only on-demand scanners and files which are executed are checked, but not all accessed files (as a ‘traditional’ on-access guard would work).

Reality:  In-the-cloud doesn’t mean moving all hash values into the cloud. Normally, in-the-cloud can be divided into three parts: light-weight cloud agent, local server, and data center.  As previously mentioned the cloud agent includes a light-weight and intelligent signature database. Each pattern inside can detect polymorphic malware belonging to the same family. Also, the emulator can be embedded inside the desktop agent or local server. Behavior patterns will be used to scan the behavior information coming from the emulator. The local scan server always keeps the latest local pattern files from the data center. Therefore, in-the-cloud can still support on-access scanning module.

Trend Micro has to wrestle with DDoS attacks as part of our antivirus business as well as our hosted security business (shameless sales plug: check out InterScan Hosted Messaging Security for hosted/SaaS email security offerings).   I checked with some of our CTOs and architects to get their thoughts on the Bitbucket episode, and got an education on the tough problem that is posed by DDoS.

Vendors and Software-as-a-Service(SaaS)/Infrastructure-as-a-Service (IaaS) providers can use smoke and mirrors to protect themselves from negative news, but from a technology perspective there is no defense once the incoming network has been saturated by a DDoS attack.  While there is no way to “architect” to avoid DDoS attacks, you can architect to mitigate attacks.  This is not something that you “set and forget” but is more about developing good working relationships with upstream providers and working with them in real-time to mitigate attacks.

Most network countermeasures cannot protect against DDoS attacks as they cannot stop the deluge of traffic and typically cannot distinguish good content from bad.  Intrusion Prevention Systems (IPS)are effective if the attacks are identified and have pre-existing signatures but are ineffective if  there is legitimate content with bad intentions.  Similarly, firewalls typically have simple rules that allow or deny protocols, ports or IP addresses.  DDoS attacks easily bypass firewalls and IPS devices since they are designed to send legitimate traffic, such as HTTP requests to a web server, and attacks generate so much traffic from so many distinct hosts that a server, or more often its internet connection, cannot handle the traffic.

While I suspect this sort of attack will be relatively rare since most attacks today are undertaken to make an illicit profit and DDoS are generally conducted for notoriety or revenge, they still present a concern for customers, IaaS vendors and ISPs alike.  Whichever bad guy compromised the machines used in this DDoS attack just identified those compromised machines, and ISPs will have to start the painful task of notifying their subscribers or shutting down the compromised machine.  ISPs notifying thousands of subscribers will not be done quickly or easily.

All of this is irrelevant if you are deploying a non-mission critical application into the cloud.  You can head to the pub until the DDoS attack blows over and your app is accessible. 

The story is different if you’re deploying a mission-critical application into the cloud because you need to architect the application for resilience from Day 1.  That means spreading the application among multiple IaaS providers and replicating data between those IaaS providers.  That also means dealing with the challenge of latency between different IaaS providers.  Cloud computing and SaaS/IaaS is great stuff, but enterprises & application architects need to think carefully about security before flying into the cloud.