Hacked: The True Confession of a Security Pro

Casting my ego aside for a moment: getting hacked can be a good thing. How? Well, it makes for a fantastic learning experience in several ways, which I'll share below. First, here's the back-story...

Not too long ago, a few of my WordPress blogs, managed by me and hosted on a small hosting company's shared servers (the economy plan), got hacked.  And not just once, but repeatedly.

Over the course of a month, every few days I'd get the dreaded email from the hosting support team that my site(s) were compromised, locked down by them to prevent the spread of infection, and that I needed to fix it.

The cleanup works was tedious. The configuration verification and back-end tweaking were time consuming. In those few weeks, I learned more about WordPress security than I wanted to know.

And despite my best efforts, it kept happening. I felt like a chump!

I tried everything they recommended: CloudFlare, premium hosting (where every WordPress install isn't share a common directory structure), numerous firewall and security plugins, blocking numerous IP addresses from countries that are notorious for hacking, locked down the .htaccess file, etc.

And in the end it was all for naught. I either wasn't cleaning out infected files properly, which would soon cause re-infection, or an unknown hack was getting through my defenses. And my hosting company wasn't much help.

Either way, it sucked. With enough time, I'm confident I could have solved the root cause, but after the 5th or 6th time, I was ready to walk away from being the administrative duties of the sites. Fortunately they were for small side projects that didn't have any major impact when they went down.

Lesson 1: After that experience, I can't say that I'm still a fan of (self-hosted) WordPress - I don't recommend it anymore, even though it has amazing flexibility and community open-source support.

What else did I learn? Lesson 2: that unless you truly  enjoy the burden of managing your infrastructure configuration, to opt instead to use Managed Services that someone else is responsible for!

Microsoft Azure's Shared Responsibility Model

In terms of cloud, this means going for SaaS over Paas, and PaaS over Iaas. With the "shared security" model that cloud providers use, they have SOME security responsibilities.

And the more they "own" the more they have to worry about security and patching. In the "highest level" SaaS model, the customer has almost NO security responsibility. The cloud provider takes on almost all of it!

This might explain the rise of SaaS software offerings, Identity as a Service offerings, and Serverless Computing. When someone else does the install, patching, maintenance, management, and protection of a system / server, you are relieved of that burden. It frees you to focus on what you love.

Of course, sharing the responsibility with a cloud provider is no guarantee you won't get hacked, so you have to make sure your infrastructure and websites are being consistently backed up, and that you have a restoration plan and incident response plan in place.

But with that said, transferring some of the risk and headaches to some other party is frequently worth the investment!

As a cloud security geek, I usually love security challenges. That's what I get paid for. But unless we have similar titles, I'm betting you DON'T love dealing with security. So the best security advice I can give you is hand off the challenge to someone else.

For example, did you notice this blog is on Google's Blogger platform, and not self-hosted WordPress? Sure, I don't get as many features, and I relinquish some control, but I also don't have to worry about securing my blog and the servers it runs on. That frees me up to focus on my employer's and clients' security issues, which suits me just fine!

To reduce your security and management headaches, use managed services whenever feasible, but make sure you don't abdicate your responsibility to ensure that the cloud provider is doing their job.

As a business owner or employee, you ALWAYS have overall responsibility and oversight for your security situation.

I hope this lands on some receptive ears. If so, please let me know your thoughts, struggles, or questions on this topic.

-Ryan

How Amazon Can Make AWS More Secure

As a certified AWS architect and SysOps engineer, I've spent a lot of time learning the ins and outs of cloud computing on their platform.

And in that time I've come to understand that done right, cloud computing on AWS is FAR MORE SECURE than most corporate implementations.

In my opinion, if you put in the same amount of time on AWS as you do for your on-premise network and app security, you'll be significantly more secure on AWS than you would be on your own datacenter!

Why? Because of the vast array of cutting edge security tools (both AWS-native and 3rd  party vendor apps and appliances) that offer deep and useful security insights that were just pipe dreams a few short years ago.

You owe it to yourself to go check them out.

I believe this "security is better in the cloud" stance holds true for Microsoft's Azure as well, which is a platform I am currently getting certified on, but back to AWS - I spotted a glaring and simple way that Amazon could make their security even better, and it relates to penetration testing and vulnerability scanning.

Penetration testing is widely understood to be one of the best ways to find security vulnerabilities before the bad guys do, and every organization should be doing it - regularly.

And yes, AWS let's you do penetration testing. However, it requires their approval first, and herein lies the problem. When I "subscribed" to a Sophos UTM 9 appliance that I wanted to learn, and use to do some scanning on my test network, I had to fill out a form. And in that process, I had to list out the hosts and the IP addresses of both the  scanning tool, and the scanner appliance, and list the times of the tests, etc.

This makes sense because AWS needs to (presumably) disable alarms and "reactive" technologies so you don't get shut down when they detect the scanning / pen testing taking place on their network.

BUT, this process could be significantly streamlined, and made much easier so that more customers commit the time and energy to conduct penetration tests.

In short, it was a clunky, time consuming process to fill out a form to request permission to do the scan.

I had to go look up I.P. addresses, and gather info on the specific instance ID's of my servers and list them out. And while not a horrible process, it definitely introduced "friction" and would discourage some people from bothering with the process in the first place. And that's not good.


A recent pen-test email confirmation from AWS

My recommendation (are you listening, Amazon?) is to allow cloud administrators to request port scanning by simply adding a checkbox next to the the instance names, so that all hosts to be scanned simply have their box checked. It would be easy to add a date/time feature where the admin could select a time window for scanning the "checked" hosts, etc. Let's call this an "in-line pen testing request".

My logic is that by reducing the amount of effort required to conduct pen testing on AWS, more customers would take advantage of this ability.

They could even "up the anty" by integrating this request with the products or appliances of some 3rd party security vendors, like Sophos, so security-conscious AWS customers could try out new tools with ease.

This could be a boon to those vendors, and give customers access to an array of tools that they are unfamiliar with.

I'm sure there are numerous other ways to facilitate pen testing on AWS, and I'd love to hear your ideas!

-Ryan

PS: If you have an Amazon account, here's the link to learn more about pen testing, and to submit the form to do your own penetration testing. Use it! Find (and fix) your network weaknesses, even if it's a clunky process.

PPS: If you know anyone inside Amazon who might like to hear this idea, please share the link with them.

You Paid for Endpoint Protection, But Is It Actually ON and Protecting You?

A report from Threat Stack came out recently that showed that a huge chunk of cloud-related security problems are caused by not doing the basics, such as ensuring your servers are patched on a regular basis.


From experience, I can tell you that this applies to non-cloud environments as well.

Along the same lines, I'd like to share about the importance of monitoring your endpoint protection tools and services running on your endpoints to make sure they are actually on! Seems like a no-brainer, doesn't it?

Yet I have worked for many organizations that run Windows desktops and rarely have I seen them use notification and monitoring tools (they are already paying for) to ensure that endpoints are continuously running the services that protect the endpoint. I do my best to fix that wherever possible, or at least bring the risks to the attention of higher level decision makers.

Is your defender even on the field?

Whether your endpoint protection is HIPS, HIDS, antivirus, or any other related service, if its running on the endpoint, it must STAY running in order to be effective. Pretty basic stuff.

Yet time and time again, I see system with basic protections turned off, and the end user and the organization they work for, are unaware of this. And that's when very bad things can happen

For example, users might be downloading risky files from the Internet, thinking that antivirus (A/V) scans are taking place, protecting them, determining what files are OK and which are malicious, when in fact they are not being scanned at all.

Keep in mind that most end users are not sophisticated enough to check to see what services are running on a Windows system, and even if they know how, why would they think to go look, if nothing seems amiss?  No pop-up warning = no problems, right? Wrong...

Granted, many endpoint protection software schemes are designed to prevent being turned off, either by the end user or by a malicious actor or malware. But what if that fails for whatever reason?

Ironically, you're probably paying for tools that have this monitoring capability built right on, but its likely you're failing to take advantage of these features.

Instead of providing a list of ways to programatically ensure that endpoint protection services are always on and running, I will ask you to analyze what services SHOULD be running, and what protections AND notifications you have in place to guarantee that they stay running.

Here are some ideas, tools, or approaches you can use to make sure endpoint protection mechanisms are actually on, all the time:

• Group Policy
• SCCM and other enterprise monitoring systems
• SIEM
• Startup Scripts
• Login Scripts
• Powershell

I recommend getting with your I.T. cohorts and brainstorming ways to ensure your endpoints are ALWAYS being protected, which will also ensure you're fully leveraging your investment in cybersecurity defenses.

Happy hunting!

-Ryan

What is AWS Shield & How Can It Save Your Website from DDoS Attacks?

The 3 Types of Attacks & Is It Worth Paying for Advanced DDOS Protection?

If you run a website that helps you turn a profit in business, then your website availability has a direct impact on your financial health. That's not news, but the number attacks against website availability are growing very quickly and getting more sophisticated every week, and every single website is a potential target.

Yours is no exception, so you must be proactive in defending your website's availability.

Now, notice I said "availability" as opposed to "uptime". Uptime is more a measure of a web server being on and running, which is from your internal perspective, but "availability" is from your external customers perspective. Just because your web servers are "up" and on doesn't mean that they're functional for your customers or end users.

And that's why Distributed Denial of Service(DDOS) attacks hurt you - they prevent your customers from resolving your domain name or URL to an I.P. address of your web servers.

If this happens, in the best case scenario, your site won't load. Worst case scenario, attackers redirect your URL to their own bogus servers, which at that point is a site hijack, not a DDOS attack, but that's a blog post for another day.

There are 3 types of DDOS attacks: Volumetric, meaning your servers get overwhelmed with DNS query traffic - drowned in a flood of DNS lookups.

Then there are Application Layer attacks, in which valid, but malicious, HTTP queries also overwhelm the server, causing it to not be able to respond to valid requests from real users.

And then there are State Exhaustion attacks, described by Amazon as "abuse of stateful protocols that causes stress on firewalls and load balancers by consuming large numbers of per-connection resources". Think "Ping of Death" attacks, which send very large chunk ping requests that overwhelm your servers.

Most attacks are volumetric: 60-68%, with the other two types each getting 16-20% of market share (depending on who is reporting the stats). It's a constantly evolving landscape of threats.

So how to prevent a DDOS attack? Well, start by realizing that you need to protect yourself from all three kinds of attacks. Yes, that's more work for you and for the security pros your employ or contract with. But now for some good news...

If you're a customer of Amazon AWS, and you host your static files on S3, or use Route 53 DNS services, then Amazon is already helping you, for free!

At re:Invent 2016, Amazon announced the roll out of AWS Shield which is a set of tools to help prevent DDOS attacks.

AWS Shield at the standard level of service is a free new tool from AWS that is "already on" helping to protect your site. According to Amazon "AWS Shield Standard will protect you from 96% of the most common attacks today, including SYN/ACK floods, Reflection attacks, and HTTP slow reads. This protection is applied automatically and transparently to your Elastic Load Balancers, CloudFront distributions, and Route 53 resources.”

But what about the other 4% of attacks, you ask? 

Well, you could upgrade to the Advanced (paid) level of shield, but that's where performing a risk/reward analysis comes into play.  

As a site owner, it's your responsibility to figure out what level of risk is acceptable to you, and how much you want to invest in reducing that risk. Failure to undertake this analysis is a failure that rests squarely on your shoulders, so don't avoid it.

How to properly perform risk analysis is too big of a topic to cover in this short article, but think about how much revenue comes in via your website.

Now imagine losing all that traffic for a day or two, and think about whether or not you are willing to risk that revenue.

For some companies, we could be talking about millions of dollars. And if advanced DDoS protection like Shield Advanced only costs you a couple of hundred dollars per month, then that's some pretty cheap insurance!

Here's how they describe the Advanced level of Shield "...provides additional detection and mitigation against large and sophisticated DDoS attacks, near real-time visibility into attacks, and integration with AWS WAF, a web application firewall."

It also provides "additional detection and mitigation against large and sophisticated DDoS attacks plus additional protection over AWS Shield Standard including intelligent DDoS attack detection for network layer (layer 3), transport layer (layer 4) and application layer (layer 7).  In addition, customers also get access to 24x7 DDoS response team during a DDoS attack and additional real-time metrics and reports. The advanced service also provides cost protection for your Elastic Load Balancing resources CloudFront and Amazon Route 53 hosted zones".

For more information about both level of AWS Shield and how they can help you against DDos attacks, here's the link to Amazon's official page about the new services:

And for more information on DDoS attacks and security threats in the wild, here's a link to Akamai's 2016 Q2 State of the Internet / Security report. it has more great stats and trends on these types of threats to your website, so it's worth reading.

-Ryan