3 Best Practices for Perfect Security: A Story

Over the last year, I've been fortunate to consult on securing some important and highly targeted networks. I know they're highly targeted because they were attacked multiple times. So they needed perfect security. I know in the cyber security business we say that perfect security is impossible and even pretty good security can be ridiculously hard to scale, even on an enormous budget. Which is why I get asked a lot how to do it. So I'm writing this to share with all of you how companies get nearly perfect security in just 3 easy steps!

Step 1 – Get To Know Your Assets

Walk around and write down the assets you see. Talk to people doing their jobs and write down the assets you don't see. Do asset management. Most people think assets are the money or the stock in the warehouse.

The Internet is a hostile environment. You need your cybersecurity to function there.

And sometimes it is. But it's also the things that make you money that are the assets. Even your security servers and security software are assets because they are profit enablers. So think of it this way: if your business is volcano tours, then flameproof underwear is both your security and a necessity for you to make tours and, therefore, profit-making in a hostile environment. The Internet is a hostile environment. You need your cyber security to function there. Therefore, cyber security is a necessity for you to make money there.

Step 2 – Apply Best Practices

Install and configure the following: a firewall, antivirus software, web application firewall, and an intrusion detection system. Then set up regular vulnerability scanning, penetration testing, and patch management.

Write a security policy and have management sign it because, you know if they don't sign it too, then they aren't taking security seriously.

Write an acceptable-use employee policy and have everyone sign it. Bug them until they do. Use threats if you have to. Give token prizes to those who hand them in.

Get a good backup system running on the databases. Do full back-ups as often as possible.

Keep a good record of how quickly you can roll out patches and respond to virus and malware outbreaks.

Turn the logs from the IDS, WAF, and AV into metrics of how many bad things were caught and stopped.

Set up an online security awareness class and track that all the employees watch the videos and answer the questions to completion. Use a system that reminds them to finish any partially viewed or completed courses. Get involved and threaten them if you have to because more important than them completing the course is just knowing they're being watched.

Hire a penetration testing company to find your weaknesses. Make the few configuration changes they suggest like removing old, renamed files off the web server. Be happy the report rates your security high. Use it for your PCI-data compliance. Make metrics of your progress and present it regularly to management.

Step 3 – Get Brutally Hacked

And not just brutally hacked but embarrassingly, brutally hacked. They put company emails and files on Pastebin and delete data from the web servers. They transfer money out of the payroll bank account. They make you look asleep on the job.

Spend time answering to management how it happened. Feel a huge, gaping hole in your chest. Feel utter helplessness. Feel sick to your stomach. Hide a while in your office as you keep from crying. Tell everyone who asks you that you’ve got dust allergies.

Realize you need to do something. Get angry. Drop everything to investigate how it happened. Look at events and logs until your head pounds and you get a sharp pain right behind your eyes. Keep stopping the investigation to answer questions from management. Give up investigating. Keep telling everyone asking that you don't really know how it happened.

Management asks if you should contact the police. You don't actually know. You contact the police, but they don't really know what to do. Tell management that they should contact partners and customers who might be affected. Tell them to avoid the journalists. They ask you what they should say. You don't know. Say to tell everyone they should change their passwords.

Focus on cleaning the suspected areas of the breach and running scans on the rest. Change all the passwords everywhere. Read online what everyone's saying about your breach. Feel sick to your stomach. Find a little comfort in security pros online saying everyone gets hacked eventually. Go home to sleep but spend the night in bed awake and worrying to the point of nausea.

Come in extra early because you get called in for an emergency. The web server pages have been tagged. Discover the database is corrupt. Be happy you've invested in a kick-ass backup solution for it. See the last back-up actually recoverable is a month old and kick yourself for never testing the recovery system. Feel stupid. Get angry. Hit the backup machine. Feel stupider for hitting a machine. Hide in your office while you run scans again.

Come out of hiding long enough to report to management. Avoid eye contact on the way there. Tell management you're still investigating and cleaning up. Take their offer on the bigger budget. Don't tell them it's too late but you think it. You think it a lot.

You think about what you would have done differently with more money. Tell yourself you would have bought the best firewall on the market. Think which one you would have bought. Then tell yourself it wouldn't matter because everyone gets hacked eventually. Get angry. Go home driving angry. Drive over your own lawn just because. Watch crappy television shows until you pass out on the sofa. Spend the next week doing this.

Wake up to your phone buzzing in your pocket. Find out that the web servers got tagged again. Go to work in the clothes you slept in. Realize that other employees avoid you in the hallways. Have a co-worker from your team tell you that you look like shit. Don't cry. Take a few deep breaths and hide in your office to fix the web pages. Realize the latest database back-ups are of the tagged pages from the week before. Scream into your coat. Pass out on your desk while analyzing logs and waiting for scans to finish.

Wake up determined to fix this all. Get coffee. Avoid stares. Go to the whiteboard in your office and think about where attacks can come from. Make simple, logical network maps. Erase maps. Draw logical maps. Draw your security coverage. Make arrows through the map to each security system and software with the type of attack that could beat that type of coverage. Learn nothing. Erase the maps. Draw threat trees. Learn nothing. Erase threat trees. Draw the logical map again. Make arrows to each place that someone could get in. Make arrows where data goes out. Realize you have no perimeter and everything goes in and out pretty much everywhere. Feel sick to your stomach.

Take a deep breath and start again. Think about where the protections are for everything going in and out. Realize you only have filters, not walls. The firewall is your only wall and that is set up like a filter. Take another deep breath. Tell yourself it's good that now you're learning.

Realize you have no perimeter and everything goes in and out pretty much everywhere.

Realize you’re late to your meeting with management. Call up and ask them to postpone because you're on to something. Realize you got into this field for a reason. You used to like security. Fire up Wireshark and take a look at what's running on your network. Try to make sense of the mess. And it is a mess.

Get your team to stop what they're doing and start separating out the traffic. The ones out and about dealing with malware on desktops should just shut off those systems and give them a spare or a laptop for the day to focus on this. You build a map of what you have. You segment. You separate. Shut down all remote administration and remote connection services everywhere. Deal with all complaints from IT. You go home at a reasonable hour and actually manage a few hours of nightmare-free sleep.

Come in the next morning and look at the new map you made with your team. It feels wrong. You should be doing something else like buying token card two-factor authentication or encrypting all the workstations. But to do that you think you should have a handle on what packets go where. So you have your team keep segmenting and separating. People start to complain that they can't share documents between their desktops anymore. White-list them through to file servers. Build more file servers. People complain their system updates and antivirus updates are failing now. Leave it for another day. Go home and eat an actual meal. Go to bed early. You sleep badly and have nightmares about the hack.

Every day you come in and watch the network traffic to make changes to systems and the maps. You fire up Wireshark in the DMZ and sort through the mess. Too much of it is encrypted traffic. Have your team set up an SSL termination point in front of all web servers, then have them set up a Bastion Host to white-list and discretely manage all remote access and administration. It takes weeks to get it right, but you notice that the web server tagging has stopped. Let management know. Let yourself feel good about it.

Spend the rest of the week sorting through the packet mess while cleaning and separating and removing everything you don't need. Give management their first positive news of improvement. Get approval to use some money to install a hybrid SIEM with SOC capability to help you analyze the network mess. Pick a fast, elastic one that also analyzes trusts between interactions. Upon firing it up, quickly learn that you still know so little about company processes and interactions.

Go with your team and sit with as many employees as you can to learn what they do, where they do it, and how they do it. Learn about external partners with bi-directional access to systems through browsers and external administration access to power companies and telephone companies and accounting firms that you knew nothing about. Learn about all the mobile devices they are using to do their jobs that you knew nothing about. Make more segmentation changes. Fine-tune the SIEM. Spend months just doing this. When friends and co-workers show you other companies getting hacked and reminding you that everyone gets hacked just nod in understanding.

One morning you get to work and find your web servers had been attacked by the new buffer overflow exploit that became known earlier that week. Check with your SOC and sure enough find your WAF and IDS let it through. Call the vendor to complain and get told you've configured them wrong. Remind them that they configured it. They deny it. Get angry. Go meet with management about booting them. Find out from management that you still have over two years left on your contract with that vendor. Find out from someone on your team that the vendor is the cousin of the CEO. Eat chocolate to deal with it.

Run your vulnerability scanner against the web servers and find out they're apparently free of vulnerabilities. Get frustrated. Try out an exploit testing service for your security controls. Watch the screen as it sends exploit after exploit through your security defenses and to virtual honeypots in your DMZ and internal network. It tells you 45% of exploits are blocked. Deny it to yourself. Notice this feels really uncomfortable. Tell yourself you don't want to know this. Eat more chocolate.

But now you do know. Think about how much worse you felt when you didn't know. Because you didn't know. Thinking about that huge hack still makes you nauseous. So you go to the CFO who approves the exploit testing service as it fits within compliance objectives of continual oversight. You set it to test once a week. Now you know how the latest known exploits affect your network. You make it your yardstick for reporting risk.

You figure if an exploit can get through your standard defenses then you need to be able to stop it some other way. So you subscribe to a vulnerability database service to get advanced and deeper knowledge of new exploits tailored to your specific systems, especially the ones that the testing service shows made it through.

You spend a few days going through all this new information on the vulnerabilities that can get through your defenses. Many that have no patches and workarounds that say you should just remove it. Worry a lot about this. You know there's no way you can get the whole organization to accept removing some of those things. Feel lost. Go home and fall asleep in front of the television with a bag of chips on your lap.

Wake up and feel terrible. You have all this security intelligence to tell you what your security defenses can't ever do. Feel like a fraud for having a degree and being certified in information security and not knowing how to really secure anything. Look in the mirror, see the shell of the person you once were, and decide to do something different. Clean the chip crumbs off your shirt and throw the rest of the bag in the garbage. You take a shower and think about how to change.

You arrive at work and start to research why security doesn't work. This brings you to rants and ravings from security professionals also sick of it. You find articles outside of the mainstream of security and eventually you find the OSSTMM.

Spend a few days reading through it. You copy/paste paragraphs of ideas and send them to your team. Then you challenge your team to think of how you can change your network security architecture to address each vulnerability and exploit instead of waiting for a patch. Watch the team rise to the challenge.

Over the next few weeks, you make huge improvements to your infrastructure with this technique and you see that you’re protecting against whole classes of exploits. Wonder why this practice isn't mentioned in any best practices.

Start to feel like you're really starting to get security now. You sleep in your bed and get up feeling rested. You eat meals that are actually healthy. You stop buying soft drinks. And you notice you now go into work with a purpose in the morning rather than out of fear of something going wrong.

Day by day you address the old and new applications. The developers slowly get the hang of the mapping and include it in their development process.

The SIEM shows a lot of strange activity on your web applications. Meet with the web app developers to talk to them about security. See their eyes glaze over. You change your approach. You take the time to map out the applications with them. You break it down into processes and controls for those processes just like you did with the network. You show them how to map out the attack surface.

Day by day you address the old and new applications. The developers slowly get the hang of the mapping and include it in their development process.

You take a look at the inventory of applications in use and in development and panic when you realize none of it has been security tested. None of it. You start to look at who from your team you can put on it and you realize you can't do it. The task is way too big to audit it all properly with consistency.

Talk to management about investing in automation for an application security governance program. Get the money. Spend a few hours with the vendor implementing a cloud-based assessment program. Spend a few weeks with the development teams setting up the process for continual application improvement. Integrate the application governance program's metrics with the SIEM. And at that moment, everything seems to be coming together. You feel like you're actually getting ahead. You spend the weekend not thinking about work all the time.

On Monday morning, you instruct your team to go back around to the employees and sit with them. Have the employees show you how they use their computer. Listen to them about problems or strange things their computer does. Don't grill them about how it happened. Just encourage them to always speak up when something odd or different happens.

Make lots of notes. Make lots of changes to the desktop environment. Harden by reducing interactive services, separating privileges, and subjugating security they can't say no to in a prompt.

Talk the employees through changes. Force them to use separate browsers for separate environments. Help them find alternatives for those they think they do need but you'd rather not have installed. Explain the consequences of what could happen and how it happens. Notice a small crowd of employees listening. Explain phishing and how and why humans are designed to fall to fraud. See something click in their eyes. You realize this is what needs to be done all the time. This is how people learn. Scrap the automated security awareness videos with pop quizzes.

Spend more time talking to employees individually and in groups. You find yourself listening to their concerns and actually thinking about how to do what they want to do securely. Despite policy that says you won't, you start helping them understand their own gadgets and explaining why they might not want their mail there or most work documents for their own liability reasons. Don't fight with those who don't understand; instead, try to see it from their point of view and why they might want to. Get them to compromise and accept some security changes on their device to do what they want more safely.

One morning, you realize over breakfast that it's been over a year since the big hack attack. It still stings to think about it, but different. Now it stings like betrayal and no longer like helplessness.

Go into work and notice that there are no incidents on the board. Go through the archives and see how the help desk hasn't flagged any urgent fires to put out for weeks.

You notice little by little the office seems to be running smoother and your meetings with management are quicker. You no longer feel like you have to dance around your metrics. Instead, you've stopped making the reports about how fast you've patched machines, how many viruses the AV caught, how many vulnerabilities the scanners found, and how many attacks the IDS blocked. And you realize that management hasn't missed them either.

On the way back to your office, someone in the hallway stops you and tells you that you're looking good. Smile and thank them. Note that you do feel good.

Your security team reminds you that you never addressed the desktop patching and antivirus updating issues you caused by the network segmentation changes months ago. And despite that, there's been no problems. So you move the patching to the IT team doing change control. This way they can decide on the upgrades they want since you're no longer in the patch race against the criminals. You even consider scrapping the antivirus to reduce your attack surface, but you can't because it's required for compliance reasons.

Days go by with little stress. You keep an eye on your network traffic and you send someone from your team to take a look whenever anyone thinks their computer or device is acting funny. You make small security improvements every day.

As the emergencies grind to a halt and you're no longer racing to put fires out, you find yourself working more with IT on bringing newer technologies into the office securely. You spend your time planning ahead and create a network segment for beta testing. You bring new security tools and systems in to try. Your team seems to enjoy that much more too. Even the normally moody Linux admin seems to be in a good mood these days. Mostly, you realize you actually like working in security.

One day, you're reading the news, because you have time now to read the news, and you find an article on how to perfectly secure your data in three easy steps. It says put up a firewall, install antivirus on all your machines, and use 24/7 automated updating and patching. You laugh out loud and water comes out your nose.

Related Content

5 Best Practices in Data Breach Incident Response

Planning for Failure - Forrester Research