5 things you wish you had in place when you got hacked
– By Ivo Pooters – Team Lead Northwave SOC
When you read this, the unimaginable has already happened to your organisation, is happening right now or will happen in the near future; Your organisation’s IT environment will be breached in some way. The scope and damage of the data breach will be determined by the value of your assets, your adversary, detection measures and…your ability to respond to the breach.
Imagine the following: You receive a disturbing call on Friday 4pm from your system administrator. Strange activity on an administrator account in the network was observed. Files were copied and weird software was detected on servers. The sysadmin does not know what is going on, but knows that it is definitely not right…
Images of a presentation about the new data breach notification law and headlines on major news sites flash through your mind. You need to know what happened and contain the incident…fast!
At this point, you may decide to investigate the incident yourself or hire a CERT (Computer Emergency Response Team) to do it. Either way, the speed and success of the investigation is determined by availability of the right information; The best CERT in the world will not be effective when they have no data to investigate. Basic measures taken in advance (before an incident occurs) will increase your capability to effectively respond to a data breach.
At Northwave CERT, we help our clients manage IT-security incidents and we have investigated many data breaches. Based on our experience, here’s our top 5 of proactive measures that will enhance your investigative capabilities during incident response.
Collect audit Logs
Security audit logs are an important source of information for an incident responder while analyzing malicious user activity. Audit logs help the investigator determine what activity has taken place regarding a user account or user group.
The windows event audit log is often a very important source of information when a breach in a Windows environment has taken place. The active directory servers and Windows member servers are capable of extensive audit logging. For effective audit settings, we advise consulting the Centre for Internet Security benchmarks for various Windows platforms. These can be found here: https://benchmarks.cisecurity.org/downloads/latest/.
Network devices are another class of devices for which audit logs are relevant. Administrator activity can often be logged locally (or remotely) in an audit log. When an investigator needs to investigate a possible compromise of such a device or administrator abuse, these logs are helpful.
Perimeter devices (e.g. VPN gateways, webmail portals, other jump hosts) are an interesting entry point for hackers, since they can usually be reached from anywhere on the internet. Investigations of data breaches often lead back to a logon onto a perimeter device.
As a first step, the log settings of abovementioned classes of devices should be reviewed. This will provide you with an overview of current audit settings and indication of where additional configuration is required. Large volumes of generated audit logs can be offloaded to a SIEM or log retention server.
Collect network traffic logs
Malicious activity almost always takes place over a network. This is most true for the following phases of the cyber kill chain: reconnaissance, delivery, command-and-control and possible exfiltration. Logging of network traffic helps the investigator determine activity from the attackers in the network and scope a data breach. Organisations often lack any form of network traffic logging, due to unawareness of the relevance or lack of retention capability for the logs.
A common use-case for network traffic is investigating C&C traffic to a certain (or multiple) known evil server. Having netflow, DNS and/or HTTP logs available, will help scope which hosts in the network are infected. Also, when pivoting the investigation on a host in the network, network logs will tell a lot about the malicious activity on that host: is it beaconing? Which other hosts did it attempt to connect to? Where did the compromise originate from?
Firewalls and routers are a typical source for network logs; netflow-like data, blocked/allowed TCP/IP packets and DNS traffic logs may be generated by your firewall. Next-generation firewalls may also allow logging of TLS session content and network traffic that is flagged as suspicious based on intrusion-detection signatures. Most organisations make use of a local HTTP proxy server. Proxy servers can be leveraged to produce (more extensive) HTTP logging.
Full packet capture provides the ultimate network traffic log for investigations. It will provide the investigator full insight into content of network streams. However, a permanent full packet capture facility often requires large storage and disk IO for the capture appliance. This often leads to the tradeoff between completeness of network traffic logs and costs.
Network security monitoring appliance can be used to provide a (more centralized) network logger. Such an appliance is independent of the network devices in place and can provide a wealth of metadata on network traffic. Bro* is a great example of free and open-source network security monitoring software.
The cost for deploying and maintaining a network logger depends on the size and distribution of an organisation’s network. Examining and leveraging the logging features of existing security and network devices in the networks is a good first step.
Actual Asset Inventory
An actual inventory of devices, networks, data and software will help create context and prioritize in case of an incident. A configuration management database (CMDB) in some form is essential to know your landscape in case of a data breach.
- An asset inventory can help investigators answer relevant questions during an investigation:
- Which systems contain PII data?
- Which routers are in place and what software is running on the host?
- What kind of system is 10.22.42.5 and who can provide access?
Asset discovery tools can help asset management by automatically scanning the network for devices and identifying the software on the device. As a bonus, it will identify devices which are not (yet) in the cmdb.
It helps if asset information is available in a single centralized location in a structured format. Often, the information is spread over different databases managed by different service providers. For example, network devices may be listed in a different inventory managed by network service provider, while the virtual machines are listed in the Microsoft Hyper-V inventory. This situation is not ideal since it fragments the asset information and provides no guarantee of normalised information storage.
The form and size of the CMDB will depend on the size of the IT-environment of the organisation. A basic excel file or a simple open-source configuration management tool can effectively facilitate an asset inventory tool for small sized IT-environments. Larger environments may require more robust and elaborate CMDB setup.
Service Level Agreements with 3rd party service providers
Outsourcing system administration of IT-environment is more common every day. Organizations will contract 3rd party service providers to manage appliances, network devices and security devices. Service level agreements will cover the service aspects of system management, however this does not per se cover the case of an IT-security incident (with exception of continuity incidents).
When an incident occurs it is paramount that the responders have access to the required IT-systems and data in a very short term. When systems or data are involved that are outsourced, access to the data will go through a service provider or require access by the service provider. Having clear agreements on the service that can be expected from the providers in this context will make the collaboration during an incident go more smoothly.
Some points that should be considered in context of incident when working with 3rd party service providers:
- Availability of log data
- Availability of designs and diagrams about the managed IT-infra
- Response time to data and information requests
- Audit trail and change process
- Escalation path for high priority incidents
- Access to devices administered by service provider
Also, with external service providers connecting to the organisation’s network and having privileged access, the service provider itself becomes a potential security risk. Data breaches have occurred where attackers accessed the network through systems of an external service provider. In this context, it is worth to consider ensuring a service provider’s cooperation in the investigation where their systems are subject to investigation.
An incident response plan
When the technical and legal measures are taken to be prepared for a data breach, it is important that the right people in the organisation know about these measures and how to leverage them. An up-to-date incident response runbook will make a smoother response when an incident happens.
Here are a few examples of questions that can be answered upfront instead of after the data breach:
- What should we do with the system that is possibly compromised?
- Who is our partner for emergency response?
- How much does this partner know about our environment?
- Who is our contact for Windows system administration?
- How should we communicate internally and externally about a data breach?
The size and content of the plan will be determined by the tasks that will be performed by the organisation and the tasks that will be outsourced to external parties. Here are three elements that we suggest to include in the plan:
- Contact information of external and internal parties that might be required during an incident. E.g. who should an incident responder contact for access to the firewall logs?
- Service level agreements with various providers. What service can the incident responder expect from external service providers?
- Pointers to important requirements following from data breach notification law. Depending on the country or industry, other legislation may be relevant as well.
But most importantly: practice, practice, practice; A plan on paper is not worth much when the people involved do not know how to execute it or perhaps don’t even know of it’s existence. Also, practicing will improve the plan by using the feedback to make it more useful and effective.
*The Bro Network Security Monitor. https://www.bro.org/