In our previous article, we discussed the importance of a good understanding of step-by-step guideIf you or your service provider have noticed any unusual activity on your Linux servers, there are a few simple ways to determine if your data is compromised.
When you are certain that a hack has occurred, it is time to take action. In this guide we will discuss some basic steps that you can take to protect yourself after a breach, such as isolating your computer from the rest of the network or making copies of drives.
You’ll learn how to implement some changes that will prevent future attacks.
Step 1. Step 1: Isolate the compromised server from the network
You should isolate your Linux server from the network if you suspect an unauthorized user or access. Isolating the server will stop it from spreading to other computers on the network if malware has been installed.
If you are at the server you can disconnect the Ethernet cable. You can disable specific network interfaces using the ifconfig command if you access it via SSH.
sudo ifconfig <interface_name> down
Be sure to replace ‘<interface-name>’ with that of the specific interface, e.g., ‘eth0′. Run:
ls = /sys/class/net
If your server is wireless-capable, you can block these connections by:
All block with sudo-rfkill
Step 2. Step 2.
For legal compliance, and to prevent hacking incidents like this from happening again, it is vital that you document and log everything on the server.
This documentation can be used as valuable evidence in the event that malicious or rogue processes are still active on the compromised server. Save a list of all processes using the following command:
ps aux > process_snapshot.txt
This file can be viewed using a text editor such as GNU nano The following are some examples of how to use Vim.
Step 3. Step 3.
It is crucial to document the server hack by logging the processes that are currently running. To conduct an in-depth investigation, cybersecurity experts may require a secure copy of the server’s disks.
This can be done by connecting a blank external hard drive formatted with the server. The drive should have a size that is at least equal to the sum of the drive sizes on the server, if not greater.
To begin, open Terminal and run the following command to list all drives on your server.
df
You can create copies of disks using a variety of utilities. From the command line, ‘dd’ is usually the simplest. If you want to copy /dev/sdc1’s contents to a drive mounted externally at /dev/sdb1, then run the following command:
sudo dd if=/dev/sdc1 of=/dev/sdb1/server_image.dd bs=4M status=progress
After the copying process is complete, you can check the integrity and quality of the disk image by generating a cryptographic SHA256 hash or MD5, e.g.:
md5sum server_image.dd
Save the hash of the drive and then unmount the drive safely. Store it in a secure place.
Step 4. Step 4.
Seek advice from professionals in cybersecurity if your server is compromised. This step is crucial, especially if your server holds information that’s subject to GDPR regulations.
A trained cybersecurity professional can help you document the incident correctly and assess the attack vector through the analysis of the offline copy on the server drives. This action will increase your protection from an attack next time.
Step 5. Step 5.
If you have been regularly backing up your server, you can restore it to the point where it was before being compromised.
This is a quicker and easier way to recover than to search every file, port, user or folder on your current server.
The steps to restore from a Linux backup vary depending on the Linux distribution and server provider. Even so, you can minimize the chances of residual malware by verifying your backup’s integrity and isolating the server from the rest of the network when restoring it, e.g. by booting into Recovery Mode.
If you have a GRUB bootloader installed on your OS, then you can restore it along with the backup of the operating system via:
grub-install /dev/sdX
Make sure to replace ‘/dev/sdX’ with the name of the server’s boot partition. After restoring your backup, run a complete system update.
Step 6. Step 6.
Even if you’ve used a fresh backup, your Linux server could have been compromised over time. You should scan your newly recovered Linux server using trusted antivirus tools to check for malware.
ClamAV ClamAV is still one of the most effective malware scanners available for Linux servers. ClamAV can be installed on Ubuntu Server if it is not already there.
Install clamav with sudo apt get
ClamAV is available for Red Hat users via:
sudo dnf install clamav
ClamAV already installed? Update the virus database by:
sudo freshclam
You can now perform a recursive malware scan and remove any infected file:
sudo clamscan -r –remove /
We also recommended installing and running either ‘chkrootkit’ or ‘rkhunter’ to scan for and remove rootkits, as these can reinfect servers that have been restored from a backup.
Step 7. Step 7.
If you restore your server using a backup it will include login data such as usernames and passwords. Once this data has leaked, there is nothing that can stop hackers from gaining access to your system.
You can mandate a password reset for specific user accounts using the ‘chage’ command by configuring the -d flag (password expiry time) to 0. To do this for user ‘barnowl’, for instance, just run:
sudo chage -d 0 -M 0 -I -1 -E -1 barnowl
If you have a large number of users on the server, you can use a ‘for’ loop to iterate a mandatory password change for all accounts:
Do not enter the username into $(cut d: f1 /etc/passwd);
sudo chage -d 0 -M 0 -I -1 -E -1 $username
The following are some of the ways to get started:
Review the current password policies by:
sudo nano /etc/security/pwquality.conf
Here you can specify password requirements. Add the following to specify that a password must have a minimum of 12 characters.
minlen = 12
Step 8. Step 8.
You can then determine the last attack vector that was used to compromise your server after your cybersecurity professional has completed their analysis. After they have completed their analysis, you can determine the attack vector used last time your server was compromised.
For instance, they may advise you to protect the server against ‘brute force’ password attacks by installing and configuring ‘fail2ban’ to detect the IP addresses of multiple failed login attempts and block them accordingly.
Cybersecurity experts might also advise you to implement two-factor authentication (2FA) for users connecting via SSH via common packages like libpam-google-authenticator.
The tool will be run from the user’s account and codes generated using an authenticator app.
It may be necessary to update configuration files and install additional security patches. Do not ignore the advice given by your cybersecurity experts.
Resilience is the key to recovery
Remember that recovering from a compromised system is only the first step. You can not only restore the integrity of your system, but also strengthen your defenses to future threats by following these steps.
Keep your security up-to-date and be aware that the cyber landscape is always changing. With the help of partners and appropriate precautions, you can secure your Linux environment.
