Maintaining a secure system involves monitoring file system activity, especially tracking file deletions, creations, and other modifications. This blog post explores how to leverage two powerful tools, auditd and process accounting with /usr/sbin/accton (provided by the psacct package), to gain a more comprehensive understanding of these events in Linux.
Introduction
Tracking file deletions in a Linux environment can be challenging. Traditional file monitoring tools often lack the capability to provide detailed information about who performed the deletion, when it occurred, and which process was responsible. This gap in visibility can be problematic for system administrators and security professionals who need to maintain a secure and compliant system.
To address this challenge, we can combine auditd, which provides detailed auditing capabilities, with process accounting (psacct), which tracks process activity. By integrating these tools, we can gain a more comprehensive view of file deletions and the processes that cause them.
What We’ll Cover:
1. Understanding auditd and Process Accounting
2. Installing and Configuring psacct
3. Enabling Audit Tracking and Process Accounting
4. Setting Up Audit Rules with auditctl
5. Simulating File Deletion
6. Analyzing Audit Logs with ausearch
7. Linking Process ID to Process Name using psacct
8. Understanding Limitations and Best Practices
Prerequisites:
1. Basic understanding of Linux commands
2. Root or sudo privileges
3. Auditd package installed (installed by default on most of the distros)
1. Understanding the Tools
auditd: The Linux audit daemon logs security-relevant events, including file system modifications. It allows you to track who is accessing the system, what they are doing, and the outcome of their actions.
Process Accounting: Linux keeps track of resource usage for processes. By analyzing process IDs (PIDs) obtained from auditd logs and utilizing tools like /usr/sbin/accton and dump-acct (provided by psacct), we can potentially identify the process responsible for file system activity. However, it’s important to understand that process accounting data itself doesn’t directly track file deletions.
2. Installing and Configuring psacct
First, install the psacct package using your distribution’s package manager if it’s not already present:
# For Debian/Ubuntu based systems
sudo apt install acct
# For Red Hat/CentOS based systems
sudo yum install psacct
3. Enabling Audit Tracking and Process Accounting
Ensure auditd is running by checking its service status:
sudo systemctl status auditd
If not running, enable and start it:
sudo systemctl enable auditd
sudo systemctl start auditd
Next, initiate recording process accounting data:
sudo /usr/sbin/accton /var/log/account/pacct
This will start saving the process information in the log file /var/log/account/pacct.
4. Setting Up Audit Rules with auditctl
To ensure audit rules persist across reboots, add the rule to the audit configuration file. The location of this file may vary based on the distribution:
For Debian/Ubuntu, use /etc/audit/rules.d/audit.rules
For Red Hat/CentOS, use /etc/audit/audit.rules
Open the appropriate file in a text editor with root privileges and add the following line to monitor deletions within a sample directory:
-w /var/tmp -p wa -k sample_file_deletion
Explanation:
-w: Specifies the directory to watch (/path/to/your/sample_directory: /var/tmp)
-p wa: Monitors both write (w) and attribute (a) changes (deletion modifies attributes)
-k sample_file_deletion: Assigns a unique key for easy identification in logs
After adding the rule, restart the auditd service to apply the changes:
sudo systemctl restart auditd
5. Simulating File Deletion
Create a test file in the sample directory and delete it:
touch /var/tmp/test_file
rm /var/tmp/test_file
6. Analyzing Audit Logs with ausearch
Use ausearch to search audit logs for the deletion event:
sudo ausearch -k sample_file_deletion
This command will display audit records related to the deletion you simulated. Look for entries indicating a “delete” operation within your sample directory and not down the the process id for the action.
# ausearch -k sample_file_deletion ... ---- time->Sat Jun 16 04:02:25 2018 type=PROCTITLE msg=audit(1529121745.550:323): proctitle=726D002D69002F7661722F746D702F746573745F66696C65 type=PATH msg=audit(1529121745.550:323): item=1 name="/var/tmp/test_file" inode=16934921 dev=ca:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 objtype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 type=PATH msg=audit(1529121745.550:323): item=0 name="/var/tmp/" inode=16819564 dev=ca:01 mode=041777 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:tmp_t:s0 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 type=CWD msg=audit(1529121745.550:323): cwd="/root" type=SYSCALL msg=audit(1529121745.550:323): arch=c000003e syscall=263 success=yes exit=0 a0=ffffffffffffff9c a1=9930c0 a2=0 a3=7ffe9f8f2b20 items=2 ppid=2358 pid=2606 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="rm" exe="/usr/bin/rm" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="sample_file_deletion"
As you can see in the above log, the user root(uid=0) deleted(exe=”/usr/bin/rm”) the file /var/tmp/test_file. Note down the the ppid=2358 pid=2606 as well. If the file is deleted by a script or cron, you would need these to track the script or cron.
7. Linking Process ID to Process Name using psacct
The audit logs will contain a process ID (PID) associated with the deletion. Utilize this PID to identify the potentially responsible process:
Process Information from dump-acct
After stopping process accounting recording with sudo /usr/sbin/accton off, analyze the captured data:
sudo dump-acct /var/log/account/pacct
This output shows various process details, including PIDs, command names, and timestamps. However, due to the nature of process accounting, it might not directly pinpoint the culprit. Processes might have terminated after the deletion, making it challenging to definitively identify the responsible one. You can grep the ppid or pid we received from audit log against the output of the dump-acct command.
sudo dump-acct /var/log/account/pacct | tail grotty |v3| 0.00| 0.00| 2.00| 1000| 1000| 12000.00| 0.00| 321103| 321101| | 0|pts/1 |Fri Aug 14 13:26:07 2020 groff |v3| 0.00| 0.00| 2.00| 1000| 1000| 6096.00| 0.00| 321101| 321095| | 0|pts/1 |Fri Aug 14 13:26:07 2020 nroff |v3| 0.00| 0.00| 4.00| 1000| 1000| 2608.00| 0.00| 321095| 321087| | 0|pts/1 |Fri Aug 14 13:26:07 2020 man |v3| 0.00| 0.00| 4.00| 1000| 1000| 10160.00| 0.00| 321096| 321087| F | 0|pts/1 |Fri Aug 14 13:26:07 2020 pager |v3| 0.00| 0.00| 2018.00| 1000| 1000| 8440.00| 0.00| 321097| 321087| | 0|pts/1 |Fri Aug 14 13:26:07 2020 man |v3| 2.00| 0.00| 2021.00| 1000| 1000| 10160.00| 0.00| 321087| 318116| | 0|pts/1 |Fri Aug 14 13:26:07 2020 clear |v3| 0.00| 0.00| 0.00| 1000| 1000| 2692.00| 0.00| 321104| 318116| | 0|pts/1 |Fri Aug 14 13:26:30 2020 dump-acct |v3| 2.00| 0.00| 2.00| 1000| 1000| 4252.00| 0.00| 321105| 318116| | 0|pts/1 |Fri Aug 14 13:26:35 2020 tail |v3| 0.00| 0.00| 2.00| 1000| 1000| 8116.00| 0.00| 321106| 318116| | 0|pts/1 |Fri Aug 14 13:26:35 2020 clear |v3| 0.00| 0.00| 0.00| 1000| 1000| 2692.00| 0.00| 321107| 318116| | 0|pts/1 |Fri Aug 14 13:26:45 2020
To better understand what you’re looking at, you may want to add column headings as I have done with these commands:
echo "Command vers runtime systime elapsed UID GID mem_use chars PID PPID ? retcode term date/time" "
sudo dump-acct /var/log/account/pacct | tail -5
Command vers runtime systime elapsed UID GID mem_use chars PID PPID ? retcode term date/time tail |v3| 0.00| 0.00| 3.00| 0| 0| 8116.00| 0.00| 358190| 358188| | 0|pts/1 |Sat Aug 15 11:30:05 2020 pacct |v3| 0.00| 0.00| 3.00| 0| 0| 9624.00| 0.00| 358188| 358187|S | 0|pts/1 |Sat Aug 15 11:30:05 2020 sudo |v3| 0.00| 0.00| 4.00| 0| 0| 10984.00| 0.00| 358187| 354579|S | 0|pts/1 |Sat Aug 15 11:30:05 2020 gmain |v3| 14.00| 3.00| 1054.00| 1000| 1000| 1159680| 0.00| 358169| 3179| X| 0|__ |Sat Aug 15 11:30:03 2020 vi |v3| 0.00| 0.00| 456.00| 1000| 1000| 10976.00| 0.00| 358194| 354579| | 0|pts/1 |Sat Aug 15 11:30:28 2020
Alternative: lastcomm (Limited Effectiveness)
In some cases, you can try lastcomm to potentially retrieve the command associated with the PID, even if the process has ended. However, its effectiveness depends on system configuration and might not always be reliable.
Important Note
While combining auditd with process accounting can provide insights, it’s crucial to understand the limitations. Process accounting data offers a broader picture of resource usage but doesn’t directly correlate to specific file deletions. Additionally, processes might terminate quickly, making it difficult to trace back to a specific action.
Best Practices
1. Regular Monitoring: Regularly monitor and analyze audit logs to stay ahead of potential security breaches.
2. Comprehensive Logging: Ensure comprehensive logging by setting appropriate audit rules and keeping process accounting enabled.
3. Timely Responses: Respond quickly to any suspicious activity by investigating audit logs and process accounting data promptly.
By combining the capabilities of auditd and process accounting, you can enhance your ability to track and understand file system activity, thereby strengthening your system’s security posture.