Troubleshooting suspected hacked sites
One of the most popular features of Watchful is that site administrators receive email or RSS notifications when critical files are modified. These files include:
Unexpected modifications of these and other sensitive files may be an indication of unauthorized access.
However, please note that Watchful only regularly monitors files that are of particular interest to hackers and spambots (your entire site — not just the critical files — may also be scanned using Watchful's Site Audit). If your web site has indeed been hacked, modifications to these files are not likely to be the source of the hack. Rather, your site is likely to have been hacked elsewhere, and the hack was then used to modify some of your sensitive files.
Thus, we recommend that any unexpected modification detected by Watchful be followed up by a full investigation.
As a general outline, we recommend the following workflow when investigating unexpected file modification notices that you receive from Watchful.
Please note that if you are not confident in your ability to compare large file sets (as described below) or if this article sounds confusing or is hard to understand, we recommend that you hire a professional to investigate changes in your site.
1. Enquire with your coworkers, colleagues, contractors, employer or anyone with access to your web site and/or your web server to determine if someone has made the change without your knowledge.
2. If you still suspect unauthorized access following Step 1, disable public access to your Joomla web site by navigating to Site > Global Configuration and enabling Offline mode. We also recommend that you enable HTPASSWD protection — sometimes called directory protection — on your entire site during the troubleshooting steps below.
You may need to contact your hosting provider for help setting up an HTPASSWD, though it is a common feature in most hosting control panels.
3. Create a backup of your site using your backup solution of your choosing (we recommend Akeeba Backup).
4. Compare or DIFF the backup created in Step 3 to your most recent backup (ideally, these are stored at an offsite location). Many text and HTML editors will include a DIFF feature, but as of this writing we recommend DiffMerge since it is free and runs on Mac OS, Windows and Linux.
If you are using Akeeba Backup, we also recommend SiteDiff.
When the DIFF process is complete, examine any files that have been changed. You should find at least one file different between the two backups: the file noted in the Watchful notification email. If this file is not identified as different, it is likely that the hack occurred prior to your most recent backup.
5. Repeat Step 4 with progressively older backups until the file noted in the Watchful notification email is identified as different between the two backups. By noting the date of the backup that first locates the file change identifed in your Watchful notification, you can determine the earliest possible date on which you site was hacked.
Note that on active sites you might find a lot of new images between the old backup and the most recent one. These may be completely normal and expected, but also note that corrupt images can also be used to exploit some vulnerabilities. You may also find scripts uploaded to your site.
7. With some luck, after all this you will have found a source of the attack, perhaps a file or script that provides a backdoor for hackers. However, it may not be clear how this backdoor was added to your site in the first place.
If the hacker has not carefully hidden his tracks, you may be able to use the filename of the backdoor to search the server logs (obtained from your web host or perhaps your web hosting control panel) and further determine what requests were sent to the server that allowed the backdoor to be uploaded. This may then reveal the true source of the hack, a vulnerability in an add-on or in Joomla, for example.
8. Based on the results of your investigation, take any and all appropriate action to (i) remove any dangerous files added to your site, (ii) restore hacked or deleted files to their original state and (iii) patch the true source of the problem.
You may also want to repeat the entire process using the very oldest backup available (preferably from the day your site was launched), but for active sites this task quickly becomes time and cost prohibitive.
9. Once you are satisfied that you have found and removed all the hacked files and backdoors, it’s time to re-publish your site by disabling Offline mode in the Global Configuration and disabling the HTPASSWD (see Step 2, above).
Before proceeding to publish your site, we recommend that you read the Joomla Security Checklist as well as adding HTPASSWD protection to the /administrator/ folder in your Joomla web site.
- Configuring remote backups with Akeeba Backup
- How do I generate reports for my clients?
- How do I use the Site Audit?
- How do I use the Watchful REST API?
- How often does Watchful monitor my web sites?
- How to backup a site in Watchful
- How to create a Watchful App
- How to make add-on/extension updates compatible with Watchful
- Installing the Watchful client
- Managing admin access with Single Sign-on