During this interview Akshat explains what happened during the BlogVault security incident, how he and his team found out about it, its aftermath, and how did the public react to their announcements and transparent approach.
A lot of noise is made when a popular WordPress website or service is hacked, but not much is done to cover the aftermath and the recovery process. On the 6th of February BlogVault CEO Akshat Choudhary announced that their online WordPress backup service suffered a security breach. Akshat and his BlogVault team were very transparent about what happened and during the recovery process. They;
- alerted all their customers via email of the breach,
- announced it publicly,
- issued an update for their plugin to address the issue within hours,
- cleaned the websites of those customers’ who had an infected website,
- opened a chat channel and were readily available to answer and assist all their customers,
- kept the public posted of the progress and released all possible details.
It turned out that the BlogVault service was not hacked, but a number of WordPress websites were hacked through a vulnerability in their plugin. The good news is that within a few days Akshat and his team ironed out all issues. I have been working in WordPress security (and generic IT security) for a number of years and very few of those who suffered a security breach were so transparent and open about it. Typically they announce that they will be, but they do not, and the incident is forgotten because people easily forget.
I’ve been using BlogVault from their early days and has endorsed them ever since, so in light of all of the above, I interviewed Akshat to learn more about the technical details of the whole incident. How did BlogVault notice that something was wrong, what steps did they take to recover from the attack, how did the public and his customers’ react, and last but not least, what can everyone in the WordPress ecosystem, especially those who have an online service learn from this security breach?
Interview with BlogVault CEO Akshat Choudhary
Akshat, how did you notice that there was a possible hack or intrusion?
A couple of our customers emailed us saying that they noticed malware on their websites. They also said that the only link between the affected websites was that they had the BlogVault plugin.
Around the same time our in-house malware scanner and cleaner that we had been building, was also identifying a higher number of hacked sites, than was usual. The Rest API vulnerability in WordPress version 4.7 was going around that week, and so we wanted to be sure to eliminate other such possibilities.
When we cross checked these sites, we verified that the sites had actually been infected with a common virus, and launched a preliminary investigation. It was then that we noticed the possibility of a hack or intrusion.
Can you walk us through the process of how you identified the issue and the source of the problem?
Once we had ascertained that there was a link between the websites getting hacked and BlogVault, we had to understand the exact cause. We knew that the hack could have taken place in one of two ways;
The first possibility, was that there was a breach of our security infrastructure. This was the worst case situation; however at the time we thought this is what had happened.
The other possibility was that there was a vulnerability in our plugin. We analysed data on both fronts to ascertain the cause.
To address the first possibility, we got down to analysing our systems and logs thoroughly. This is a very deliberate and time-consuming process. However, given the grave nature of the scenario we didn’t want to keep our customers guessing, so we reached out and informed them that we may have been breached. Only at end of our investigation were we sure that we were not breached. At this point we sent out a mail stating the same. This probably wasn’t a very wise decision from a PR-perspective but we knew what was at stake and we didn’t want to risk our customers in any way.
The other possibility– a vulnerability in our plugin, turned out to be the source of the issue in the end. We inspected the code of our plugin one line at time, sifting for vulnerabilities. While we were doing this, a customer reported a vulnerability in our plugin. We released an update for the plugin on the same day with a patch for this vulnerability.
While this vulnerability was the only one which was exploited, we could not be certain of this at the time. It was only after analysing our own logs and the logs of some of our customers websites could we sure.
What were you looking for in the logs? i.e. what other WordPress website / service administrators can look for in such cases? Any tips?
As I mentioned our internal systems’ logs were audited systematically and thoroughly for signs of intrusion.
Many of them who reached out to us, were very helpful during the investigations and shared logs with us. With the logs of their sites, we were able to identify common requests and IP addresses. With the daily backups of the sites we maintained, we were able to track exactly the changes the virus was making to the affected websites.
To answer the second part of the question, while this process was specific to our case; generally when looking for malicious requests, I’d advise looking for ‘POST requests’ rather than ‘GET requests’. It can be time-consuming but will point you in the right direction. This is because, the payload related to the malware is often present in a POST request. For us though, having our own scanner and cleaner proved to be invaluable in identifying the malware on sites.
Can you give us a “technical” summary of what happened during the hack?
The hacker exploited PHP unserialize function; which was employed by our plugin, to inject malware onto sites. As I mentioned, it should not be used on unvalidated inputs.
The attack was very similar to the one described in this article: https://malware.expert/vulnerability/common-php-object-injection-vulnerability-in-backup-restore-dropbox/.
Can you walk us through the process of how you fixed the issue and how you ensured everything was clean?
One of the first steps we took, was to reset the passwords for all BlogVault accounts. We never stored passwords and we follow best practices of storing only the bcrypt hashes; nonetheless we reset passwords of all our customers.
We tried to be as communicative as possible with our customers. We reached out to them to let them know that they could contact us and that we would clean the affected websites.
Again, our automatic scanner and cleaner was of big help. It helped us identify the malware and undertake large-scale cleanups of websites. Without the scanner it would have been difficult for us to clean as many sites within the time frame in which we did. Once we cleaned the sites, we performed multiple scans at regular intervals to ensure that they remained clean.
Lastly, I have to say that most of our customers were very cooperative and patient with us throughout the entire incident.
I followed this incident from the beginning. First of all, kudos for being transparent with the public and customers. Nowadays it is not a common thing to see unfortunately. Considering you’ve been so transparent, what was the people’s reaction? Can you give us a few (anonymous) examples of what people said, expected or how they reacted?
Mostly people wanted to be kept in the loop and we tried our best to do so, as and when it was possible.
We had some customers who were understandably upset and we expected that. What we didn’t expect was anyone to express confidence in us or be supportive in such a time. However, many customers; even those whose websites were affected, said that they trusted BlogVault and knew that we would solve the issue. In fact, some went out of their way to message on chat or even write emails showing support.
Frankly, knowing that people trusted us helped us in a big way to get through what was a very strenuous marathon; both, for them and us. It may sound corny, but we were humbled and encouraged.
What security measures did you implement in the aftermath, to ensure this does not happen again?
During the course of our investigation we had released a couple of more security updates after patching the ‘PHP unserialize’ function, as we were still looking at other possibilities of intrusion at the time. We are now committed to having regular audits of our plugin by external security experts as well.
Apart from that; after the hack, we had undertaken a wider project of making architectural changes to secure our systems for the future. While our systems were not affected by this issue, we have performed a host of changes to our systems to ensure that if at all our systems are intruded in the future, the damage is minimal.
Should it happen again (hopefully not), are there any things you would do differently?
Let’s hope that it does not! As I said, we have taken numerous steps to avoid such issues; and continue to be committed to evaluating our systems’ security regularly.
The one aspect that I wish that we could have handled better, was our communication with our customers. The moment we announced the issue we received many queries. In such situations customers rightly expect quick; if not immediate, responses.
However, we did not have the right tools. We are a very small team and we were working round-the-clock but we weren’t always able to keep up.
We have implemented changes on this end as well.
Any tips and recommendations for the readers, especially to those who have an online service / WordPress business like you?
WordPress is an open source project that brings many benefits. As a company operating in the WordPress ecosystem, we have to constantly improve and provide better products and services which are secure and show continuous vigilance. We also strongly recommend that you get plugins audited regularly.
Last but not least, where is BlogVault heading? Any new features coming up?
We are looking forward in a big way. I mentioned our automatic hack scanner and cleaner. Ideally if this issue had not occurred we would be working on fine-tuning that and doing a broad release.
We have experienced its value first-hand during this process, and have definitely found it to be invaluable. We are proud of it as there has not been an automatic hack cleaner that is so powerful for the end-user, and we are pushing now for the broad release.
Along with that we are pushing to expand the capability of the BlogVault dashboard to remotely manage WordPress sites. So, you can backup, scan, and manage WordPress from a single dashboard.
