In this post, I'll share two fascinating hacking stories I've experienced: one involving a sophisticated scam that targeted a major U.S. Fortune 500 conglomerate, and another detailing the implementation of honeypots by a renowned U.S. think tank that went wrong.
Hacking Story 1
I was consulting for a large, U.S. multinational, multi business, conglomerate, Fortune 500 company. I had been brought in because they had been badly hacked for the third time.
They had been fined over $100 million for the second hack, so this hacking incident was likely to be even more expensive and result in many heads rolling (i.e., firings).
We had no idea how the hacking happened. We were clueless. Because of this we decided that the only thing we could do is completely rebuild the network from the ground up. Every account would be disabled and every password would be changed. But it would not be easy because you cannot do that all at once in a huge, multinational company without significantly impacting revenues. It had to be strategic and planned.
Because we had no clue how the hacking had occurred or even what the hackers had compromised and were still in control of, the incident response team (which numbered nearly 20 individuals) decided the only safe option was to use brand new equipment on brand new networks. So, we bought each team member a new laptop, new Wi-Fi routers, created and used new networks, and even met offsite in a new building location unrelated to the current company. We were taking no chances.
So, in our new, secure location, the team met to discuss our response and how to rebuild the network from scratch. One of the major decisions was how and when to reset everyone’s password. We needed to do it globally all at once to minimize the chances the hackers could get back in. We decided that all accounts would be disabled, all passwords reset, and that each legitimate person needing to login after that event would get their new temporary password in person from their boss.
That way, every person getting a new password would be physically vouched for by their boss. This process of disabling, resetting, and giving out new passwords would be a huge, separate project by itself. Although all accounts would be disabled and all passwords would be reset at once, the process of allowing people back in would be at least an all-day process, “following the sun”, as business daytime zones occurred globally around the world.
This part of the project was so big that we decided it needed its own code word so that we could refer to it without anyone listening in to understand what we were talking about. We decided the code word would be ‘picnic event’. This was because the day all passwords would be reset coincided with a company-wide employee picnic event. We figured there would be fewer questions from random people who overheard us talking about the picnic event on the same day as the actual event.
One of the big challenges was how to minimize operational disruption stemming from resetting all the admin accounts. The hackers were likely in control of one or more admin accounts, and for true security, we had to reset all of them. We then wanted to only allow legit admins back in. That might sound easy, but a large, global company literally has thousands of admins, most of whom the top network administrators have never met, do not know, and do not really trust. This is pretty normal in a very large company, insane as it sounds.
So, we decided that on the day we disabled all accounts (the picnic event day), we would have all admins fill out a form proving their identity and telling us why they needed admin rights and permissions. We spent a few hours creating the form on one person’s laptop displaying it to the rest of the team members using the local data display projector we had brought along and remotely using a telecommunication’s service that was a popular precursor of Zoom. We probably got 90% of the way through the form, but had not finished it. It was late, so we called it a day.
As usual, I showed up early the next day in the remote team meeting room about an hour before the rest of the team did, although the company’s project leader was always already there. He came in about 30 minutes before I did. We both appreciated the “quiet time” where we usually worked in silence or brought up critical issues to each other without the full team there.
Usually, I walked in, said hello and got right to work. This time the project leader said, “Hey, we got our first two picnic event forms in!” I said, “What!?” The project leader had a grin telling me that he was as befuddled as I was.
The “picnic event form” was the document we had been working on the previous day and had not even finished that admins would fill out and submit in the future after we cut off their access. But the picnic event was months away and we had not finished the forms, much less, sent any out.
But, indeed, there were two fully filled out picnic event forms. In the area where the admin was to justify why they needed their admin access restored, they had written, “Because our passwords were reset due to the picnic event.”
It was filled out by two Chinese employees who we did not know, although they were real employees from one of the company’s China-based locations.
So, how did two people, not on the incident response team, get a form that was only located on a single person’s laptop, fill it out, and then submit it?
Well, we had our hacker spies or at least some of our hacker spies identified. That was the easy part.
But we were flummoxed as to how those two individuals had gotten the picnic event form from our leader’s laptop, which was on a separate, isolated, stand-alone network, not connected to the real corporate network in any way. We were stumped!
It took us a few hours to figure it out and I am proud to say that my hunch led to the answer. I finally realized that the only possible link back to the original corporate offices that we had were our data display projector and our “Zoom link”. I realized that the fancy data display projector we were using allowed for remote logins and indeed when we checked the projector’s digital logs, two IP addresses from China had been logging into the data display projector and watching the entire time.
As another test, I asked the team leader to print the real picnic event form on his laptop and then I compared it to the two newly arriving picnic event forms we had received. They were identical in every way except one line was off by one row. They had made a single small mistake…well, on top of their rather large one.
The Chinese hackers were good, but I am assuming the language barrier prevented them from understanding that the picnic event form which they saw on our screen was something we had not planned to send out for another two months. They had missed that one vital fact and then scurried to make sure their own, unauthorized access was justified on their picnic event forms.
Later, after extensive research, we discovered this company had a ton of Chinese hacker spies in it, not only in the data display projector system, but also in their international telephone system. In the end, all Chinese employees were let go from the company and it learned it could not trust Chinese employees and locations. The risk was just too great.
Takeaway: The biggest lesson learned was that it is really hard to have true, new network isolation if you are not truly doing everything brand new.
Watch the full video here:
Hacking Story 2
I had been brought in to install honeypots into a large, popular (you would recognize the name) U.S. think tank. Honeypots are fake systems that are intended to capture hackers and malware. Because a honeypot is a fake system, nothing should be trying to log into them and anything logging into them is likely malicious.
I was one of the world’s leading experts on honeypots. I even wrote a book on them. The client had suspected they were being hacked from the outside and wanted to install multiple honeypots to find out how the intruder was getting in.
So, I set up the honeypots, instructed the staff on how to monitor and maintain them, and waited. The staff was very concerned that I had done nothing special to attract the hacker to the new honeypots. “How would the hacker find the honeypots?”, they wanted to know. I told them that if there were hackers, they would find the honeypots and not to worry. I was not wrong.
The next day, we detected a hacker connecting to one of the fake web servers. But it was not an outside hacker. It was an insider. We quickly traced the hacking to a young, beautiful employee who was working in accounts payable as a payroll clerk. There was already a video camera in the room, so we were able to watch what she did. She was an expert hacker.
We were not sure what she was interested in, so we created a few different web servers. One was a web server that mimicked the U.S. space shuttle program, although all we did was copy the publicly accessible real-world space shuttle website down to our fake website, but made the names and directories seem top secret-like.
We made a second website that purported to discuss Middle East policies, which was a specialty of the thinktank. And we made a third website that ran a popular game. At the time, many IT employees were illegally using servers to run unauthorized versions of this popular game. We wanted to see if the hacker was interested in playing games or wanted more valuable information.
The hacker went straight for the space shuttle and Middle East servers and left the gaming server alone. We had our answer. This was a real, legit hacker.
We watched her for a few days, recording everything she did. We learned that she had even placed an unauthorized wireless network card (which was rare for that day and time) into our work computer and was sending captured data into a public meeting room a hundred or so feet away. That public conference room was “rented” by hundreds of different groups and would have dozens to hundreds of different people in it all the time. Well, one of those people was a spy siphoning off data.
Finally, corporate security and members of our team confronted the inside hacker. As soon as we walked into her work room, she threw her hands up and said her computer was doing something she did not understand. She was crying and whining that none of the hacking was her…that it was someone else in control of her computer. Had we not watched her for days, I would have believed her crocodile tears. She could have won an Academy Award.
The funny thing is that the thinktank had hired her (and others) from a local accounting temp job firm. She was just temporary “help”. And supposedly, she was so bad with computers that the think tank had sent her to keyboarding school to improve her computer skills.
We were never able to learn who the other hacker was in the public conference room, although me and the incident response team had bought a mobile, portable “sniffer” wireless antenna that could have tracked that person down. The think tank’s lawyers shut down our investigation into that hacker at the last second, as we were headed into the room, because of legal issues.
Turns out that not only was the young woman a hacker, but her entire accounting firm was a Russian foreign hacker entity. It was only in existence to spy on U.S. companies. I alerted my company, and we did indeed learn that one of the employees was in our company acting as a spy. This particular spy had been stealing our source code.
Many months later, I was able to see the young woman I had helped catch, along with other “co-workers” of hers, boarding a commercial flight home to Russia in an international prisoner exchange.
One of the other spies had been caught in an even higher profile, public hacking event. She had been arrested, detained and was being deported in a U.S.-Russian spy prisoner exchange. The young woman I had helped catch was part of the exchange and ended up on the television next to the more popular hacker compatriot.
Takeaway lesson: Not all hackers break in from the outside.
Here's how it works:
