Four months of phone calls and two prior embassy visits had led Popov to this point. Now he met with an FBI assistant legal attaché to present his passport and make final arrangements. A short time later, he plowed through the wintry cold of Grosvenor Square to a hotel room the embassy had secured for him. He opened both his laptop and the hotel minibar and read his email while downing tiny bottles of whiskey until he passed out. The next day, January 19, 2001, Popov and an FBI escort boarded a TWA flight to the US.
Popov was nervous but excited. He’d left behind his parents and everything else familiar to him, but in the US he would be more than a dutiful son and student. Popov was also a wanted man involved in international intrigue, like a character in one of the cyberpunk novels he loved. Now he would reinvent himself by selling his computer security expertise to the government for a decent salary, then transition to an Internet startup and make himself wealthy.
When the plane landed, though, it was clear the arrangement was going to work a little differently. The once-friendly FBI agents threw Popov in an isolation room, then returned an hour later with a federal prosecutor, a defense attorney, and a take-it-or-leave-it offer: Popov was going to be their informant, working all day, every day, to lure his crime partners into an FBI trap. If he refused, he’d go to prison.
Popov was shocked. He’d been played for a durak—a fool. He was placed under 24-hour guard at an FBI safe house in Fair Lakes, Virginia, and instructed to talk to his friends in Russian chat rooms while the bureau recorded everything. But Popov had some tricks of his own. He pretended to cooperate while using Russian colloquialisms to warn his associates that he’d been conscripted into a US government sting. When agents finally got the logs translated three months later, they angrily pulled Popov from his comfortable safe house and threw him in a small county jail to face charges for his past cybercrimes. Popov armored himself in defiance. “Fuck you,” he said. “You have no idea what you’re dealing with.” But he was scared. Prosecutors around the country were lined up to indict him. There seemed no escape from a future of endless jail cells and anonymous American courtrooms.
Except that in a backwater FBI office in Santa Ana, California, an up-and-coming agent named Ernest “E. J.” Hilbert saw that the government needed Popov more than anyone knew.
Hilbert recognized that the US was at a crucial moment in computer crime. Throughout the 1990s, hacking had mostly been a recreational sport. But in 2000 the first tremors of change began radiating out of Eastern Europe. The signs were everywhere if you knew to look: the types of websites being hacked, the volume of spam and phishing email, the first uptick in credit card fraud losses after years of reliable decline. Hacking was evolving into a professional and profit-driven enterprise.
In 2001, Ukrainian and Russian hackers debuted a website called CarderPlanet that introduced an even more ominous property to the underground: scalability. CarderPlanet was a thieves’ market for buying and selling hacked credit card numbers, passwords, stolen bank accounts, and identities. It featured paid advertising, an eBay-like review system, and an organized message board. For the first time, an aspiring identity thief could source all the raw material for a crime wave on a single site. Thousands of users signed up.
Hilbert figured he had a shot at cracking this world. But first he’d have to crack a pissed-off hacker who’d already tricked the FBI once.
Weaned on cyberpunk fiction and the 1995 movie Hackers, Popov knew two things from the start: He was going to be a computer outlaw, and he was going to make money at it. He found plenty of fellow mercenaries in the Russian-speaking regions of the Internet. In the late 1990s, former Soviet states were as flush with smart young programmers as they were impoverished of high tech career opportunities. Cadres of hackers were bootstrapping their own dotcom gold rush, stealing credit card numbers from US ecommerce sites.
Popov wasn’t as technical as many in his cohort, but he had a talent for managing and manipulating people and a gift for language. He began making money by “cashing out” stolen credit card numbers, using nearly flawless English to phone in fraudulent orders to US cell phone and computer retailers. It was a good business for about a year, but the stores eventually grew wary of Eastern European shipping addresses, and the scheme dried up.
Popov would steal customer data, then contact the company and offer his services as a “security consultant.”
At the same time, local gangsters learned of Popov’s online scamming and began showing up at his apartment to strong-arm him for cash. Popov decided to try his own hand at extortion. He and his crew would crack a company’s computers and steal customer data, then Popov would contact the company and offer his services as a “security consultant” to keep the intrusions a secret in exchange for money.
In July 2000 they cracked E-Money, a now-defunct electronic payment provider based in Washington, DC, and stole credit card data on 38,000 customers. They hit Western Union’s website for another 16,000 customer names, addresses, passwords, and credit cards. Popov approached the companies and offered to put a stop to the intrusions and bury the stolen data in exchange for consulting fees that ranged from $50,000 to $500,000.
The results were inauspicious. E-Money strung him along while secretly calling in the FBI, and Western Union publicly announced the breach, obliterating Popov’s hope for hush money. His efforts amounted to nothing, even as the pressure from neighborhood thugs escalated. Popov felt trapped in Zhytomyr, in his life of middling scams and looming violence. He began contemplating a bold move: turning himself in to the American police. He would escape from Ukraine, he figured, and reboot himself as a reformed hacker and computer security expert in the Land of Opportunity.
Now he found himself stuck in a St. Louis jail near Western Union’s offices. At least until Agent Hilbert came looking for him.
A straitlaced family man with the air of a 1950s sitcom dad, Hilbert looked every inch a Fed, with an earnest gaze and neat brown hair combed into a crisp part. He had walked away from a career as a high school history teacher at the age of 29 to pursue his childhood dream of wearing an FBI badge. His very first case established him as a cybercrime agent, when he linked a computer intrusion at an Anaheim, California, company to a prolific hacker in Russia’s Ural Mountains, then helped engineer a sting that lured the suspect to Seattle so the FBI could arrest him. Hilbert understood hackers. As a suburban kid growing up near San Diego, he’d done some innocuous hacking himself, adopting the name Idolin—his take on an ancient term for ghost or spirit.
Hilbert knew that as a native Russian speaker and experienced cyberthief, Popov could go places the FBI couldn’t, moving through underground chat rooms and message boards, forging relationships, and feeding the bureau much-needed evidence and leads. The trick would be to manage Popov carefully, stroking his ego and showing deference to his skills.
Hilbert discussed his plan with a prosecutor in Los Angeles who had a case pending against Popov, and the two were soon sitting across from Popov and his lawyers at the US attorney’s office in St. Louis. They laid out a deal. Popov would get time served on the Missouri case, and the government would consolidate all the others in Southern California, where Popov could work off his charges by going undercover for the FBI.
This time Popov wouldn’t be expected to set up his friends. His targets would be strangers to whom Popov owed no loyalty. Hilbert called it an intelligence-gathering mission, like something James Bond might do. “I truly respect your skill set,” Hilbert said. Popov signed a plea deal accepting the government’s offer in March 2002, and Hilbert had his mole.
Still, it was a relief in August when Hilbert and another agent collected Popov for his first day at work. In a procedure that would become an almost daily routine, the agents kept Popov shackled and handcuffed as they led him to their car. After a short drive, they pulled up to the back door of an office building and escorted Popov to a small room stuffed with desks, a table, and a handful of Windows machines seized in a piracy raid. Hilbert ankle-cuffed Popov to a computer table in front of a Cyrillic keyboard. Popov was ecstatic. Compared with jail, the drab workspace was the Oval Office. He could accomplish anything here.
They called the operation Ant City. Now that he was back online, Popov adopted a new identity and began hanging out in underground chat rooms and posting on CarderPlanet, portraying himself as a big-time Ukrainian scammer with an insatiable hunger for stolen credit cards. His first big target was at the top of CarderPlanet’s rigorous hierarchy: a mysterious Ukrainian then known only as “Script.” Popov made contact in early September, and the two began talking privately over ICQ, the instant messenger favored in Eastern Europe. Two weeks later Popov negotiated a deal to buy $400 worth of stolen credit card numbers. By sending the contraband to Popov in California, Script committed a federal crime in a US jurisdiction. Hilbert’s evidence would eventually help persuade Ukrainian police to arrest Script, though the hacker would be released after six months in jail.
That kind of “controlled buy” of credit card data was key to Hilbert’s strategy: Spreading a little money around was an easy way for Popov to make contacts, and with the cards in hand, Hilbert could work with the credit card companies to identify the source of the breach. Popov moved down the ladder to rank-and-file vendors and hackers, striking deals and collecting intelligence.
Some days were short, others stretched out to 10 hours. But regardless of Popov’s successes, each ended the same way, with Hilbert returning to his home and family, and Popov going back to his crummy jail cell. But on Thanksgiving, Hilbert prepared a surprise for his prized asset. When Popov arrived for work, he found a projector set up and pointed at the wall. Hilbert hit a few keys on a laptop and the screen filled with the opening credits of The Fellowship of the Ring, fresh out on DVD. For lunch, Hilbert brought out a complete Thanksgiving meal: turkey, stuffing, cranberry sauce, sweet potatoes, even pumpkin pie. Popov was moved. Hilbert had chosen to spend part of the holiday with him, instead of with his own family.
As word of Ant City filtered through the bureau, Hilbert began fielding requests from other FBI offices to look into specific hacks. February 2003 saw the biggest yet: an intrusion into the credit card payment processor Data Processing International that had exposed 8 million cards. Popov began asking around about DPI, and one of his contacts, a 21-year-old Russian student called “RES,” volunteered that he knew the three hackers responsible and could broker a deal.
One thing Popov had always known about Eastern European hackers: All they really wanted was a job.
Popov boldly declared that he intended to buy all 8 million cards for $200,000, but he wanted a small sample first. The sample would let Hilbert confirm that the cards really came from the DPI breach. But RES scoffed at the offer. Popov’s relatively small purchases up to that point had offered no evidence that he had $200,000 in his bank account.
Hilbert came up with a solution. Popov dressed in street clothes and, with an entourage of FBI agents for security, was shuttled to a nearby bank that had agreed to cooperate. In a back room, bank workers brought out $200,000 in hundred-dollar bills from the vault and arranged it on a table. Hilbert uncuffed Popov and shot a video of the hacker from the neck down as he riffled through the wads of cash.
“So look, I am showing the dough,” Popov said in Russian. “The dough is fucking real, no fucking blathering. I’ll be transferring it to my account.” He snatched a bill from a stack and held it close to the camera. “All the fucking watermarks, all the shit is here. I am showing it to you at point-blank range.” He tossed the bill disdainfully to the table. “So call your mob, and let us settle this fucking business.”
The video satisfied the Russian. Identifying RES was even easier. Popov mentioned to the hacker that some of his money came from a day job he held with a company called HermesPlast that was in the credit card printing business. Suggesting that the Russian apply for work there himself, he pointed RES to the company’s website and shared the email address of his purported boss, “Anatoly Feldman.”
RES sent Feldman an application the same day, with a copy of his résumé and a scan of his Russian national ID card.
HermesPlast, of course, was a fake company set up by Hilbert and Popov. Now the FBI had RES’ real name, date of birth, and address. It was a surprisingly simple ploy that would work again and again. One thing Popov had always known about Eastern European hackers: All they really wanted was a job.
Twenty-eight months after he had boarded a flight to the US, Popov was set free in the middle of Orange County, California, 8 miles from Disneyland and a world away from Zhytomyr. But his immigration status was complicated. He had no green card or Social Security number and no way to get a legitimate job or a driver’s license. Hilbert arranged for the FBI to rent Popov an apartment near the beach and pay him a $1,000-a-month stipend to continue working on Ant City. But Popov couldn’t adjust to life in a suburban swelter of freeways and strip malls. In July he was waiting at a bus stop near his probation office when a man walked up to him, drunk and angry and talking shit. Popov hit the guy hard enough to knock him to the pavement. He called the FBI in a panic, already imagining his return to prison. If he got out of this, he decided, he was going home.
Popov got permission from Judge Carter to visit Ukraine, provided he return to California by August 18 to serve out the remainder of his three years of supervised release. Hilbert drove him to the airport and said good-bye, knowing full well he wouldn’t see Popov again.
Ant City closed down for good. By Hilbert’s count, the operation had taken some 400,000 stolen credit cards off the black market and alerted over 700 companies that they’d been breached by Eastern European hackers. Ten suspects would eventually be charged, including Script, but none extradited.
On New Year’s Eve 2004, Hilbert’s cell phone rang. “Hey, you know what?” Popov said in his smooth, tumbling accent. “I got something new here.” There had been a big breach, he explained. And, remarkably, the FBI itself was a victim.
Popov had been monitoring a Russian hacker gang that specialized in a pre-Internet networking technology called X.25, which had powered the first public packet-switched networks in the ’70s and ’80s. By 2004, X.25 was the Betamax to the Internet’s VHS, but the legacy networks were still running and thousands of corporations and government agencies around the world were still connected.
The Russians were spelunking in these ancient networks and burrowing into US companies left and right. But one target was particularly alarming. Hackers had breached an AT&T data center in New Jersey where the telecom ran, under contract, the email servers for a number of US government agencies. One of these was the FBI’s, giving the Russians access to the email of every agent with an FBI.gov address.
Hilbert hung up and called his boss. Soon he was on a plane to Washington, DC, to lead the investigation. Hilbert arranged for the FBI to pay Cycmos $10,000 to retrieve any stolen material and identify the hackers involved. Popov came through, handing over two documents he said were plucked from an FBI inbox: a confidential 11-page dossier the government had compiled on a CarderPlanet kingpin called King Arthur and a spreadsheet of FBI and Secret Service cybercrime targets, broken down by jurisdiction.
The target list was dated six months earlier and marked “Law Enforcement Sensitive” and “Do not transmit over the Internet.” It was a potential gold mine to the underground, containing the handles—and in some cases the real names—of over 100 hackers in the government’s crosshairs, with a smattering of notes like “top-level target” or “currently cooperating with the government.” The White House was notified, raising the stakes even higher. Hilbert asked Popov for more.
Then Popov got a scoop. He directed Hilbert to an underground chat room where he could find the Russian leader of the X.25 gang. Hilbert was soon conversing with Leonid “Eadle” Sokolov, an engineering student in Saint Petersburg, Russia. Under Hilbert’s questioning, Sokolov admitted to the AT&T intrusion and the document theft. Hilbert had him. It would be the biggest case of his career.
But there was a hiccup. On February 10, 2005, Hilbert was summoned into a conference room in the J. Edgar Hoover Building, with five supervisors sitting around the table and an angry federal prosecutor on speakerphone.
It turned out other corporations had also been hit in the X.25 hacking spree, and Popov had been reaching out to them to offer his assistance. One victim was the Boston-based multinational EMC, where intruders had stolen the source code for the company’s ubiquitous virtualization software, VMware. If the code got out, hackers everywhere could plumb it for security holes. VMware’s purpose is to allow a single server to house multiple virtual computers, each walled off from the others. So in the worst-case scenario, a hacker might find a way to “escape” from a virtual machine and seize control of the underlying system.
Using his standard business alias “Denis Pinhaus,” Popov had approached EMC and warned it about the hack. For the right price, he promised, he could prevent the stolen source code from leaking and provide EMC a detailed technical analysis of the breach. As he’d done before, Popov gave EMC the name and contact information of an FBI agent who could vouch for his credibility: E. J. Hilbert.
EMC apparently viewed the pitch as an extortion attempt and reported it to the US attorney’s office in Boston. It fell on the desk of Stephen Heymann, a tough cybercrime prosecutor who would later gain notoriety for his pursuit of Internet activist Aaron Swartz.
Agent Hilbert learned what everyone else seemed to already know: He was under investigation by the Justice Department.
Now Heymann was on speakerphone demanding answers—who was this Pinhaus? Hilbert explained that Pinhaus was an FBI asset who was helping with an urgent investigation. “I need this guy out there right now,” he said. Heymann wasn’t moved. He wanted to charge the Ukrainian with extortion. He demanded that Hilbert give up his source’s real name.
Hilbert refused. Heymann was free to build a case against Pinhaus under his alias and go through channels to get his real identity from the FBI. But he wasn’t going to get it from Hilbert.
It was the wrong thing to say to a prosecutor from Boston, where the stink of the FBI’s most infamous informant scandal still hung in the air of the federal building. Heymann’s office had sent a former FBI agent to prison for protecting a murderous South Boston mob boss for decades to maintain him as an informant. “This is another Whitey Bulger situation,” the prosecutor growled.
A supervisor ordered Hilbert out of the room. Hilbert went to his computer and messaged Popov to steer clear of EMC. “Knock it off on that side, all right?” Hilbert recalls writing. “It’s important. Everybody’s looking into this situation. You have to knock it off.”
Hilbert turned back to the AT&T case. Sokolov was charged in a sealed indictment in New Jersey, and a confidential Interpol Red Notice was issued for his arrest, should he ever leave Russia for a country that extradites to the US. Popov was paid and given a commendation letter on FBI stationery to display on Cycmos’ website: “We acknowledge and express our appreciation for the assistance you have provided.”
The entire matter sank into the murk of the FBI’s hidden past. The only public notice of the FBI.gov email breach was a 2005 Newsweek story, and the bureau downplayed the incident, claiming that no sensitive information was stolen.
The dispute with the Boston prosecutor receded in Hilbert’s mind. But four months later the FBI abruptly ordered Hilbert to cut off all contact with Popov and to hand over the 600 pages of logs he’d kept from 18 months of their online chats. Soon after, he transitioned off cybercrime and moved to a counterterrorism detail.
Hilbert threw himself into the new assignment, but over time he noticed something was wrong. He was snubbed for incentive awards, and agents he’d known for years stopped talking to him. In August 2006 he applied for a supervisor slot in the Los Angeles field office. When the job posting reached headquarters, Hilbert was dropped from the candidate list and told not to reapply. “What the hell’s going on?” Hilbert asked his supervisor. That’s when he learned what everyone else seemed to already know: He was under investigation. For a year, the Justice Department’s Office of the Inspector General had been investigating Hilbert on suspicion of conspiracy, fraud against the government, and leaking confidential law enforcement information—the warning to Popov about the EMC probe.
Hilbert was devastated. The FBI was his dream job, but a criminal investigation would put a dead stop to his rise in the bureau, and he had two children at home and a third on the way. He began quietly casting around for job opportunities in the private sector, and in February 2007 he walked into his boss’s office, plunked his gun and badge on the desk, and quit. Thanks to his breakthrough case, his eight-year career with the FBI was now over.
“He called me up to thank me for the way I treated him and for his time in jail and the way it was handled,” Hilbert told me over lunch at a family restaurant in Orange County early in 2013. “Now he’s gone home and changed his life, and he’s got a family now, and he owes me everything—his words.”
The call from Popov only served to stir up Hilbert’s sense of mistreatment by the government. Even after he’d left the bureau, the inspector general’s office continued to investigate him; at one point it had even sent agents to Hilbert’s workplace to try to question him. Finally, in 2009, Hilbert was cleared when the Justice Department formally declined to indict.
When Popov approached EMC for the balance of his $70,000 “consulting” fee, the company refused. It looked like EMC execs wanted to pretend the whole thing never happened.
In my first conversations with Popov, he told me the same story of redemption that he shared with Hilbert. But eventually, a different narrative emerged. Popov had been nursing grievances of his own in the EMC affair. At the time of his call to Hilbert, he had just resolved them.
In addition to contacting Heymann, EMC had quietly made a deal with Popov in 2005, he said, paying him $30,000 by wire transfer and promising a second payment, of $40,000, in four years if the stolen VMware source code didn’t leak. He kept his part of the bargain. The code never leaked, and the fact that the sensitive blueprints for VMware were in the hands of overseas hackers remained a secret from customers and shareholders alike.
But years after that hack, when he approached EMC for the balance of his $70,000 “consulting” fee, the company refused, he says. (EMC declined to comment). By then EMC had spun out VMware as its own company. To Popov, it looked like EMC executives wanted to pretend that the whole thing had never happened.
The sheer disrespect galled him and he wanted revenge. Popov crafted a new identity— “Hardcore Charlie,” a self-described Russian hacktivist aligned with Anonymous. And on April 23, 2012, nearly eight years after it was taken, the stolen VMware code’s first 520 lines appeared on the web.
Despite the age of the code, the leak alarmed the technology world and galvanized the staff at VMware’s offices in Palo Alto, California. The 2004 breach had long faded from VMware’s institutional memory, and some of the stolen kernel code was still in the company’s current product. Security chief Iain Mulholland, a onetime officer in the British Army, mounted a staggering damage-control operation, recruiting every security auditor he could lay his hands on to search for weaknesses in the kernel code. The company pushed out the first of multiple security updates 10 days later. By the time Popov released a larger tranche of source code in November 2012, the critical security holes were all patched.
“He was the only friend I had,” Popov says about Hilbert. “I’m still a blackhat, and I never changed. But who cares? I still love him.”
This hardly sounded like the efforts of a conventional security consultant. Pressed, Popov finally confirmed what by then had become obvious: The EMC intrusion and the FBI email hack hadn’t really been the work of a random Russian hacker.
“Technically, we were the ones who did it,” Popov tells me in a late-night phone call.
Sokolov, the Saint Petersburg student charged in the FBI breach, had been working with Popov from the start to squeeze money from the X.25 hacks. “He is the best of the best,” Popov says. When they cracked the AT&T data center, Popov figured the telco would easily fork out $150,000 to learn the details and protect its government contracts. It was only when AT&T refused that Popov phoned Hilbert to report the breach, hoping the FBI would pay for the information.
Once he had a deal with Hilbert, Popov persuaded Sokolov to talk to the agent in a chat room so Hilbert could “solve” the crime. Popov says Hilbert wasn’t in on the scam. “I think he suspected something, really,” Popov says. “But it wasn’t that obvious at the time.”
I can’t confirm whether or not Hilbert suspected something, because by the time of Popov’s confession Hilbert had stopped talking to me, concerned that a story about Ant City would harm him in his new post as a director of cybersecurity and privacy at the Big Four accounting firm PricewaterhouseCoopers.
For his part, Popov, now 35, comes across as alternately weary and defiant. He has no regrets about hacking the FBI. But his swagger fades a bit when I ask him about the role his double-dealing played in ruining Hilbert’s FBI career.
Popov still remembers Thanksgiving 2002, the turkey meal, and The Lord of the Rings. “He was the only friend I had,” Popov says about Hilbert. “I still love him, even if he’s getting kind of distant from me now because of my new stuff. I’m still a blackhat, and I never changed. But who cares? I still love him.”
Hilbert’s work with Popov was the first attempt to really crack this world, though in many ways it was just a new twist on a timeworn law enforcement strategy. When a federal law enforcement agency confronts a vast criminal machine, it invariably tries to sabotage the clockwork from the inside. And to do that, the agency must become a working component in the very criminal apparatus it hopes to destroy. The tactic always strikes a fraught balance, and Ant City would not be the last time it backfired. In another case soon after, a Secret Service informant named Albert Gonzalez covertly joined with Russian hackers in a crime spree—which compromised 160 million credit cards and inflicted losses in the hundreds of millions of dollars—before he was caught and sentenced to 20 years in prison in 2010. The prosecutor, assistant US attorney Heymann, had asked for 25.
Some operations end in arrests and award ceremonies, others in embarrassed silence. The only constant is the Eastern European underground, which grinds on, like any machine, tireless and indifferent and, for the most part, simply looking for work that pays.