Update: New 25 GPU Monster Devours Passwords In Seconds

Editor’s note: I’ve updated the article with some new (and in some cases) clarifying detail from Jeremi. I’ve left changes in where they were made. The biggest changes: 1) an updated link to slides 2) clarifying that VCL refers to Virtual OpenCL and 3)  that the quote regarding 14char passwords falling in 6 minutes was for LM encrypted – not NTLM encrypted passwords. Long (8 char) NTLM passwords would take much longer…around 5.5 hours. 😉  – Paul

There needs to be some kind of Moore’s law analog to capture the tremendous advances in the speed of password cracking operations. Just within the last five years, there’s been an explosion in innovation in this ancient art, as researchers have realized that they can harness specialized silicon and cloud based computing pools to quickly and efficiently break passwords.

Password Cracking HPC
Gosney’s set-up uses a pool of 25 virtual AMD GPUs to brute force even very strong passwords.

A presentation at the Passwords^12 Conference in Oslo, Norway (slides available here – PDF), has moved the goalposts, again. Speaking on Monday, researcher Jeremi Gosney (a.k.a epixoip) demonstrated a rig that leveraged the Open Computing Language (OpenCL) framework and a technology known as Virtual OpenCL Open Cluster (VCL) to run the HashCat password cracking  program across a cluster of five, 4U servers equipped with 25 AMD Radeon GPUs and communicating at  10 Gbps and 20 Gbps over  Infiniband switched fabric.

Gosney’s system elevates password cracking to the next level, and effectively renders even the strongest passwords protected with weaker encryption algorithms, like Microsoft’s LM and NTLM, obsolete.

In a test, the researcher’s system was able to churn through 348 billion NTLM password hashes per second. That renders even the most secure password vulnerable to compute-intensive brute force and wordlist (or dictionary) attacks. A 14 character Windows XP password hashed using LM NTLM (NT Lan Manager), for example, would fall in just six minutes, said Per Thorsheim, organizer of the Passwords^12 Conference.

[Note of clarification from Jeremi: “LM Is what is used on Win XP, and  LM converts all lowercase chars to uppercase, is at most 14 chars long, and splits the password into two 7 char strings before hashing — so we only have to crack 69^7 combinations at most for LM. At 20 G/s we can get through that in about 6 minutes. With 348 billion NTLM per second, this means we could rip through any 8 character password (95^8 combinations) in 5.5 hours.” ]

“Passwords on Windows XP? Not good enough anymore,” Thorsheim said.

Tools like Gosney’s GPU cluster aren’t suited for an “online” attack scenario against a live system. Rather, they’re used in “offline” attacks against collections of leaked or stolen passwords that were stored in encrypted form, Thorsheim said. In that situation, attackers aren’t limited to a set number of password attempts – hardware and software limitations are all that matter.

The clustered GPUs clocked impressive speeds against more sturdy hashing algorithms as well, including MD5 (180 billion attempts per second, 63 billion/second for SHA1 and 20 billion/second for passwords hashed using the LM algorithm. So called “slow hash” algorithms fared better. The bcrypt (05) and sha512crypt permitted 71,000 and 364,000 per second, respectively.

Benchmarks - Fast Hash Cracking
Published benchmarks against common hashing algorithms using the 25 GPU HPC cluster

In an IRC chat with Security Ledger, Gosney said he has been working on CPU clustering for about five years and GPU clustering for the last four years.

“Then we just started trying to build the biggest GPU rigs we could, packing as many GPUs into a single server as possible so that we wouldn’t have to deal with clustering or distributing load,” Gosney wrote.

He started developing the new platform since stumbling on VCL in April, after trying his hand at pooling traditional CPUs for password cracking.

“I was extremely disappointed that setting up a clustered VMware instance wouldn’t allow me to create a VM that spanned all the hosts in the cluster. E.g. if i had five VMware ESX hosts with 8 processor cores, I wanted to be able to create a single vm with 40 cores and use all nodes in the cluster,” he wrote.

Then he came across VCL, or Virtual Open Cluster, a small and heretofore little recognized project from the scientists who manage the MOSIX distributed operating system first released in the 1970s.

“It did just what I wanted, not with an entire OS per se, but with an entire OpenCL application. and that’s good enough for me.”

After playing around with VCL for a while, Gosney approached Prof. Amnon Barak, one of Mosix’s creators. Gosney was interested in adding features to VCL that would allow it to run the HashCat password cracking tool.

“Once we convinced Amnon  that we did not aspire to turn the world into one giant botnet, he was very cooperative in working with (us) to resolve issues with VCL that was preventing it from working 100% with hashcat,” he said.

VCL makes load balancing across the cluster – once an arduous task that required months of custom scripting – a trivial matter. As a result, Gosney said that his team is at a point where their implementation of Hashcat on VCL could be scaled up far above the 25GPU rig he has created – supporting “at least 128 AMD GPUs.

“I always had these dreams of doing very simple and very manageable grid/cloud computing,” Gosney wrote. “It really is the marriage of two absolutely fantastic programs, which allows us to do unprecedented things,” he wrote.

Gosney is no stranger to password cracking. After 6.4 million Linkedin password hashes were leaked online, Gosney was one of the first researchers to decrypt them and analyze the findings. He and a partner were ultimately able to crack between 90% and 95% of the password values.

Gosney’s GPU cluster is just the latest leap forward in password cracking in a year that has already seen prominent encryption algorithms deemed compromised by an onslaught of cheap compute power. In June, Poul-Henning Kamp, creator of the md5crypt() function used by FreeBSD and Linux-based operating systems was forced to acknowledge that the hashing function is no longer suitable for production use – a victim of GPU powered systems that could perform “close to 1 million checks per second on COTS (commercial off the shelf) GPU hardware,” he wrote.  Gosney’s cluster cranked out more than 70 times that number –  77 million brute force attempts per second against MD5crypt.

Recent years have also seen the launch of services like Moxie Marlinspike’s WPACracker and then CloudCracker, a cloud-based platform for penetration testers that can do lookups of password hashes and other encrypted content against a dictionary of over hundreds of millions – or even billions – of potential matches — all for under $200.  And if that price is too rich, a team of U.S. based researchers have shown how you can do the same thing – on the cheap – by leveraging Google’s MapReduce and cloud based browsers. Then, in 2011, researcher Thomas Roth, who developed the Cloud Cracking Suite (CCS) – a tool that leveraged eight Amazon EC2-based Nvidia GPU instances to crack the SHA1 encryption algorithm and dispense with tens of thousands of passwords per second.

Gosney said he plans to “make a bit of money” off his invention, either by renting out time on it or by offering it as a paid password recovery and domain auditing service. “I have way too much invested in this to not get some kind of return out of it,” he wrote.

Spread the word!


  1. Pingback: The Geek’s Reading List – Week of December 7th, 2012 | thegeeksreadinglist

  2. Pingback: LINK: Update: New 25 GPU Monster Devours Passwords in Seconds | Kansas City Network Professionals

  3. What’s the cost of this monster? It’s just the 25 GPUs + 1 server rack mount?

  4. This is really just old news under a new guise. Brute force on new machine cracks old password algorithm. Remember: physical access to the hardware = compromised system. Limiting actual password attempts to something like 3-10 eliminates brute force attack. Anyone who needs to get in ‘should’ know the correct passwords already.

    • The news is really about using VCL to do GPU clustering, to get around the 8-GPU limit imposed by the AMD drivers and the physical limitations of many motherboards and BIOS. This is taking the most advanced password cracking software to date, and applying it to grid computing.

      Also, we support all attack modes and all 45+ algorithms supported by both oclHashcat-plus and oclHashcat-lite, not just brute force, and not just LM & NTLM.

      You do not need physical access to get password hashes, I’m not sure how you got that impression. Especially in the cases of web applications where some vulnerability may provide limited read access, and cracking password hashes is the only way to escalate privileges.

      You’re also neglecting the legitimate side of all this as well, which is penetration testing and password auditing.

  5. Pingback: 25 GPUs AMD Radeon unidas podem descobrir suas senhas em poucos minutos « News Technology

  6. finally the nuclear plant in Dimona can be supported in nuclear analisys and nuke emulations using a cluster of GPU and not obsolete CPU, saving space and energy. Good job, Barak! 🙂

  7. There is some confusion on the part of the author and the person who did the presentation- there is no such thing as an “NTLM hash”. There are LM hashes, and there are NT hashes. There are no “NTLM” hashes.

    • You are correct, the actual name of the hash is “NThash”; however, the names of the algorithms used in the presentation are the names of the algorithms as used within Hashcat. Most password cracking tools — and most password crackers themselves — refer to the algorithm that produces the NThash as “NTLM.” This is pretty much standard throughout the community, and this is the accepted name that is used for the algorithm. However, it does create a bit of a namespace collision as the average person assumes we are talking about NetNTLM, and not NThash.

  8. Wow. Imagine what you could do with a set of Quadro, Teslla, or FirePro cards! (Not to mention a larger budget.)

    • For the purpose of password cracking, Quadro and Tesla cards are much slower at password cracking than their GTX equivalents. Further, Nvidia GPUs are much slower than AMD GPUs. It is a similar story on the AMD side, with almost all of the Radeons being significantly faster than the Firepro — with the sole exception of the new Firepro S1000.

  9. Pingback: GPU-stuffed monster cracks Windows passwords in minutes | Technophile

  10. It’s too bad you need to make money on it now. There are lots of flawed computing algorithms still in use. You could test all sorts of things. E.g., instead of testing hashes for weaknesses, you could test hashes to see which have the lowest collisions. Low-collision hashes are used for hash-tables. Many hash tables are becoming huge in size. It’s likely that some hashes which are great with binary are lousy with ASCII. So, you could test hashes to see which are best, low-collision, with ASCII, EBCDIC, or Unicode. You could also test hashes used in PKE and PGP etc. Since it’s so fast, you could even use it to design new hashes simply through brute-force testing and ranking of random or semi-random algorithms.

  11. Pingback:   Daily Blog Post ITGS (weekly) by ITGS Online

  12. Pingback: The NetophilesThe Netophiles – Episode 11 – Lessons From The Tragic Clown |

  13. Pingback: É com esse hardware que hackers descobrem senhas em minutos « Besteiras da Internet

  14. Pingback: Newest GPU Monster Breaks 14 Character Windows XP Password In Less Than 6 Minutes! | Security Beacon

  15. Pingback: 25个GPU的设备轻松破解各种哈希密码

  16. Pingback: 25个GPU的设备轻松破解各种哈希密码_系统安全 - Z的新闻

  17. Pingback: 25个GPU的设备轻松破解各种哈希密码 - Yours苹果网((IPHONE4 IPAD2 MAcBOOK ios IPOD ITOUCH 苹果公司 乔布斯)

  18. Pingback: 25个GPU的设备轻松破解各种哈希密码 | 帮你资源

  19. Pingback: How often should you change your passwords? « RMK Consulting

  20. Pingback: Every standard Windows passwords hacked in less than 6 hours | Cayo Buay

  21. My password is 24 chracters long, and fully random. I don’t memorise it. I use Aladdin to type it for me. http://igg.me/aladdin-key

    Aladdin is trying to improve the current situation of people using simple or identical passwords everywhere by removing the need to memorise passwords. Aladdin works with Windows, Mac, Linux as well as Android and iPad.

    Aladdin is a USB key(board). No software needed.

  22. Pingback: Cluster equipado com 25 GPUs quebra senhas do Windows XP em seis minutos | Limbotech

  23. Pingback: Password Strategies for an Internet Minefield « Security Advancement

  24. Pingback: Password-cracking feats at blistering speed shown in Oslo - I Hate Paypal

  25. Pingback: Nightline Tech Talk 2012-12-11 : nystedt.org

  26. Pingback: Nightline Tech Talk 2012-12-11 | dxb TECH RADIO SHOW

  27. Pingback: Welcome to Garland Group, the Continuous Compliance experts. » Modern Passwords Now Less Secure

  28. Pingback: Un monstruo de 25 GPU revienta contraseñas fuertes en segundos [ING]

  29. Pingback: Cracking passwords with 25 GPUs - Stormfront

  30. Pingback: skh:tec

  31. Pingback: #397 Video Conversion for iOS, Lantronix XPrint Server, MonoPrice Inexpensive Gear, MacBook Pro with Retina, Curves Explained « Nosillacast

  32. Pingback: 25 GPUs devour password hashes at up to 348 billion per second | Mon Web News

  33. clipfish Video runterladen online

    Thank you for sharing your thoughts. I really appreciate
    your efforts and I will be waiting for your further write ups thanks once again.