Update: New 25 GPU Monster Devours Passwords In Seconds

Editor’s note: I’ve updated the article with some new (and in some cases) clarifying detail from Jeremi. I’ve left changes in where they were made. The biggest changes: 1) an updated link to slides 2) clarifying that VCL refers to Virtual OpenCL and 3)  that the quote regarding 14char passwords falling in 6 minutes was for LM encrypted – not NTLM encrypted passwords. Long (8 char) NTLM passwords would take much longer…around 5.5 hours. 😉  – Paul

There needs to be some kind of Moore’s law analog to capture the tremendous advances in the speed of password cracking operations. Just within the last five years, there’s been an explosion in innovation in this ancient art, as researchers have realized that they can harness specialized silicon and cloud based computing pools to quickly and efficiently break passwords.

Password Cracking HPC
Gosney’s set-up uses a pool of 25 virtual AMD GPUs to brute force even very strong passwords.

A presentation at the Passwords^12 Conference in Oslo, Norway (slides available here – PDF), has moved the goalposts, again. Speaking on Monday, researcher Jeremi Gosney (a.k.a epixoip) demonstrated a rig that leveraged the Open Computing Language (OpenCL) framework and a technology known as Virtual OpenCL Open Cluster (VCL) to run the HashCat password cracking  program across a cluster of five, 4U servers equipped with 25 AMD Radeon GPUs and communicating at  10 Gbps and 20 Gbps over  Infiniband switched fabric.

Gosney’s system elevates password cracking to the next level, and effectively renders even the strongest passwords protected with weaker encryption algorithms, like Microsoft’s LM and NTLM, obsolete.

In a test, the researcher’s system was able to churn through 348 billion NTLM password hashes per second. That renders even the most secure password vulnerable to compute-intensive brute force and wordlist (or dictionary) attacks. A 14 character Windows XP password hashed using LM NTLM (NT Lan Manager), for example, would fall in just six minutes, said Per Thorsheim, organizer of the Passwords^12 Conference.

[Note of clarification from Jeremi: “LM Is what is used on Win XP, and  LM converts all lowercase chars to uppercase, is at most 14 chars long, and splits the password into two 7 char strings before hashing — so we only have to crack 69^7 combinations at most for LM. At 20 G/s we can get through that in about 6 minutes. With 348 billion NTLM per second, this means we could rip through any 8 character password (95^8 combinations) in 5.5 hours.” ]

“Passwords on Windows XP? Not good enough anymore,” Thorsheim said.

Tools like Gosney’s GPU cluster aren’t suited for an “online” attack scenario against a live system. Rather, they’re used in “offline” attacks against collections of leaked or stolen passwords that were stored in encrypted form, Thorsheim said. In that situation, attackers aren’t limited to a set number of password attempts – hardware and software limitations are all that matter.

The clustered GPUs clocked impressive speeds against more sturdy hashing algorithms as well, including MD5 (180 billion attempts per second, 63 billion/second for SHA1 and 20 billion/second for passwords hashed using the LM algorithm. So called “slow hash” algorithms fared better. The bcrypt (05) and sha512crypt permitted 71,000 and 364,000 per second, respectively.

Benchmarks - Fast Hash Cracking
Published benchmarks against common hashing algorithms using the 25 GPU HPC cluster

In an IRC chat with Security Ledger, Gosney said he has been working on CPU clustering for about five years and GPU clustering for the last four years.

“Then we just started trying to build the biggest GPU rigs we could, packing as many GPUs into a single server as possible so that we wouldn’t have to deal with clustering or distributing load,” Gosney wrote.

He started developing the new platform since stumbling on VCL in April, after trying his hand at pooling traditional CPUs for password cracking.

“I was extremely disappointed that setting up a clustered VMware instance wouldn’t allow me to create a VM that spanned all the hosts in the cluster. E.g. if i had five VMware ESX hosts with 8 processor cores, I wanted to be able to create a single vm with 40 cores and use all nodes in the cluster,” he wrote.

Then he came across VCL, or Virtual Open Cluster, a small and heretofore little recognized project from the scientists who manage the MOSIX distributed operating system first released in the 1970s.

“It did just what I wanted, not with an entire OS per se, but with an entire OpenCL application. and that’s good enough for me.”

After playing around with VCL for a while, Gosney approached Prof. Amnon Barak, one of Mosix’s creators. Gosney was interested in adding features to VCL that would allow it to run the HashCat password cracking tool.

“Once we convinced Amnon  that we did not aspire to turn the world into one giant botnet, he was very cooperative in working with (us) to resolve issues with VCL that was preventing it from working 100% with hashcat,” he said.

VCL makes load balancing across the cluster – once an arduous task that required months of custom scripting – a trivial matter. As a result, Gosney said that his team is at a point where their implementation of Hashcat on VCL could be scaled up far above the 25GPU rig he has created – supporting “at least 128 AMD GPUs.

“I always had these dreams of doing very simple and very manageable grid/cloud computing,” Gosney wrote. “It really is the marriage of two absolutely fantastic programs, which allows us to do unprecedented things,” he wrote.

Gosney is no stranger to password cracking. After 6.4 million Linkedin password hashes were leaked online, Gosney was one of the first researchers to decrypt them and analyze the findings. He and a partner were ultimately able to crack between 90% and 95% of the password values.

Gosney’s GPU cluster is just the latest leap forward in password cracking in a year that has already seen prominent encryption algorithms deemed compromised by an onslaught of cheap compute power. In June, Poul-Henning Kamp, creator of the md5crypt() function used by FreeBSD and Linux-based operating systems was forced to acknowledge that the hashing function is no longer suitable for production use – a victim of GPU powered systems that could perform “close to 1 million checks per second on COTS (commercial off the shelf) GPU hardware,” he wrote.  Gosney’s cluster cranked out more than 70 times that number –  77 million brute force attempts per second against MD5crypt.

Recent years have also seen the launch of services like Moxie Marlinspike’s WPACracker and then CloudCracker, a cloud-based platform for penetration testers that can do lookups of password hashes and other encrypted content against a dictionary of over hundreds of millions – or even billions – of potential matches — all for under $200.  And if that price is too rich, a team of U.S. based researchers have shown how you can do the same thing – on the cheap – by leveraging Google’s MapReduce and cloud based browsers. Then, in 2011, researcher Thomas Roth, who developed the Cloud Cracking Suite (CCS) – a tool that leveraged eight Amazon EC2-based Nvidia GPU instances to crack the SHA1 encryption algorithm and dispense with tens of thousands of passwords per second.

Gosney said he plans to “make a bit of money” off his invention, either by renting out time on it or by offering it as a paid password recovery and domain auditing service. “I have way too much invested in this to not get some kind of return out of it,” he wrote.

133 Comments

  1. I think you are missing two “not”s in this paragraph:

    “Tools like Gosney’s GPU cluster are suited for an “online” attack scenario against a live system. Rather, they’re used in “offline” attacks against collections of leaked or stolen passwords that were stored in encrypted form, Thorsheim said. In that situation, attackers are limited to a set number of password attempts – hardware and software limitations are all that matter.”

    • yes. thanks. made that change.

      • dude, this has to be the best article with the absolute worst editing i have ever seen in my life. i was thinking about it pretty hard and i swaer I can’t think of a single article that i’ve read with not only so many errors but also an obvious complete lack of understanding of the subject matter.

        just thought id mention it. 🙂

  2. And you never closed the parenthesis in this sentence:

    “The clustered GPUs clocked impressive speeds against more sturdy hashing algorithms as well, including MD5 (180 billion attempts per second, 63 billion/second for SHA1 and 20 billion/second for passwords hashed using the LM algorithm.”

  3. “A 14 character Windows XP password hashed using NTLM (NT Lan Manager), for example, would fall in just six minutes, said Per Thorsheim.”

    This is incorrect. This speed is possible only when the password is hashed with LM (not NTLM), and when using the 62 alphanumeric characters only (no special characters). Given these circumstances, the two 7-character LM halves can be bruteforced in 2*(62^7)/20e9/60 = 5.9 minutes.

    • Yes. LM not NTLM. I corrected that and added a clarifying comment from Jeremi. Thanks for pointing this out.

    • Mrb, both 7-character halves are cracked in one pass, you don’t have to multiply by 2. The calculation that is stated in the article of ~6 minutes is based on using a space of 69 characters. If you use a space of 62 chars, the cracking will be around 3 minutes.

  4. Heh, yeah, there are some inaccuracies in this article. I’ve already made Paul aware, hopefully he can get them corrected. The slides from the presentation tell the real story though, and the video of the talk will be up shortly from what Thorsheim tells me.

    mrb, you’re correct that the 6-minute time is for LM, not NTLM; however, LM is 69^7, not 62^7, and that is including alpha, digit, and special chars — essentially (95 – 26)^7

  5. actually i just realized the link to the slides in this article is incorrect as well; the correct link is http://heim.ifi.uio.no/hennikl/passwords12/Jeremi_Gosney_Password_Cracking_HPC.pdf

    • Object not found!

      The requested URL was not found on this server. The link on the referring page seems to be wrong or outdated. Please inform the author of that page about the error.

      If you think this is a server error, please contact the webmaster.

      Error 404

      heim.ifi.uio.no
      Thu 06 Dec 2012 03:22:32 PM CET
      Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8j DAV/2 mod_fastcgi/2.4.6 mod_wsgi/3.3 Python/2.5.2 PHP/5.4.7 mod_perl/2.0.4 Perl/v5.8.8

  6. If you are coming here from Slashdot or Hacker News — please just read the slides, and all will be made much more clearly. It’s rare that reporters ever get things exactly right. The link to the slides above is incorrect; the actual slides can be found at:

    heim.ifi.uio.no / hennikl / passwords12 / Jeremi_Gosney_Password_Cracking_HPC.pdf

    (modified URL to bypass moderation filter)

    • Will update the link.

      • Link is still broken, missing the “h” in http.

      • Cuenta sabiduria en calculos hay por aqui..Bueno al final solo es cuestion de minutos arriba o abajo.¿no?
        Y alguien me podria decir si esto esto no es más bien un problema de etica no de lo que se puede o no se puede hacer.
        Me explico, robar a una ancinita es facil (pero no lo hacemos por que es una cobardia),pues para mi todo funciona igual.
        Quiero decir que depende de el fin para el que se hagan las cosas algo es licito o no.Quizas un poco maquiavelico pero, me explico.Robar el bolso a una viejecita porque sabes que lleva una bomba que va a matar a un monton de gente puede ser algo bueno.Al igual que creo que el exigir transparencia e información es algo licito y que si el hackeo nos ayuda a desvelar atrocidades hechas por alguien pues les quita el factor impunidad.Pero el descifrar contraseñas de particulares porque si.. es como, lo de la viejeciita pero sin bomba.. Espero que se me entienda

  7. Why not store the results of the hashing in a large service, and offer quick lookups for cracking passwords? + also compute and store new hasheas paralelly? 🙂

    • Generating hashes is one thing. Searching them is another. Large dataset searches, no matter how you do them, indexed or not, are computationally intensive tasks. The difficulty arises in matching. Every iteration of a check has to processed.
      If I’ve got a GUID and I want to check against a list of ten billion (a very tiny number in that key space as it’s a possibility of 2^122 unique values). That’s an effort expressed in polynomial time. (look that up on Wikipedia)

      • @UnfeignedShip If your 10 billion potential matches are in a sorted list, the search problem is expressed in logarithmic (and not polynomial) time. Just use a simple algorithm called binary search. Of course the sorting is done in O(N*Log(N)) but you only have to do that once.

  8. Pingback: New 25 GPU Monster Devours Passwords In Seconds

  9. Instead of VMware, he should have tried ScaleMP. VMware partitions resources while ScaleMP aggregates.

    • So, this portion of the article is a bit out of context, and was just given as historical background information on how we evolved to this point.

      What I was referring to in that quote were my aspirations for virtualization solutions that aggregate resources and permit generic, unmodified applications to utilize all resources in that pool transparently.

      The VMware quote was referring to my first experience with VMware ESX back in 2007, when I was working for Intel at the time. I was disappointed to discover that when setting up a clustered instance, the hypervisors themselves could not be used to form an HPC cluster, only an HA cluster. Mind you this was during a time period when most people had not even heard of ESX, let alone were fully aware of what it was and was not capable of. I had the same hopes for Eucalyptus a year later, which were similarly dashed.

      Thus, we had to settle for MPI at the time. But MPI is not a generic solution, your applications have to be MPI-aware. We had much bigger aspirations, but bit the bullet until technology caught up to our dreams.

      And so with that said, ScaleMP did not exist at the time any of this was occurring. Well, they existed as a company, but had yet to release any products. By the time ScaleMP had begun releasing products, we had already moved on to GPUs.

      Then it became a matter of building denser GPU systems, or writing custom scripts to distribute the load across smaller systems (simple enough for brute force, not so simple when dealing with wordlists, rules, hybrid attacks, etc.)

      And finally we arrive at Virtual OpenCL, which does everything I could have ever hoped it could do.

      • Have looked every where for a webpage ore mail where it is possible to contact Jeremi Gosney for time and price on password recovery of files, the article says : “make a bit of money” off his invention, either by renting out time on it or by offering it as a paid password recovery and domain auditing service” anyone know if this is possible

        • my contact details are available in the slidedeck, and i can be reached via email as well, epixoip at bindshell.nl

          • Sorry epixoip have tried to mail you trough epixoip at bindshell.nl but get permanent error and cant find anything at slidedeck 🙁

  10. I recall an lecture on this issue claiming that simply requiring 3 separate simultaneous passwords per username would push the computing power for brute force attacks beyond anything remotely possible.

    • Brute force is just one method of offline password recovery that we employ. In fact, for most algorithms, brute force is a last resort. We have dozens of tricks up our sleeves that allow us to yield a very high percentage of user-selected passwords in a very short amount of time.

  11. You guys ever hear of Bitcoin mining?

    • yes, of course. the majority of our used gpus (not just those featured in this cluster) have been purchased from miners who have stopped mining. and we actually do mine for bitcoin in our spare gpu cycles.

      • That’s cool. My first thought when I saw this was ‘what a waste of a bitcoin miner…’

        • it’s unlikely that gpu mining will continue to be profitable, with both the reward being halved and asics hitting the market. most gpu miners started selling off their hardware a few months ago.

    • Lol, my thoughts exactly. Could you just do an hour to this address:
      16NsKf9kde8AFHV9viU1MX2ERkq2eSedTK

      🙂

  12. I appreciate the amount of computing power this system has, but IMHO this article is sort of misleading people. I wrote an article titled “The Brute Force Misconception” earlier this year, and it explains (with some calculations) why brute forcing passwords shouldn’t be given the hype that it gets sometimes.

    http://security.nathanbowman.us/2012/04/the-brute-force-misconception.html

    • While the article itself may focus on brute force, we sure do not. For most algorithms, brute force is a last-ditch effort for us. Keep in mind there are far more methods of offline password recovery than just brute force.

      • You mean rainbow tables? Those take just as long to generate…

        • … No, I absolutely do not mean rainbow tables. There are more attacks than just brute force and rainbow tables, you know. Wordlist, wordlist + rules, hybrid wordlist + mask, combinator, combinator + rules, Markov chains…

          • I see, I sort of regard those methods as the “first thing I do that rarely works” and therefore don’t refer to them often. I don’t see much success in recovering modern complex passwords using those methods.

            • Then you’re doing it wrong 🙂 With large lists you should be able to recover the first 50-60% using nothing but wordlists + rules. Brute force helps you identify passwords that you don’t have in your wordlists and cannot find through other means, so that you can then add those words to your wordlists, create rules, etc. And even then we rarely do an exhaustive brute force, as Markov attacks are much better.

    • I see that you mention cryptohaze.com in your blog post there — what you may not know is that Bitweasil, creator of Cryptohaze, is part owner of this cluster.

  13. How well does it do with passwords hashed using blowfish? I’ll assume not very well..

    • With a key size of up to 448 bits, blowfish would probably take a while. I think the general consensus of the blowfish algorithm is that it has endured years of cryptanalysis, and is therefore secure.

      • I assume you mean bcrypt, not Blowfish. It is mentioned in the article: “The bcrypt (05) and sha512crypt permitted 71,000 and 364,000 per second, respectively.” bcrypt is memory hard and not GPU-friendly, and therefore cannot be accelerated much.

        Keep in mind that the key size and its cryptographic strength really have no bearing on whether or not it is a good password storage algorithm.

        • No, I meant 448 bits

          https://en.wikipedia.org/wiki/Blowfish_(cipher)#The_algorithm

          Also, you are right, the key size doesn’t need to go larger than that.

          “…given the slow initialization of the cipher with each change of key, it is granted a natural protection against brute-force attacks, which doesn’t really justify key sizes longer than 448 bits.”

          • What you are referring to is irrelevant for password storage. In fact, you are reading the wrong article entirely. You should be reading http://en.wikipedia.org/wiki/Bcrypt

            • I didn’t mean it in terms of password storage. I was referring to Cody’s question that asked about reversing blowfish. Blowfish’s resistance to known plaintext attacks makes it cost prohibitive (in relation to time due to it’s slow initialization of the cipher with each change of key ) to brute force. Bcrypt is good to read up on also, being that it creates an irreversible hash with a blowfish encrypted salted password.

              • The question was, “How well does it do with passwords hashed using blowfish?” and not about reversing blowfish encryption.

                • I took “how well does it do” to mean how well does your setup do with cracking passwords encrypted with the blowfish cipher. My mistake, I didn’t know there was a distinction.

  14. Couldn’t these attacks be even more massively parallelized by distributing the task better?

    What I’m thinking is if you had 19 of these 25 GPU clusters, and each one handles ONLY guesses starting with 5 assigned characters…ie: one cluster starts all guesses with only Q, q, W, w, E; the next does e, R, r, T, t, etc.

    You wouldn’t need any internetwork communication at all with this method I’d think…a machine simply brute forces through all of its assigned work, if it gets the password it moves on, and if it marches through every possibility without result, it moves on too.

    Then just set up a light network where machines can flag passwords that they’ve already correctly guessed, that way other machines aren’t wasting time running through their permutations and can move on to the next results.

    • You’re thinking only in terms of brute force, and we do much more than just brute force. The slidedeck talks about why we chose Infiniband, and why it is a good fit for this project.

      You’re also missing the bigger picture on VCL — it is a generic OpenCL virtualization platform, capable of running any unmodified* OpenCL application. And that is the real story here. The fact that I’ve applied it to password cracking is really just a footnote.

      • Also, with this method, we only see about 1.3% overhead while doing brute force attacks with fast hashes. So no, there’s not much to be gained by doing custom distributing. This is also a far more manageable and flexible solution, that doesn’t require any dev time.

  15. Pingback: 25 GPUs devour password hashes at up to 348 billion per second | News Trading or Technology

  16. Er, FreeBSD is definitely not a form of Linux, so a hashing algorithm “used by FreeBSD and other Linux-based operating systems” is jarring. This is like saying “Mustang and other GMC-based cars…..” Sure, they have wheels and drive on roads, but there’s some fundamental differences, too.

    Perhaps you mean “used by FreeBSD and Linux-based operating systems” instead.

  17. Pingback: 25 GPU System Unveiled at Passwords^12 Conference Eats Password Hashes for Breakfast | SiliconANGLE

  18. Pingback: Researcher Pushes The Limits Of Password Cracking Operations With 25 Concurrent GPUs | The Tech Journal

  19. I bet Mass Effect would look awesome on this machine…

  20. how long do you think it would take your rig to crack the wikileaks insurance file password?

    • Encryption is not the same as hashing, and the software we use (Hashcat) does not support encrypted documents, only password hashes. Some software, like John the Ripper, does support a few encrypted document formats. However, I think only a few have been ported to GPU, and I don’t think JtR has any multi-GPU support yet.

  21. Pingback: Update: New 25 GPU Monster Devours Passwords In Seconds | The Security Ledger | robot son link blog

  22. Pingback: Hashing fast and slow: GPUs and 1Password | Agile Blog

  23. Pingback: Researcher Pushes The Limits Of Password Cracking Operations With 25 Concurrent GPUs | OMG-Gurubootcamp!! Singapore

  24. Pingback: Linux Mint Czech - Monštrózne zostavenie z 25 GPU prelomí heslo za krátky čas

  25. Pingback: Geslo z 14 znaki razbito v samo 6 minutah

  26. Pingback: Все теоретически возможные пароли WinXP можно перебрать за 6 минут | Gramnacha IT

  27. Pingback: The Hardware Hackers Use to Crack Your Passwords | Exploit Archive

  28. Pingback: This is the machine hackers use to crack your security. | Management Insights

  29. Pingback: Ultimate Password Cracking Rig - jhack.co.uk

  30. I wonder if it can be used to crack the Indus Valley language of Pakistan, that has defied all attempts to crack the code?

  31. Pingback: [mark]25个GPU组成的设备轻松破解各种哈希密码 « NeverBest!我还能做的更好 – 007boy | im007boy

  32. Pingback: 25 GPUs brute force 348 billion hashes per second to crack your passwords

  33. Pingback: How Often Should I Change My Passwords?

  34. Pingback: How Often Should I Change My Passwords? | Tips for the Unready

  35. Pingback: WOW, and just last night, I was thinking I needed to go upgrade all my passwords… | auTONYmous!

  36. Pingback: Ask LH: How Often Should I Change My Passwords? | Lifehacker Australia

  37. Pingback: The Hardware Hackers Use to Crack Your Passwords | Gadget News, Reviews and Guides

  38. Pingback: The Hardware Hackers Use To Crack Your Passwords | Gizmodo Australia

  39. Pingback: É com esse hardware que hackers descobrem senhas em minutos | Gato Folgado | Notícias e entretenimento

  40. Pingback: Indagadores |Seguridad informatica |Seguridad en internet » 25 GPUs de fuerza bruta los hashes de 348 mil millones por segundo para reprimir sus contraseñas

  41. Pingback: Cluster con 25 GPUs devora contraseñas por millones

  42. Pingback: MySQL or Yours? | TechSNAP | Jupiter Broadcasting

  43. Pingback: Episode 813 – BIND 9.9.2, BlackHole & Chrome, 25 GPU Monster, SMB Medical Offices | InfoSec Daily

  44. That being said…my question is, nowadays, if you are design a new password storage system what should you use ?

    1) Adaptive Hashing algorithms such as Bcrypt/Script ?

    or….

    2) Encryption (yeah…encryption AES, for example), storing the keys in secure HSM ?

    Thanks…

    • the former, for sure. you definitely do not want to use reversible encryption. what you want to use is scrypt, bcrypt, pbkdf2, or any of the modern crypt(3) algorithms.

  45. Pingback: Must be tools people use to hack systems in less than 10 minutes. | Tyler Merrick

  46. I am looking to make a single rig. Would it be possible to get a hardware breakdown? Obviously the graphics cards are listed but the information of the other hardware would be useful.

    What motherboard, PSU, case/chassis, processor, RAM etc? 🙂

    • If you want to build one like the one in the picture, pick up a TYAN FT77B7015 barebones. If you want something a bit smaller, grab a Chenbro RM41300-FS81 chassis and Gigabyte GA-990FXA-UD7 motherboard. CPU does not matter one bit, get the cheapest one you can find. Memory is cheap, get at least 8GB. Do not go cheap on the PSU, go 1200W – 1600W 80plus Gold or 80plus Platinum. You many need more than one depending on how many GPUs you run.

  47. Yes very intresting… but it sounds to that its really been built as an excuse so Jeremi Gosney can play Crysis and something like FSX at full graphics detail levels. What would the 3Dmark score be for his system?

  48. Pingback: GPU-stuffed monster cracks Windows passwords in minutes | Security Watch

  49. Pingback: ste williams » GPU-stuffed monster cracks Windows passwords in minutes