This has been a very very long time coming, however lastly, after a marathon effort, the model new Have I Been Pwned web site is now reside!
Feb final yr is once I made the primary commit to the general public repo for the rebranded service, and we soft-launched the brand new model in March of this yr. Over the course of this time, we have utterly rebuilt the web site, modified the performance of just about each net web page, added a heap of recent options, and at the moment, we’re even launching a merch retailer 😎
Let me discuss you thru simply a few of the highlights, strap your self in!
The Search
The signature characteristic of HIBP is that huge search field on the entrance web page, and now, it is even higher – it has confetti!
Effectively, not for everybody, solely about half the individuals who use it is going to see a celebratory response. There is a cause why this response is deliberately jovial, let me clarify:
As Charlotte and I’ve travelled and frolicked with so many various customers of the service world wide, a theme has emerged time and again: HIBP is a bit playful. It isn’t a scary place emblazoned with hoodies, padlock icons, and fearmongering about “the darkish net”. As a substitute, we intention to be extra consumable to the plenty and supply factual, actionable info with out the hyperbole. Confetti weapons (sure, there are a number of, they usually’re animated) lighten the temper a bit. The choice is that you just get the pink response:
There was a really transient second the place we thought of a extra light-hearted therapy on this web page as nicely, however one way or the other a little bit of unhappy trombone actually did not appear applicable, so we deferred to a extra demure response. However now it is on a timeline you may scroll by means of in reverse chronological order, with every breach summarising what occurred. And in order for you extra data, now we have an all-new web page I am going to discuss in a second.
Only one little factor first – we have dropped username and telephone quantity search assist from the web site. Username searches have been launched in 2014 for the Snapchat incident, and telephone quantity searches in 2021 for the Fb incident. And that was it. That is the one time we ever loaded these courses of knowledge, and there are a number of good the explanation why. Firstly, they’re each painful to parse out of a breach in comparison with electronic mail addresses, which we merely use a regex to extract (we have open sourced the code that does this). Usernames are a string. Telephone numbers are, nicely, it relies upon. They don’t seem to be simply numbers as a result of should you correctly internationalise them (like they have been within the Fb incident), they’ve additionally received a plus on the entrance, however they’re incessantly all over when it comes to format. And we won’t ship notifications as a result of no one “owns” a username, and telephone numbers are very costly to ship SMSs to in comparison with sending emails. Plus, each different incident in HIBP aside from these two has had electronic mail addresses, so if we’re asking “have I been pwned?” we are able to at all times reply that query with out loading these two hard-to-parse fields, which normally aren’t current in most breaches anyway. When the previous web site supplied to just accept them within the search field, it created confusion and assist overhead: “why wasn’t my quantity within the [whatever] breach?!”. That is why it is gone from the web site, however we have saved it supported on the API to make sure we do not break something… simply do not anticipate to see extra knowledge there.
The Breach Web page
There are lots of causes we created this new web page, not least of which is that the search outcomes on the entrance web page have been getting too busy, and we needed to palm off the small print elsewhere. So, now now we have a devoted web page for every breach, for instance:
That is largely info we had already (albeit displayed in a way more user-friendly vogue), however what’s distinctive in regards to the new web page is far more focused recommendation about what to do after the breach:
I just lately wrote about this part and the way we plan to determine different companions who’re capable of present applicable providers to individuals who discover themselves in a breach. Identification safety suppliers, for instance, make a variety of sense for a lot of knowledge breaches.
Now that we’re reside, we’ll additionally work on fleshing this web page out with extra breach and user-specific knowledge. For instance, if the service helps 2FA, then we’ll name that out particularly quite than depend on the generic recommendation above. Identical with passkeys, and we’ll add a bit for that. A current dialogue with the NCSC whereas we have been within the UK was round including localised knowledge breach steering, for instance, displaying people from the UK the NCSC emblem and a hyperlink to their useful resource on the subject (which recommends checking HIBP 🙂).
I am certain there’s far more we are able to do right here, so should you’ve received any nice concepts, drop me a remark under.
The Dashboard
Over the course of a few years, we launched an increasing number of options that required us to know who you have been (or no less than that you just had entry to the e-mail handle you have been utilizing). It started with introducing the idea of a delicate breach through the Ashley Madison saga of 2015, which meant the one approach to see your involvement in that incident was to obtain an electronic mail to the handle earlier than looking out. (Sidenote: There are lots of good the explanation why we do not do this on each breach.) In 2019, when I put an auth layer across the API to deal with abuse (which it did fantastically!) I required electronic mail verification first earlier than buying a key. And extra issues adopted: a devoted area search dashboard, managing your paid subscription and earlier this yr, viewing stealer logs to your electronic mail handle.
We have now unified all these completely different locations into one central dashboard:
From a look on the nav on the left, you may see a variety of acquainted options which are fairly self-explanatory. These mix related issues for the plenty and people which are extra business-oriented. They’re now all behind the one “Signal In” that verifies entry to the e-mail handle earlier than being proven. Sooner or later, we’ll additionally add passkey assist to keep away from needing to ship an electronic mail first.
The dashboard strategy is not nearly shifting present options underneath one banner; it is going to additionally give us a platform on which to construct new options sooner or later that require electronic mail handle verification first. For instance, we have usually been requested to offer individuals with the flexibility to subscribe their household’s electronic mail addresses to notifications, but have them go to a distinct handle. Many people play tech assist for others, and this could be a genuinely helpful characteristic that is smart to put at a degree the place you have already verified your electronic mail handle. So, keep tuned for that one, amongst many others.
The Area Search Characteristic
Extra time went into this one characteristic than many of the different ones mixed. There’s quite a bit we have tried to do right here, beginning with a a lot cleaner checklist of verified domains:
The search outcomes now give a a lot cleaner abstract and add filtering by each electronic mail handle and a hotly requested new characteristic – simply the newest breach (it is within the drop-down):
All these searches now simply return JSON from APIs and the entire dashboard acts as a single-page app, so every little thing is actually snappy. The filtering above is completed purely client-side in opposition to the complete JSON of the area search, an strategy we have examined with domains of over 1 / 4 million breached electronic mail addresses and nonetheless been workable (though arguably, you actually need that knowledge through the API quite than scrolling by means of it in a browser window).
Verification of area possession has additionally been utterly rewritten and has a a lot cleaner, easier interface:
We nonetheless have work to do to make the non-email verification strategies smoother, however that was the case earlier than, too, so no less than we’ve not regressed. That’ll occur shortly, promise!
The API
First issues first: there have been no adjustments to the API itself. This replace would not break something!
There is a dialogue over on the UX rebuild GitHub repo about the appropriate approach to do API documentation. The overall consensus is OpenAPI and we began happening that route utilizing Scalar. In reality, you may even see the work Stefan did on this right here at haveibeenpwned.com/scalar:
It’s totally cool, particularly the best way it paperwork samples in all kinds of various languages and even has a take a look at runner, which is successfully Postman within the browser. Cool, however we simply could not end it in time. As such, we have saved the previous documentation for now and simply styled it so it seems to be like the remainder of the positioning (which I reckon continues to be fairly slick), however we do intend to roll to the Scalar implementation after we’re not underneath the duress of such an enormous launch.
The Merch Retailer
You recognize what else is superior? Merch! No, severely, we have had so many requests over time for HIBP branded merch and now, right here we’re:
We truly now have a real-life merch retailer at merch.haveibeenpwned.com! This was in all probability the worst potential use of our time, contemplating how a lot mechanical stuff we needed to do to make all the brand new stuff work, nevertheless it was a little bit of a ardour mission for Charlotte, so yeah, now you may truly purchase HIBP merch. It is all finished by means of Teespring (the place have I heard that identify earlier than?!) and every little thing listed there may be at value worth – we make completely zero {dollars}, it is only a enjoyable initiative for the neighborhood 🙂
We did check out their choice for stickers too, however they fell nicely in need of what we already had up with our little one-item retailer on Sticker Mule so for now, that is still the go-to for laptop computer decorations. Or simply go and seize the open supply paintings and get your individual printed from wherever you please.
The Nerdy Bits
We nonetheless run the origin providers on Microsoft Azure utilizing a mix of the App Service for the web site, “serverless” Features for many APIs (there are nonetheless a couple of async ones there which are known as as part of browser-based options), SQL Azure “Hyperscale” and storage account options like queues, blobs and tables. Just about all of the coding there may be C# with .NET 9.0 and ASP.NET MVC on .NET Core for the online app. Cloudflare nonetheless performs a huge position with a variety of code in employees, knowledge in R2 storage and all their good bits round WAF and caching. We’re additionally now solely utilizing their Turnstile service for anti-automation and have ditched Google’s reCAPTCHA utterly – huge yay!
The entrance finish is now newest gen Bootstrap and we’re utilizing SASS for all our CSS and TypeScript for all our JavaScript. Our (different) man in Iceland Ingiber has simply finished a fully excellent job with the interfaces and exceeded all our expectations by a large margin. What now we have now goes far past what we anticipated after we began this course of, and an enormous a part of that has been Ingiber’s skill to take a easy requirement and switch it right into a factor of magnificence 😍 I am very glad that Charlotte, Stefan and I received to spend time with him in Reykjavik final month and share some beers.
We additionally made some measurable enhancements to web site efficiency. For instance, I ran a Pingdom web site velocity take a look at simply earlier than taking the previous one offline:
After which ran it over the brand new one:
So we lower out 28% of the web page measurement and 31% of the requests. The load time is far of a muchness (and it is extremely variable at that), however having strong measures for all of the values within the column on the appropriate is a really pleasing end result. Think about additionally the commentary anybody in net dev would have seen over time about how a lot greater net pages have develop into, and right here we’re shaving off strong double-digit percentages 11 years later!
Lastly, something that would remotely be construed as monitoring or advert bloat simply is not there, as a result of we merely do not do any of that 🙂 In reality, the one actual visitors stats now we have are based mostly on what Cloudflare sees when the visitors flows by means of their edge nodes. And that 1Password product placement is, because it’s at all times been, simply textual content and a picture. We do not even monitor outbound clicks, that is as much as them in the event that they wish to seize that on the touchdown web page we hyperlink to. This truly makes discussions comparable to we’re having with identification theft corporations that need product placement a lot more durable as they’re used to getting the types of numbers that invasive monitoring produces, however we would not have it every other method.
The AI
I needed to make a fast notice of this right here, as AI appears to be both continuously overblown or denigrated. Both it may clear up the world’s issues, or it simply produces “slop”. I used Chat GPT particularly actually extensively throughout this rebuild, particularly within the last days when time received tight and my mind received fried. Listed below are some examples the place it made an enormous distinction:
I am utilizing Bootstrap icons from right here: https://icons.getbootstrap.com/
What's a great icon for example a heading known as "Index"?This was proper on the eleventh hour after we realised we did not have time to implement Scalar correctly, and I wanted to shortly migrate all the prevailing API docs to the brand new template. There are over 2,000 icons on that web page, and this strategy meant it took about 30 seconds to seek out the appropriate one, every time.
We killed off some pages on the previous web site, however earlier than rolling it over, I needed to know precisely what was there:
Write me a PowerShell script to crawl haveibeenpwned.com and write out every distinctive URL it findsAfter which:
Now write a script to take all of the paths it discovered and see in the event that they exist on stage.haveibeenpwned.com
It discovered great things too, just like the safety.txt file I would forgotten emigrate. It additionally discovered stuff that by no means existed, so it is the standard “belief, however confirm” state of affairs.
And only a gazillion little issues the place each time I wanted something from some CSS recommendation to configuring Cloudflare guidelines to idiosyncrasies within the .NET Core net app, the proper reply was seconds away. I would say it was proper 90% of the time, too, and should you’re not utilizing AI aggressively in your software program improvement work now (and I am certain there are a lot better methods, too) I am fairly assured in saying “you are doing it unsuitable”.
The Journey Right here
It is onerous to elucidate how a lot has gone into this, and that goes nicely past simply what you see in entrance of you on the web site at the moment. It is seemingly little issues, like minor revisions to the phrases of use and privateness coverage, which required many hours of time and 1000’s of {dollars} with attorneys (simply minor updates to how we course of knowledge and a mirrored image of recent providers such because the stealer logs).
We pushed out the brand new web site within the wee hours of Sunday morning my time, and nearly every little thing went nicely:
One or two little glitches that we have mounted and pushed shortly, that is it. I’ve truly waited till now, 2 days after going reside, to publish this submit simply so we may iron out as a lot stuff as potential first. We have pushed greater than a dozen new releases already since that point, simply to maintain iterating and refining shortly. TBH, it has been a bit intense and has been an enormously time-consuming effort that is dominated our focus, particularly over the previous few weeks main as much as launch. And simply to drive that time dwelling, I actually received a well being alert very first thing Monday morning:
Nothing like empirical knowledge to make a degree! That final weekend after we went reside was particularly brutal; I do not suppose I’ve devoted that a lot high-intensity time to a software program launch for many years.
Have I Been Pwned has been a ardour for 1 / 4 of my life now. What I in-built 2013 was by no means supposed to take me this far or final this lengthy, and I am kinda shocked it did if I am trustworthy. I really feel that what we have constructed with this new web site and new model has elevated this little pet mission right into a severe service that has a brand new degree of professionalism. However I hope that in studying this, you see that it has maintained every little thing that has at all times been nice in regards to the service, and I am so glad to nonetheless be right here writing about it at the moment in the 205th weblog submit with that tag. Thanks for studying, now go and revel in the brand new web site 😊
Edit (a couple of hours after initially posting): Let me increase on Cloudflare’s Turnstile because it’ll clarify some idiosyncrasies some individuals have seen:
That is an anti-automation strategy that does not contain palming visitors to Google (like reCAPTCHA did), and it can be carried out utterly invisibly. There are extra invasive implementations of it, however we’re making an attempt to be seamless right here. It entails some Cloudflare script operating within the browser and offering a problem, which is then submitted with the HTTP request and verified server facet. We have had it on HIBP in a single type or one other since 2023, and it can be superior… till it is not. If the problem fails, what occurs subsequent? It relies upon.
On types the place we actually want to dam the robots (for instance, any that ship electronic mail), a failed Turnstile problem was initially simply displaying a pink error. It now says this:
Our anti-automation course of thinks you are a bot, which you are clearly not! Attempt behaving like a human and clicking the button once more and if it nonetheless misbehaves, give the web page a reload.
We have usually discovered a second click on or a web page reload solves the issue, so hopefully this sends individuals in the appropriate course. If it would not, we’ll want to take a look at extra in-your-face implementations of Turnstile that present a widget it’s essential to work together with. To have a go your self and see it in motion, attempt the dashboard register web page.
The opposite place Turnstile options closely is on the primary search web page on the root of the positioning. We do not need that API being hit by bots, so it is a will need to have there. Right here, like on the opposite pages of the brand new web site, we’re asynchronously posting to API endpoints and sending the problem token together with the request. What we’re doing in another way on the entrance web page, nonetheless, is that if the problem fails and returns HTTP 401 when posted to the HIBP endpoint (you will additionally see a response physique of “Invalid Turnstile token”), we have been meant to be falling again to a full web page submit. That wasn’t occurring within the new web site after we first launched it. However it’s now 🙂
When the complete web page submit again happens, Cloudflare will current a managed problem. That is far more invasive, nevertheless it’s additionally far more dependable and can then serve the identical end result as you’ll have seen anyway, albeit through a full web page load. We implement the identical managed problem logic on the deep-linked account pages, which you’ll see right here: https://haveibeenpwned.com/account/take a look at@instance.com
In keeping with the Cloudflare stats, about 82% of all our issued challenges are efficiently solved:
Of the 18% that are not, many shall be attributable to bots stopped by Turnstile doing precisely what it is meant to do. It is doubtless a single-digit share of requests which are actual people being impeded, and we have to take a look at methods to get that quantity down, however no less than the fallback positions are improved now. In the event you have been having issues, give the positioning a great refresh, see the way you go and go away your suggestions within the feedback under.
