Secure-Erase method option for supported ATA and NVMe devices#687
Secure-Erase method option for supported ATA and NVMe devices#687Knogle wants to merge 6 commits intomartijnvanbrummelen:masterfrom
Conversation
…ating
drive-internal sanitization facilities across ATA and NVMe devices. The method
implements a unified workflow:
1. Detect the drive type (ATA, SAS-over-ATA, NVMe)
2. Execute a hardware-level secure erase operation using the appropriate
system tooling:
- ATA/SAS: hdparm --security-erase
- NVMe: nvme format -s1
3. Perform a full-device verification pass to confirm that the media reads
back as all zeros after the hardware erase has completed.
This aligns nwipe’s functionality with modern storage devices that provide
dedicated sanitization mechanisms far more reliable and performant than
software overwrites.
- New method function: `nwipe_secure_erase()`
- New method label and GUI integration (“Secure Erase (ATA/NVMe)”)
- CLI method selection via: `--method=secure_erase`
- Help text updated to reflect the new method
- Infrastructure added for automatic method-to-label mapping
- Real secure erase execution implemented for:
**ATA / SATA / SAS (ATA pass-through)**
Using:
`hdparm --yes-i-know-what-i-am-doing --user-master u --security-erase NULL <device>`
**NVMe namespaces**
Using:
`nvme format <device> -s1`
These operations are invoked through `system()` on glibc systems and expect
the presence of `hdparm(8)` and `nvme(1)` in the user’s PATH.
After completion of the hardware secure erase operation, nwipe performs a
full-device verification pass:
- a single “zero pattern” is defined (`0x00`)
- `nwipe_static_verify()` is executed with full progress tracking
- GUI progress correctly displays the single verification phase
- Verification guarantees that the device reports a post-erase all-zero state
- If the device type is unsupported, Secure Erase fails early with a clear
log message, preventing accidental misuse.
- If hdparm/nvme report an error or return a non-zero exit status, the method
aborts before verification.
- system() invocation failures (INSUFFICIENT MEMORY, missing /bin/sh,
PATH errors, etc.) are treated as hard failures.
- All secure erase steps and verification steps include detailed logging.
- The secure erase helper functions are:
`nwipe_execute_ata_secure_erase()`
`nwipe_execute_nvme_secure_erase()`
- Each helper contains full English-language documentation and describes the
rationale for not re-implementing low-level ioctl-based ATA/NVMe mechanisms.
- The new method is fully integrated across:
- method.h
- method.c
- gui.c
- options.c
- CLI help and GUI help panes
The helper functions are structured such that direct ATA taskfile commands or
direct NVMe admin commands (NVME_IOCTL_ADMIN_CMD, NVME_IOCTL_IO_CMD) can be
added later without modifying the high-level workflow of `nwipe_secure_erase()`.
This commit significantly expands nwipe’s capabilities by enabling hardware-
level sanitization, while preserving the existing method interface and GUI/CLI
behaviour.
Knogle
changed the title
|
I'll take a look at this once the DIRECT I/O has been merged tomorrow. If it works ok, I'll probably merge it as is, with the view to maybe adding some additions to the GUI drive completion status. Hdparm provides an estimated wipe time, I always thought the estimated time provided by hdparm could be used in the GUI so instead of % completion it shows T-2:35:00 and counts down in real time from the estimated time provided by hdparm. If it reaches T-00:00:00 and hdparm still hasn't finished then it goes positive T+00:00:01 T minus to be used for ATA secure erase because we really don't know how long it will take and if we used percentage it would go over 100% and make it look broken. Overall ETA would also need looking at how it deals with an estimated time. However, let me check this PR out as it is before you add anything further. |
|
That's an interesting point, tbh didn't think about the estimated time issue! |
|
Just ran an actual test. It worked great! 
And one with a disk not supported. |
|
Wow! Never noticed this! Perhaps update the people here as well: #641 |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@Knogle this looks amazing. Some additional questions:
|
fthobe
left a comment
fthobe
left a comment
There was a problem hiding this comment.
Added some notes regarding output on failure and an additional comment regarding interuption during the process.
| */ | ||
| if( rc != 0 ) | ||
| { | ||
| nwipe_log( NWIPE_LOG_ERROR, "ATA Secure Erase: hdparm reported failure on %s (rc=%d).", c->device_name, rc ); |
There was a problem hiding this comment.
I'd be more verbose:
Replace "ATA Secure Erase: hdparm reported failure on %s (rc=%d)."
with "Error: hdparm reported failure on %s (rc=%d), run hdparm directly from terminal for a more detailed report."
There was a problem hiding this comment.
@Knogle is there any verbose switch to catch the hdparm / nvme-cli and sg Output and Print it on screen / Output in Report
| if( rc != 0 ) | ||
| { | ||
| nwipe_log( | ||
| NWIPE_LOG_ERROR, "NVMe Secure Erase: nvme format reported failure on %s (rc=%d).", c->device_name, rc ); |
There was a problem hiding this comment.
I'd be more verbose:
Replace "NVMe Secure Erase: nvme format reported failure on %s (rc=%d)."
with "Error: NVME-CLI reported failure on %s (rc=%d), run nvme-cli directly from terminal for a more detailed report."
| * | ||
| * This method does NOT write patterns itself. Instead it: | ||
| * 1) issues the appropriate secure-erase operation for the bus type | ||
| * 2) performs a full-device zero verification pass using nwipe_static_verify() |
There was a problem hiding this comment.
-
- Even if the drive is unplugged, the process will continue till completion or failure if the drive is reattached.
There was a problem hiding this comment.
Crypto erase has no stop once initiated, from what I've read.
There was a problem hiding this comment.
Crypto erase has no stop once initiated, from what I've read.
That is indeed the expected behaviour. It varies though according to revisions of the implemented SATA standard (you can read more about revisions inside the mega long thread you linked)
|
Hey @Knogle , this is amazing work, I'd consider also adding sg_utils as thanks to your work it became low hanging fruit and considering that SAS still has a significant market share. sg_sanitize also allows to poll progress. An alternative would be to use openSeaChest, we had a longer discussion with @vonericsen if you remember. It would allow to have an abstraction layer for all three methods. @vonericsen can probably also tell you what possibilities there are to track progress. |
sdparm and sg_utils are for SAS, hdparm for sata. I don't know if you are aware of it so I throw it in. |
|
@PartialVolume What's the desired report output here? |
At least we do a verify-zero pass afterwards, I think there is at least somewhat more confidence than just using the secure erase. |
I absolutely agree, but the intransparent behaviour of broken sectors remains a problem. I'd feel extremely hazy on this one and would really like to know what blanco does here as I believe it's clusterf**** and nobody really came up with a solution for this other than failing the report. |
With regards to the secure erase and optional verify zeros as being one selectable method, we will also have a method that consists of secure erase, followed by prng pass followed by verification on prng pass. In my mind this will attempt to mitigate against bad drive firmware not doing it's job properly hence why we use a prng pass. Why prng and not zeros or ones? Random data is pretty much impossible to compress, so the drive's firmware can't utilise any fancy algorithms to fool you into thinking you wiped it's memory when it's just marked the sector as wiped when it hasn't. Because the random data has to be reproduced exactly it's a lot harder for it to utilise any tricks to appear wiped when it didn't really wipe the memory. Verification is another complication when using secure erase. It's fine when the secure erase resets the drive to all zeros but I've come across some SSD's that are perfectly functional but when you secure erase it you get a 0x3F occurring once in every sector, at the same location in that sector, where the rest of the sector is zeros. I don't know how common this is. But a zeros verification would fail, yet there is nothing actually wrong with the drive. A secure erase, prng pass plus last pass verification will correctly pass the drive. So depending on how common a secure wiped drive's firmware generates a non zero output it may be necessary for a additional smart verification to be written. One that recognises a repeating reproducible pattern. In terms of over-provisioned memory in the drive, a secure erase, then prng pass + verification is the closest you will probably get to being sure all data is destroyed without taking out the chips and putting them in a reader to see what's left over. There might also be the possibility the drive firmware could also be tampered with so again it appears to have wiped but didn't really, another reason why a prng pass is a must in my opinion. I'm thinking state level messing with firmware. Depending on the level of risk you can accept. The only 100% guarantee is to vaporise the drive which is a big shame. I hate to see perfectly functional hardware destroyed but your level of paranoia may well depend on the drive application. When 100% guarantees are required, as in military, security services, physical destruction is the only option but for everything else it's what level of risk is acceptable. |
I fully agree, but we should mention that in the report in the likes of: Analogue to NIST 800-88r2, 3.1.1. Clear Sanitization Method, P.9
|
So like two options? Like one with verify-zero, and another "enhanced" one with PRNG pass? |
For me that doesn't make a difference as the problem lies in Firmware response: |
…fy for enhanced security.
|
This is how seagate currently categorizes sanitize. Everything that passes purge is currently defined as resonable Secure by NIST for non-overly sensitive data. It would not pass the thresholds for state secrets, but can be perfectly acceptable for regular business data. For clarity:
|
Yes, exactly. |
|
@Knogle one last question: any reason you went with secure erase over sanitize? |
I wasn't aware about those options! Maybe we can implement sanitze as well. |
|
ahhh ok, I will give you a crash course: secure erase does not fully delete all data on all drives as outlined here: Below an overview to verify sanitize support and execution with links to more detailed description outlining the different types like crypto scramble and all 0. Secure erase is ancient and does not warrant the cancellation of data, but sometimes only drops tables.
|
|
Ok, if we now even get a warning on ssd that people should check with vendor regarding compliance I'd be fully happy, @Knogle would have crushed it and new year was blessed 👯 |
Sure we can do so! Have you got some kind of idea or suggestion where we should display such a warning? |
|
@DaanSelen can you build and test this with SAS / NVME? |
|
@DaanSelen do you people intend to sign the certificate? |
Not really, may I ask why? |
@Knogle secure erase/sanitise screenshots are looking good 🥇 |
no problem! For USB it can be a bit of a mess. SATA USB: With older adapters you can run into various limitations. Some only support the 28bit command set and single sector transfers which means Sanitize won't work, but ATA security erase might. Generally, the main focus in the early adapters was ATA identify data and basic SMART data, so you don't always get much more than that. If the adapters are old enough (or just using old enough firmware) that they only respond to vendor unique CDBs, then it depends on which vendor unique method it is for what works. Some are robust enough to support sending sanitize and others are not. This linux kernel wiki (archived) mentions some issues specifically about ATA security erase. I have no idea how they got a bricked drive from this, but I have never seen this happen myself (we use all kinds of erase methods in our return sites for USB and haven't had this happen). I have no doubt it happened, but I would say it is not a common failure from my experience. NVMe USB: NVMe is basically in the early days of ATA over USB right now. SAS USB: Other weird things on USB: |
|
@vonericsen Sorry, going slightly off topic here, but I've a quick question re openSeaChest versions. I'm just about to release the latest ShredOS with openSeaChest upgraded to 25.05.3, the build goes fine, but during final testing and audit of versions I notice that each program within openSeaChest has a different version. Is there some command that tells me I'm running 25.05.3 as the overall package version or if not is there somewhere that tells me what individual package versions are included in 25.05.3. I'm probably being overly pedantic but I do like to check these things. |
No problem, this is a great question! Currently openSeaChest's tools (openSeaChest_Basics, _Erase, etc) use semantic versions that come out of the We do have one issue open around the build date though and will be implementing reproducible build dates in the future according to reproducible-builds.org I have considered numerous times changing from semantic versions in each tool to matching the calendar versioning the full package uses instead. If you have any suggestions or thoughts on this, please let me know. I know this is something we need to address to resolve confusion as you are not the first person to ask me this. P.S. I think later this month a new openSeaChest release will come out with the big feature being JSON formatted output. We have a little more work to do, but it is close. |
Thanks for the quick response, in my case the build date shows my build date as 8 Jan as I'm building from source
Not to worry, I just thought there might be some txt file which listed the versions included in v25.05.03 |
Add two new device-internal sanitize methods in addition to existing Sanitize Crypto Erase: - Sanitize Block Erase (ATA/SCSI/NVMe) - Sanitize Overwrite (ATA/SCSI/NVMe) Both methods trigger the drive's internal SANITIZE feature where supported and do not perform read-back verification, as post-sanitize read patterns are device-specific and may not be deterministic. The implementation selects the appropriate backend automatically based on the detected bus type (ATA, SCSI, NVMe). These methods are intended for secure, standards-aligned media sanitization using device-native mechanisms.
|
Okay just compiled the program and was checking out the new features. Couple things I wanted to ask:
Is there a way to verify if all these methods passed correctly since its all dependent on compliant firmware? Just some "reassurances" |
The concept of cancellation is different here:
If you cancel the key the data is inaccessible, this right now is considered "Purge" according to nist. Keep in mind though that today's definition of secure might change (e.g. if tomorrow a new quantum computer solves any calculation of any prime number, any definition of security by eliminating keys goes out of the window). Also consider that people tend to overestimate wear on drives. If you run one full cycle for a rewrite, it's still just one cycle. == UPDATE == |
Thanks I was familiar with the crypto erase. But if possible I want to check if the data is actually unreadable with a crypto erase. P.S. The Secure Erase + PRNG + Verify has crashed on my laptop (it was 3000KB/S syncing anyway). I am going to try again. Do we want the files/reports/dmesg/logs of my runs? |
|
Always post them when you have failures |
|
Can you poll Support for sanitize commands? I think that's a sata > m.2 and not real nvme, secure erase should work on this one and sanitize block should work. @Knogle errors are not handled correctly as the PDF Shows erased in the Header. |
I don't have a SSD ready at home (maybe with some luck) but I will return to this on monday! Unless I can do this on HDD's. |
|
Currently testing on my WD SN7100. |
That because if the drive doesn't encrypt by it's own the strategy can't work, but there's still Block erase and overwrite which should Both do the Job. |
Would it be possible to either disable unsupported methods or at least let the person know if something has not succeeded? |
Yes, but:
One way would be to trigger and Abort an unrestricted sanitize (sanitize can not be aborted and resumes even After a disconnect from power when ran as restricted). |
|
With the release of v0.40 I am curious to the feedback given and the process further :) I DO NOT INTENT THIS TO PUT PRESSURE JUST CURIOUS |
My focus over the next 24hrs will be the release of ShredOS, built with the latest 0.40 nwipe. Everything is complete and tested. Probably the largest release notes for ShredOS I have ever written has been completed. Uploading eight .isos and .IMG files with checksums is where I'm at, and it's truly tediously slow at uploading these images to GitHub. I'm not sure if it's my fibre connection (unlikely) or whether it's GitHub throttling the uploads. Anyway fingers crossed by tomorrow I'll release ShredOS. As for nwipe, going forward, I'll be looking at @Knogle 's SSD PR. |
|
@PartialVolume man, I so much appreciate what to do but I believe getting some funding from the OSS community and public grants through incorporating as an association would be a step forward to distribute workloads better. You know I come from refurbishment sales, I know like ten companies that for removing blanco would help funding this |
Maybe a organisation/non-profit for some priority support? |
|
Hi, I’m joining the discussion a bit late, but I wanted to suggest an enhancement to how the new firmware‑based secure erase feature could be integrated.
Where: yes: run the appropriate firmware erase/sanitize operation for the disk, then continue with the chosen overwrite method even if the firmware step encounters an error. Any failure should raise a warning in the logs and appear in the PDF report. force: run the firmware erase, but do not continue to any overwrite method if the firmware sanitize step fails. This would allow users to combine firmware‑level sanitization with traditional overwrite methods. Providing one report and one run compliancy with NCSA-FI guidelines |
I do plan on having SSD/NVMe as not only standalone but incorporated into methods. So for instance we could have a method called
Optionally you could select last pass or ALL verification although I'm not sure about the verification on the sanitise operation if you selected ALL, as there are some edge case drives that produce non zero output. So initially we might ignore verification on the SSD and just verify the two PRNGs, assuming you selected Verification=ALL. So yes, it would produce just the one report. |
|
Tbh i am not really satisfied with this one yet, i think it's far too large already to properly troubleshoot it, apart from simple secure erase, nothing works reliably accross SSDs (or maybe it's the expected behaviour due to firmware blabla?) In my opinion we have to re-evaluate this thing here, and maybe just go for a few simple options, and leave this sanitize stuff for specialized tools. What do you think guys? @PartialVolume @fthobe @desertwitch |
|
Where do you struggle? |
|
To integrate SSD/NVMe sanitisation into nwipe I always thought would take a huge number of changes to methods, contexts and the GUI. I can see how it's going to work in my head but I've not documented the changes that need to take place yet. The drive selection screen will change so frozen drives are immediately visible requiring nwipe to send the system to sleep for a few seconds if a specific drive is selected for secure erase., I posted a mockup of the new drive selection some time ago. It maybe that where we select wipe using the spacebar this field toggles through WIPE then SERA, SERA being secure erase rather than a traditional wipe. This is obviously necessary as the system may have one SSD and a traditional hard drive so you may want to wipe different drives with different methods at the same time. There will be a new selection screen/tab for selecting SSD/NvMe options relevant to SERA selected drives. Nwipe currently applies only one method to all drives, this will have to change so that two or even three different methods can be run concurrently. I'd prefer we wrote low level code, at least for ATA secure erase but I guess it depends how many manufacturers deviate from the standard. The problem for me with this PR is the secure erase is not selectable individually for each drive. Toggling WIPE/SERA with the spacebar would solve that but requires a fundamental change in that two different methods need to run at the same time. Also lots of work would need doing to the progress percentage as there is no progress data from the drive so we have to estimate completion. I think it's doable or are you talking about issue closer to the drives, i.e just not behaving correctly with hdparm? Originally my plan was not to use hdparm but write our own low level code as I did for accessing HPA/DCO without using hdparm. |
|
That's what i would like to go for as well! Having our own code in place for those erase options, at least for those widely accepted ones. I'd consider this PR more a PoC than a real implementation tbh at this point, because as you pointed out, major changes have to be made to how nwipe works. |
|
I'd generally tend to favour low level code as well, as opposed to depending on (vendor) tool X or Y. I'm not an expert on the topic (nor have a real need for such a feature myself), but if a standard exists then I think we should support that one first and foremost. If a vendor chooses not to implement that standard then that's on them and people will need to use their tools instead (which we can ship inside ShredOS, rather than making them a dependency of I also see this requiring a larger rewrite of many parts of the program, but I have no specific vision about it in my mind and I think it's something that we should carefully test/plan so we don't cause degraded quality of the previous core "selling points" of We should also consider if Is it more a nice to have or something where we're meaning to fill a substantial need? Are we able to provide a better experience than what is already available? In my mind a program that does a few things very well seems generally favourable to a jack of all trades which does some things well and others poorly. The latter especially being a risk if there's no consensus on a standard (among vendors) and we're basically playing catch-up all the time. If realistically out of 10 NVMe wipes, 6 fail because of vendor-specific quirks... not sure if it's worth it (now) over being revisited at a point when the industry has decided on a common standard (if that ever happens). |
7 participants
This commit introduces a new wipe method named "Secure Erase", integration drive-internal sanitization facilities across ATA and NVMe devices. The method implements a unified workflow:
New method function:
nwipe_secure_erase()New method label and GUI integration (“Secure Erase (ATA/NVMe)”)
CLI method selection via:
--method=secure_eraseReal secure erase execution implemented for:
ATA / SATA / SAS (ATA pass-through) Using:
hdparm --yes-i-know-what-i-am-doing --user-master u --security-erase NULL <device>NVMe namespaces Using:
nvme format <device> -s1These operations are invoked through
system()on glibc systems and expect the presence ofhdparm(8)andnvme(1)in the user’s PATH.After completion of the hardware secure erase operation, nwipe performs a full-device verification pass:
a single “zero pattern” is defined (
0x00)nwipe_static_verify()is executee GUI progress correctly displays the single verification phaseVerification guarantees that the device reports a post-erase all-zero state
If the device type is unsupported, Secure Erase fails early with a clear log message, preventing accidental misuse.
If hdparm/nvme report an error or return a non-zero exit status, the method aborts before verification.
system() invocation failures (INSUFFICIENT MEMORY, missing /bin/sh, PATH errors, etc.) are treated as hard failures.
All secure erase steps and verification steps include detailed logging.
The helper functions are structured such that direct ATA taskfile commands or direct NVMe admin commands (NVME_IOCTL_ADMIN_CMD, NVME_IOCTL_IO_CMD) can be added later without modifying the high-level workflow of
nwipe_secure_erase().