[Hotstorage-chairs] [HotStorage 2026] Rejected, but invited as poster, submission #213 "Regular: Towards 100 Million IOPS for..."

[HotStorage 2026 HotCRP]

Dear authors,

The program committee for the 2026 ACM Workshop on Hot Topics in Storage 
and File Systems (HotStorage 2026) regrets to inform you that your 
submission 213 was rejected to appear in the proceedings of the 
workshop.

However, your work incited exciting discussion among the program 
committee members, and we would like to invite you to present your work 
during the poster session. Please let us know by August 12, 2026 
(Wednesday) whether or not you accept or decline the poster invitation.

* Title: Regular: Towards 100 Million IOPS for GPU-initiated I/O
* Site: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fhotstorage26.hotcrp.com%2fpaper%2f213&c=E,1,7EaNUSFzGMJH1xMO5wDKM-iav_pS2tAClzfmmdF69aUMSJgrWpvaLAqtoG9HVsQ3_8p-cGg5ufXH60QRNiz3YpBJSpMjgEZyuMyXmvcz&typo=1

23 of 78 submissions were accepted.

Visit the submission site for reviews, comments, and related 
information. Reviews and comments are also included below.

Contact chairs26 at hotstorage.org with any questions or concerns.

Sincerely,
Young-ri Choi and Bryan S. Kim
HotStorage 2026 Program Co-Chairs

Review #213A
===========================================================================

Overall merit
-------------
2. Weak reject

Reviewer expertise
------------------
3. Knowledgeable

Paper summary
-------------
The paper "Towards 100 Million IOPS for GPU-initiated I/O" presents an NVMe-SSD emulator, which directly runs on a GPU and investigates, based on the emulator, overheads of the BAM GPU file system and proposes and investigates NVMe extensions, which can help to improve overall performance of GPU file systems, so that they can break the 100 Million IOPS boundary.

Comments for authors
--------------------
Thank you for submitting this paper to HotStorage 2026. I appreciate the idea of directly running an NVMe-SSD emulator on a GPU, so that it becomes possible to investigate GPU FS overheads. I also like the performance breakdown of latencies inside the BaM protocol, so that it becomes possible to provide better implementations and protocols. 

I have some smaller comments on the paper:

- I would have expected that Section 3.4 also validates the correctness of the SSD emulation and the corresponding timing model presented in Section 3.3. Instead, it only seems to check whether the target performance seems to match the emulated performance of the drive. It would have been helpful to spend some more explanations on the validation of the emulator.
- I agree with the authors that the proposed NVMe protocol extensions could improve overall performance. However, I had hoped for a discussion whether it is also possible to adapt the BaM software itself to decrease overheads without changing the protocol, so that protocol changes are only performed if absolutely necessary. 
- It would have also been helpful if the authors would have discussed in more detail the overhead of direct completion (Section 4.3) in real implementations over PCIe.


* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


Review #213B
===========================================================================

Overall merit
-------------
3. Weak accept

Reviewer expertise
------------------
2. Some familiarity

Paper summary
-------------
This paper argues that the NVMe protocol, whose submission and completion queues were designed for a handful of CPU cores that each own a queue, becomes the bottleneck once thousands of GPU threads share those queues for GPU-initiated I/O. The key insight is that at GPU thread counts the shared completion queue, not the SSD, dominates per-I/O cost, so the authors first build WarpSSD, a GPU-resident NVMe emulator that runs on the same GPU as the workload and lets them measure GPU-side protocol overhead in isolation. Using it, they attribute most of that overhead to the shared completion queue and propose three redesigns, per-request completion flags, batched vector submissions, and a completion-wait hardware primitive, that together aim to close the gap to 100M IOPS.

Comments for authors
--------------------
#### Strengths

S1. Thank you for submitting this paper! I enjoyed reading it and found the diagnosis work pretty sharp and and convincing.

S2. WarpSSD is a useful artifact.

#### Weaknesses

W1. The whole evaluation is emulated, and the redesign speedups are measured at zero device latency as protocol-processing ceilings, not as achievable storage throughput. The results quantify how much GPU-side atomic and polling overhead each design removes, which is fair and useful, but they are not IOPS a real GPU-plus-SSD system would deliver. The abstract and contributions present these as steps that "close the gap to 100M IOPS" without making the ceiling framing explicit, which somewhat overstates what is shown.

W2. The device-side cost of the redesigns is assumed away. However, this is exactly what the emulator cannot model. Direct Completion is called a firmware-only change and Vector SQE a new opcode with no ring or doorbell changes, but WarpSSD's emulated controller performs both instantly, so the paper never accounts for the SSD-side work.

W3. The headline 2.1X over SwarmIO compares artifacts that are not really comparable. SwarmIO emulates SSDs on a dedicated 86-core CPU server, whereas WarpSSD emulates on the GPU while co-hosting the workload, so the two target different purposes.


* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


Review #213C
===========================================================================

Overall merit
-------------
3. Weak accept

Reviewer expertise
------------------
3. Knowledgeable

Paper summary
-------------
This paper studies whether the NVMe protocol can support GPU-initiated I/O at the scale of future 100M IOPS SSDs. It develops WarpSSD, a GPU-resident SSD emulator, identifies the shared completion queue as a major bottleneck, and proposes a few changes to the NVMe protocol and GPU hardware primitive to reduce protocol overhead.

Comments for authors
--------------------
This is an interesting and forward-looking paper. It examines an important question as GPU-initiated storage access becomes increasingly common and SSD IOPS continue to scale.  

I have a few questions regarding the motivation in Section 2. In general, a cluster with 128 GPUs is unlikely to rely on only a few SSDs, and its aggregate IOPS demand would typically be distributed across many storage nodes or absorbed by local caches. In addition, not every GPU thread necessarily participates in I/O simultaneously. I would love to see a better justification of the per-device concurrency and IOPS assumptions using realistic workload traces or system configurations.

I also have some questions about the proposed redesigns. Eliminating the shared completion queue removes substantial contention, but the paper should discuss possible side effects, such as completion ordering. Wait-on-Write requires a new GPU hardware primitive. I left wondering how practical such a change would be and what architectural support would be required from GPU vendors.


* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


Review #213D
===========================================================================

Overall merit
-------------
1. Reject

Reviewer expertise
------------------
4. Expert

Paper summary
-------------
This work presents WarpSSD, a GPU-resident NVMe SSD emulator that runs on the same GPU as the I/O workload. Using WarpSSD, the work reports that completion-queue processing dominates GPU-side NVMe overhead and proposes three optimizations: (i) Direct Completion, (ii) Vector SQE, and (iii) Wait-on-Write. Direct Completion replaces the shared completion queue with per-request completion records, Vector SQE batches submissions across a warp, and Wait-on-Write allows GPU threads to yield while waiting for external completion. The first two proposals are evaluated in WarpSSD, reaching up to 297 MIOPS under the evaluated configuration.

Comments for authors
--------------------
## Strengths
+ The work investigates a timely question as GPU-initiated I/O and SSD request rates continue to scale.
+ The bottleneck analysis highlights potentially important GPU-side queue-management overheads.
+ The proposed mechanisms provide concrete directions for improving the resource efficiency of GPU-initiated I/O.

## Weaknesses
- The evaluation does not substantiate the central claim that the NVMe protocol must change.
- WarpSSD’s level of SSD and PCIe emulation is insufficiently explained and validated.
- Direct Completion leaves important queue-management and real-PCIe costs unclear.
- Vector SQE assumes workload behavior that may not hold for real BaM-style applications.

## Detailed Comments
Thank you for submitting the work to HotStorage 2026. The manuscript raises an interesting question: whether an NVMe interface designed around CPU-core-owned queues remains efficient when thousands of GPU threads initiate storage requests. The reported completion-queue overhead and the proposed optimizations are potentially useful.

Nevertheless, I found that the current evaluation does not support the manuscript’s strong conclusion that the NVMe protocol itself must be redesigned. Given the substantial amount of necessary modifications, I believe the manuscript is not ready for publication. I provide more detailed comments on the critical weaknesses I found in the current submission, which I hope help the authors strengthen the work.

### The Necessity of NVMe Protocol Changes
GPU-initiated I/O fundamentally exploits massive GPU concurrency to tolerate SSD latency and generate enough outstanding requests to drive storage throughput. The proposed Direct Completion and Vector SQE mechanisms may reduce the number of GPU threads, queues, or SM cycles required to reach a given request rate, but this is different from showing that existing NVMe semantics fundamentally prevent GPU-initiated I/O from reaching the target throughput.

The evaluated baseline couples one NVMe queue pair to one CUDA thread block, which unnecessarily couples hardware queue count with GPU requester parallelism. The main analysis uses 256 queue pairs and 256 threads per block, but the separate 512-queue experiment already reaches 106 MIOPS for the control path alone. This suggests that 100 MIOPS may not be a fundamental protocol ceiling.

To support the claim that NVMe must change, the work should fix the queue-pair count and queue depth to realistic device limits, decouple the number of requester threads from the number of hardware queues, and provide sufficient GPU threads and outstanding requests behind those queues. It should also evaluate an optimized standards-compliant implementation, for example, with hierarchical submission and dedicated completion-consumer warps, rather than requiring every requester to scan the shared CQ.

Moreover, 100 MIOPS is a per-device target in the manuscript’s future-SSD framing. A GPU-side stack may need to sustain the aggregate throughput of several such devices (e.g., 400 MIOPS with 4 SSDs). The current proposals reach 297 MIOPS only in the zero-latency, same-GPU setup, so the work neither establishes the ceiling of standard NVMe nor demonstrates that the redesign meets the broader aggregate GPU-side requirement.

### Unclear WarpSSD’s Contribution and Fidelity
WarpSSD appears closer to a GPU-resident NVMe queue and latency generator than to a detailed SSD emulator. The emulator warp reads SQ entries, copies data from an HBM-resident namespace, assigns configured completion times, and posts CQ entries. However, the manuscript provides little detail about the modeled controller scheduling, channel and die contention, request reordering, FTL behavior, latency distributions, or other device-level mechanisms.

The validation mainly shows that WarpSSD follows a configured target IOPS. This demonstrates high-rate request generation, but not SSD-model fidelity. The manuscript should either narrow the contribution to a GPU-side protocol stress platform or provide substantially more design detail and validation against a real SSD or the original CPU timing model.

Running the controller and workload on the same GPU also removes important aspects of the real path, e.g., device DMA, PCIe ordering, transaction formation, and SSD-controller firmware costs. These omissions directly affect the proposed protocol changes. In addition, WarpSSD posts completions strictly in submission order, whereas real SSDs may complete requests out of order; this choice may affect the CQ contention behavior the work aims to study.

### Direct Completion Needs Further Clarification
Direct Completion can eliminate per-request CQ scanning, but its real implementation has trade-offs that the same-GPU experiment cannot capture.

A standard CQ places completions in consecutive ring positions regardless of command ID or completion order, allowing the SSD to batch adjacent CQ entries into efficient PCIe writes. Direct Completion instead targets a per-command record, potentially producing scattered writes. The manuscript should evaluate whether its advantage survives real PCIe transaction costs.

The queue-reclamation mechanism is also under-specified. The manuscript states that Direct Completion eliminates the CQ and CQ doorbell entirely, while leaving the SQ interface unchanged. However, the standard CQ also communicates submission-queue consumption progress. If the CQ is truly removed, the design needs an alternative method to determine which SQ entries can be reused, handle out-of-order completions and wraparound, and synchronize queue-space reclamation across requesters. If a background consumer still drains a standard CQ for this purpose, the contribution is more accurately a change to completion polling than complete CQ elimination. The manuscript should explain the mechanism and include its synchronization cost in the evaluation.

### Vector SQE Relies on a Strong Workload Assumption
Vector SQE assumes that 32 lanes in a warp simultaneously produce 32 independent I/O requests. This is compatible with the evaluated synthetic workload, but it may not hold in real BaM-style applications. Warp-level access coalescing, cache hits, and control divergence can result in substantially fewer than 32 unique storage requests.

If the design always reserves 32 SQ positions, it may waste SQ capacity and reduce the number of useful outstanding I/Os. A practical implementation would need to identify active request lanes, reserve only the required number of positions, and compact requests into consecutive entries. The overhead and benefit would then depend on the number of actual requests per warp. The manuscript should evaluate Vector SQE under realistic per-warp request distributions rather than only full-warp participation.

The mechanism itself is also unclear. One description reserves 32 ordinary SQ slots and rings one doorbell, which appears implementable without a new opcode. Another introduces one Vector Read command pointing to a descriptor array. These are different designs with different device-side costs, and the manuscript should clarify which one is evaluated.


Comment @A1 by Reviewer A
---------------------------------------------------------------------------
Dear Authors, thank you for submitting your work to HotStorage 2026. Your paper has been discussed in the PC meeting. The reviewers agree that providing an emulator inside the SSD and the performance breakdown of latencies inside the BaM protocol are interesting research contributions. The main reason for not accepting the paper has been that the central claim of the paper (the NVMe protocol must change) has not been substantiated enough and may mislead the reader in several aspects. We therefore would like to encourage you to resubmit the paper with better explanations / justifications to another venue.