[Hotstorage-chairs] [HotStorage 2026] Rejected submission #1 "EvolveFTL: An OpenEvolve-Based Framework..."

Sat Jul 25 20:10:07 EDT 2026

아쉬운 결과인데…예상보다 리뷰가 critical한것 같다.

개선할 부분 긴 버전에 잘 반영하고
archive에 올릴지 검토하도록 하자.

—-
김지홍 드림

(Sent from a mobile phone)

2026년 7월 25일 (토) 오후 12:06, 'HotStorage 2026 HotCRP' via professor <
jihong at davinci.snu.ac.kr>님이 작성:

> Dear authors,
>
> The program committee for the 2026 ACM Workshop on Hot Topics in Storage
> and File Systems (HotStorage 2026) regrets to inform you that your
> submission 1 was rejected to appear in the proceedings of the workshop.
>
> * Title: EvolveFTL: An OpenEvolve-Based Framework for Autonomous FTL
>   Optimization
> * Site: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fhotstorage26.hotcrp.com%2fpaper%2f1&c=E,1,TkjNEaMQQYOBkuzuaGdoSirAacV34m2nCE-Z5wnhufmabXOq-DFBr-z33RX5bxabnjx4W7Mdc9IN8vvJiF5E9AmqqYMtcvIvGJ2fCj2VPjLjOaDW3P6zDTPcO0o,&typo=1
>
> 23 of 78 submissions were accepted.
>
> Visit the submission site for reviews, comments, and related
> information. Reviews and comments are also included below.
>
> Contact chairs26 at hotstorage.org with any questions or concerns.
>
> Sincerely,
> Young-ri Choi and Bryan S. Kim
> HotStorage 2026 Program Co-Chairs
>
> Review #1A
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 3. Knowledgeable
>
> Paper summary
> -------------
> This paper presents EvolveFTL, an OpenEvolve-based framework to optimize
> FTL design in an autonomous way. The motivation behind the design of
> EvolveFTL is that, according to the authors, FTL design is largely "manual"
> and "intuition driven", so this contribution has the objective to automate
> the design of FTL through the use of coding agents. The authors extended
> OpenEvolve with two mechanisms: 1) an LLM-driven feature engineering stage
> to identify policy driven features, and 2) and FTL-aware prompt structuring
> scheme that searches for the candidate FTL.
>
> Comments for authors
> --------------------
> Strengths of the paper
> - Exploring AI-driven research for systems in the case of FTL design is
> interesting
> - The topic is timely and good for HotStorage
> - Some interesting result discussions
>
> Main weaknesses
> - It seems like there is a prerequisite for understanding the paper
> - Non convincing/awkward motivations
> - No evaluation against "manual" FTLs
> - Several Important aspects remained unclear
>
> Discussion
> - It seems like there is a prerequisite for understanding the paper: a
> first read of the paper was difficult, so I got back digging into
> OpenEvolve framework explanations. Once understood, I went back to read
> this paper and it was clearer. This is to say that the paper is not self
> contained and have as a prerequisite the understanding of the OpenEvolve
> framework; which is not favorable.
>
> - Non convincing/awkward motivations: the motivations of this paper were
> given in the intro section, they were mainly related to the current design
> of FTLs that the authors find "manual" et "intuition-driven". Do the
> authors have any evidence about this? to what extent are "manual" and
> "intuition-driven" FTLs bad or not sufficient?
>
> - No evaluation against "manual" FTLs: the evaluation part focuses on the
> difference between directly using OpenEvolve or its FTL-specific extension
> EvolveFTL. However, no SOTA FTL has been evaluated.
>
> - Several Important aspects remained unclear:
>         - during my first read, at the middle of the paper, I was still
> wondering what was the contribution, was it an FTL? an FTL generator? a
> system that changes dynamically the FTL online? I needed to go back and
> learn about OpenEvolve and read the paper evaluation to understand the
> output of EvolveFTL.
>         - several terms were used but not rigorusly defined: what is an
> "FTL state", and a "workload signal" those two seem important to understand
> the contributin but the reader is left to guess their meanings.
>         - same for "FTL-aware prompt", "policy evolution", "diverse
> scenarios" ... all those words used in the abstract without definition
> makes it hard to understand.
>         - it seems to me that the undiscolsed idea behind this paper is
> that FTL design is solely about code generation. I guess this needs to be
> explicit in the paper and discussed.
>         - The authors classified FTLs according to a binary choice:
> throughput driven and latency-driven, I guess the authors need to provide
> evidence about this, at least strong references
>         - the assumption behind this paper is that an SSD model is solely
> built for one application. I do not necessarily agree, at least the authors
> need to discuss this clearly.
>         - I did not find the explanations about the design clear enough,
> what is the role of the simulator? to what extent the result depends on the
> exhaustivity of the FTL libraries? it is not clear how the output solution
> is interfaced with real systems.
>         - what is "evolution memory" ?
>         - what is "target region" ?
>         - what is the relation between Figure 2 and Figure 1, it is not
> explicit enough.
>         - Is explainability an issue in such design? how is the system
> validated?
>
> I found the main idea of the paper interesting, however, several apsects
> remain unclear
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #1B
> ===========================================================================
>
> Overall merit
> -------------
> 4. Accept
>
> Reviewer expertise
> ------------------
> 2. Some familiarity
>
> Paper summary
> -------------
> The paper proposes EvolveFTL, a framework that adapts AI-driven research
> for systems, specifically OpenEvolve-style evolutionary code generation, to
> flash translation layer policy optimization. The core idea is that generic
> LLM-based search is insufficient for FTL design because it relies on
> manually exposed features and aggregates feedback across scenarios.
> EvolveFTL addresses this with two additions: an LLM-driven
> feature-engineering stage that identifies and instruments FTL-relevant
> signals, and a prompt-structuring scheme that preserves per-scenario
> feedback across iterations.
>
> Comments for authors
> --------------------
> Thank you for submitting your work to HotStorage ’26. I appreciate that
> the paper is well-motivated. FTL policy design is a plausible target for
> automated policy search because the design space is large,
> workload-specific, and often relies on expert intuition. The distinction
> between feature discovery and policy generation is also useful: exposing
> the right SSD-internal signals is often as important as the policy logic
> itself.
>
> There is obviously a lot of uncertainty regarding the physical properties
> of media and how they change between generations, but if we look solely at
> a fixed set of operations to optimize, I believe this approach could be
> very interesting, especially when considering the tight end-to-end
> integration found in hyper-scalers' infrastructure.
>
> Note that the research, especially around GC, burstiness, etc., could be
> explored in the context of SSDs that support Zoned Namespaces. They allow
> the SSD to maintain control over the media but leave all data placement
> decisions to the host. Thus, EvolveFTL could use this to continuously try
> to optimize within a narrower workload context.
>
> The largest issue, I see, is that the paper does not sufficiently
> establish generality. The evaluation uses two policy tasks, but both are
> still relatively narrow and simulator-based. The conclusion notes that
> broader SSD configurations and workloads remain future work, but this
> limitation is central rather than secondary. It is hard to know whether
> EvolveFTL is learning robust FTL principles, simulator-specific heuristics,
> or workload-specific decision rules.
>
> Another issue is that the comparison against OpenEvolve may not fully
> isolate the value of the proposed method. OpenEvolve is presented as the
> baseline ADRS method, but the paper does not compare against simpler
> non-LLM optimization strategies, manually tuned expert policies,
> reinforcement learning-based SLC cache policies, or search over a
> parameterized policy space. Since the paper’s claim concerns autonomous FTL
> optimization, the relevant baselines are not only generic OpenEvolve but
> also existing automated or semi-automated FTL tuning approaches.
>
> In summary, I believe EvolveFTL represents a promising exploration toward
> automated FTL policy optimization, combining LLM-driven feature engineering
> and scenario-specific feedback structuring. While the framework shows
> potential, broader evaluation across diverse SSD configurations and
> comparison with established baselines are needed to understand its
> generality and practical impact.
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #1C
> ===========================================================================
>
> Overall merit
> -------------
> 3. Weak accept
>
> Reviewer expertise
> ------------------
> 4. Expert
>
> Paper summary
> -------------
> EvolveFTL is a LLM-driven framework that extends OpenEvolve to optimize
> certain FTL policies (e.g., GC scheduling).
>
> Comments for authors
> --------------------
> I believe the paper has merit in terms of adding AI/ML to optimizing
> heuristics in the FTL. However, I have a few questions / concerns about
> both the design and the paper itself:
>
> - ADRS is not a well-known term. While it is explained to some extend, the
> abstract and introduction are written to assume they are common knowledge.
> - Related to the above, the OpenEvolve baseline is also niche. I would
> like to see benchmarks against more standard frameworks too.
> - I would like to see more experimentation to highlight local optima in
> conflicting scenarios.
> - The paper assumes QLC and does not touch other media (e.g., TLC). I do
> not believe the techniques described here are media-specific. Also, not all
> QLC SSDs use SLC caching.
> - It should be more clear that EvolveFTL targets new heuristics through
> the SS stage. In the beginning I thought it was writing firmware directly.
> It should also me more clear that the heuristics apply to specific hooks in
> the FTL.
> - There is no comment on guardrails or validation of the policies
> generated on the SS stage. How do we protect against hallucinations or bad
> heuristics in production? This is my main concern from the design point. A
> full paper should address this in detail.
> - I would not use LLM-specific nomenclature. I believe this is AI/ML
> application.
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #1D
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 3. Knowledgeable
>
> Paper summary
> -------------
> In this work the authors apply the idea of ADRS to the FTL design space.
> They identify that just blindly applying OpenEvolve (ADRS exploration) to
> the FTL is not sufficient as it lacks understanding that (i) FTL gc policy
> exploration is not stateless and relevant state can be scattered across
> many FTL components; (ii) FTL design objectives are multi-dimensional and
> instead of using a single aggregate matrix, it proposes to retain the
> history of evolution across multiple dimensions objectives. With this the
> search can pick different components that maximize the different objectives
> from different historical runs to come up with a single best
> implementation. The implemented system, EvolveFTL, compares favorably with
> the current policy implementation in a MQSim, a popular SSD simulator.
>
> Comments for authors
> --------------------
> Thank you for submitting your work to HotStorage 2026. The premise of the
> paper is interesting and contributes to this long line of recent interest
> in ADRS systems. The paper shows that the concept can work which is nice.
> However, like other recent systems in this area, it has limited discussion
> on useful/transferrable insights beyond “we put it together and it works”.
> Here are some comments on the current paper draft:
>
>   * One of the claims for the work is that with the use of LLMs, we can
> enhance the design space explored for algorithms and features that we can
> reason about. However, there is limited evidence presented for this in the
> evaluation. What kind of EvolveFTL design did the ADRS system come up with
> that was evaluated in section 4? There is no information presented for it.
> “Why” were there performance gaps, what did EvolveFTL do better than the
> others (the “reasoning” part).
>
>   * Section 3.2 - I would assume that the overall quality of the work is
> primarily driven by how good feature abstraction and workload signal
> extraction is. There is limited detail presented for this. Are these manual
> processes? What will happen to the performance on an unseen
> workload/skewness, how bad the performance can get here? Are there
> guard-rails to identify pathological bad cases where some “default” sane
> policy would be a good fallback.
>
>
>   * 3.2 - how big are the changes to the overall simulation code base to
> integrate it with Open-Evolve (as described for step-3)?
>
>
>   * 3.3 - in the solution space exploration - is there human intervention
> for quality control needed?
>
>   * For any such ADRS project - (i) what was the overall **time** and
> **cost** of exploration? (ii) how explainable and maintainable is the
> code?; (iii) What kind of model-capabilities one would need to make such
> exploration feasible, i.e., do we always need frontier models to do such
> exploration; (iv) what is the per iteration code changes, are there bounded
> limited small hyper-parameter tuning changes or completely not
> implementation of critical functions?
>
>
>   * 3.3 - there is limited detail regarding how things are done but more
> description of high-level flow. Between the three steps, there are a lot of
> design details missing for me, which makes evaluation of such work hard(er)
> as it comes across “we put things together and it worked”. For example - in
> dynamic trace, what kind of key informational signal accumulated/processed
> to make it useful? Who defines the boundaries for the static context,
> human?
>
>
>   * Section 4.1 - there is a missing SoTA FTL comparison. Why not compare
> the work with https://github.com/platformxlab/LeaFTL -- the current
> evaluation shows that the performance is comparable with some sane/default
> policy implemented in a simulation, not SoTA/SoTP FTL designs.
>
>   * Section 4.2 - how are the hyper-parameters chosen? “Nine independent
> search runs”, “30 iterations”.
>
>   * How is the OpenEvolve FTL policy that mainly relies on block-level GC
> signals different from what is implemented in MQSim to outperform in the
> first 5 (A-E) scenarios?
>
>   * 4.2.2 - How do we reason about the gain achieved? 5.2-17.7% range is
> what we can expect from such a system? What is the optima here? If we were
> to run the exploration longer, would it have discovered a high performance
> setup?
>
>   * What was the exploration success rate? Among the 9 runs how did the
> performance improve over the 30 iterations? On what factors did this depend
> on. For me this is the most interesting question: how do frontier models
> “reason” about this FTL design space.
>
> Overall I like the project idea and the application of ADRS to the large
> design space of FTL design. But there are pieces missing from this write up
> and evaluation.
>
>
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jihong+unsubscribe at davinci.snu.ac.kr.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fsl.cs.sunysb.edu/pipermail/hotstorage-chairs/attachments/20260726/ed2191c7/attachment-0001.htm>