[Hotstorage-chairs] [HotStorage 2026] Rejected submission #120 "Regular: FlexiPool: Towards Flexible..."

[Avani (અવની)]

There wasn't much that the reviewers disliked that we didn't expect, so
let's go and improve the overhead discussion.  Onwards and upwards!

On Fri, Jul 24, 2026, 8:06 PM HotStorage 2026 HotCRP <
noreply-hotstorage26 at hotcrp.com> wrote:

> Dear authors,
>
> The program committee for the 2026 ACM Workshop on Hot Topics in Storage
> and File Systems (HotStorage 2026) regrets to inform you that your
> submission 120 was rejected to appear in the proceedings of the
> workshop.
>
> * Title: Regular: FlexiPool: Towards Flexible Cache Policy Switching
> * Site: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fhotstorage26.hotcrp.com%2fpaper%2f120&c=E,1,5u1uCLBvKOJu9Psfek4uFIGtQBOQw2IvPX1yDPJTNu0h0yYW7oAmj6uHrtpWmMMj9OfogYKe_qYvUk98S2pse2nE9uqrCSN22bar53gHmsxlfwbzEBR4SQ,,&typo=1
>
> 23 of 78 submissions were accepted.
>
> Visit the submission site for reviews, comments, and related
> information. Reviews and comments are also included below.
>
> Contact chairs26 at hotstorage.org with any questions or concerns.
>
> Sincerely,
> Young-ri Choi and Bryan S. Kim
> HotStorage 2026 Program Co-Chairs
>
> Review #120A
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 4. Expert
>
> Paper summary
> -------------
> The paper proposes an automated cache policy switching framework called
> FlexiPool. FlexiPool have the ability to switch between arbitrary policies
> (not necessarily previously predefined). It uses an LLM-based agent to
> synthesize the switching rules. Three challenges were identified for which
> a solution has been proposed: 1) maintaining multiple eviction strategies
> meta data, 2) "switch logic"  design, and 3) overfitting of the switch
> logic.
> The results show some benefit as compared to SOTA work.
>
> Comments for authors
> --------------------
> Strengths of the paper
>         - using agents to explore cache eviction policy selection is
> interesting
>         - the paper addresses an impactful topic about cache strategy
> selection
>         - the paper falls clearly within the topics of HotStorage
>         - the authors evaluated exhaustively against several workloads
>
> Main weaknesses
>         - the overheads are not studied
>         - the gains seem modest
>         - the motivation could be enhanced
>         - the writing could be enhanced
>
>
> Discussion
> We would like to thank the authors for their contribution
>
>         - the overheads are not studied: one important aspect in deploying
> a cache strategy is its overhead. Even if the authors ackowledged this
> aspect in the introduction no overhead related to the simulations, to the
> agent were given. This is a real issue as the reader does not have any idea
> about the benefit in terms of BHR with regards to the added overhead. Is it
> worth it to run a voluminous LLM for such performance gain ? what is the
> simulator overhead, how does it scale with the number of policies ?...
>
>         - the gains seem modest: the authors show in the evaluation part
> that their policy gives BHR percentage gains that are mostly under 3%. It
> would be interesting to discuss how such an enhancement is worth the
> deployment.
>
>         - the motivation could be enhanced: in the introduction, the
> motivation behind this study could be enhanced. It is not clear what is the
> limitation in previous work and what is the impact in terms of performance
> of those limitations. Why is it so important to embed more than 10 caching
> strategies?
>
>
>         - the writing could be enhanced: even if HotStorage is a topical
> workshop, there are several terminology used without being defined:
>                 - BHR acronym is used without being defined,
>                 - "switching logic" is a concept used without being
> properly defined, the challenges are given but the reader do not understand
> how were they identified ....
>                 - Table 1 reports some numbers but the reader does not
> have any clue to know which metric the authors are talking about
>                 - Figure one is not clear for the reviewer
>                 - the authors suppose that the reviewer has full knowledge
> of past work they discuss in the paper
>                 - Several paragraphs are not clear (e.g. the paragraph on
> the agent in the intro, eq 1, ...)
>
>         - the strategies FlexiPool is evaluated against are quite old.
>
>         - Section 2.1 : "... the next workload phase", how is this
> prediction done ?
>
>         - For the simulation part, it is not clear on which time window
> the simulation is performed: is it on time windows N-1 during time window N
> ?
>
>         - Why is the choice of cache policies the same as the cache
> policies on libCacheSim ? should the choice of the caching strategies be as
> random as that ?
>
>         - an overview figure of the contribution could be helpful.
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #120B
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 3. Knowledgeable
>
> Paper summary
> -------------
> This paper proposes FlexiPool, a framework that dynamically switches among
> multiple cache eviction policies to adapt to changing workloads. It reduces
> online switching overhead by simulating candidate policies in the
> background and preserving their respective cache states and metadata. The
> paper also introduces an LLM-based coding agent that automatically
> generates interpretable and editable switching rules from historical byte
> hit rates. Experiments show that FlexiPool generally outperforms several
> state-of-the-art cache eviction methods and matches or surpasses the best
> static policy on most traces.
>
> Comments for authors
> --------------------
> Thank you for submitting this paper to HotStorage 2026. The paper presents
> FlexiPool, which simulates multiple cache policies in the background and
> uses an LLM-based coding agent to generate executable switching rules. The
> paper has two main strengths.
>
> First, it addresses an important problem with clear motivation: different
> cache policies may perform best during different workload phases, while
> existing approaches are often limited to predefined policy sets or fixed
> switching rules.
>
> Second, its use of LLMs is relatively novel. Rather than producing
> black-box decisions, the agent generates executable, inspectable, and
> editable code, making the switching algorithm itself automatically
> generated and refinable.
>
> However, several issues prevent me from leaning toward acceptance.
>
> First, the paper lacks essential system-overhead experiments. The
> evaluation focuses mainly on BHR, but does not quantify the CPU, memory,
> storage-bandwidth, switching-latency, or tail-latency costs of background
> simulation and state migration. The cost of LLM-based code generation is
> also not reported. More importantly, the paper does not show whether these
> overheads offset the BHR gains or how they affect end-to-end throughput and
> latency.
>
> Second, FlexiPool’s overall gains are modest. On several Cloudflare
> traces, it only matches the best static policy. On CF1, the gain is 0.12
> percentage points, compared with an Oracle gap of 1.02 points, while the
> gains on CF5–CF8 are near zero. Given the additional complexity introduced
> by background simulation and LLM-based code generation, it is unclear
> whether these modest BHR improvements justify the proposed design.
>
> Third, the LLM-based coding agent is one of the key novelties of the
> paper. However, it is not yet clear why an LLM is the appropriate solution
> for this problem, rather than simpler rule-generation or program-synthesis
> techniques. A discussion of what unique capabilities the LLM provides, and
> why these capabilities are essential for generating effective switching
> policies, would better justify this design choice.
>
> Finally, the LLM-based coding agent is not described clearly enough. The
> paper provides no concrete examples of generated rules and lacks details
> about prompts, interfaces, feedback formats, stopping criteria, and the
> multi-round refinement process. This makes the core method difficult to
> understand and reproduce. The paper would benefit from representative
> generated rules and a clearer description of the agent workflow.
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #120C
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 3. Knowledgeable
>
> Paper summary
> -------------
> This paper presents FlexiPool, a cache-policy switching framework that
> supports an extensible pool of eviction policies. FlexiPool runs only the
> active policy on the request path, while periodically simulating candidate
> policies in the background and retaining their cache states and metadata
> for switching. The paper further uses an LLM-based coding agent to
> synthesize executable and editable switching rules from recent per-policy
> byte-hit-rate histories. Experiments using different traces show that
> FlexiPool generally matches the best static policy and outperforms
> baselines. On the Cloudflare traces, the agent-generated rules improve over
> recent-best Search, although the gains are generally small.
>
> Comments for authors
> --------------------
> The idea of separating request serving from background policy simulation
> is useful, and preserving each simulated policy’s warm state makes
> switching more flexible.
>
> However, the novelty of using an LLM to generate switching rules feels
> incremental, and the evaluation does not yet demonstrate a compelling
> benefit. Further, the agent is developed separately for each trace and
> cache-size configuration using training and validation segments, so the
> evaluation demonstrates offline workload-specific rule generation, rather
> than the claimed ability to adapt automatically to a new workload at
> runtime.
>
>
> * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
>
>
> Review #120D
> ===========================================================================
>
> Overall merit
> -------------
> 2. Weak reject
>
> Reviewer expertise
> ------------------
> 3. Knowledgeable
>
> Paper summary
> -------------
> This paper presents a dynamic switching eviction cache policy at runtime.
> It solves three challenges: 1) for the memory overhead of running multiple
> policies, it just run one policy for online requests, 2) for an expressive
> switching logic, it uses a LLM-based coding agent, 3) for removing bias
> toward specific phases or transient behavior, it uses a window with a
> train/validation split and selects the candidate with the best held-out
> validation performance.
>
> Comments for authors
> --------------------
> Strengths:
> • As workload behavior changes over time, the policy must be dynamically
> adjusted. This dynamic adjustment is clearly defined and adequately
> demonstrates the impact of policy changes.
> • It compares the work with the most related work, such as SOPA.
> • It considers cache replacement, memory‑cost pressure, and LLM agents for
> systems design, which is valuable.
>
> Weaknesses:
> • The overhead of the proposed approach is not presented in terms of CPU
> and memory utilization.
> • The evaluation is not complete. It does not show the switch latency
> between policies or the resulting impact on hit rate.
> • Some policies require specific metadata and data structures, and
> therefore switching to other policies requires high operational cost. This
> work does not address such policies.
> • The work does not discuss how to select the window time.
> • The improvement is low compared to SOPA (Table 2), and this paper does
> not discuss the overhead of their approach relative to SOPA.
> • The paper does not compare their approach against recent work such as
> "3L-CACHE: Low Overhead and Precise Learning‑based Eviction Policy for Web
> Caches" or "Learning Cache Replacement with CACHEUS".
> •  The paper focuses on trace-driven simulations using libCacheSim. There
> is no evaluation on a real-world workloads that have complexities.
> • The evaluation uses relatively short windows (100K requests) and
> specific traces. Generalization to completely different workload types or
> much longer time horizons has not been explored.
> • All experiments use only a 0.1% cache size relative to working-set size.
> This is extremely small and may favor certain policies; results may not
> generalize to larger, more typical cache sizes where behavior differs
> significantly.
>
> Comments for Authors:
> • It would be better to measure CPU and memory utilization and compare
> their approach against state‑of‑the‑art articles.
> • It would be better to implement their approach in an enterprise cache
> module (e.g., OpenCAS or EnhanceIO) to evaluate a real system rather than
> relying on simulation alone.
> • It would be better to classify policies that share the same metadata and
> data structures so that they can switch between them at runtime without
> requiring extra metadata.
> •   Including experiments on a live cache system or evaluating the
> framework under larger policy pools and more diverse workload
> characteristics would provide stronger evidence of FlexiPool's scalability
> and practical applicability.
> • Although the reported byte hit rate improvements are consistent,
> discussing their impact on end-to-end system performance (e.g., latency
> reduction, backend traffic, or infrastructure cost savings) would better
> demonstrate the practical value of the proposed approach.
> • Figure 1 is referenced but is not described in details in the
> manuscript. The caption alone is not descriptive.
> • In this manuscript only byte-miss cost is considered. In real systems,
> switching may involve CPU bursts, lock contention, or application stalls.
> Modeling or measuring these temporal costs would make the evaluation more
> realistic.
>
>
> Comment @A1 by Reviewer B
> ---------------------------------------------------------------------------
> Dear Authors,
>
> All reviewers agree that the paper addresses an important problem.
> However, there is broad consensus that the novelty and motivation for the
> LLM-based approach are not yet sufficiently justified, the reported
> performance gains are relatively modest, and the evaluation needs to be
> strengthened. Overall, the reviewers do not believe the current submission
> provides sufficient evidence to support acceptance. We encourage the
> authors to address these concerns in a future revision.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fsl.cs.sunysb.edu/pipermail/hotstorage-chairs/attachments/20260724/8044a88a/attachment-0001.htm>