Skip to content

[Feature] Implement withFilter for PartitionsTable and FilesTable #7792

@heye1005

Description

@heye1005

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Search before asking

  • I had searched in the issues and found no similar issues.

Motivation

PartitionsRead.withFilter and FilesRead.withFilter are currently no-op
stubs marked // TODO. As a result, predicates on system tables such as
SELECT * FROM t$files WHERE schema_id = 0 rely entirely on the upstream
engine to filter the result, even though the source could prune rows
much earlier.

Solution

Apply the predicate as a post-filter on the materialized row before projection.

For FilesRead, leaves referencing partition / bucket / level need to
be stripped because they are already pushed down by FilesScan — re-evaluating
them on the materialized row may throw ClassCastException due to type mismatch
(e.g. a String literal vs the BinaryString in the row).

PartitionsRead keeps the predicate as-is because PartitionsScan does not
push anything down and PartitionsTable schema does not contain bucket
/ level columns.

POC ready (already implemented locally with tests, will open PR once assigned).

Anything else?

No.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions