Skip to content

Add top-level Text function for BeautifulSoup-style text extraction - #596

Open
ChrisJr404 wants to merge 2 commits into
PuerkitoBio:masterfrom
ChrisJr404:text-toplevel-func
Open

Add top-level Text function for BeautifulSoup-style text extraction#596
ChrisJr404 wants to merge 2 commits into
PuerkitoBio:masterfrom
ChrisJr404:text-toplevel-func

Conversation

@ChrisJr404

Copy link
Copy Markdown

This adds goquery.Text(s *Selection, opts *TextOptions), a top-level counterpart to the Selection.Text method that gives control over how text is extracted, so callers can get clean, readable text out of a document without hand-rolling a node walk.

This is the feature discussed in #443, where the request was to offer something similar to Python BeautifulSoup's get_text. As suggested there, it is a package-level function rather than a Selection method (to keep the method set aligned with the jQuery API, like NodeName and OuterHtml), and it is "general enough" via a small options struct:

  • Separator is inserted between the contents of consecutive text nodes.
  • Trim strips leading/trailing whitespace from each text node and drops the ones that become empty (the insignificant whitespace from source indentation).
  • Keep is an optional predicate called per text node; returning false excludes it, which is how a caller drops the text of <script>/<style> elements by inspecting the node's parent.

A nil (or zero-value) TextOptions makes Text behave exactly like Selection.Text, so the default is unsurprising and the traversal semantics match the existing method.

Example:

text := goquery.Text(doc.Find("#content"), &goquery.TextOptions{
    Separator: " ",
    Trim:      true,
    Keep: func(n *html.Node) bool {
        if p := n.Parent; p != nil && p.Type == html.ElementNode {
            return p.Data != "script" && p.Data != "style"
        }
        return true
    },
})

Tests cover the nil-options equivalence to Selection.Text, the separator/trim behaviour, the Keep filter, and a multi-node selection; a runnable ExampleText is included as documentation. go test ./..., go vet ./..., and gofmt are all clean.

Closes #443.

Adds goquery.Text(s, *TextOptions), a package-level counterpart to the
Selection.Text method that gives control over how the text of distinct
text nodes is joined (Separator), whether each node is trimmed of
surrounding whitespace (Trim), and which text nodes are included (Keep).
This covers the common need to extract clean, readable text from a
document - for example joining fragments with a space and dropping the
text of script/style elements - without hand-rolling a node walk.

Passing a nil TextOptions keeps the behaviour identical to Selection.Text.
Includes tests and a runnable example.
@mna

mna commented Aug 22, 2026

Copy link
Copy Markdown
Member

Hello Chris,

Thanks for this PR! Took a quick glance this morning and it looks great, I will wait to merge for when I have a bit more time to dive into it. In the meantime, I noticed some edge cases that should probably be covered by a test, if you have a moment to update it - I'm thinking of calling goquery.Text with an empty selection (e.g. doc.Find that doesn't match anything) and one where every node is filtered-out by the Keep option.

Thanks again,
Martin

@ChrisJr404

Copy link
Copy Markdown
Author

Added the two cases you asked for, empty selection and everything filtered out by Keep. Thanks, Martin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to achieve the effect of BeautifulSoup get_text?

2 participants