Add string built-ins for multi-line text formatting: ?indent, ?dedent, ?wrap, ?pad_lines

[gdisirio]

This adds four string built-ins that make it easier to format multi-line text, which is useful when generating source code, configuration files, documentation comments, and similar structured output.

• ?indent(prefix) — prepends prefix to each (non-empty) line.
• ?dedent(prefix) — removes prefix from the start of each line that has it (the inverse of ?indent).
• ?wrap(width, firstPrefix[, restPrefix]) — word-wraps the string to the given column width, with configurable per-line prefixes (handy for wrapped comment blocks).
• ?pad_lines(width[, fillChar]) — pads each line on the right to the given column. Unlike ?right_pad, which pads the string as a whole, this operates per line, which is useful for aligning multi-line text.

All four operate on the string they're applied to, have no side effects, and require no new configuration or language mode. Line breaks (LF, CR, CRLF) are recognized and preserved.

Backward compatibility

Purely additive — these are new built-in names, so existing templates are unaffected.

Testing & docs

• JUnit coverage in IndentAndWrapBuiltInTest.
• FreeMarker Manual reference entries added for each built-in (marked @since 2.3.35).
• ./gradlew check and ./gradlew manualOffline both pass.

Background

These were developed for a code-generation workflow (generating embedded C in ChibiOS, via FMPP), but they're generally useful for any multi-line output.

[ddekany]

I'm primarily commenting just to signtal that I have seen this and will eventually deal with it in depth. indent/dedent/wrap is something that we can certainly add as funcitonality.

pad_lines functionality feels a bit too special... which is not necessarily a blocker. But can you tell some use-case? Also I think since we have ?left_pad/?right_pad already, it should be called ?right_pad_lines. (Yes, there's no ?left_pad_lines, or not yet, but still, for consistency.)

The main doubt there is handling tabs, and even non-breaking-space characters. dedent assumes that you can provide exact prefix as string, but that's not very robust in face of inperfect input. Even indent can be affected by that approach, like some possibly wants specify the indentation width, and there tabs can play a role. The reason to think about these is that if we want to make this more advanced later, backard-comaptibility constratints can complicat that.

[gdisirio]

Thanks for the read! All three points are fair, here's how I'd address them.

On renaming ?pad_lines to ?right_pad_lines — yeah, you're right that's the consistent name with ?right_pad/?left_pad. Will do. On whether it's needed at all: it does something ?right_pad can't, which is pad each line of a multi-line string independently. So "a\nbb"?right_pad_lines(5) gives you "a \nbb \n" (each line padded to 5), whereas ?right_pad would treat the 4-char string as one and pad it to 5. It's the difference between aligning a column of values vs. padding a single value, and it shows up a lot when generating tabular output.

On tabs and non-breaking spaces: widths in all four built-ins are counted in Java chars (UTF-16 code units), not visual columns — same as ?right_pad/?left_pad. So a tab counts as one character, not as "advance to next multiple of 8" or whatever. I'll make that explicit in each built-in's manual entry — something like "widths are in characters, not display columns; expand tabs to spaces first if you need visual alignment."

Non-breaking spaces are an interesting case: ?wrap splits on \s+, and Java's \s doesn't include U+00A0, so non-breaking spaces stay inside a word and aren't break points. That's actually exactly what you'd want from a non-breaking space, but it's by happy accident rather than by design — I'll mention it in the docs so it's visible behavior.

On ?dedent — yeah, requiring an exact prefix match is fragile. I'll add a no-argument form: ?dedent (with no args) finds the longest common leading whitespace across all non-empty lines and removes that. Same semantics as Python's textwrap.dedent, which is the well-known reference for "do the obvious thing with imperfect input." The explicit ?dedent(prefix) form would stay for the cases where you actually do want exact control — it's not redundant, just less robust by design.

---

One related thought, since you brought up the tab handling: it might be useful to have built-ins for normalizing tabs and spaces explicitly. Something like ?expand_tabs(tabWidth) for column-aware tab-to-spaces conversion (semantics of Python's str.expandtabs / Unix expand), and an inverse for converting leading-whitespace runs to tabs at multiples of tabWidth (Unix unexpand-style, leading-only — otherwise you'd break alignment).

For code generation that's a real need — if the target project uses tabs, the generator needs to emit tabs; if it mandates 4-space indent, ditto. Having primitives for that means generators don't each have to roll their own. Not proposing them in this PR — just floating the idea, since it's the underlying capability you'd want behind the "expand tabs first" docs note. Happy to do a follow-up PR if it sounds worth having.

I'll push the revisions for the three current points in a day or two and ping you. Thanks again for engaging on this.

[gdisirio]

Pinging as promised — the revisions are pushed (commits ecfa044, 3abd7f0, 0b6e617), one per point you raised:

• ecfa044 — rename ?pad_lines → ?right_pad_lines (and the camelCase alias, the class, the manual section, and the index link).
• 3abd7f0 — adds the no-argument ?dedent() form with Python textwrap.dedent semantics (longest common leading whitespace across non-empty lines, removed from each). 8 new tests covering uniform/mixed/blank-line/tabs/no-common-prefix cases. The explicit ?dedent(prefix) form is kept for exact-control cases.
• 0b6e617 — manual notes on ?right_pad_lines and ?wrap: widths in Java chars (not visual columns), and that \s+ doesn't include U+00A0 so non-breaking spaces correctly stay inside words in ?wrap.

./gradlew check passes. No rush on the review — just letting you know it's ready when you get to it. The offer from my previous comment (a follow-up PR for ?expand_tabs / ?unexpand_tabs if of interest) still stands; happy to do that separately whenever.

[ddekany]

Have you signed an ICLA at Apache (see https://www.apache.org/licenses/contributor-agreements.html)? That will be needed.

And again, sorry for the slow review... Will try to take a close look this weekned.

[ddekany]

One thing I want to change is that currently we have ?dedent() and ?dedent(prefix), insead of the more idiomatic ?dedent and ?dedent(prefix). Like if you check ?url, that has the same logic. BUT, do not touch anything, I'm on it... will hopefully finish after work tomorrow.

[ddekany]

I was, perhaps naively, pushed into this branch my adjustments... hoping it will somehow magically do what I meant to do in the PR (i.e. adding a commit to it), but no. Anyway, see if you can bring that into your repo, and move the PR head on it: 4ab4d64

Now, the functional changes I did:

• ?dedent() => ?dedent, and ?dedent() is error
• ?wrap(width) is now valid (prefix defaults to "")

But, I have doubts about the designed (i.e. deliberate) behavior of some of the build-ints. Can you make your case regarding these? Like, show what the practical use-case is, and why this behavior is better than what I think would be more intuitive (we try operate on the principle of least surprise).

• ?dedent(prefix): Currently, only the lines that start with the whole parameter prefix are dedented, others are left as is. But this is anti-intutive, if you consider how dedent works in text editros for example. I think, intuitively, the longest matching prefix of the prefix should be removed from all lines. Like, for ?dedent(s, "---") all these lines will become to "a": "a", "-a", "--a", "---a". It's not obviously what the user wanted, but since we don't want to throw an exception, we have to do the least surprising wrong thing we can. This also produces less messed up output, if someone expects ?dedent(prefix)/?indent(prefix) roundtrips (because as it is now, dedent for some lines doesn't remove any indenation, while indent aways adds indentation, and so not just end up with a partially flattened indentation hierarchy, but with a mess where lines were originally less indented can become the most indented).

Update: Removed concerns regarding ?right_pad_lines and ?indent of an 0-length string. The idea is clearly that you have 0 lines there. But see concerns with empty lines in the next comment.

[ddekany]

My doubt with the designed behavior of ?right_pad_lines and ?indent is how they handle empty lines.

?indent will leave empty lines as is, but if you have something like ?indent("# "), that is, you want to comment out a block of text, that's probably not what you want. If the indentation is white-space only though, that's different, then maybe you want that. What you really want is probably always adding the indentation prefix, but then right-trim the resulting line. Because in the case of "# ", for empty lines you only want a "#", as the space in "# " was meant to be a separator in case the line is not empty. So maybe we want 2nd boolean argument to turn right trimmin on/off, and by default it should be true (which is a somewhat surprising behavior, but gives the (naively) expected output for most).

I have similar issue with ?right_pad_lines, but here I'm not sure what the practical use-case is. But at very least if the I use non-whitepsace filler, let's say "_", then I think you even want empty lines to be filled up with that. But again, please show what the practical use-cases are, otherwise it's hard to tell.

Also, when it comes to input line emptiness, especially for source code, you often want to treat white-space-only lines as "", because the white-space in it is always almost accidental there. So a multi-line text build-in not very robust in everyday use, if it creates a practical edge-case from whether a line is truly empty or has a few spaces or tabs.

Lastly, one thing that's surely missing is that ?right_pad_lines should support any non-0 length of filler, just like ?right_pad and ?left_pad does.

[gdisirio]

Thanks for doing the work rather than just describing it — the _CoreStringUtils split is clearly the right structure, and having the FTL-level tests separated from the transformation tests makes the argument-handling cases much easier to see.

The PR head is now on your commit.

One thing to flag, in case it bites you elsewhere: _CoreStringUtilsTest.java is missing the ASF license header, so ./gradlew check fails on the rat and ratDistSrc tasks with "1 unapproved license". _CoreStringUtils.java itself has it, so it looks like just the one file got missed. I've added the header in a follow-up commit; nothing else in check failed.

On the four design questions:

?dedent(prefix) — agreed, your semantics are better

You're right, and the roundtrip argument is what convinces me. Since ?indent always adds the prefix unconditionally, the current all-or-nothing ?dedent doesn't just flatten the hierarchy unevenly — it can invert it, leaving an originally-less-indented line as the most indented one. That's worse than either flattening or leaving everything alone.

The everyday case makes it clearer than the "---" example does: with ?dedent(" ") over a block where one line happens to carry two spaces, the current behaviour silently leaves that line untouched while dedenting everything around it. Removing the longest matching prefix of the prefix from every line is what an editor does and what people will expect.

My original reasoning was "exact prefix, exact control" — but that argument doesn't survive contact with imperfect input, which is exactly the case the parameter form is for. Please go with your version.

?indent and empty lines — agreed, including the default

Add the prefix unconditionally, then right-trim, with a boolean to turn the trimming off. This is neater than what I had, because one rule gives the right answer in both cases: "# " on an empty line produces "#" rather than a line with trailing whitespace, and a whitespace-only prefix on an empty line produces "", which is what happens today.

So defaulting the trim to true isn't a behaviour change for whitespace indentation at all — it only changes the case that's currently wrong. Agreed on true as the default.

Whitespace-only lines treated as empty — agreed, and it should be uniform

Agreed, and for generated source in particular. A line containing three spaces is almost always an accident of whichever loop emitted it, and making behaviour depend on that is exactly the kind of edge case that makes a built-in feel unreliable.

Worth noting the family is already inconsistent about this: the no-argument ?dedent ignores whitespace-only lines when computing the common prefix, while ?indent treats them as non-empty. Applying "whitespace-only counts as empty" across all four would settle it.

?right_pad_lines — the use case, and where I'd push back

The use case is column alignment in generated source. You build a block of lines, pad each to a common column, then append something per line:

list fields as f
  decls = decls + f.type + " " + f.name + ";\n"
endlist
emit decls?right_pad_lines(24) // then append per-line comments

giving

uint8_t  status;        /* Device status. */
uint32_t base_address;  /* Base address.  */

The same pattern shows up for aligning #define values, register bit-field tables, and assembly operands and comments. ?right_pad can't do it because the block is one string with embedded newlines — it would pad the whole thing to a single width, not each line to a column.

Filler length: agreed, it should accept any non-zero-length filler for consistency with ?right_pad/?left_pad. No reason for this one to be special.

Empty lines: here I'd argue for leaving them alone, which I realise cuts against your "_" intuition. The reason is that in the alignment use case a blank separator line inside the block is common, and padding it produces a line of 24 trailing spaces — which most code-style checks flag, and which shows up as noise in diffs of generated files. Generated code that emits trailing whitespace tends to come back as a bug report.

The "_"-as-ruler case is real but different in shape: it's normally a single line, not a block with blank separators in it, so leaving empty lines alone costs it nothing. If you'd rather have the choice available, this could be the same boolean-argument treatment as ?indent — but I'd default it to leaving empty lines empty.

Happy to implement whichever of these you want in this PR, or to leave them to you if it's faster — just say which.

[gdisirio]

On ?right_pad_lines — you asked what the practical use case is, so here's real code rather than my invented example. This is from the ChibiOS board-header generator, which predates these built-ins entirely.

The generated output looks like this (os/hal/boards/ST_STM32F3_DISCOVERY/board.h):

#define VAL_GPIOA_MODER             (PIN_MODE_INPUT(GPIOA_BUTTON) |         \
                                     PIN_MODE_INPUT(GPIOA_PIN1) |           \
[...]
                                     PIN_MODE_ALTERNATE(GPIOA_SPI1_SCK) |   \
                                     PIN_MODE_ALTERNATE(GPIOA_SPI1_MISO) |  \
                                     PIN_MODE_ALTERNATE(GPIOA_SPI1_MOSI) |  \
                                     PIN_MODE_INPUT(GPIOA_PIN8) |           \

A 16-line C macro, one line per GPIO pin, with the trailing \ continuations aligned to a common column. The template does it like this:

${(line + " |")?right_pad(76, " ") + "\\"}

That idiom appears 121 times across 19 board templates. Aligning trailing continuations in generated C macros isn't a niche case — it's a large part of what a header generator does, and the same shape shows up for trailing comments and for aligning #define values.

I should be straight about what this does and doesn't prove, though. Here the lines are produced one at a time inside a #list over the pins, so single-line ?right_pad is sufficient — this particular code wouldn't be shortened by ?right_pad_lines.

Where the multi-line form earns its place is when the block arrives as a single string rather than as a loop: the rendered body returned by a macro, the content of a multi-line literal, or the output of ?dedent. In those cases the only way to align it today is to split on newlines, loop, pad, and rejoin — reconstructing by hand exactly the loop that ?right_pad_lines would have made unnecessary. So the alignment requirement is pervasive and well evidenced; the multi-line form specifically is for when you have a block and don't want to destructure it just to pad it.

That also sharpens my earlier point about empty lines. In the trailing-comment variant of this pattern you pad a block and then append per line; a blank separator line inside the block, padded to column 76, gives you 76 trailing spaces in generated C. Hence my preference for leaving empty lines untouched by default.

One more thing from the same library, relevant to your empty-lines question rather than to ?right_pad_lines. Its hand-written indentation macro (tools/ftl/libs/libccode.ftl) decides emptiness like this:

[#list lines as line]
  [#local s = line?chop_linebreak]
  [#if !line?is_first || (s?trim?length > 0)]
    [#if s?trim?length > 0]
      [#if s[0] == "#"]
${s}
      [#else]
${indent + s}
      [/#if]
    [#else]

    [/#if]
  [/#if]
[/#list]

So it treats whitespace-only lines as empty (s?trim?length > 0, not s == ""), and emits blank lines with no prefix at all — which is what your add-prefix-then-right-trim rule produces for a whitespace prefix. That was written years before this PR and without reference to it, so I'd offer it as independent support for your position rather than mine.

It also shows one limit worth knowing about: that macro forces lines starting with # to column 0, because indenting preprocessor directives would be wrong. A general ?indent can't express that, so in real generators these built-ins tend to be building blocks inside a wrapper rather than the whole answer. Not an argument against any of your proposals — just why I wouldn't try to make ?indent cover every case.

[ddekany]

Note that I haven't done the proposed behavioral changes. I did some cleanup in my commit, but I half-assumed as PR maker you will want to do the other changes, so, go ahead.

[ddekany]

"In those cases the only way to align it today is to split on newlines, loop, pad, and rejoin" - Typically, you shoundn't do the rejoining though. You just print the lines one-by-one inside a #list, which allows you to use all the existing built-ins that doesn't consider multi-line input. If you were using ?right_pad_lines, then how do you even add the trailing / at the end of each line (after the padding)? So I wonder if we want ?right_pad_lines at all, or it's better just split and loop for a such "specialist" use-case.

[gdisirio]

Pushed as e0f32f00f — all three behavioral changes, plus the removal.

?dedent(prefix) now removes from each line the longest prefix of the parameter that the line actually starts with, so your "---" example gives "a" for all of "a", "-a", "--a", "---a".

?indent(prefix) now adds the prefix to every line and then right-trims each resulting line, with an optional 2nd boolean argument to switch the trimming off, defaulting to true.

Whitespace-only lines are now treated as empty in ?indent and in both ?dedent forms.

?right_pad_lines is gone — built-in, _CoreStringUtils methods, tests, manual section, index entry, and the changelog mention.

You're right about it, and the argument that settled it is the one you made: you don't rejoin, you print inside the #list. Once the lines are separate you have every single-line built-in available, and appending the trailing \ after the padding is trivial — whereas a whole-block built-in can't express it at all. That's fatal for the case I offered as the motivating one, which rather makes the point. It's a small function to write locally if anyone does want the block form.

Two things I decided while implementing, both of which you may want to look at:

I left ?wrap alone regarding whitespace-only lines. It splits on \s+ across the whole input, so the line structure is gone before wrapping starts and "a whitespace-only line" has no meaning there. Applying the rule uniformly would have meant inventing behavior, so I applied it where the concept exists.

The no-argument ?dedent now normalizes whitespace-only lines to empty, where it previously passed them through verbatim. That's slightly beyond what you asked for, but "whitespace-only counts as empty" isn't true otherwise, and it's what textwrap.dedent does — which the docs cite as the reference. Easy to revert if you'd rather keep it conservative.

Also added getBooleanMethodArg / getOptBooleanMethodArg to BuiltIn, next to the existing string and number variants, since ?indent's new argument needed one.

./gradlew check and ./gradlew manualOffline both pass. _CoreStringUtilsTest is at 41 tests and IndentAndWrapBuiltInTest at 16, with the new cases covering your "---" example, "# " on a blank line, the trimming switched off, non-breaking-space preservation, and an ?indent/?dedent roundtrip.

[ddekany]

My new commint in this branch (please integrate into PR as previosuly): 621a8ac

This time I have read through the text processing algorithms, and cleaned them up (feelt quite AI-ish at places). There's no functional change.

Now, what's left:

• I think it's odd that ?dedent(prefix) doesn't trim blank lines, because the others do that. Like it makes an assymetry with ?indent, which does that. So s?indent(" ")?dedent(" ") is not the same as s?dedent(" "). This behavior is emphasized by testDedentWhitespaceOnlyLineIsNotTrimmed (which matches the current behavior). Maybe, ?dedent(prefix) should have the optional trim argument like ?indent, also defaulting to true.

• The test cases in _CoreStringUtils Test should be reviewed by human eyes. Also, they should become parameteric tests (we are on JUnit 4, but that still have that), so that we test for all 3 line ending types, and probably even with and without a line-break at the end of the input string.

[gdisirio]

This contribution is AI-assisted so that is not surprising, I hope this was clear from beginning. I will review your points.

[gdisirio]

Drafted by Claude (an AI coding assistant) working on this branch with me. The decisions described below — the empty-prefix no-op, the test structure, and dropping ?right_pad_lines — were discussed with me and are mine; the implementation and this write-up are its work.

---

Your commit is in, and both points are done — pushed as 99bcddc62.

?dedent(prefix) now trims too, with an optional 2nd boolean argument to switch it off, defaulting to true, exactly as ?indent. You were right that the asymmetry was the real problem: a whitespace-only line loses only as much whitespace as the prefix is long, so with a 4 character prefix a line of 5 spaces kept 1, and ?indent(p)?dedent(p) didn't round-trip for such lines. testDedentWhitespaceOnlyLineIsNotTrimmed is now ...IsTrimmed, with the old expectation kept as a separate test for the false case.

The tests are parametric now. _CoreStringUtilsTest is split into two nested classes under Enclosed:

• IndentAndDedentTest is @RunWith(Parameterized.class) over the 3 line-break types × with/without a trailing line-break. Test data is written with "\n" and no trailing break, and a small lb() helper adapts both the input and the expected value, so assertIndent(" a\n b", "a\nb", " ") also covers the CRLF and CR forms and the trailing-break variants. 31 methods × 6 combinations = 186 tests.
• WrapTest stays plain: ?wrap collapses all whitespace including line-breaks, so its output can't depend on the line-break type of the input. Parameterizing it would have run the same assertions 6 times for nothing.

This subsumed the tests that existed only to pin a specific line ending or the presence of a trailing break, including the one you marked !!T. Worth noting the new coverage found no bugs — CR and CRLF were already handled correctly, including in the no-argument ?dedent; what was missing was the evidence.

One decision I'd like you to check, as it's a behavior change you didn't ask for. While adding the trim argument I'd made an empty prefix stop being a short-circuit, so ?indent("") and ?dedent("") became pure right-trimmers. That seemed consistent — "trimming happens unless disabled" — but it's surprising for something called indent: ?indent("") shouldn't quietly modify the string. So an empty prefix is now a no-op for both, and doesn't trim either. Say if you'd rather have it trim.

On the tests being reviewed by human eyes: fair, and they have been now. Two things came out of it beyond the empty-prefix decision. The 4th assertion in testWrapWithNbsp asserts that an NBSP-joined token is kept whole even when it overflows width — that's intended, since you can't wrap what can't be split, but it is behavior-defining rather than obviously right, so it's worth your eye too. And in testDedentWhitespaceOnlyLineIsTrimmed the expected value ends with a line-break even though the convention in that class is to write data without one; that's correct, as its last line is genuinely empty, but it reads like a violation of the convention.

./gradlew check and ./gradlew manualOffline both pass.