Skip to content

fix: mentions for users with UTC-16 characters#222

Merged
highesttt merged 2 commits into
mainfrom
highest/plat-38040
Jul 24, 2026
Merged

fix: mentions for users with UTC-16 characters#222
highesttt merged 2 commits into
mainfrom
highest/plat-38040

Conversation

@highesttt

Copy link
Copy Markdown
Collaborator

No description provided.

@linear-code

linear-code Bot commented Jul 24, 2026

Copy link
Copy Markdown

PLAT-38040

@indent-zero

indent-zero Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor
PR Summary

Fixes LINE mention offset handling so MENTIONEES.S/E are treated as UTF-16 code units (matching the LINE Chrome Extension / JavaScript String.length semantics) instead of UTF-8 bytes. Without this, any emoji or multi-byte BMP character (CJK, Thai, etc.) before a mention corrupted the resulting formattedBody link target on receive and produced wrong offsets on the LINE wire on send.

  • Add pkg/connector/text_offsets.go with utf16OffsetToByteIndex / byteIndexToUTF16Offset helpers that walk runes, count supplementary-plane code points as 2 UTF-16 units, and return false for negative, mid-surrogate, or mid-rune inputs, plus a resolveMentionRange helper that parses S/E strings and returns validated byte indices.
  • Update both user and @room branches in convertLineMessage (handle_message.go) to use resolveMentionRange when converting incoming S/E for HTML link building and @room body substitution.
  • Update both room-mention and per-user-mention paths in buildMentionMetadata (send_message.go) to translate the byte positions returned by strings.Index back into UTF-16 offsets before writing MENTIONEES.
  • Add unit tests covering emoji + Thai/CJK bodies for the receive path (user mention and @room), plus outgoing user and @room mentions with mixed surrogate-pair and multi-byte BMP characters.

Issues

All clear! No issues remaining. 🎉

1 issue already resolved
  • Duplicated mention-offset conversion between the user (M) and @room (A == "1") branches in convertLineMessage — the two nearly identical 8-line blocks only differ in the mxid field, and any future change (e.g. clamping/logging) will need to be applied twice. Consider factoring into a small helper like resolveMentionRange(unwrappedText, ment.S, ment.E) (start, end int, ok bool). (fixed by commit 54180d2)

CI Checks

All CI checks passed on 54180d2.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1541d3b0-3fc2-440e-bbaf-fb649df260b9

📥 Commits

Reviewing files that changed from the base of the PR and between ffe73d9 and 54180d2.

📒 Files selected for processing (3)
  • pkg/connector/handle_message.go
  • pkg/connector/handle_message_test.go
  • pkg/connector/text_offsets.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/connector/handle_message_test.go
  • pkg/connector/handle_message.go
📜 Recent review details
⏰ Context from checks skipped due to timeout. (4)
  • GitHub Check: build-docker
  • GitHub Check: Lint with 1.25
  • GitHub Check: Lint with 1.25
  • GitHub Check: build-docker
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go

📄 CodeRabbit inference engine (AGENTS.md)

**/*.go: Use go fmt for code formatting across all Go files
Use goimports with -local "github.com/highesttt/matrix-line-messenger" flag to group project-local imports correctly
Use zerolog for logging throughout the codebase
Do not use Msgf in logging; use Msg with structured fields instead
Use Stringer interface where applicable in Go code

Files:

  • pkg/connector/text_offsets.go
**/!(ltsm)/**/*.go

📄 CodeRabbit inference engine (AGENTS.md)

**/!(ltsm)/**/*.go: Run staticcheck on all Go files excluding pkg/ltsm package (transpiled WASM code)
Run go vet on all Go files excluding pkg/ltsm package (transpiled WASM code)

Files:

  • pkg/connector/text_offsets.go
🔇 Additional comments (1)
pkg/connector/text_offsets.go (1)

3-21: LGTM!

Also applies to: 23-51, 53-77


📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Fixed LINE mention handling to use UTF-16 code-unit offsets, ensuring mention links work correctly when messages include emoji or other non-BMP characters.
    • Corrected mention range conversion for received and sent messages, preventing mismatched start/end positions.
    • Improved validation so only well-formed mention ranges are converted into mention links.
  • Tests
    • Added unit tests covering UTF-16-based mention offset conversion for both user and room mentions.

Walkthrough

LINE mention metadata now uses UTF-16 code-unit offsets consistently. New conversion helpers support outbound metadata generation and inbound mention parsing, with tests covering user mentions, room mentions, emoji, MXID mapping, and formatted HTML links.

Changes

LINE mention offset handling

Layer / File(s) Summary
UTF-16 offset conversion helpers
pkg/connector/text_offsets.go
Adds conversions between UTF-16 offsets and UTF-8 byte indices, rejecting invalid or mid-rune boundaries.
Outbound mention metadata
pkg/connector/send_message.go, pkg/connector/send_message_test.go
User and room mention metadata now records UTF-16 offsets, with tests validating serialized offsets and mention targets.
Inbound mention parsing
pkg/connector/handle_message.go, pkg/connector/handle_message_test.go
Incoming LINE mention offsets are converted to byte ranges before creating mention entries, with coverage for text containing a non-BMP emoji.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • beeper/line#218: Both changes modify convertLineMessage mention handling for multibyte text.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Description check ❓ Inconclusive No actual PR description was provided, so there is no meaningful description to assess. Add a brief PR description summarizing the UTF-16 mention-offset fix and its impact.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main change: mention handling for UTF-16-based offsets.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch highest/plat-38040

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/connector/send_message.go`:
- Around line 966-974: Update the room-mention matching logic surrounding
byteIndexToUTF16Offset to perform case-insensitive searches directly against
body rather than a lowercased copy. Preserve the original body byte indexes for
`@room`, `@all`, and `@everyone` matches so mentionEntry generation is not dropped for
UTF-8 content.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a3eda935-c2c4-4272-a88a-65d4ff4fea53

📥 Commits

Reviewing files that changed from the base of the PR and between 412eda7 and ffe73d9.

📒 Files selected for processing (5)
  • pkg/connector/handle_message.go
  • pkg/connector/handle_message_test.go
  • pkg/connector/send_message.go
  • pkg/connector/send_message_test.go
  • pkg/connector/text_offsets.go
📜 Review details
⏰ Context from checks skipped due to timeout. (4)
  • GitHub Check: build-docker
  • GitHub Check: Lint with 1.25
  • GitHub Check: build-docker
  • GitHub Check: Lint with 1.25
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go

📄 CodeRabbit inference engine (AGENTS.md)

**/*.go: Use go fmt for code formatting across all Go files
Use goimports with -local "github.com/highesttt/matrix-line-messenger" flag to group project-local imports correctly
Use zerolog for logging throughout the codebase
Do not use Msgf in logging; use Msg with structured fields instead
Use Stringer interface where applicable in Go code

Files:

  • pkg/connector/text_offsets.go
  • pkg/connector/handle_message_test.go
  • pkg/connector/handle_message.go
  • pkg/connector/send_message_test.go
  • pkg/connector/send_message.go
**/!(ltsm)/**/*.go

📄 CodeRabbit inference engine (AGENTS.md)

**/!(ltsm)/**/*.go: Run staticcheck on all Go files excluding pkg/ltsm package (transpiled WASM code)
Run go vet on all Go files excluding pkg/ltsm package (transpiled WASM code)

Files:

  • pkg/connector/text_offsets.go
  • pkg/connector/handle_message_test.go
  • pkg/connector/handle_message.go
  • pkg/connector/send_message_test.go
  • pkg/connector/send_message.go
🔇 Additional comments (5)
pkg/connector/text_offsets.go (1)

5-58: LGTM!

pkg/connector/send_message.go (1)

1009-1017: LGTM!

pkg/connector/send_message_test.go (1)

4-79: LGTM!

pkg/connector/handle_message.go (1)

483-503: LGTM!

pkg/connector/handle_message_test.go (1)

167-202: LGTM!

Comment on lines +966 to +974
start, startOK := byteIndexToUTF16Offset(body, pos)
end, endOK := byteIndexToUTF16Offset(body, pos+matchLen)
if startOK && endOK {
mentionees = append(mentionees, mentionEntry{
S: strconv.Itoa(start),
E: strconv.Itoa(end),
A: "1",
})
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
python3 - <<'PY'
text = "K `@ROOM`"
lower = text.lower()
assert text.encode().index(b"@") == 4
assert lower.encode().index(b"@") == 2
print("Lowercasing changed the UTF-8 byte coordinate of @.")
PY

Repository: beeper/line

Length of output: 201


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the relevant function and inspect the surrounding lines.
file="pkg/connector/send_message.go"
sed -n '930,1025p' "$file" | cat -n

Repository: beeper/line

Length of output: 3476


Keep room-mention matching in body’s byte space. strings.ToLower(body) can change UTF-8 byte offsets before the @room/@all/@everyone match, so byteIndexToUTF16Offset may reject the position and silently drop the room mention. Match case-insensitively on body directly so the indexes stay aligned.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/connector/send_message.go` around lines 966 - 974, Update the
room-mention matching logic surrounding byteIndexToUTF16Offset to perform
case-insensitive searches directly against body rather than a lowercased copy.
Preserve the original body byte indexes for `@room`, `@all`, and `@everyone` matches
so mentionEntry generation is not dropped for UTF-8 content.

Comment thread pkg/connector/handle_message.go
@highesttt
highesttt merged commit 52c22da into main Jul 24, 2026
10 checks passed
@highesttt
highesttt deleted the highest/plat-38040 branch July 24, 2026 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant