feat(upload): add multipart uploader with resumable chunked transfer#577
feat(upload): add multipart uploader with resumable chunked transfer#577thetapilla wants to merge 7 commits into
Conversation
- Add a chunked resumable uploader driving /api/fs/multipart with three concurrent ascending chunk requests - Resume interrupted sessions transparently from init's received ranges and fall back to Stream for single-chunk files - Treat flow control (429/409) and transient network errors as retryable, probing session status before declaring failure - Stop sending remaining chunks once the server reports the session completed (rapid upload) - Register the uploader gated by the multipart_enabled public setting
- Add a Retry button on failed rows that reuses the kept File handle, resuming multipart uploads from the breakpoint - Disable "add as task" while the multipart uploader is selected, since sessions are synchronous pipelines
- Report progress and speed from a high-water mark: a chunk rejected by flow control or a network hiccup drops its in-flight bytes from the running sum until it is resent, which made the displayed speed go negative and the progress bar jump backwards - Rejected chunks now show as a progress plateau with speed falling to zero
- The chunk size setting accepts any positive integer now; mention the 8-chunk per-session server buffer instead of the removed 1-90 range
| } catch (_) { | ||
| // network hiccup, or the server fast-rejected without draining the | ||
| // body (flow control) and the browser reported it as an error | ||
| resp = undefined | ||
| } |
There was a problem hiding this comment.
Axios 拦截器使 catch 块对网络错误失效
Axios 的响应拦截器错误处理器将网络错误转换为 resolved 值返回：
OpenList-Frontend/src/utils/request.ts
Lines 33 to 45 in 1e059b1
这意味着 r.put(...) 对于网络错误不会 reject，而是 resolve 为 { code: undefined, message: "Network Error" }。于是 multipart.ts 中的 catch 块几乎不可达：
try {
resp = await r.put(...)
} catch (_) {
// 这段代码对 Axios 拦截的错误不会被触发
resp = undefined
}由于 resp.code 是 undefined（不是 429 也不是 409），直接 throw 导致整个上传中止：
if (resp && resp.code !== 429 && resp.code !== 409) {
throw new Error(resp.message)
}这破坏了代码中原设计用于瞬态网络故障时的重试逻辑。对于快速上传完成后的网络错误，也无法正确降级。
建议修复：将 try/catch 改为检查 resp.code：
// 方案 A：修改 catch 逻辑以兼容 Axios 拦截器的行为
try {
resp = await r.put(...)
} catch (_) {
resp = undefined
}
// 改为显式检查
if (!resp || typeof resp.code !== 'number' || resp.code === undefined) {
resp = undefined
}或者更彻底的方案：让错误拦截器区分 HTTP 业务错误（保留原样）和网络错误（仍然 reject）。
There was a problem hiding this comment.
错误成立。逐调用点排查同类错误，发现同一病灶实际有四处，已在 72e26dc 一并修复：
sendChunk（review 指出）：网络错误落进code !== 429/409分支即硬失败，导致无法自动重发；- init 的重试循环：catch 永不触发，重试等于不存在；
- complete 的
.catch(() => undefined)：CDN 掐断 complete 是设计上的预期场景，但轮询兜底整段不可达——未勾秒传（无哈希）时 receiving 会话不可复用，手动重试等于全量重传，这处影响最大； - 终态轮询的
catch { continue }：收尾阶段一次断连会误杀整个上传。
实际修法与建议的方案 A 有两点偏差，说明原因：
- 判别式没有用
typeof resp.code !== 'number'：CDN/网关产生的 502/524 会带真实 HTTP 状态码进错误拦截器（error.response?.status是数字），按 typeof 判别会把它们当成业务错误硬失败，而这恰是最需要自动重试的错误类型。改为按"是否服务器 envelope"判别：业务响应恒为 HTTP 200、且Resp的data字段无 omitempty 序列化，故typeof code === "number" && "data" in resp即 envelope，拦截器产物只有{code, message}必然落空；另特判code === -1（取消）终止而非无限重试。决定统一复用为一个mpRequesthelper 并应用于全部六个调用点。 - 没有采纳"让拦截器区分网络错误重新 reject"：全站所有调用方目前都遵循"永不 reject"，改
request.ts影响不可控，修复保持控制在 multipart.ts 内。
测试：8 个回归场景（正常上传/流控 429/业务 400/秒传/小文件回退/断点续传/重灌/取消）与修复前行为一致；4 个故障场景（chunk 网络抖动、chunk 502、complete 响应被吞、init 抖动）旧代码硬失败，新代码全部自动恢复。测试使用故障注入手动复现服务端错误。
| const fallbackThreshold = | ||
| Math.max(1, getSettingNumber("multipart_chunk_size", 10)) * 1024 * 1024 |
There was a problem hiding this comment.
X-Chunk-Size 未发送给服务端
前端读取了 multipart_chunk_size 设置用于计算回退阈值，但没有将其作为 X-Chunk-Size 头发送给服务端。而服务端 multipartChunkSize() 使用自己的默认配置（也可能是 10MB）。如果前端的设置与服务端不一致，分片阈值判断可能出现偏差。虽然实际的切片逻辑使用服务端返回的 session.chunk_size，数据完整性不受影响，但用户配置无法生效。
建议修复：在 initHeaders 中加入 X-Chunk-Size：
if (getSettingNumber("multipart_chunk_size", 10) > 0) {
initHeaders["X-Chunk-Size"] = getSettingNumber("multipart_chunk_size", 10) * 1024 * 1024
}There was a problem hiding this comment.
已在 041572a 补上：initHeaders["X-Chunk-Size"] = fallbackThreshold
回退阈值本身就是字节数的"建议分片大小"，于是直接复用同一个数。
前端 getSettingNumber("multipart_chunk_size") 与服务端 setting.GetInt(conf.MultipartChunkSize) 读的是同一个 PUBLIC 设置，正常不存在"前端与服务端不一致"；切片一致性也始终由 init 回传的 session.chunk_size 保证。这个 header 有实质收益当且仅当场景为设置刚被修改、旧标签页尚未刷新的窗口期上传。服务端给建议值的逻辑为 OpenListTeam/OpenList#2723 里 server/handles/multipart.go 的这个函数：
const multipartMinChunkSize = int64(1) << 20 // 1MB
// multipartChunkSize resolves the effective chunk size. The admin setting is
// the ceiling: a client may suggest a smaller chunk via X-Chunk-Size but never
// a larger one — the server buffers a window of several chunks per session, so
// an unbounded client suggestion would translate directly into server-side
// disk usage.
func multipartChunkSize(requested int64) int64 {
ceiling := int64(setting.GetInt(conf.MultipartChunkSize, 10)) << 20
if ceiling < multipartMinChunkSize {
ceiling = multipartMinChunkSize
}
size := ceiling
if requested > 0 && requested < ceiling {
size = max(requested, multipartMinChunkSize)
}
return size
}令 F = 标签页缓存值 = 前端发送的 requested，S = 服务端当前设置 = ceiling：
| 场景 | 回传 chunk_size | 与不发的差异 |
|---|---|---|
| F = S（稳态，绝大多数时间） | requested == ceiling → 取 ceiling | 无差异 |
| F > S（管理员刚下调） | 超上限 → 返回 S | 无差异 |
| F < S（管理员刚上调） | requested < ceiling → 采纳 F | 有实质差异 |
F < S 举例：标签页缓存 10MB、服务端已上调为 100MB，上传 50MB 文件——标签页按 10MB 阈值判定"该走分块"，不发 header 时服务端会按 100MB 切出单分片（一个 50MB 请求体外加 init/complete 两个陪跑请求）；发 header 后按 10MB 切 5 片。
另外没有沿用建议片段里的 if ... > 0：发送值直接复用上文的 fallbackThreshold 并已经有 Math.max(1, …)，且这个设置经管理端校验只能保存 ≥1 的整数、服务端对建议值还有 [1MB, 管理值] 的限制。
The shared axios instance resolves transport failures into a bare
{code, message} instead of rejecting, so every try/catch in multipart.ts
was unreachable: a network hiccup on a chunk failed the whole file, the
init retry loop never engaged, the CDN-cut fallback for complete (poll
to the verdict) was dead code, and a blip during the final poll killed
an upload that was finishing fine.
Route every request through a helper that tells genuine server
envelopes (numeric code with a data key, always HTTP 200) apart from
transport-layer failures (no data key; CDN-level errors carry a numeric
HTTP status), returns undefined for the latter so the existing
probe/retry paths finally engage, and turns cancellation (-1) into a
terminating error.
Send X-Chunk-Size with the same value the multipart-vs-stream fallback decision was made with. The server clamps it to [1MB, admin setting], so this is a no-op in steady state; when an admin has just raised the setting and a stale tab still holds the old value, it keeps the session's chunking consistent with the tab's own fallback reasoning instead of degenerating into a single oversized chunk.
| { | ||
| name: "Multipart", | ||
| upload: MultipartUpload, | ||
| available: () => getSettingBool("multipart_enabled"), | ||
| }, |
There was a problem hiding this comment.
Done in 7d55451.
或许可以问问其他 Member 该不该放第一项，我肯定是没意见的。
不过小文件也能回退到 Stream 就是了
2 participants
Summary / 摘要
为上传界面新增 Multipart 上传方式（配套后端分块上传 API），解决经 CDN 反代（如 Cloudflare 免费计划 100MB 请求体上限）时网页端无法上传大文件的问题。
用户可感知的变化：
Multipart选项（由后端multipart_enabled公共设置门控，后端未启用或版本过旧时不显示，自动向后兼容）实现要点：
src/pages/home/uploads/multipart.ts：3 并发升序分片调度；init 返回的已收区间直接驱动续传；秒传完成信号使剩余分片停发兼容性说明：不改变既有 Stream / Form / HTTP Direct 上传方式的任何行为（对现有代码的修改均以"选中 Multipart"为条件门控）；重试按钮对所有上传方式的失败行生效（非 multipart 方式点击重试等效于重新上传该文件）。
/ 此 PR 包含破坏性变更。
/ 此 PR 修改了公开 API、配置、存储格式或迁移行为。
/ 此 PR 需要关联仓库同步修改。
Related repository PRs / 关联仓库 PR:
Related Issues / 关联 Issue
Relates to OpenListTeam/OpenList#460
Testing / 测试
测试平台：macOS arm64，Node 26 / pnpm 11。
go test ./...不适用前端仓
自动化检查：
pnpm build构建通过pnpm lint（tsc）：本 PR 未引入任何新类型错误（仓库现存 4 个与本 PR 无关的既有错误，干净分支上同样存在）浏览器手动验证（对接本地后端配套分支联调）：
multipart_enabled后下拉框不再显示 Multipart 选项Checklist / 检查清单
/ 我已阅读 CONTRIBUTING。
/ 我确认此贡献符合仓库许可证、贡献规范和行为准则。
gofmt,go fmt, orprettierwhere applicable./ 我已按适用情况使用
gofmt、go fmt或prettier格式化变更代码。/ 我已在适用情况下请求相关维护者或代码所有者审查。
AI Disclosure / AI 使用声明
/ 此 PR 包含 AI 辅助内容。
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Refactoring / 重构
Documentation / 文档
Tests / 测试
Translation / 翻译
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
/ 我已审核并验证此 PR 中的所有 AI 辅助内容。
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution./ 我已确保所有 AI 辅助提交都包含
Co-Authored-By归属信息。I can reproduce all AI-assisted content included in this PR without any AI tools.
/ 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。