Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "DataPoly Dev (JDK8 + MySQL8 + PG14)",
"name": "DataPoly Dev (JDK25 + MySQL8 + PG14)",
"dockerComposeFile": "docker-compose.yml",
"service": "dev",
"workspaceFolder": "/workspace",
Expand Down
4 changes: 2 additions & 2 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
version: '3.8'

# DataPoly 开发/冒烟环境(Dev Containers):
# dev —— JDK8+Maven 开发容器(仓库挂载 /workspace,复用宿主 ~/.m2 缓存),三节点在本容器内启动
# dev —— JDK25+Maven 开发容器(仓库挂载 /workspace,复用宿主 ~/.m2 缓存),三节点在本容器内启动
# mysql —— 8.0,mysql_native_password(免三期冒烟时的 ALTER USER 步骤)
# postgres —— 14(PG 元库冒烟用)
# 用法见 AGENTS.md 第六节「devcontainer 冒烟方法」。

services:
dev:
image: maven:3.8.7-eclipse-temurin-8
image: maven:3.9-eclipse-temurin-25
container_name: datapoly_dev
command: sleep infinity
volumes:
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,21 @@ concurrency:

jobs:
test:
name: Build & Test (JDK 8)
name: Build & Test (JDK 25)
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4

- name: Set up JDK 8
- name: Set up JDK 25
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '8'
java-version: '25'
cache: maven

- name: Build and run tests
run: >
mvn -B -ntp test
-pl datapoly-common,datapoly-template,datapoly-core,datapoly-executor,datapoly-gateway,datapoly-manager
-pl datapoly-test
-am
20 changes: 14 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,22 @@

> 安全弱点上报见 SECURITY.md;内部评审细节不入库。

Maven 多模块,BSD-3-Clause。本机构建与 CI 统一 JDK 8(temurin/zulu 均可),产物即成 Java 8 字节码(无需本机安装 JDK 25);宿主机扩展构建(build-extension.sh)JDK 8 优先,无 8 时允许 8 以上(编译目标钉在 1.8,产物仍为 Java 8 字节码)。CI(temurin 8)只跑 common/template/core/executor/gateway/manager 六模块的测试
- lombok 1.18.46、groovy 4.0.33(org.apache.groovy)为钉版勿回退(JDK 8 与 25 下均验证可用);注解处理器依赖必须走 `annotationProcessorPaths`。
- 三服务经 Eureka 互联:manager(8090,Liquibase 唯一迁移执行方)、executor(8092)、gateway(8091 唯一对外入口)。前端 datapoly-manager-ui 非 Maven:manager resources 下的 `index.html` 与 `static/` 为构建产物 **不入库**(已被 .gitignore 排除,勿提交/勿 git add -f),打包前由根目录 `build-ui.sh`(node:23-alpine 容器,本机无需 Node)生成——`build.sh` 与 `docker-maven-build.sh` 已在它之前前置 `build-extension.sh`(宿主扩展装配:配置 `DATAPOLY_EXTENSION_GIT_URL` 或 `datapoly-extension` 目录存在即构建,否则无操作)及该 UI 步骤;纯 `mvn package` 的 jar 不含 UI。`build-ui.sh debug` 产出 devtools 可用的调试构建(`DATAPOLY_UI_ENV=debug`,NODE_ENV=development),仅限本机联调、勿随发行版发布;`build.sh`/`docker-maven-build.sh` 会透传首参给 build-ui.sh。
Maven 多模块,BSD-3-Clause。本机构建与 CI 统一 JDK 25(LTS;temurin/homebrew 均可),产物即成 Java 25 字节码;宿主机扩展构建(build-extension.sh)同样要求 JDK ≥25(编译目标钉在 25,低于 25 直接中止)。CI(temurin 25)经 `mvn test -pl datapoly-test -am` 运行测试:各模块测试统一集中在 datapoly-test 模块(JUnit 4 + 手写 fake,包名与被测模块保持同包以访问 package-private 成员),其余模块不含 src/test
- lombok 1.18.48、groovy 4.0.33(org.apache.groovy)为钉版勿回退(JDK 25 基线验证可用);注解处理器依赖必须走 `annotationProcessorPaths`。
- 三服务经 Eureka 互联:manager(8090,Liquibase 唯一迁移执行方)、executor(8092)、gateway(8091 唯一对外入口)。前端 datapoly-manager-ui 非 Maven:manager resources 下的 `index.html` 与 `static/` 为构建产物 **不入库**(已被 .gitignore 排除,勿提交/勿 git add -f),打包前由根目录 `build-ui.sh`(node:24-alpine 容器,本机无需 Node)生成——`build.sh` 与 `docker-maven-build.sh` 已在它之前前置 `build-extension.sh`(宿主扩展装配:配置 `DATAPOLY_EXTENSION_GIT_URL` 或 `datapoly-extension` 目录存在即构建,否则无操作)及该 UI 步骤;纯 `mvn package` 的 jar 不含 UI。`build-ui.sh debug` 产出 devtools 可用的调试构建(`DATAPOLY_UI_ENV=debug`,NODE_ENV=development),仅限本机联调、勿随发行版发布;`build.sh`/`docker-maven-build.sh` 会透传首参给 build-ui.sh。
- 许可头:新改文件只写 BSD 许可声明行,勿写个人 Copyright 头;vendored 文件(如 io.modelcontextprotocol.*)保留原版权声明。

## 模块速览与常用命令

SQL/DSL → RESTful API 的数据访问中间件(Boot 3.5.x LTS + Cloud 2025.0.x,jakarta 命名空间)。Maven 模块:common(通用定义)、mcp(LLM MCP 协议)、template(SQL 内容模板)、persistence(数据库持久化)、core(接口核心实现)、cache(执行缓存)、executor/gateway/manager(三服务,见下)、test(集中全部测试)、dist(发行打包);`datapoly-manager-ui` 为前端(非 Maven);`drivers/` 装配 20+ 数据库 JDBC 驱动;`build-docker/` 镜像与 compose 一键安装。改动前按需读 docs/{zh,en}/ 下 overview.md、build-deploy.md、data-task.md(涉及 DataTask 必读)。API 文档注解用 springdoc/swagger v3(io.swagger.v3.oas.annotations),勿引入 springfox。

- 全量测试(CI 同款):`mvn test -pl datapoly-test -am`
- 发行构建:`./build.sh`(前置 build-extension.sh 与 build-ui.sh 再 mvn package);容器内构建:`./docker-maven-build.sh`

## 一、网络分段(必须遵守)

- 仅 gateway 对外发布端口;manager 与 executor 除 Eureka 外无 HTTP 直连,新增直连须同步更新白名单与本文件。
- manager 内置两个 MCP Server(均走 `?token=` 查询参数鉴权,`/mcp/**` 豁免会话拦截器):数据 MCP `/mcp`·`/mcp/sse`(任意 MCP 令牌,调用数据 API tools);管理 MCP `/mcp/admin`·`/mcp/admin/sse`(仅 `mcp_client.manage_flag=1` 令牌,`dp_{entity}_{action}` 工具覆盖数据源/模块/分组/API/客户端等实体增删改查,见 `McpAdminTools`)。manage 令牌等同管理员(可读 appSecret 与 MCP 令牌明文),创建/授予须走已认证会话。
- executor `GatewaySourceFilter`(`/*`):`trusted-cidrs` 仅 IPv4 CIDR(K8s 探针须加入);`auth-token` 设置后必须带头 `X-DATAPOLY-Gateway-Token`(恒定时间比较,gateway 配 `DATAPOLY_GATEWAY_TOKEN` 注入)。gateway 未设 token 时注入哨兵 `UNSET`,勿改回空默认(SCG 启动会失败)。
- 来源判定只用 `getRemoteAddr()`,勿信任何转发头。

Expand All @@ -28,7 +36,7 @@ Groovy 沙箱默认启用但不是 JVM 隔离:勿把脚本编写权开放给
## 五、其他约束

- 新增 JDBC 代码资源必须 try-with-resources;firewall 规则行被删时网关按"全拒绝"处理(fail-closed,属预期)。
- DataTask 投递 Sink 仓库内置零实现,外部以 Spring Bean / `META-INF/services` 注册(SPI `com.cs.common.datatask.DataTaskSink`);宿主可自行维护本地扩展:在顶层 `datapoly-extension/`(已被 .gitignore 排除,独立 git 仓库)下用 `backend/` 放 Maven 扩展模块(依赖钉版在模块自身 pom、不进根 reactor,由入库脚本 build-extension.sh 在宿主机 JDK 8 构建后投放 lib-extra/,随发行版装配进各服务 classpath)、`front/` 放扩展 UI;API 扩展点 `ApiAssignmentPostProcessor` 注册方式相同、须同步执行且保持轻量。详见 docs/*/data-task.md。
- 默认前端扩展目录 `datapoly-extension/front`(同被 .gitignore 排除)经 datapoly-manager-ui 编译期装配:webpack `@extension` 别名自动探测该目录 `src/index.js`(见 build/webpack.base.conf.js)、`src/extension-stub` 为缺省回退、扩展路由与 i18n 词条在 manager-ui 入口深合并——这四处钩子文件(build/webpack.base.conf.js、src/extension-stub、src/router、src/main.js)勿移除或改名;目录不存在时 CI 与普通构建不受影响。front 自带 `package.json` 可直接启动(`npm run dev`,复用宿主 webpack 链,前置为宿主 node_modules 已安装、Node 23——dev server 与生产构建均已实测)。
- 宿主扩展 jar 经根目录 `lib-extra/` 投放点进入发行版 `lib/common/`(`package.xml` 打包该目录 `*.jar`;目录只占位入库,jar 永不入库)。扩展为独立 git 仓库(内部 GitLab,front+backend 一体):`build-extension.sh` 按环境变量 `DATAPOLY_EXTENSION_GIT_URL`(真实地址不入库,CI 注入)+ `DATAPOLY_EXTENSION_GIT_REF`(默认 master)浅克隆到 `datapoly-extension/`(仍被 .gitignore 排除),目录已存在则按本地工作区构建(宿主机 JDK 8 优先、无 8 时 8 以上可用)、`DATAPOLY_EXTENSION_FORCE_SYNC=1` 强制覆盖本地改动;`build.sh`/`docker-maven-build.sh` 会先调用该脚本,未配置且目录不存在时无操作(纯开源构建零影响)。本地环境变量注入(env.sh)与防误提交钩子集中在被忽略的 `dev-local/`。
- DataTask 投递 Sink 仓库内置零实现,外部以 Spring Bean / `META-INF/services` 注册(SPI `com.cs.common.datatask.DataTaskSink`);宿主可自行维护本地扩展:在顶层 `datapoly-extension/`(已被 .gitignore 排除,独立 git 仓库)下用 `backend/` 放 Maven 扩展模块(依赖钉版在模块自身 pom、不进根 reactor,由入库脚本 build-extension.sh 在宿主机 JDK 25 构建后投放 lib-extra/,随发行版装配进各服务 classpath)、`front/` 放扩展 UI;API 扩展点 `ApiAssignmentPostProcessor` 注册方式相同、须同步执行且保持轻量。详见 docs/*/data-task.md。
- 默认前端扩展目录 `datapoly-extension/front`(同被 .gitignore 排除)经 datapoly-manager-ui 编译期装配:webpack `@extension` 别名自动探测该目录 `src/index.js`(见 build/webpack.base.conf.js)、`src/extension-stub` 为缺省回退、扩展路由与 i18n 词条在 manager-ui 入口深合并——这四处钩子文件(build/webpack.base.conf.js、src/extension-stub、src/router、src/main.js)勿移除或改名;目录不存在时 CI 与普通构建不受影响。front 自带 `package.json` 可直接启动(`npm run dev`,复用宿主 webpack 链,前置为宿主 node_modules 已安装、Node 24——dev server 与生产构建均已实测)。
- 宿主扩展 jar 经根目录 `lib-extra/` 投放点进入发行版 `lib/common/`(`package.xml` 打包该目录 `*.jar`;目录只占位入库,jar 永不入库)。扩展为独立 git 仓库(内部 GitLab,front+backend 一体):`build-extension.sh` 按环境变量 `DATAPOLY_EXTENSION_GIT_URL`(真实地址不入库,CI 注入)+ `DATAPOLY_EXTENSION_GIT_REF`(默认 master)浅克隆到 `datapoly-extension/`(仍被 .gitignore 排除),目录已存在则按本地工作区构建(宿主机 JDK 25 优先、低于 25 不可用)、`DATAPOLY_EXTENSION_FORCE_SYNC=1` 强制覆盖本地改动;`build.sh`/`docker-maven-build.sh` 会先调用该脚本,未配置且目录不存在时无操作(纯开源构建零影响)。本地环境变量注入(env.sh)与防误提交钩子集中在被忽略的 `dev-local/`。
- 一次性 token 在校验时即消费(含查库兜底路径;2026-09 修复兜底不消费导致的重放);并发首用竞态下多 executor 仍可能各放行一次(无分布式锁,已知限制)。
12 changes: 6 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Thanks for your interest in contributing!

1. Fork the repository and create your branch from `main`.
2. Set up the environment:
- JDK 8 (the project targets Java 8)
- JDK 25 (the project targets Java 25, LTS line)
- Maven 3.6+
- Docker (for the database containers and the front-end build)

Expand All @@ -21,15 +21,15 @@ mvn -B -ntp test -pl datapoly-common,datapoly-template,datapoly-core,datapoly-ex
## Front-end (built-in manager UI)

The built-in UI (`datapoly-manager-ui`) is a Vue 2 project built with a
webpack 5 toolchain. The canonical toolchain is the pinned `node:23-alpine`
container (the production build is verified on Node 23), so build it with
webpack 5 toolchain. The canonical toolchain is the pinned `node:24-alpine`
container (the production build is verified on Node 24), so build it with
the provided script (works on any host with Docker, no local Node required):

```bash
sh ./build-ui.sh
```

This builds the UI inside a `node:23-alpine` container and syncs
This builds the UI inside a `node:24-alpine` container and syncs
`dist/index.html` + `dist/static/` into `datapoly-manager/src/main/resources/`
(wiping the previous copies first so stale hashed files do not linger).

Expand All @@ -44,7 +44,7 @@ To rebuild the UI without the wrapper script (for UI-only development), the
raw container command is:

```bash
docker run --rm -v $PWD/datapoly-manager-ui:/app -w /app node:23-alpine \
docker run --rm -v $PWD/datapoly-manager-ui:/app -w /app node:24-alpine \
sh -c "npm config set registry https://registry.npmmirror.com && \
npm install --no-audit --no-fund --no-package-lock --legacy-peer-deps && \
npm run build"
Expand All @@ -55,7 +55,7 @@ If you build this way, copy `dist/index.html` and `dist/static/` into

## Local smoke testing

The `.devcontainer/` directory contains a dev container (JDK 8 + Maven,
The `.devcontainer/` directory contains a dev container (JDK 25 + Maven,
MySQL 8, PostgreSQL 14) for end-to-end smoke tests:

```bash
Expand Down
12 changes: 6 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Use of this source code is governed by a BSD-style license

# ============================================================================
# 单 Dockerfile 全自动构建(多阶段):UI(node:23) → Maven 打包(temurin 8) → 运行时(temurin 8 JRE)
# 单 Dockerfile 全自动构建(多阶段):UI(node:24) → Maven 打包(temurin 25) → 运行时(temurin 25 JRE)
# 产物与 build.sh / docker-maven-build.sh / build_and_push_image.sh 一致,两条链路并行可用。
#
# 用法(在仓库根目录):
Expand All @@ -19,9 +19,9 @@
# npmmirror / 阿里云镜像下载 npm 与 Maven 依赖,之后由层缓存与 cache mount 加速。
# ============================================================================

# ---- Stage 1: 构建内置管理端 UI(Vue2 + webpack5,与 build-ui.sh 同为 node:23) ----
# ---- Stage 1: 构建内置管理端 UI(Vue2 + webpack5,与 build-ui.sh 同为 node:24) ----
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM node:23-alpine@sha256:a34e14ef1df25b58258956049ab5a71ea7f0d498e41d0b514f4b8de09af09456 AS ui
FROM node:24-alpine@sha256:a34e14ef1df25b58258956049ab5a71ea7f0d498e41d0b514f4b8de09af09456 AS ui
WORKDIR /opt/app
# 拷贝整个仓库(与 build-ui.sh 挂载仓库根一致):webpack 的 @extension 别名会探测
# 仓库根下 datapoly-extension/front/src,缺省回退 stub;仅生产构建,debug 变体仍走 build-ui.sh
Expand All @@ -31,9 +31,9 @@ RUN npm config set registry http://mirrors.cloud.tencent.com/npm/ \
&& npm install --no-audit --no-fund --no-package-lock --legacy-peer-deps \
&& npm run build

# ---- Stage 2: Maven 打包(与 docker-maven-build.sh 同为 temurin 8,支持 arm64) ----
# ---- Stage 2: Maven 打包(与 docker-maven-build.sh 同为 temurin 25,支持 arm64) ----
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM maven:3.9-eclipse-temurin-8@sha256:d8b1b22e93012cd0257d37b3fd6d7bec3d0f7ae611730fc3ba0657b9ebe4e7ca AS build
FROM maven:3.9-eclipse-temurin-25@sha256:0e3f1c4394674505f73525188b8005f521ef3c786c651cb61432b2db3517087b AS build
# CI 可传 MAVEN_ARGS=-Dmaven.test.skip=true 加速;默认与 docker-maven-build.sh 一致(跑测试)
ARG MAVEN_ARGS=""
WORKDIR /src
Expand All @@ -51,7 +51,7 @@ RUN --mount=type=cache,target=/opt/maven/localRepository \

# ---- Stage 3: 运行时镜像(与 build-docker/datapoly/Dockerfile-* 同基线) ----
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM eclipse-temurin:8-jre-jammy@sha256:06641b36281c1ac815c33f3f3528cfea1c6fc41ddc60d261746e4343d19cbe65
FROM eclipse-temurin:25-jre-jammy@sha256:20a695e74d47fb29cda1cbad5d9ee6cfad4ac6e88a8e048ed6265cede1e71f5e
ENV TZ=Asia/Shanghai
# 非 root 运行(K8s 安全上下文友好);用户需在 COPY --chown 前存在;
# curl 仅供 HEALTHCHECK 使用
Expand Down
2 changes: 1 addition & 1 deletion build-docker/datapoly/Dockerfile-executor
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM eclipse-temurin:8-jre-jammy@sha256:06641b36281c1ac815c33f3f3528cfea1c6fc41ddc60d261746e4343d19cbe65
FROM eclipse-temurin:25-jre-jammy@sha256:20a695e74d47fb29cda1cbad5d9ee6cfad4ac6e88a8e048ed6265cede1e71f5e

ENV TZ=Asia/Shanghai

Expand Down
2 changes: 1 addition & 1 deletion build-docker/datapoly/Dockerfile-gateway
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM eclipse-temurin:8-jre-jammy@sha256:06641b36281c1ac815c33f3f3528cfea1c6fc41ddc60d261746e4343d19cbe65
FROM eclipse-temurin:25-jre-jammy@sha256:20a695e74d47fb29cda1cbad5d9ee6cfad4ac6e88a8e048ed6265cede1e71f5e

ENV TZ=Asia/Shanghai

Expand Down
2 changes: 1 addition & 1 deletion build-docker/datapoly/Dockerfile-manager
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# 基镜像钉 digest(tag 可被上游覆盖,digest 不可变);升级基镜像时须同步刷新
FROM eclipse-temurin:8-jre-jammy@sha256:06641b36281c1ac815c33f3f3528cfea1c6fc41ddc60d261746e4343d19cbe65
FROM eclipse-temurin:25-jre-jammy@sha256:20a695e74d47fb29cda1cbad5d9ee6cfad4ac6e88a8e048ed6265cede1e71f5e

ENV TZ=Asia/Shanghai

Expand Down
Loading
Loading