[fix] AS-5322: Fix gcompat for alpine image stability - #809
Open
hshankar31 wants to merge 8 commits into
Open
Conversation
hshankar31
changed the title
hshankar31
changed the title
Remove the apachepulsar/glibc-package approach (dual-libc injection) which causes JVM crashes due to conflicting allocators between glibc and musl. Replace with gcompat (musl-native glibc ABI shim) + LD_PRELOAD, aligning with upstream apache#23762 and the 4.0_ds branch. Changes: - docker/pulsar/Dockerfile: - Remove glibc-package stage (FROM apachepulsar/glibc-base:2.38) - Add gcompat, libgcc, libstdc++, libuuid, coreutils to apk install - Upgrade JVM stage to amazoncorretto:17-alpine3.21 (removes manual APK key) - Replace python-deps build stage with inline pip3 install (--only-binary grpcio,pulsar-client to avoid musl build issues) - Add ENV LD_PRELOAD=/lib/libgcompat.so.0 - Add ENV ROCKSDB_MUSL_LIBC=true - Add ENV PULSAR_PID_DIR=/pulsar/logs - Add apk upgrade for latest CVE fixes - docker/glibc-package/: deleted entirely (no longer needed) - docker/pulsar-all/Dockerfile: - Add KPL 1.0.4 Alpine-compatible binary via PULSAR_IO_KINESIS_KPL_IMAGE ARG - Add PULSAR_IO_KINESIS_KPL_PATH env and runtime deps (libstdc++, libgcc) - docker/kinesis-producer-alpine/: new directory - Dockerfile, build-alpine.sh, musl patch, README for KPL 1.0.4 on Alpine - pulsar-io/kinesis-kpl-shaded/pom.xml: pin to 1.0.4 - docker/pulsar/test_script.sh: Alpine stability checks (gcompat, LD_PRELOAD, libgcc, ROCKSDB_MUSL_LIBC, Python gRPC stub loading, JVM crash detection)
hshankar31
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace
glibc-packagewithgcompatfor Alpine image stabilityMotivation
The
3.1_dsAlpine image previously injected real glibc into the container viaapachepulsar/glibc-base:2.38— a pattern that installslibc.so.6andld-linux-x86-64.so.2alongside Alpine's native musl libc. Running two libcimplementations in the same process is inherently unsafe: memory allocated by
one allocator and freed by the other causes heap corruption, manifesting as JVM
crashes in Netty native transport, Conscrypt/BoringSSL, and RocksDB under load.
This issue was identified and fixed in upstream
apache/pulsar#23762 and is
already absent from the
4.0_dsbranch. This PR brings3.1_dsintoalignment.
Changes
docker/pulsar/Dockerfileglibc-packagemulti-stage build (FROM apachepulsar/glibc-base:2.38) and all associatedCOPY/apk addstepsgcompat— a musl-native shim that implements the glibc ABI on top of musl using a single allocator, eliminating the dual-libc conflictENV LD_PRELOAD=/lib/libgcompat.so.0so Netty/Conscrypt native libraries resolve glibc symbols correctly at runtimeENV ROCKSDB_MUSL_LIBC=trueto enable RocksDB's musl code pathFROM amazoncorretto:17-alpine3.21python-depsbuild stage with an inlinepip3 install --only-binary grpcio,pulsar-clientto avoid C-extension compilation against muslapk upgradeto pick up the latest security fixes at build timeENV PULSAR_PID_DIR=/pulsar/logsto fix PID file write permissions for uid 10000docker/glibc-package/docker/pulsar-all/DockerfilePULSAR_IO_KINESIS_KPL_IMAGEbuild ARG and copies thekinesis_producernative executable with required musl-compatible runtime depsdocker/kinesis-producer-alpine/docker/pulsar/test_script.shgcompatpresence,LD_PRELOAD,ROCKSDB_MUSL_LIBC, absence of glibc artifacts, Python gRPC stub loading, and JVM crash-free startupTesting
Validated end-to-end on a single-node minikube cluster (podman driver): all
pods reached
1/1 Running, ZooKeeper, BookKeeper, broker admin API, and a fullproduce/consume round-trip all passed.