Repository navigation
Conversation
…ng powers-of-ten tables A decimal type's scale can be any int32_t value and the scale of a decimal string comes from its exponent, but IncreaseScaleBy(), ReduceScaleBy(), GetWholeAndFraction() and the floating-point PowerOfTen() index fixed-size tables and only check the index with a DCHECK. - Truncating casts from decimal to integer, decimal to decimal and string to decimal bound the scale change: reducing it by more than kMaxScale gives zero, increasing it by more than kMaxScale is an error for a non-zero value. - product and hash_product reject a decimal scale outside [0, kMaxScale]. - Decimal ToReal() does not split the value into integer and fractional parts above kMaxScale, and FromReal() uses the approximate algorithm with LargePowerOfTen() there.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A decimal type's scale can be any
int32_tvalue and the scale of a decimal string comes from its exponent, butIncreaseScaleBy,ReduceScaleBy,GetWholeAndFractionand the floating-pointPowerOfTenindex fixed-size tables of powers of ten and only check the index with aDCHECK. Truncating casts,product/hash_productandDecimal::ToReal/FromRealpass them scale changes computed from such scales, so release builds read out of bounds (see the issue: pyarrow 26.0.0 segfaults on a truncating cast of"1E-100000000"todecimal128(38, 0)).What changes are included in this PR?
scalar_cast_numeric.cc: the truncating (allow_decimal_truncate) casts from decimal to integer, decimal to decimal and string to decimal compute the scale change in 64 bits and go through two helpers,UnsafeIncreaseScaleByandUnsafeReduceScaleBy. Reducing the scale by more thankMaxScalegives zero, which is exact since every value of the type is below 10^(kMaxScale + 1). Increasing it by more thankMaxScalereturnsInvalidunless the value is zero: a non-zero value would overflow, andallow_decimal_truncateis about dropping digits, not about overflow.productandhash_product: a decimal type whose scale is outside[0, kMaxScale]is rejected withNotImplementedwhen the kernel is initialized (MultiplyTraits::CheckType).MultiplyTraitsrescales by the type's scale, and the multiplicative identity cannot be represented at a negative scale or a scale abovekMaxScale.util/decimal.cc:ToRealno longer splits the value into integer and fractional parts when the scale is abovekMaxScale: the integer part is always zero then, and the split looked upGetWholeAndFraction(scale).FromRealuses the approximate algorithm when the scale is abovekMaxScale(as it already did for negative scales), and the approximate algorithm usesLargePowerOfTen, which falls back tostd::pow, instead of the table-onlyPowerOfTen.DCHECKs on the scale range inToRealandFromRealare removed since any scale is now handled, andLargePowerOfTentakes anint64_tso that negating a scale ofINT32_MINdoes not overflow.Decimal::Rescaleis handled by #50965 (GH-50964) and the conversion order ofDecimal32narrowing casts by #53327 (GH-53288); this PR does not change either, it only keeps the truncating paths' table lookups in bounds.Are these changes tested?
Yes, with new tests for all decimal widths, at scales just past
kMaxScaleand atINT32_MIN/INT32_MAX:Cast.DecimalToIntTruncateScaleOutOfRange,Cast.DecimalToDecimalTruncateScaleOutOfRangeandCast.StringToDecimalTruncateScaleOutOfRangeTestDecimalProductKernel.ScaleOutOfRangeandGroupBy.ProductDecimalScaleOutOfRangeTestDecimalToReal.TestScaleOutOfRangeandTestDecimalFromReal.TestScaleOutOfRangeRun against the unpatched libraries, each of them reports a
global-buffer-overflowor an out-of-bounds index under ASan/UBSan, or fails. With the fix,arrow-compute-scalar-cast-test,arrow-compute-aggregate-test,arrow-acero-hash-aggregate-testand the decimal tests ofarrow-utility-testpass in an ASan/UBSanRelWithDebInfobuild.Are there any user-facing changes?
kMaxScalereturns zero, and one that increases it by more thankMaxScalereturns an error for a non-zero value, instead of reading out of bounds.productandhash_producton a decimal type with a scale outside[0, kMaxScale]returnNotImplementedinstead of a wrong result or a crash.Decimal32/64/128/256::ToRealandFromRealaccept any scale.This PR contains a "Critical Fix". These operations read out of bounds of static tables, and
productreturned wrong results, for decimal types and strings with out-of-range scales. Such scales can come from untrusted data, for example an IPC schema, or a string column cast to decimal; pyarrow 26.0.0 segfaults on the cast above.Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by:
An AI coding assistant found the issue, wrote the fix, the regression test and this description, and ran the tests; a separate AI pass reviewed the diff.