Nested UDF array signatures are unwieldy

[Jefffrey]

Is your feature request related to a problem or challenge?

datafusion/datafusion/expr-common/src/signature.rs

Lines 526 to 541 in 70c26a0

	#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
	pub enum ArrayFunctionSignature {
	/// A function takes at least one List/LargeList/FixedSizeList argument.
	Array {
	/// A full list of the arguments accepted by this function.
	arguments: Vec<ArrayFunctionArgument>,
	/// Additional information about how array arguments should be coerced.
	array_coercion: Option<ListCoercion>,
	},
	/// A function takes a single argument that must be a List/LargeList/FixedSizeList
	/// which gets coerced to List, with element type recursively coerced to List too if it is list-like.
	RecursiveArray,
	/// Specialized Signature for MapArray
	/// The function takes a single argument that must be a MapArray
	MapArray,
	}

datafusion/datafusion/expr-common/src/signature.rs

Lines 565 to 577 in 70c26a0

	#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
	pub enum ArrayFunctionArgument {
	/// A non-list or list argument. The list dimensions should be one less than the Array's list
	/// dimensions.
	Element,
	/// An Int64 index argument.
	Index,
	/// An argument of type List/LargeList/FixedSizeList. All Array arguments must be coercible
	/// to the same type.
	Array,
	// A Utf8 argument.
	String,
	}

There are two current use cases which aren't well supported by our current array signature code:

• Can't coerce to a specific element type
• Can't handle arrays of different element type

For the first point, see how for cosine_distance it needs to define signature via user defined in order to ensure input lists come as list<f64>:

datafusion/datafusion/functions-nested/src/cosine_distance.rs

Lines 100 to 139 in 70c26a0

	fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
	let [_, _] = take_function_args(self.name(), arg_types)?;
	let coercion = Some(&ListCoercion::FixedSizedListToList);
	for arg_type in arg_types {
	if !matches!(arg_type, Null | List(_) | LargeList(_) | FixedSizeList(..)) {
	return plan_err!("{} does not support type {arg_type}", self.name());
	}
	}
	// If any input is `LargeList`, both sides must be widened to `LargeList`
	// so the runtime dispatch in `cosine_distance_inner` sees a homogeneous
	// pair. Follows the pattern in `ArrayConcat::coerce_types`.
	let any_large_list = arg_types.iter().any(|t| matches!(t, LargeList(_)));
	let coerced = arg_types
	.iter()
	.map(|arg_type| {
	if matches!(arg_type, Null) {
	let field = Arc::new(Field::new_list_field(DataType::Float64, true));
	return if any_large_list {
	LargeList(field)
	} else {
	List(field)
	};
	}
	let coerced = coerced_type_with_base_type_only(
	arg_type,
	&DataType::Float64,
	coercion,
	);
	match coerced {
	List(field) if any_large_list => LargeList(field),
	other => other,
	}
	})
	.collect();
	Ok(coerced)
	}

For the second point, an example is arrays_zip which can accept lists of different type (though technically since it can accept an arbitrary number of arguments this is also another limitation)

Describe the solution you'd like

See if we can refine the array signature code so more of our nested UDFs can use it, instead of resorting to user defined

Describe alternatives you've considered

No response

Additional context

Potentially related:

• Write Signature method for list datatypes. #6559