CompiledRunner Functions Bug

[Synforge]

Hello,

First of all thanks for producing this library very useful for some applications I'm looking at right now.

I'm trying to use the CompiledRunner with some additional functions defined, short example below:

{
    'lower': celpy.celtypes.FunctionType,
    'upper': celpy.celtypes.FunctionType
}

def lower(lhs: celpy.celtypes.StringType) -> celpy.celtypes.StringType:
    return celpy.celtypes.StringType(lhs.lower())

def upper(lhs: celpy.celtypes.StringType) -> celpy.celtypes.StringType:
    return celpy.celtypes.StringType(lhs.upper())

env = celpy.Environment(annotations=declarations, runner_class=CompiledRunner)

rule = "size(items) == 1 && meta.device.lower() == 'tablet'"
ast = env.compile(rule)

prgm = env.program(ast, functions={
    'lower': lower,
    'upper': upper
})
prgm.evaluate({'items': [1,2,3], 'meta': {'device': 'tablet'}})

This works normally when using the InterpretedRunner, however with the compiled runner I get the following:

TypeError: <class 'celpy.evaluation.CELEvalError'> CELEvalError(*('no such overload', <class 'TypeError'>, ('<class \'celpy.evaluation.CELEvalError\'> CELEvalError(*(\'undeclared reference to "name \\\'__main__\\\' is not defined" (in container ())\', <class \'NameError\'>, ("name \'__main__\' is not defined",))) and <class \'celpy.evaluation.CELEvalError\'> CELEvalError(*(\'undeclared reference to "name \\\'__main__\\\' is not defined" (in container ())\', <class \'NameError\'>, ("name \'__main__\' is not defined",)))',))) or <class 'celpy.evaluation.CELEvalError'> CELEvalError(*('undeclared reference to "name \'__main__\' is not defined" (in container ())', <class 'NameError'>, ("name '__main__' is not defined",)))

I dumped out the source from the compiled runner:

# conditionaland:
ex_2_l = lambda activation: celpy.evaluation.bool_eq(celpy.evaluation.function_size(activation.items), celpy.celtypes.IntType(1))
ex_2_r = lambda activation: celpy.evaluation.bool_eq(__main__.lower(activation.meta.get('device')), celpy.celtypes.StringType('tablet'))
ex_2 = lambda activation: celpy.celtypes.logical_and(celpy.evaluation.result(activation, ex_2_l), celpy.evaluation.result(activation, ex_2_r))
CEL = celpy.evaluation.result(base_activation, ex_2)

Internal error: CELEvalError(*('undeclared reference to "name \'__main__\' is not defined" (in container ())', <class 'NameError'>, ("name '__main__' is not defined",)))

This also happens when you use a lambda rather than referencing a function.

Looks like this may not be referencing the function correctly? Happy to assist if you want to point me in the right direction for resolution, or if this is an error on my part please let me know.

Thanks

[Synforge]

Traced this back a little more, it's due to how globals is being passed into exec within the Transpiler.evaluate method:

https://github.com/cloud-custodian/cel-python/blob/main/src/celpy/evaluation.py#L3092

Here the globals are being taken from celpy.evaluation.result.__globals__

However, if you pass in additional functions then the packages that these functions reside in will not be in the globals and cannot be resolved. It seems likely that any functions that are passed in to the activation should have their packages resolved and added to the globals here, something along the lines of:

def resolve_fn_packages(self):
  result = {}
  for _, f in self.base_activation.functions.items():
      package = f.__module__.split('.', 1)[0]
      if package in result:
          continue
  
      result[package] = sys.modules[package]
  
  return result

def __init__(
    self,
    ast: TranspilerTree,
    activation: Activation,
) -> None:
  ...
  fn_packages = self.resolve_fn_packages()

  # Global for the top-level ``CEL = result(base_activation, ...)`` statement.
  evaluation_globals = (
      celpy.evaluation.result.__globals__ | self.fn_packages
  )  # the ``evaluation`` moodule

[slott56]

Thanks! I see the PR for this. Please complete the EasyCLA so I can merge this.

Also. There should perhaps be a documentation change to explain that lambda names can't be resolved easily.

[Quantum0uasar]

The author's diagnosis looks correct — the exec call in Transpiler.evaluate only includes the function name in globals, not its parent module, so any qualified names (__main__.<fn>) fail to resolve.

The proposed fix of calling resolve_fn_packages to populate the execution context with the function's module globals should work. Happy to look into this further and open a small PR with the fix + a regression test case if the original PR has stalled.

[slott56]

I closed PR #146 because the tests were failing.

I think the changes from this PR need to be re-merged, and a new PR created. (I'm unclear on how to restart the CI tests for the old PR, so I gave up on it in frustration. I'm sure @Synforge was pretty unhappy with my failure, also.)

If you can resurrect these changes, that would be very helpful!