Interface factory and two copies of parameters.

[rozyczko]

How a refnx object is created and kept in sync with a Parameter

This writeup is a description of why and how we use the interface bindings and what functionality needs to remain during the potential rewrite, as proposed in #7 and #160.

The easyscience Parameter is the persistent, user-facing "source of truth"
The refnx object is transient backend state,
The binding layer - a property stored in Parameter._callback is the only bridge between them. It is rebuilt on calculator switch.

1. The static picture

Example: a Material('Si', sld=6.335). Its two parameters are mirrored into one reflect.SLD object, keyed by the material's unique_namein the wrapper's storage dict. Note the name translation: easyscience sayssld/isld, refnx says real/imag`.

flowchart LR
  subgraph ES["easyscience model"]
    M["Material 'Si'"]
    P1["Parameter sld<br/>_scalar = 6.335 (value, unit, variance)<br/>min / max / fixed<br/>_callback ●"]
    P2["Parameter isld<br/>_scalar = 0.0<br/>_callback ●"]
    M --> P1
    M --> P2
  end
  subgraph BIND["binding layer"]
    IC["ItemContainer<br/>link_name = material.unique_name<br/>name_conversion: sld→real, isld→imag<br/>getter = wrapper.get_material_value<br/>setter = wrapper.update_material"]
  end
  subgraph RX["refnx backend"]
    ST["RefnxWrapper.storage['material'][unique_name]"]
    SLD["reflect.SLD<br/>real = 6.335<br/>imag = 0.0"]
    ST --> SLD
  end
  P1 -- "_callback = property(fget, fset)" --> IC
  P2 -- "_callback" --> IC
  IC --> ST

Loading

Parameter._callback is set in InterfaceFactoryTemplate.generate_bindings (core: src/easyscience/fitting/calculators/interface_factory.py:170) from the ItemContainer returned by CalculatorBase.create (easyreflectometry: calculators/calculator_base.py:57).
The name map {'sld': 'real', 'isld': 'imag'} is Refnx._material_link (calculators/refnx/calculator.py:14).

2. Creation - generate_bindings

The refnx object is born empty (placeholder zeros) and only receives real values when each parameter's freshly attached callback pushes its current value across. That final fset is what populates the backend.

sequenceDiagram
  autonumber
  actor U as User
  participant P as Parameter sld
  participant F as InterfaceFactory
  participant C as Refnx calculator
  participant W as RefnxWrapper
  participant R as refnx.reflect

  U->>P: Material('Si', sld=6.335)
  Note over P: _scalar = 6.335<br/>_callback = property() - empty, no backend yet
  U->>F: generate_bindings(material)
  F->>C: create(material)
  C->>W: create_material(unique_name)
  W->>R: reflect.SLD(0, name=unique_name)
  Note over W,R: storage['material'][unique_name] = SLD<br/>placeholder zeros!
  C-->>F: [ItemContainer(unique_name, sld→real, getter, setter)]
  loop for each linkable Parameter (sld, isld)
    F->>P: v = value_no_call_back
    Note over P: read cached _scalar directly -<br/>callback not wired yet
    F->>P: _callback = item.make_prop('sld')
    F->>P: _callback.fset(v)
    P->>W: update_material(unique_name, real=6.335)
    W->>R: SLD.real.value = 6.335
    Note over R: placeholder replaced with real value
  end

Loading

Steps 3-8: interface_factory.py:188-208. Steps 5-6: calculator_base.py:67-78 and refnx/wrapper.py:32-40. make_prop builds the property with the sld→real key translation built in its closures (interface_factory.py:244-268).

generate_bindings in detail

The full method, from src/easyscience/fitting/calculators/interface_factory.py:170-208:

def generate_bindings(self, model, *args, ifun=None, **kwargs) -> None:
    class_links = self.__interface_obj.create(model)          # (1)
    props = model._get_linkable_attributes()                  # (2)
    props_names = [prop.name for prop in props]
    for item in class_links:                                  # (3)
        for item_key in item.name_conversion.keys():          # (4)
            if item_key not in props_names:
                continue
            idx = props_names.index(item_key)
            prop = props[idx]

            # Should be fetched this way to ensure we don't get value from callback
            if hasattr(prop, 'value_no_call_back'):           # (5)
                prop_value = prop.value_no_call_back
            else:
                prop_value = prop.value

            prop._callback = item.make_prop(item_key)         # (6)
            prop._callback.fset(prop_value)                   # (7)

Step by step:

1. self.__interface_obj.create(model) - asks the current calculator (e.g. the Refnx class) to create its backend representation of model. For a Material this calls wrapper.create_material(unique_name), which stores reflect.SLD(0, name=...) in storage['material'] with placeholder zeros. The return value is a list of ItemContainer tuples; each one bundles everything needed to talk to that backend object:

   • link_name: the storage key (the model object's unique_name),
   • name_conversion: dict mapping easyscience parameter names to backend attribute names
     (e.g. {'sld': 'real', 'isld': 'imag'} from Refnx._material_link),
   • getter_fn / setter_fn: bound wrapper methods (get_material_value /
     update_material).

2. model._get_linkable_attributes() - collects the model's Parameter/Descriptor objects that are eligible for binding, so they can be matched by name against the name_conversion keys.

3. Outer loop over class_links - a single create call can return several ItemContainers (a Model returns containers for the model itself plus what it recursed into), so each container is processed in turn.

4. Inner loop over name_conversion keys - each easyscience-side name (sld, isld, thickness, ...) is looked up among the model's linkable attributes. Names not present on this model are skipped.

5. value_no_call_back - reads the parameter's cached scipp scalar directly, bypassing _callback.fget. This matters in two cases: the parameter may still hold a callback into the previous backend (during a calculator switch), or the new callback is not attached yet. Either way, reading through value here would return stale or wrong data. Plain Descriptor objects have no callback machinery, hence the hasattr fallback to .value.

6. item.make_prop(item_key) - builds the actual bridge: a Python property whose closures capture the container's link_name, name_conversion, and the wrapper's getter/setter (see ItemContainer.make_prop, interface_factory.py:244):

   • fget() calls getter_fn(link_name, name_conversion[item_key]), e.g.
     get_material_value(unique_name, 'real'),
   • fset(value) calls setter_fn(link_name, **{name_conversion[item_key]: value}), e.g.
     update_material(unique_name, real=value).

   This property is assigned to prop._callback, replacing the empty property() the parameter was constructed with.

7. prop._callback.fset(prop_value) - the initial push: the parameter's current value is written into the backend object, replacing the placeholder zero from step 1. From this point on, the two sides stay in sync through the set/get paths below.

generate_bindings is idempotent in practice: CalculatorBase.create checks if key not in storage before creating backend objects, so re-running it refreshes callbacks without duplicating backend state. It is also what switch() relies on: after swapping
calculators, update_bindings/generate_bindings replays this sequence against the new wrapper, and step 7 carries the parameter values across.

3. Update and read - the callback at work

After binding, every value access goes through the callback. A set writes both homes (local scalar first, then push to refnx). A get trusts the backend: it pulls from refnx and re-syncs the local scalar if the backend drifted - which happens when refnx mutates its own parameters during a fit.

sequenceDiagram
  autonumber
  actor U as User / minimizer
  participant P as Parameter sld
  participant W as RefnxWrapper
  participant R as reflect.SLD

  rect rgba(120,120,120,0.08)
    Note over U,R: SET - material.sld.value = 4.2
    U->>P: value = 4.2
    Note over P: clamp to [min, max]<br/>_scalar.value = 4.2
    P->>W: _callback.fset(4.2) → update_material(unique_name, real=4.2)
    W->>R: SLD.real.value = 4.2
    Note over P: notify observers
  end

  rect rgba(120,120,120,0.08)
    Note over U,R: GET - material.sld.value
    U->>P: value
    P->>W: _callback.fget() → get_material_value(unique_name, 'real')
    W->>R: read SLD.real.value
    R-->>P: 4.2
    Note over P: if backend ≠ _scalar: _scalar.value = backend value<br/>(backend wins - re-sync on drift)
    P-->>U: 4.2
  end

Loading

Set path: src/easyscience/variable/parameter.py:599-615 (clamp, write scalar, fset, notify). Get path: parameter.py:573-577 (fget, drift re-sync). During a fit, wrapper.calculate (refnx/wrapper.py:169) rebuilds a ReflectModel from storage and evaluates reflectivity - the values it sees are whatever the callbacks last pushed.

Two locations

	easyscience Parameter	refnx object
Lifetime	survives calculator switch, serialization, undo/redo	destroyed and rebuilt on switch()
Knows about	units, variance, bounds, fixed flag, dependencies	only the raw numbers its kernel needs
Role	source of truth for the user and the fitter	compute state for the reflectivity kernel

Because calculators are swappable (refnx ↔ BornAgain), the model cannot live inside any one
backend. switch() throws the old wrapper away and update_bindings / generate_bindings
replays diagram 2 against the new one - the Parameters carry the values across.

[rozyczko]

Potential solutions to avoid this complexity:

1. Proxy parameters, where backed delegates to core

The wrapper could hand the external calculator a subclass whose value reads directly from the core's Parameter

class ProxyParameter(refnx.analysis.Parameter):
    def __init__(self, es_param, name):
        super().__init__(es_param.value, name=name)
        self._es = es_param

    @property
    def value(self):
        return self._es.value_no_call_back

    @value.setter
    def value(self, v):          # backend writes (constraints, refnx-side fits)
        self._es.value = v       # land directly in the core parameter

This assumes that external calculator is Python and we can subclass its state.
Only works for backends whose parameters are pluggable Python objects like refnx.

[rozyczko]

2. Stateless rebuild - no persistent backend object

We could also drop the persistent storage dict and build the refnx structure from easyscience model at evaluation time (as it was done at some point with the CrysFML calculator in diffraction)

def calculate(self, q, model):           # model = easyscience Model, not a name
    structure = reflect.Structure([
        reflect.Slab(l.thickness.value, l.material.sld.value, l.roughness.value)
        for l in model.layers
    ])
    return reflect.ReflectModel(structure, scale=model.scale.value, ...)(q)

We could then get rid of the entire binding thing - ItemContainer, generate_bindings, _callback, switch etc.
Costs: object construction on every minimizer iteration, and we lose backend-side state that might be expensive to rebuild.
Mitigation: a dirty flag: we already have Parameter._notify_observers, so the wrapper can subscribe, rebuild structure only when things change, and just refresh values otherwise.

This loses the bidirectionality, though.

[damskii9992]

I also thought about having a seperate _calculate method, which doesn't create a new calculator, which is then the method called in each iteration of the fit, to avoid having to rebuild the calculator every iteration.
With this idea, a new calculator is still created whenever the fit is initialized and whenever the user calls calculate, i.e. to view the data or change something, but we avoid having to rebuild the backend object every iteration.
We should probably profile how long it takes to build a backend calculation object.

[rozyczko]

The bidirectionality can be achieved with post-run pull, explicitly updating the core Parameters from the calculator object values.
This might actually be even better, since it's a well defined action (at the end of run) instead of a hidden side effect of every property read as it is now.

This would most likely mean a need to extend the Parameter class rather than add more stuff to the setter method.

def update_from_calculator(self, value: float) -> None:
    if self._updating_from_calculator:      # re-entry guard
        return
    self._scalar.value = value             # no push back to backend, so no .self.value assignment
    self._notify_observers()                # dependents and GUI

[rozyczko]

I also thought about having a seperate _calculate method, which doesn't create a new calculator, which is then the method called in each iteration of the fit, to avoid having to rebuild the calculator every iteration.

Yes, we had that very approach in the old(er) diffraction library impl. Makes sense to split the behaviour

[damskii9992]

I still would really like to know more about how refnx can update parameters during a fit. By my understanding, our backends are only Calculators i.e. given a set of input parameters and a list of x-values, they use the parameters to calculate the corresponding y-values, i.e. they are JUST complex functions.

This is a one-way communication, I don't really understand why and how refnx is different.

[rozyczko]

I still would really like to know more about how refnx can update parameters during a fit. By my understanding, our backends are only Calculators i.e. given a set of input parameters and a list of x-values, they use the parameters to calculate the corresponding y-values, i.e. they are JUST complex functions.

This is a one-way communication, I don't really understand why and how refnx is different.

All right, looked closer at possible cases and it seems in reflectometry we don't actually NEED the bidirectionality. I assumed things like recomputed SLDs would be brought back but all we get is the reflectivity profiles and the SLD profile, which are functions of model parameters. The parameters are not being changed, as expected.

This said, in the old version of diffraction library, we had symmetry snapping functionality. We could create a Site at, say, (0.001, 0, 0) in a space group where that's a special position, assign it to a phase and cryspy snaps the coordinate on the engine side. The next read of site.fract_x.value pulls the snapped value through fget and overwrites easycore's Parameter. Multiplicity and symmetry-constraint flags change the same way.

However, we no longer have that version of library and I will leave to @AndrewSazonov to comment on this requirement

[rozyczko]

Then, I would propose to implement solution 2. Stateless rebuild - no persistent backend object
If the back update is ever needed, we could potentially have update_from_calculator as a method on the Parameter class.