Suggestion: Plugin full example with input of n cols and output of m cols

[oscar6echo]

@MarcoGorelli
As mentionned today after your very good talk at pydata Paris, this tutorial repo is so useful 👍

My suggetion is to add an example of a function taking as input n columns and returning m colums.
It seems to me as generic as can be for row-wise functions.

The steps would be:

• python: create a column col_in of type struct {col_in_1: _, col_in_1: _, ..., col_in_n: _ } from cols col_1, col_2,... col_n
• plugin: create a plugin with Rust code unpacking the struct col in n variables with their own type
• Rust: apply a function with
  • input Struct{col_in_1, col_in_1, ..., col_in_n}
  • output Struct{col_out_1, col_out_1, ..., col_out_m}
  • Remark: this function can be from an existing crate e.g. a back scholes option pricer - closer to real life example than toy local function
• plugin: return the Struct so that polars creates a col_out of type struct
• python: convert column col_out in m columns col_out_1, col_out_2, ..., col_out_m

The benefits would be - pls correct if/where wrong:

• 100% generic for a row-wise functions
• efficient as creating/expanding a struct column from/into regular columns is cheap in polars

The next and last question would be how to take in all data in one go as a vect (length L) of struct from the Rust side (I mean as opposed to row by row) and return a L-long vect of struct to create a full polars series. Unless I miss something this would literally cover all cases ! It would be a template for all potential plugins "row-wise" or "whole". It would be invaluable !!

The "glue" code between "polars Rust" and "generic Rust" is the difficult, least documented part, so the harder to discover for users willing to expand polars with plugins.

What do you think ?

[oscar6echo]

Finally I rolled my sleeves up and did it.
See https://github.com/oscar6echo/polars-plugin-option-pricing

It is quite efficient (10m black scholes prices in 1s, and 10m implied volatility calculations in 3.2s, on my machine).

But the option pricing thing is just a pretext here.
The idea was to use a generic formula with n cols as input and m cols as output.
The actual calculations are delegated to a third party crate.

I used 2 ways to access the same rust function:

• one stuct column
• several columns

Almost zero difference in implementation: In the second case the struct is build in Python before running the rust function. The output is a struct col so needs be unnested - trivial in polars.

I found that writing the Rust function with struct as input and output quite convenient. And it is definitely generic: All func(n params)->(m params) can fold into this mold, it seems to me.
Once the "glue" code between "polars Rust" and "generic Rust" is written, the most difficult (because specific) part is out of the way.

REMARK (hopefully temporary): the cross compile to windows does not work yet... -- EDIT1: Fixed -- EDIT2: nope, wheel still does not work, investigating. -- EDIT3: replaced setuptools-rust with maturin, more convenient in many respects incl. cross compile. Cross compile to win should work now. To be tested soon. -- EDIT4: still does not work. Cross compile to win is real mess... -- EDIT5: now working at last. Cf discussion: In short avoid optional PolarsAllocator for win wheels (for now at least).