Hi Devs,
I am working on a specific problem right now.
So we have some data pipelines code written in Python Pandas mainly where we take some data from source (which can be DB or API) then make dataframes and then at last after transformation we put the data into redshift tables.
Now we need to build a kind of lineage tool on top of it which when we select a particular column from redshift tables shows us the complete path of how that column got calculated and stuff.
Anyone have any idea about some python library or some way to achieve this? I tried the AST graph thing but it only shows lineage for columns which are actually used in code not the dynamic one or those who never got used/mentioned in code directly.