'Azure Synapse: return value from .NET Spark (C#) Notebook to pipeline exception
I am trying to call a .NET Spark (C#) Notebook from a Synapse pipeline. Passing parameters in a relatively straight forward, but how to a return a result from the Notebook into the pipeline.
I think this should work:
using Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils;
[..]
var Result = _{ ... }_.ToString();
MSSparkUtils.Notebook.Exit(Result);
but it does not. Instead, I get an exception when I run the Notebook interactively
ExitValue: Exception of type 'Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookExitException' was thrown. at Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookAccessor.Exit(String value)
and that exception is returned to my pipeline as result (even if the Status is marked as Succeeded
):
[...]
"runStatus": "Succeeded",
"message": "Notebook execution is in Succeeded state",
[...]
"exitValue": "Exception of type 'Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookExitException' was thrown.\n at Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookAccessor.Exit(String value)",
"sessionId": 8,
"sparkPool": "pool1",
"lastCheckedOn": "2022-04-07T14:55:55.0766667Z"
[...]
Anybody come across this before? Thanks
-- (edit) In essence, this works in Python
mssparkutils.notebook.exit('hello')
And returns
ExitValue: hello
But this does not work in .NET Spark (C#)
MSSparkUtils.Notebook.Exit("hello")
-- (edit #2)
I am not going to mark this as answer, because it really shouldn't be, but what I am doing for now is this:
Build a DataFrame
that contains my result:
using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;
var Results = new List<string> { Result };
var df = spark.CreateDataFrame (Results);
df.CreateOrReplaceTempView("_Results");
And then switch over to a Python cell to extract and return that value.
%%pyspark
df = spark.sql("select * from _Results")
result = df.first()[0]
mssparkutils.notebook.exit( result )
And as painful as it looks (to me) - that works.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|