'Azure Synapse: return value from .NET Spark (C#) Notebook to pipeline exception

I am trying to call a .NET Spark (C#) Notebook from a Synapse pipeline. Passing parameters in a relatively straight forward, but how to a return a result from the Notebook into the pipeline.

I think this should work:

using Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils;

[..]

var Result = _{ ... }_.ToString();
MSSparkUtils.Notebook.Exit(Result);

but it does not. Instead, I get an exception when I run the Notebook interactively

ExitValue: Exception of type 'Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookExitException' was thrown. at Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookAccessor.Exit(String value)

and that exception is returned to my pipeline as result (even if the Status is marked as Succeeded):

[...]
    "runStatus": "Succeeded",
     "message": "Notebook execution is in Succeeded state",
[...]
    "exitValue": "Exception of type 'Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookExitException' was thrown.\n   at Microsoft.Spark.Extensions.Azure.Synapse.Analytics.Notebook.MSSparkUtils.Internal.NotebookAccessor.Exit(String value)",
    "sessionId": 8,
    "sparkPool": "pool1",
    "lastCheckedOn": "2022-04-07T14:55:55.0766667Z"
[...]

Anybody come across this before? Thanks

-- (edit) In essence, this works in Python

mssparkutils.notebook.exit('hello')

And returns

ExitValue: hello

But this does not work in .NET Spark (C#)

MSSparkUtils.Notebook.Exit("hello")

-- (edit #2)

I am not going to mark this as answer, because it really shouldn't be, but what I am doing for now is this:

Build a DataFrame that contains my result:

using Microsoft.Spark.Sql;
using Microsoft.Spark.Sql.Types;

var Results = new List<string> { Result };
var df = spark.CreateDataFrame (Results);

df.CreateOrReplaceTempView("_Results");

And then switch over to a Python cell to extract and return that value.

%%pyspark

df = spark.sql("select * from _Results")
result = df.first()[0]
mssparkutils.notebook.exit( result )

And as painful as it looks (to me) - that works.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source