Category "scala"

Visualizing dependency size via sbt assembly

I am using sbt assembly to package my project into one fat jar. I wanted to understand if I can visualize how much disk space is each dependency taking in my fa

In multi-stage compilation, should we use a standard serialisation method to ship objects through stages?

This question is formulated in Scala 3/Dotty but should be generalised to any language NOT in MetaML family. The Scala 3 macro tutorial: https://docs.scala-lang

scala add _2 s in a list of Tuple 2

I have the following mutable Hashmap in Scala: HashMap((b,3), (c,4), (a,8), (a,2)) and need to be converted to the following: HashMap((b,3), (c,4), (a,10)) I

command 'scala' not found after set up by coursier

I use coursier to set up the environment of Scala using command ./cs setup in Ubuntu 20.04 LTS. It was said that all the requirements were installed. However, w

Update a highly nested column from string to struct

|-- x: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- y: long (nullable = true) | | |-- z: array (nullable = tru

Spark - Update a nested column to string

|-- x: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- y: struct (nullable = true) | | |-- z: struct (nullable =

What causes AbstractMethodError for scala implicit val?

I have a gears library, depending on "org.typelevel" %% "cats-core" % "2.7.0", "org.typelevel" %% "cats-effect" % "2.5.4"

Can't figure out how to change Prometheus content type header

So my metrics all appear in one line at my end-point, not in new line per metric. I use micrometer, spring, prometheus and scala. My controller: @RequestMapping

Generate Kafka message with Headers using Apache Spark

I have an ETL (spark-scala). After writing in a table, a message with "header" must be sent to Kafka. I couldn't add the header in the message. I have a spark D

How to get list of all leaf folders from ADLS Gen2 path via Scala code?

We have folders and subfolders in it with year,month, day folders in it. How can we get only the last leaf level folder list using dbutils.fs.ls utility? Exampl

Create a collection of case class names

I'm working in Spark 3.1 with Scala 2.12.10. I'd like to create a collection (Seq, whatever) of case classes that have implemented a common trait, so I can exec

ChannelClosedException - Upstream Address: Not Available

Our scala/thrift service is using twitter finagle 2.12-18.10.0 with java8. The Service is working fine for single request or handful of requests but when we try

tail recursion count, split & get the prefix of a list functions

Language: Scala I'm working on some tail recursion questions in Scala. (GOAL) A function that counts the number of ones in a list. The function takes a list of

Performing a groupBy on a dataframe while limiting the number of rows

I have a dataframe that contains an "id" column and a "publication" column. The "id" column contains duplicates, and represents a researcher. The "publication"

Using K0.ProductInstances in shapeless3

I have a library with typeclasses that I am migrating to Scala 3 using shapeless-3. One of my typeclasses is: trait Parser[T] { def parse(ctx: Context): (Opt

Cannot pass arguments using sbt to gatling simulation

Regarding to Gatling SBT execute a specific simulation topic is there any way to pass argument to simulation? I've been trying passing command from any CLI like

flink cluster with zookeeper HA always shutdown: [RECEIVED SIGNAL 15: SIGTERM]

Environment: flink1.14.4 standalone application mode in kubernetes according to official steps: flink cluster: https://nightlies.apache.org/flink/flink-docs-rel

How can I use snowflake jar in Bitnami Spark Docker container?

I was able to create docker based bitnami stand alone spark instance and run spark jobs on it. However I'm not able not able to write data to snowflake from the

Extract value from ArrayType column in Scala and reshape to long

I have a DataFrame that consists of Column that is ArrayType, and the array may have a different length in each row of the data. I have provide some example cod

How to find position of substring in another column of dataframe using spark scala

I have a Spark scala DataFrame with two columns, text and subtext, where subtext is guaranteed to occur somewhere within text. How would I calculate the positio