Category "scala"

Spark scala data frame udf returning rows

Say I have an dataframe which contains a column (called colA) which is a seq of row. I want to to append a new field to each record of colA. (And the new filed

Error while running Scala code - Databricks 7.3LTS and above

I am running databricks 7.3LTS and having errors while trying to use scala bulk copy. The error is: object sqldb is not a member of package com.microsoft. I hav

Summation/multiplication of a list of tuples

I'm trying to figure out a simple operation that takes a list of (Int, Int) tuples and multiplies the tuples internally and then sums those results. Example: v

Get only Right values from Either sequence

This code prints List(1, 2, ()) def f1(i:Int): Either[String,Int] = if (i > 0) Right(1) else Left("error 1") def f2(i:Int): Either[String,Int] = if

How to convert timestamp column of Spark Dataframe to string column

I want to convert Spark dataframe all TIMESTAMP columns into String columns. Could anybody say how to do that automatically for each dataframe? The position of

Spark SQL error : org.apache.spark.sql.catalyst.parser.ParseException: extraneous input '$' expecting

I am forming a query in a String Builder like below : println(dataQuery) Execution started at 2019-10-31 02:58:24.006019 PST res245: String = " SELECT transac

Importing classes into Scala worksheet

In my scala worksheet test.sc I define : import com.tradedata.VolatilityInstance test.sc exists at src/main/scala In location src/main/scala I define : pa

No endpoint could be found for: test, please check your classpath contains the needed Camel component jar

I am trying to send and receive messages using akka-camel and created a sample example for producer and consumer like below : Producer: import akka.actor.{Act

Automatically deriving a transformer from a rich case class to a simple case class?

I have a simple case class that represents a pet: case class Pet(name: String, age: Int) Now say I have a case class that has a 1:1 mapping to Pet, but where a

scala string Tail recursion with pattern match

I am trying to write a tail recursion function to reverse string, here is the code, some reason I am not sure if the pattern match condition is preventing it fr

Scala Play Framework: cannot generate object from json with null values

I'm new to Scala and the Play Framework. I have written the following controller: @Singleton class MyController @Inject()(val controllerComponents: ControllerCo

sonarQube Findbug error and ##[error]java.lang.IllegalStateException: Can not execute Findbugs

I am really struggling from months. We are trying to scan SCALA code with SonarQube in Azure Devops which is in Databricks. We were getting around 30 error. But

Finding how many times a given string is a substring of another string using scala

what is the elegant way for finding how many times a given string is a substring of another string using scala? The below test cases that should make clear wh

dataframe Spark scala explode json array

Let's say I have a dataframe which looks like this: +--------------------+--------------------+--------------------------------------------------------------+

How to escape ( parentheses in Spark Scala?

I am trying to replace parentheses in a string (i.e. column names). It is working fine with white spaces but not with ( parentheses. I tried """, \(, \\( but I

Spark 3.0 is much slower to read json files than Spark 2.4

I have large amount of json files that Spark can read in 36 seconds but Spark 3.0 takes almost 33 minutes to read the same. On closer analysis, looks like Spark

Akka: persist to Cassandra and publish to Kafka multiple events

I need to store to Cassandra and publish to Kafka multiple events, and call some final handler() only after all events are stored and published. I came across U

cats-effect:How to transform Map[x,IO[y]] to IO[Map[x,y]]

I have a map of string to IO like this Map[String, IO[String]], I want to transform it into IO[Map[String, String]]. How to do it?

How to organize java and scala code in Play?

activator new results in: Fetching the latest list of templates... Browse the list of templates: http://lightbend.com/activator/templates Choose from these f

List All objects in S3 with given Prefix in scala

I am trying list all objects in AWS S3 Buckets with input Bucket Name & Filter Prefix using following code. import scala.collection.JavaConverters._ import