[SPARK-50917][EXAMPLES] Add SparkConnectPi Scala example to work both for Connect and Classic

[yaooqinn]

What changes were proposed in this pull request?

This PR adds SparkConnectPi Scala example to work both for Connect and Classic

Why are the changes needed?

The SparkPi example, mostly as the first step for users to get to know Spark, should be able to run on Spark Connect mode.

Does this PR introduce any user-facing change?

no

How was this patch tested?

Manually build and test

bin/spark-submit --remote 'sc://localhost' --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.13-4.1.0-SNAPSHOT.jar
WARNING: Using incubator modules: jdk.incubator.vector
25/01/23 15:00:03 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
25/01/23 15:00:03 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
25/01/23 15:00:03 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
Pi is roughly 3.1388756943784717
25/01/23 15:00:04 INFO ShutdownHookManager: Shutdown hook called
25/01/23 15:00:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/84/dgr9ykwn6yndcmq1kjxqvk200000gn/T/spark-25ed842e-5888-47ce-bb0b-442385d643cb

Was this patch authored or co-authored using generative AI tooling?

no

[yaooqinn]

cc @cloud-fan @dongjoon-hyun @HyukjinKwon, thank you!

[HyukjinKwon]

Actually here is Core examples (https://github.com/apache/spark/tree/7f1158badeccfaaa763a3f39e5fc63a66029b52e/examples/src/main/scala/org/apache/spark/examples). SQL examples should be in https://github.com/apache/spark/tree/7f1158badeccfaaa763a3f39e5fc63a66029b52e/examples/src/main/scala/org/apache/spark/examples/sql

[yaooqinn]

Actually here is Core examples (https://github.com/apache/spark/tree/7f1158badeccfaaa763a3f39e5fc63a66029b52e/examples/src/main/scala/org/apache/spark/examples). SQL examples should be in https://github.com/apache/spark/tree/7f1158badeccfaaa763a3f39e5fc63a66029b52e/examples/src/main/scala/org/apache/spark/examples/sql

make sense

[dongjoon-hyun]

I understand what you aim, but this is not SQL in a user perspective, @yaooqinn .

We should distinguish SQL vs Spark Connect because Apache Spark already has Spark SQL modules and user interfaces like JDBC and spark-sql shell. Could you revise the name, 😄 ?

[yaooqinn]

I'd rename it with the FQDN as org.apache.spark.examples.sql.connect.SparkConnectPi

[dongjoon-hyun]

Thank you. Please revise the PR title and description accordingly too.

Addressed

[cloud-fan]

How is this different from the SQL example? My understanding is that the example should just use public SQL/DataFrame APIs and then it will work for both classic and Spark Connect. We should encourage users to use Spark SQL correctly (don't rely on private APIs), and in the example we can enable or disable Spark Connect w.r.t. the arguments.

[yaooqinn]

IIUC，this example seem to be the exact thing you described. Or you were just concerning about the classname？

[dongjoon-hyun]

To @yaooqinn , the PR description seems to be outdated still~

bin/spark-submit --remote 'sc://localhost' --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.13-4.1.0-SNAPSHOT.jar