A Guide to Listing Files and Directories with (Py)Spark, or How To Summon the Beast
Different methods for traversing file-systems with PySpark. From the mundane to the occult.
pyspark
A collection of 2 posts
Increase PySpark's JDBC parallelism through predicates
Increase the parallelism of loading data through JDBC with Spark using predicates on non-numeric columns.