Skip to content

Опции

Bases: FileDFReadOptions, GenericOptions

Options for FileDFReader.

Added in 0.9.0

Examples

Note

You can pass any value supported by Spark, even if it is not mentioned in this documentation. Option names should be in camelCase!

The set of supported options depends on Spark version.

from onetl.file import FileDFReader

options = FileDFReader.Options(recursive=True)
Source code in onetl/file/file_df_reader/options.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
@support_hooks
class FileDFReaderOptions(FileDFReadOptions, GenericOptions):
    """Options for [FileDFReader][onetl.file.file_df_reader.file_df_reader.FileDFReader].

    !!! success "Added in 0.9.0"

    Examples
    --------

    !!! note

        You can pass any value [supported by Spark](https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html),
        even if it is not mentioned in this documentation. **Option names should be in** `camelCase`!

        The set of supported options depends on Spark version.

    ```python
    from onetl.file import FileDFReader

    options = FileDFReader.Options(recursive=True)
    ```
    """

    class Config:
        extra = "allow"

    recursive: Optional[bool] = Field(default=None, alias="recursiveFileLookup")
    """If `True`, perform recursive file lookup.

    !!! warning

        This disables partition inferring using file paths.

    !!! warning

        Can be used only in Spark 3+. See [SPARK-27990](https://issues.apache.org/jira/browse/SPARK-27990).
    """

    @slot
    def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader:
        """
        Apply provided format to `pyspark.sql.DataFrameReader`. [![support hooks](https://img.shields.io/badge/%20-support%20hooks-blue)](/hooks/)

        Returns
        -------
        pyspark.sql.DataFrameReader
            Reader with options applied.
        """
        options = self.dict(by_alias=True, exclude_none=True)
        return reader.options(**options)

recursive = Field(default=None, alias='recursiveFileLookup') class-attribute instance-attribute

If True, perform recursive file lookup.

Warning

This disables partition inferring using file paths.

Warning

Can be used only in Spark 3+. See SPARK-27990.

apply_to_reader(reader)

Apply provided format to pyspark.sql.DataFrameReader. support hooks

Returns

pyspark.sql.DataFrameReader Reader with options applied.

Source code in onetl/file/file_df_reader/options.py 
58
59
60
61
62
63
64
65
66
67
68
69
@slot
def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader:
    """
    Apply provided format to `pyspark.sql.DataFrameReader`. [![support hooks](https://img.shields.io/badge/%20-support%20hooks-blue)](/hooks/)

    Returns
    -------
    pyspark.sql.DataFrameReader
        Reader with options applied.
    """
    options = self.dict(by_alias=True, exclude_none=True)
    return reader.options(**options)