Skip to content

Опции

Bases: FileDFReadOptions, GenericOptions

Options for :obj:FileDFReader <onetl.file.file_df_reader.file_df_reader.FileDFReader>.

.. versionadded:: 0.9.0

Examples

.. note::

You can pass any value `supported by Spark <https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html>`_,
even if it is not mentioned in this documentation. **Option names should be in** ``camelCase``!

The set of supported options depends on Spark version.

.. code:: python

from onetl.file import FileDFReader

options = FileDFReader.Options(recursive=True)
Source code in onetl/file/file_df_reader/options.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
@support_hooks
class FileDFReaderOptions(FileDFReadOptions, GenericOptions):
    """Options for :obj:`FileDFReader <onetl.file.file_df_reader.file_df_reader.FileDFReader>`.

    .. versionadded:: 0.9.0

    Examples
    --------

    .. note::

        You can pass any value `supported by Spark <https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html>`_,
        even if it is not mentioned in this documentation. **Option names should be in** ``camelCase``!

        The set of supported options depends on Spark version.

    .. code:: python

        from onetl.file import FileDFReader

        options = FileDFReader.Options(recursive=True)
    """

    class Config:
        extra = "allow"

    recursive: Optional[bool] = Field(default=None, alias="recursiveFileLookup")
    """If ``True``, perform recursive file lookup.

    .. warning::

        This disables partition inferring using file paths.

    .. warning::

        Can be used only in Spark 3+. See `SPARK-27990 <https://issues.apache.org/jira/browse/SPARK-27990>`_.
    """

    @slot
    def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader:
        """
        Apply provided format to :obj:`pyspark.sql.DataFrameReader`. |support_hooks|

        Returns
        -------
        :obj:`pyspark.sql.DataFrameReader` with options applied
        """
        options = self.dict(by_alias=True, exclude_none=True)
        return reader.options(**options)

recursive = Field(default=None, alias='recursiveFileLookup') class-attribute instance-attribute

If True, perform recursive file lookup.

.. warning::

This disables partition inferring using file paths.

.. warning::

Can be used only in Spark 3+. See `SPARK-27990 <https://issues.apache.org/jira/browse/SPARK-27990>`_.

apply_to_reader(reader)

Apply provided format to :obj:pyspark.sql.DataFrameReader. |support_hooks|

Returns

:obj:pyspark.sql.DataFrameReader with options applied

Source code in onetl/file/file_df_reader/options.py 
58
59
60
61
62
63
64
65
66
67
68
@slot
def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader:
    """
    Apply provided format to :obj:`pyspark.sql.DataFrameReader`. |support_hooks|

    Returns
    -------
    :obj:`pyspark.sql.DataFrameReader` with options applied
    """
    options = self.dict(by_alias=True, exclude_none=True)
    return reader.options(**options)