Skip to content

Опции File Mover

Bases: GenericOptions

File moving options.

Added in 0.8.0

Examples

from onetl.file import FileMover

options = FileMover.Options(
    if_exists="replace_entire_directory",
    workers=4,
)
Source code in onetl/file/file_mover/options.py 
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
class FileMoverOptions(GenericOptions):
    """File moving options.

    !!! success "Added in 0.8.0"

    Examples
    --------

    ```python
    from onetl.file import FileMover

    options = FileMover.Options(
        if_exists="replace_entire_directory",
        workers=4,
    )
    ```
    """

    if_exists: FileExistBehavior = Field(  # type: ignore[literal-required]
        default=FileExistBehavior.ERROR,
        alias=avoid_alias("mode"),
    )
    """
    How to handle existing files in the local directory.

    Possible values:
        * `error` (default) - mark file as failed
        * `ignore` - mark file as skipped
        * `replace_file` - replace existing file with a new one
        * `replace_entire_directory` - delete directory content before moving files

    !!! success "Added in 0.8.0"

    !!! info "Changed in 0.9.0"
        Renamed `mode` → `if_exists`
    """

    workers: int = Field(default=1, ge=1)
    """
    Number of workers to create for parallel file moving.

    1 (default) means files will me moved sequentially.
    2 or more means files will be moved in parallel workers.

    Recommended value is `min(32, os.cpu_count() + 4)`, e.g. `5`.

    !!! success "Added in 0.8.1"
    """

    @root_validator(pre=True)
    def _mode_is_deprecated(cls, values):
        if "mode" in values:
            warnings.warn(
                "Option `FileMover.Options(mode=...)` is deprecated since v0.9.0 and will be removed in v1.0.0. "
                "Use `FileMover.Options(if_exists=...)` instead",
                category=UserWarning,
                stacklevel=5,
            )
        return values

if_exists = Field(default=(FileExistBehavior.ERROR), alias=(avoid_alias('mode'))) class-attribute instance-attribute

How to handle existing files in the local directory.

Possible values
  • error (default) - mark file as failed
  • ignore - mark file as skipped
  • replace_file - replace existing file with a new one
  • replace_entire_directory - delete directory content before moving files

Added in 0.8.0

Changed in 0.9.0

Renamed mode → if_exists

workers = Field(default=1, ge=1) class-attribute instance-attribute

Number of workers to create for parallel file moving.

1 (default) means files will me moved sequentially. 2 or more means files will be moved in parallel workers.

Recommended value is min(32, os.cpu_count() + 4), e.g. 5.

Added in 0.8.1