Skip to content

Конфигурация

Bases: BaseSettings

Worker application settings.

This class is used to configure various settings for the worker application. The settings can be passed in several ways:

  1. By storing settings in a configuration file config.yml (preferred).
  2. By setting environment variables matching specific keys (SYNCMASTER__DATABASE__URL == database.url).
  3. By explicitly passing a settings object as an argument of application factory function.

More details can be found in Pydantic documentation <https://docs.pydantic.dev/latest/concepts/pydantic_settings/>_.

Examples

.. code-block:: yaml :caption: config.yml

database:
    url: postgresql+asyncpg://postgres:postgres@localhost:5432/syncmaster

broker:
    url: amqp://user:password@localhost:5672/

logging: {}
encryption: {}
worker: {}
hwm_store: {}
Source code in syncmaster/worker/settings/__init__.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
class WorkerAppSettings(BaseSettings):
    """
    Worker application settings.

    This class is used to configure various settings for the worker application.
    The settings can be passed in several ways:

    1. By storing settings in a configuration file ``config.yml`` (preferred).
    2. By setting environment variables matching specific keys (``SYNCMASTER__DATABASE__URL`` == ``database.url``).
    3. By explicitly passing a settings object as an argument of application factory function.

    More details can be found in
    `Pydantic documentation <https://docs.pydantic.dev/latest/concepts/pydantic_settings/>`_.

    Examples
    --------

    .. code-block:: yaml
        :caption: config.yml

        database:
            url: postgresql+asyncpg://postgres:postgres@localhost:5432/syncmaster

        broker:
            url: amqp://user:password@localhost:5672/

        logging: {}
        encryption: {}
        worker: {}
        hwm_store: {}
    """

    database: DatabaseSettings = Field(default_factory=DatabaseSettings, description="Database settings")  # type: ignore[arg-type]
    broker: RabbitMQSettings = Field(default_factory=RabbitMQSettings, description="Broker settings")  # type: ignore[arg-type]
    logging: LoggingSettings = Field(default=DEFAULT_LOGGING_SETTINGS, description="Logging settings")
    worker: WorkerSettings = Field(description="Worker-specific settings")
    encryption: CredentialsEncryptionSettings = Field(
        default_factory=CredentialsEncryptionSettings,  # type: ignore[arg-type]
        description="Settings for encrypting credential data",
    )
    hwm_store: HWMStoreSettings = Field(default_factory=HWMStoreSettings, description="HWM Store settings")

Bases: BaseModel

Celery worker settings.

Examples

.. code-block:: yaml :caption: config.yml

worker:
    log_url_template: "https://logs.location.example.com/syncmaster-worker?correlation_id={{ correlation_id }}&run_id={{ run.id }}"

    create_spark_session_function: custom_syncmaster.spark.get_worker_spark_session
    spark_session_default_config:
        spark.master: local
        spark.driver.host: 127.0.0.1
        spark.driver.bindAddress: 0.0.0.0
        spark.sql.pyspark.jvmStacktrace.enabled: true
        spark.ui.enabled: false
Source code in syncmaster/worker/settings/__init__.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
class WorkerSettings(BaseModel):
    """Celery worker settings.

    Examples
    --------

    .. code-block:: yaml
        :caption: config.yml

        worker:
            log_url_template: "https://logs.location.example.com/syncmaster-worker?correlation_id={{ correlation_id }}&run_id={{ run.id }}"

            create_spark_session_function: custom_syncmaster.spark.get_worker_spark_session
            spark_session_default_config:
                spark.master: local
                spark.driver.host: 127.0.0.1
                spark.driver.bindAddress: 0.0.0.0
                spark.sql.pyspark.jvmStacktrace.enabled: true
                spark.ui.enabled: false
    """  # noqa: E501

    create_spark_session_function: ImportString = Field(
        "syncmaster.worker.spark.get_worker_spark_session",
        description="Function to create Spark session for worker",
        validate_default=True,
    )
    spark_session_default_config: dict[str, Any] = Field(
        default_factory=dict,
        description="Default Spark session configuration",
    )
    log_url_template: str = Field(
        "",
        description=":ref:`URL template to access worker logs <worker-log-url>`",
    )