Skip to content

graflo.connections.bulk

Tuning of a bulk load, carried on a database config as bulk.

Nothing here turns bulk on: a bulk run is asked for by the ingestion mode.

Classes

BulkConfig

Bases: BaseModel

How a bulk load folds and writes; every field has a working default.

Source code in graflo/connections/bulk.py 
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class BulkConfig(BaseModel):
    """How a bulk load folds and writes; every field has a working default."""

    model_config = ConfigDict(extra="forbid")

    chunk_size: int = Field(
        default=50_000,
        gt=0,
        description="Folded rows per chunk: the unit a writer commits and resume skips.",
    )
    parallelism: int = Field(
        default=4,
        gt=0,
        description="Chunks written at once; a backend may allow fewer.",
    )
    memory_limit: str | None = Field(
        default=None,
        description="Memory the fold may use before spilling to the staging directory "
        "(DuckDB syntax, e.g. '8GB'); DuckDB's default when unset.",
    )
    fold_threads: int | None = Field(
        default=None,
        gt=0,
        description="Threads the fold uses; DuckDB's default when unset.",
    )

Attributes

chunk_size = Field(default=50000, gt=0, description='Folded rows per chunk: the unit a writer commits and resume skips.') class-attribute instance-attribute
fold_threads = Field(default=None, gt=0, description="Threads the fold uses; DuckDB's default when unset.") class-attribute instance-attribute
memory_limit = Field(default=None, description="Memory the fold may use before spilling to the staging directory (DuckDB syntax, e.g. '8GB'); DuckDB's default when unset.") class-attribute instance-attribute
model_config = ConfigDict(extra='forbid') class-attribute instance-attribute
parallelism = Field(default=4, gt=0, description='Chunks written at once; a backend may allow fewer.') class-attribute instance-attribute