thunder.distributed.FSDPBucketingStrategy¶

class thunder.distributed.FSDPBucketingStrategy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶

Bases: Enum

Specify how we group parameters into a bucket for collective communication in fsdp.

NONE[source]¶: Disables bucketing.

LAYER[source]¶: Creates buckets per layer such as torch.nn.Linear and torch.nn.LayerNorm.

BLOCK[source]¶: Creates buckets per block such as Transformer block.

__init__(*args, **kwds)[source]¶

Attributes

`NONE`
`LAYER`
`BLOCK`