higgsfield-ai/higgsfield

Largest clones in the code

• 45 lines in 2 placespython271 tokens
  higgsfield/llama/llama.py:182–226·higgsfield/mistral/mistral.py:165–209

  182def __call__(self, batch):
  183    local_rank = int(os.environ["LOCAL_RANK"])
  184
  185    for key in batch.keys():
  186        batch[key] = batch[key].to(local_rank)
  187
  188    if self.precision == "fp16":
  189        with torch.cuda.amp.autocast():
  190            loss = super().__call__(**batch).loss
  191    else:
  

  First 10 of 45 lines.

• 49 lines in 2 placespython177 tokens
  higgsfield/llama/llama.py:121–169·higgsfield/mistral/mistral.py:103–151

  121        LlamaDecoderLayer,
  122    }
  123)
  124
  125if zero_stage == 0:
  126    sharding_strategy = ShardingStrategy.NO_SHARD
  127
  128elif zero_stage == 1:
  129    raise NotImplementedError("stage 1 is not supported. Only 0 2 3")
  130
  

  First 10 of 49 lines.

• 38 lines in 2 placespython119 tokens
  higgsfield/llama/llama.py:83–120·higgsfield/mistral/mistral.py:64–102

  83            model = LlamaForCausalLM(llama_config)
  84
  85if fast_attn:
  86    #raise NotImplementedError("Fast attention is not supported yet")
  87    model = BetterTransformer.transform(model)
  88
  89fpSixteen = MixedPrecision(
  90    param_dtype=torch.float16,
  91    reduce_dtype=torch.float16,
  92    buffer_dtype=torch.float16,
  

  64    model.resize_token_embeddings(num_embeddings)
  65
  66
  67if fast_attn:
  68    #raise NotImplementedError("Fast attention is not supported yet")
  69    model = BetterTransformer.transform(model)
  70
  71fpSixteen = MixedPrecision(
  72    param_dtype=torch.float16,
  73    reduce_dtype=torch.float16,
  

  First 10 of 38 lines, the second copy on the right.