GitHub - kyegomez/BitNet: Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

PyTorch Implementation of the linear methods and model from the paper "BitNet: Scaling 1-bit Transformers for Large Language Models" Paper link: BitLinear = tensor -> layernorm -> Binarize -> abs max quantization -> dequant "The implementation of the BitNet architecture is quite simple, requiring only the replacement of linear projections (i.e., nn.Linear in PyTorch) in the Transformer. " -- BitNe

[stealthinu]

BitNetのアンオフィシャル実装。ちゃんときれいに学習進むらしい。めちゃすごい勢いで実装が進んでいくな。

[misshiki]

“BitNet”公式リポジトリ。

[mickn]

1-bit LLM