LLM推論高速化手法が推論結果に 与える影響の分析Analysis of the effects of LLM inference acceleration methods W&B Fully Connected 2024 株式会社リクルート Megagon Labs 松田寛. Made by Hiroshi Matsuda using W&B
Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorch
Blog Accelerating Generative AI with PyTorch II: GPT, Fast This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segmen
DeepSpeed, vLLM, CTranslate2 で rinna 3.6b の生成速度を比較する
はじめに 言語モデルを用いたテキストの生成にはtransformersライブラリが広く使われていますが、transformersライブラリは幅広いモデルに対応する一方で、テキスト生成の速度やメモリ効率には十分に最適化されていません。そこでこの記事ではテキスト生成の効率を上げるためのツールを紹介します。 今回はPyPIから簡単にインストールできるDeepSpeedとvLLM、CTranslate2を比較します。 モデルはrinna/japanese-gpt-neox-3.6b-instruction-ppoを使います。プロンプトのフォーマットやトークナイザ等の使い方についてはモデルカードをご覧ください。 この記事ではColabのT4 GPUタイプを利用してテキスト生成の速度を測定しています。それぞれのツールを試すノートブックと、Colabで開けるリンクを載せているので参考にしてみてください。
Accelerating Generative AI Part III: Diffusion, Fast – PyTorch
Blog Accelerating Generative AI Part III: Diffusion, Fast This post is the third part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment
TensorFlow Model Optimization
import tensorflow as tf import tensorflow_model_optimization as tfmot model = tf.keras.Sequential([...]) pruning_schedule = tfmot.sparsity.keras.PolynomialDecay( initial_sparsity=0.0, final_sparsity=0.5, begin_step=2000, end_step=4000) model_for_pruning = tfmot.sparsity.keras.prune_low_magnitude( model, pruning_schedule=pruning_schedule) ... model_for_pruning.fit(...) TensorFlow Model Optimization
[English ver.] [Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020. - Qiita
[English ver.] [Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020.PythonDeepLearningTensorFlowPyTorchOpenVINO Japanese English - English - 1. Introduction In this article, I'd like to share with you the quantization workflow I've been workin
![[English ver.] [Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020. - Qiita](https://cdn-ak-scissors.b.st-hatena.com/image/square/b93a6311e694073988f16ec0058be64c23ed2235/height=288;version=1;width=512/https%3A%2F%2Fqiita-user-contents.imgix.net%2Fhttps%253A%252F%252Fqiita-user-contents.imgix.net%252Fhttps%25253A%25252F%25252Fcdn.qiita.com%25252Fassets%25252Fpublic%25252Farticle-ogp-background-afbab5eb44e0b055cce1258705637a91.png%253Fixlib%253Drb-4.0.0%2526w%253D1200%2526blend64%253DaHR0cHM6Ly9xaWl0YS11c2VyLXByb2ZpbGUtaW1hZ2VzLmltZ2l4Lm5ldC9odHRwcyUzQSUyRiUyRnFpaXRhLWltYWdlLXN0b3JlLnMzLmFtYXpvbmF3cy5jb20lMkYwJTJGMTk0NzY5JTJGcHJvZmlsZS1pbWFnZXMlMkYxNTQ5NDY1NjA4P2l4bGliPXJiLTQuMC4wJmFyPTElM0ExJmZpdD1jcm9wJm1hc2s9ZWxsaXBzZSZiZz1GRkZGRkYmZm09cG5nMzImcz1mMGIxYzkzMTFlZWQ3NzNhYWYwNzkyNGE4N2E1MGQyNA%2526blend-x%253D120%2526blend-y%253D467%2526blend-w%253D82%2526blend-h%253D82%2526blend-mode%253Dnormal%2526s%253D7e1fd5b995195430331529ecb6a93078%3Fixlib%3Drb-4.0.0%26w%3D1200%26fm%3Djpg%26mark64%3DaHR0cHM6Ly9xaWl0YS11c2VyLWNvbnRlbnRzLmltZ2l4Lm5ldC9-dGV4dD9peGxpYj1yYi00LjAuMCZ3PTk2MCZoPTMyNCZ0eHQ9JTVCRW5nbGlzaCUyMHZlci4lNUQlMjAlNUJUZW5zb3JmbG93JTIwTGl0ZSU1RCUyMFZhcmlvdXMlMjBOZXVyYWwlMjBOZXR3b3JrJTIwTW9kZWwlMjBxdWFudGl6YXRpb24lMjBtZXRob2RzJTIwZm9yJTIwVGVuJUUyJTgwJUE2JnR4dC1hbGlnbj1sZWZ0JTJDdG9wJnR4dC1jb2xvcj0lMjMxRTIxMjEmdHh0LWZvbnQ9SGlyYWdpbm8lMjBTYW5zJTIwVzYmdHh0LXNpemU9NTYmdHh0LXBhZD0wJnM9OWVmZTZlYTk2MmQ5OWQxZDI0OGQ2OTY5M2U4Zjg4MDY%26mark-x%3D120%26mark-y%3D112%26blend64%3DaHR0cHM6Ly9xaWl0YS11c2VyLWNvbnRlbnRzLmltZ2l4Lm5ldC9-dGV4dD9peGxpYj1yYi00LjAuMCZ3PTgzOCZoPTU4JnR4dD0lNDBQSU5UTyZ0eHQtY29sb3I9JTIzMUUyMTIxJnR4dC1mb250PUhpcmFnaW5vJTIwU2FucyUyMFc2JnR4dC1zaXplPTM2JnR4dC1wYWQ9MCZzPTIyMDgzNTMyZDRjMGQyZjUyN2JhZGQyM2EwYTY4Zjk1%26blend-x%3D242%26blend-y%3D480%26blend-w%3D838%26blend-h%3D46%26blend-fit%3Dcrop%26blend-crop%3Dleft%252Cbottom%26blend-mode%3Dnormal%26s%3D16df54595941a556626187fac3eeb882)
ONNXの最適化まとめ - ぱたへね
ONNXの最適化を一通り試してみたのでまとめ。 サポートしている最適化一覧の取得 サポートしている最適化は、get_available_passesで取得できます。 from onnx import optimizer all_passes = optimizer.get_available_passes() 大きく分けると、このように分類できます。 意味のないOpの削除 （eliminate_deadend等） 2つのOpのfusion （fuse_matmul_add_bias_into_gemm等） Convへのfusion （fuse_add_bias_into_conv等） その他 convへのfuseは全く動かず、バージョンアップ待ちです。 最適化の結果 Qiitaにそれぞれまとめました。 ONNXでeliminate_deadend 最適化 ONNXで eliminate_i
[Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020. - Qiita
[Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020.PythonDeepLearningTensorFlowPyTorchOpenVINO 日本語 English - Japanese - 1. Introduction 今回は私が半年間掛けてためてきた、学習済みモデルの量子化ワークフローをメモがてら共有したいと思います。 Tensorflow の checkpoint (.ckpt/.meta)、 FreezeGraph (.
![[Tensorflow Lite] Various Neural Network Model quantization methods for Tensorflow Lite (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization, EdgeTPU). As of May 05, 2020. - Qiita](https://cdn-ak-scissors.b.st-hatena.com/image/square/d351fa289a4988d5d2e90a565874464556e1b49a/height=288;version=1;width=512/https%3A%2F%2Fqiita-user-contents.imgix.net%2Fhttps%253A%252F%252Fqiita-user-contents.imgix.net%252Fhttps%25253A%25252F%25252Fcdn.qiita.com%25252Fassets%25252Fpublic%25252Farticle-ogp-background-afbab5eb44e0b055cce1258705637a91.png%253Fixlib%253Drb-4.0.0%2526w%253D1200%2526blend64%253DaHR0cHM6Ly9xaWl0YS11c2VyLXByb2ZpbGUtaW1hZ2VzLmltZ2l4Lm5ldC9odHRwcyUzQSUyRiUyRnFpaXRhLWltYWdlLXN0b3JlLnMzLmFtYXpvbmF3cy5jb20lMkYwJTJGMTk0NzY5JTJGcHJvZmlsZS1pbWFnZXMlMkYxNTQ5NDY1NjA4P2l4bGliPXJiLTQuMC4wJmFyPTElM0ExJmZpdD1jcm9wJm1hc2s9ZWxsaXBzZSZiZz1GRkZGRkYmZm09cG5nMzImcz1mMGIxYzkzMTFlZWQ3NzNhYWYwNzkyNGE4N2E1MGQyNA%2526blend-x%253D120%2526blend-y%253D467%2526blend-w%253D82%2526blend-h%253D82%2526blend-mode%253Dnormal%2526s%253D7e1fd5b995195430331529ecb6a93078%3Fixlib%3Drb-4.0.0%26w%3D1200%26fm%3Djpg%26mark64%3DaHR0cHM6Ly9xaWl0YS11c2VyLWNvbnRlbnRzLmltZ2l4Lm5ldC9-dGV4dD9peGxpYj1yYi00LjAuMCZ3PTk2MCZoPTMyNCZ0eHQ9JTVCVGVuc29yZmxvdyUyMExpdGUlNUQlMjBWYXJpb3VzJTIwTmV1cmFsJTIwTmV0d29yayUyME1vZGVsJTIwcXVhbnRpemF0aW9uJTIwbWV0aG9kcyUyMGZvciUyMFRlbnNvcmZsb3clMjBMaXRlJTIwJUUyJTgwJUE2JnR4dC1hbGlnbj1sZWZ0JTJDdG9wJnR4dC1jb2xvcj0lMjMxRTIxMjEmdHh0LWZvbnQ9SGlyYWdpbm8lMjBTYW5zJTIwVzYmdHh0LXNpemU9NTYmdHh0LXBhZD0wJnM9NzJmMjRjYWNiYjJjNDI4OTQwNzkxZTdmZWZkYjVlZTk%26mark-x%3D120%26mark-y%3D112%26blend64%3DaHR0cHM6Ly9xaWl0YS11c2VyLWNvbnRlbnRzLmltZ2l4Lm5ldC9-dGV4dD9peGxpYj1yYi00LjAuMCZ3PTgzOCZoPTU4JnR4dD0lNDBQSU5UTyZ0eHQtY29sb3I9JTIzMUUyMTIxJnR4dC1mb250PUhpcmFnaW5vJTIwU2FucyUyMFc2JnR4dC1zaXplPTM2JnR4dC1wYWQ9MCZzPTIyMDgzNTMyZDRjMGQyZjUyN2JhZGQyM2EwYTY4Zjk1%26blend-x%3D242%26blend-y%3D480%26blend-w%3D838%26blend-h%3D46%26blend-fit%3Dcrop%26blend-crop%3Dleft%252Cbottom%26blend-mode%3Dnormal%26s%3D332ad5d4f52234cbfb6c5faad533bfbe)
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
GitHub - huggingface/optimum-nvidia
Distributed inference
Deploy Your Local GPT Server With Triton
GitHub - NVIDIA/FasterTransformer: Transformer related optimization, including BERT, GPT
NVIDIA Triton Inference Server on AWS: Customer success stories and AWS deployment methods to optimize inference throughput, reduce latency, and lower GPU or CPU inference costs. SE31488 | GTC Digital November 2021 | NVIDIA On-Demand
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
GitHub - PINTO0309/PINTO_model_zoo: A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
GitHub - pfnet-research/chainer-trt: Chainer x TensorRT
benchmarking-hardware-for-cnn-inference-in-2018-1d58268de12a