microgpt
microgpt.py P%]� �tP� """ The most atomic way to train and run inference for a GPT in pure, dependency-free Python. This file is the complete algorithm. Everything else is just efficiency. @karpathy """ import os # os.path.exists import math # math.log, math.exp import random # random.seed, random.choices, random.gauss, random.shuffle random.seed(42) # Let there be order among chaos # Let there be
microgpt
This is a brief guide to my new art project microgpt, a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT. This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency.
Over the course of my Spring 2020 semester at Harvey Mudd College, I developed a self-hosting compiler entirely from scratch. This article walks through many interesting parts of the project. It’s laid out so you can just read from beginning to end, but if you’re more interested in a particular topic, feel free to jump there. Or, take a look at the project on GitHub. Table of contents What the pro
Supply Chain Attack on Axios Pulls Malicious Dependency from...
Update 4/1: We dug deeper into the hidden blast radius of this attack and how dependency resolution expanded its impact: https://socket.dev/blog/hidden-blast-radius-of-the-axios-compromiseA supply chain attack targeting the widely used HTTP client Axios has introduced a malicious dependency into specific npm releases, including axios@1.14.1 and axios@0.30.4. The latest version pulls in plain-crypt
Eliciting Reasoning in Language Models with Cognitive Tools Brown Ebouky IBM Research - Zurich ETH Zurich Brown.Ebouky@ibm.com Andrea Bartezzaghi IBM Research - Zurich abt@zurich.ibm.com Mattia Rigotti IBM Research - Zurich mrg@zurich.ibm.com Abstract The recent advent of reasoning models like OpenAI’s o1 was met with excited spec- ulation by the AI community about the mechanisms underlying these
This year’s Advent of Code has been brutal (compare the stats of 2023 with that of 2022, especially day 1 part 1 vs. day 1 part 2). It included a problem to solve with dynamic programming as soon as day 12, which discouraged some people I know. This specific problem was particularly gnarly for Advent of Code, with multiple special cases to take into account, making it basically intractable if you
The Alkyne GC · mcyoung
Alkyne is a scripting language I built a couple of years ago for generating configuration blobs. Its interpreter is a naive AST walker1 that uses ARC2 for memory management, so it’s pretty slow, and I’ve been gradually writing a new evaluation engine for it. This post isn’t about Alkyne itself, that’s for another day. For now, I’d like to write down some notes for the GC I wrote3 for it, and more
The simplicity of Prolog
Back to homepage Nowadays the most popular programming languages are Python, Javascript, Java, C++, C#, Kotlin and Ruby, and the average programmer is probably familiar with one or more of these languages. It's relatively easy to switch from one to another (barring any framework specific knowledge that may be needed), since they are all imperative (and for the most part object-oriented) languages,
I find blockchain fascinating because it extends open source software development to open source + state. This seems to be a genuine/exciting innovation in computing paradigms; We don’t just get to share code, we get to share a running computer, and anyone anywhere can use it in an open and permissionless manner. The seeds of this revolution arguably began with Bitcoin, so I became curious to dril
std::flip
std::flip is a little-known utility from the C++ standard library header <functional>: it is a higher-order function that accepts a Callable and returns an equivalent Callable with the order of its parameters reversed (or “flipped”). To understand how it can be useful, let’s start with a simple example. Consider the following tree node class: struct node { int value; node* parent = nullptr; node*
In October 2023, a group of authors from the Center for AI Safety, among others, published Representation Engineering: A Top-Down Approach to AI Transparency. That paper looks at a few methods of doing what they call "Representation Engineering": calculating a "control vector" that can be read from or added to model activations during inference to interpret or control the model's behavior, without
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Kalyn: a self-hosting compiler for x86-64
Eliciting Reasoning in Language Models with Cognitive Tools
Dynamic Programming is not Black Magic - Quentin Santos
A from-scratch tour of Bitcoin in Python
Representation Engineering Mistral-7B an Acid Trip