Summing columns in remote Parquet files using DuckDB

vivym/midjourney-messages on Hugging Face is a large (~8GB) dataset consisting of 55,082,563 Midjourney images - each one with the prompt and a URL to the image hosted on Discord. TLDR: Each record links to a Discord CDN URL, and the total size of all of those images is 148.07 TB - so Midjourney has cost Discord a LOT of money in CDN costs! Read on for the details of how I figured this out. Each o

[secondlife]

おおお、huggingface の　duckdb 便利なんだけど、連番get面倒と思っていたらあの手この手が。必要なカラムだけの転送で済むのも良さそうだなぁ。