dbt (data build tool) を使ってデータをテストする - CUBE SUGAR CONTAINER
ソフトウェアエンジニアリングの世界では、自動化されたテストを使ってコードの振る舞いを検証するのが当たり前になっている。 同じように、データエンジニアリングの世界でも、自動化されたテストを使ってデータの振る舞いを検証するのが望ましい。 データをテストするのに使える OSS のフレームワークも、いくつか存在する。 今回は、その中でも dbt (data build tool) を使ってデータをテストする方法について見ていく。 dbt 自体はデータのテストを主目的としたツールではないものの、テストに関する機能も備えている。 また、dbt には WebUI を備えたマネージドサービスとしての dbt Cloud と、CLI で操作するスタンドアロン版の dbt Core がある。 今回扱うのは後者の dbt Core になる。 使った環境は次のとおり。 $ sw_vers ProductName:
Weird Lexical Syntax
I just learned 42 programming languages this month to build a new syntax highlighter for llamafile. I feel like I'm up to my eyeballs in programming languages right now. Now that it's halloween, I thought I'd share some of the spookiest most surprising syntax I've seen. The languages I decided to support are Ada, Assembly, BASIC, C, C#, C++, COBOL, CSS, D, FORTH, FORTRAN, Go, Haskell, HTML, Java,
しくじり先生のように学ぶ「NFS+sqliteで苦労した話から学ぶ、問題解決の考え方」という勉強会をやってみました | Raccoon Tech Blog [株式会社ラクーンホールディングス 技術戦略部ブログ]
株式会社ラクーンホールディングスのエンジニア/デザイナーから技術情報をはじめ、世の中のためになることや社内のことなどを発信してます。 インフラLinux勉強会SQLiteNFS こんにちは、羽山です。 今回は社内で実施した勉強会をほぼそのままブログ記事にしています。 ITエンジニアたるものドキュメントを残したり勉強会を実施したり分かりやすい設計にリファクタリングしたりなど、日々 知見の伝達 を意識していると思います。主要な知見はたいていそれらの方法で満たせるのですが、しかし残念ながら中には画一的な方法では伝えにくいものもあったりします。 ところで伝統芸能や工芸の世界では技術の継承において 背中を見て学ぶ という方法が用いられることがあるようです。 これと似た手法はITエンジニアの世界にもあって、先輩エンジニアとペアプログラミングをすることで文書では伝わりにくい技術や考え方・テクニックを学ぶ
![しくじり先生のように学ぶ「NFS+sqliteで苦労した話から学ぶ、問題解決の考え方」という勉強会をやってみました | Raccoon Tech Blog [株式会社ラクーンホールディングス 技術戦略部ブログ]](https://cdn-ak-scissors.b.st-hatena.com/image/square/9477296f61e752f8b6816cb517cdb9166a028f2c/height=288;version=1;width=512/https%3A%2F%2Ftechblog.raccoon.ne.jp%2Fwp-content%2Fuploads%2F2021%2F10%2Fscience_hakase_shippai_s.png)
I want to write a post about Pitchfork, explaining where it comes from, why it is like it is, and how I see its future. But before I can get to that, I think I need to share my mental model on a few things, in this case, Ruby’s GVL. For quite a long time, it has been said that Rails applications are mostly IO-bound, hence Ruby’s GVL isn’t that big of a deal and that has influenced the design of so
Spring Boot and Cloud Native Buildpacks Hands-on Lab
以下のソフトウェアをインストールしてください。 Java 11 docker pack curl jq この資料は次のバージョンで動作確認しています。 $ java -version openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM (build 11.0.10+9-LTS, mixed mode) $ docker version Client: Docker Engine - Community Cloud integration: 1.0.9 Version: 20.10.5 API version: 1.41 Go version: go1.13.15 Git commit: 55c4c88 Built
prompts.chat
Welcome to the “Awesome ChatGPT Prompts” repository! While this collection was originally created for ChatGPT, these prompts work great with other AI models like Claude, Gemini, Hugging Face Chat, Llama, Mistral, and more. ChatGPT is a web interface created by OpenAI that provides access to their GPT (Generative Pre-trained Transformer) language models. The underlying models, like GPT-4o and GPT-o
Announcing DuckDB 1.3.0
To install the new version, please visit the installation guide. Note that it can take a few hours to days to release some client libraries (e.g., Go, R, Java) and extensions (e.g., the UI) due to the extra changes and review rounds required. We are proud to release DuckDB 1.3.0. This release of DuckDB is named “Ossivalis” after Bucephala Ossivalis, an ancestor of the Goldeneye duck that lived mil
Try Mastering Emacs for free! Are you struggling with the basics? Have you mastered movement and editing yet? When you have read Mastering Emacs you will understand Emacs. It’s that time again: there’s a new major version of Emacs and, with it, a treasure trove of new features and changes. Notable features include the formal inclusion of native compilation, a technique that will greatly speed up y
An interesting twist in my recent usage of SQLite was the fact that I noticed my research scripts and the database intertwine more. SQLite is unique in that it really lives in-process, unlike standalone database servers. There is a feature to that which does not get used very frequently, but can be indispensable in some situations. By the way, the talk about the system that made me me to explore S
Current Best: 100M rows inserts in 33 seconds. (you can check the source code on Github) Recently, I ran into a situation where I needed a test database with lots of rows and needed it fast. So I did what any programmer would do: wrote a Python script to generate the DB. Unfortunately, it was slow. Really slow. So I did what any programmer would do: went down the rabbit hole of learning more about
Does OLAP need an ORM?
TL;DR · ORMs have proven to be useful for many developers in the OLTP/transactional stack (Postgres, MySQL, etc). · OLAP/analytical databases like ClickHouse could potentially benefit from ORM abstractions. · Existing transactional ORMs probably shouldn’t be extended to OLAP due to fundamental differences in semantic meaning between OLTP and OLAP. · Moose OLAP (part of MooseStack) is an open sourc
Noble Numbat Release Notes Table of Contents Introduction New features in 24.04 LTS Known Issues Official flavours More information Introduction These release notes for Ubuntu 24.04 LTS (Noble Numbat) provide an overview of the release and document the known issues with Ubuntu and its flavours. For details of the changes applied since 24.04, please see the 24.04.2 change summary. Support lifespan
Pandas is a very popular tool for data analysis. It comes built-in with many useful features, it's battle tested and widely accepted. However, pandas is not always the best tool for the job. SQL databases have been around since the 1970s. Some of the smartest people in the world worked on making it easy to slice, dice, fetch and manipulate data quickly and efficiently. SQL databases have come such
Published on 2022-01-04, 1640 words, 6 minutes to read It is clear that most of the world has decided that they want to use JSON for their public-facing API endpoints. However, most of the time you will need to deal with storage engines that don't deal with JSON very well. This can be confusing to deal with because you need to fit a square peg into a round hole. However, SQLite added JSON function
"Get or create" is a very common operation for syncing data in the database, but implementing it correctly may be trickier than you may expect. If you ever had to implement it in a real system with real-life load, you may have overlooked potential race conditions, concurrency issues and even bloat! In this article I explore ways to "get ot create" in PostgresSQL. Illustration by Abstrakt Design Ta
An AI-native software engineer is one who deeply integrates AI into their daily workflow, treating it as a partner to amplify their abilities. This requires a fundamental mindset shift. Instead of thinking “AI might replace me” an AI-native engineer asks for every task: “Could AI help me do this faster, better, or differently?”. The mindset is optimistic and proactive - you see AI as a multiplier
Join a VS Code Dev Days event near you to learn about AI-assisted development in VS Code. Update 1.64.1: The update addresses these security issues. Update 1.64.2: The update addresses these issues. Downloads: Windows: x64 Arm64 | Mac: Universal Intel silicon | Linux: deb rpm tarball Arm snap Welcome to the January 2022 release of Visual Studio Code. There are many updates in this version that we
Investigating a backdoored PyPi package targeting FastAPI applications | Datadog Security Labs
Introduction FastAPI is a highly popular Python web framework. On November 23rd, 2022, the Datadog Security Labs team identified a third-party utility Python package on PyPI related to FastAPI, fastapi-toolkit, that has been backdoored by a malicious actor. The attacker inserted a backdoor in the package, adding a FastAPI route allowing a remote attacker to execute arbitrary python code and SQL qu
Sketch of a Post-ORM
I’ve been writing a lot of database access code as of late. It’s frustrating that in 2023, my choices are still to either write all of the boilerplate by hand, or hand all database access over to some inscrutable “agile” ORM that will become a crippling liability in the 2-3y timescale. This post is about how I want to use databases, from the perspective of an application server developer—not a DBA
Updated 2024-09-17 to reflect updated PgBouncer support for protocol-level prepared statements 🐘 To start, I want to say that I’m appreciative that PgBouncer exists and the work its open source maintainers put into it. I also love working with PostgreSQL, and I’m thankful for the incredible amount of work and improvements that go into it as well. I also think community and industry enthusiasm aro
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
So You Want To Remove The GVL?
What's New in Emacs 28.1?
Supercharge SQLite with Ruby Functions
Towards Inserting One Billion Rows in SQLite Under A Minute - blag
Ubuntu 24.04 LTS (Noble Numbat) Release Notes
Practical SQL for Data Analysis
Bashing JSON into Shape with SQLite
How to Get or Create in PostgreSQL
The AI-Native Software Engineer
January 2022 (version 1.64)
PgBouncer is useful, important, and fraught with peril