こんにちは、株式会社FP16で結構コードを書いている二宮です。 最近Webスクレイピングのコードを色々な方法で書いているので、そこで得た知見をここに残しておこうと思います。 ほぼ毎日なにかのWebスクレイピングコードを書いています。 Webスクレイピング手段 Webスクレイピングには色々な方法があります。 私が最近主に使っているのはこの5つの手段です。 cheerioでHTMLを解析 Playwrightなどで要素指定でデータを取得する APIを見つけて叩く（バックエンドとの通信を再現してデータを取得） LLMでサイト構造を解析してデータを取得する Next.jsからのレスポンスに含まれているデータを解析して取得する これが令和のWebスクレイピングのベストプラクティスだと思っています。 これらの方法を、目標に合わせて使い分けています。 使い分け方 CheerioでHTML解析 JavaS
すぐに役に立つものはすぐに陳腐化してしまうから方法ではなく設計の本を読む - API Design Patterns の読書感想文 - じゃあ、おうちで学べる
あなたがさっきまで読んでいた技術的に役立つ記事は、10年後も使えるでしょうか。ほとんどの場合でいいえ この問いに真正面から殴られたのは数か月前、リリース前夜の「とりあえずこれで走らせよう」が翌朝に別サービスを巻き込む事故になったときでした。Slackで「なんで昨日の変更でメトリクスが暴れてるのか」と聞かれ、胃がキリキリしたまま休日をつぶす羽目に。目先のテクニックだけで乗り切った仕事は、想像より早く自分を噛みに来る。だから設計そのものの筋肉を付けたい、という切実さを忘れないうちに本を開きました。 ただし、先に言っておきます。Design Patternsは設計そのものではありません。 パターンを知っているだけで良い設計ができるわけではない。10年前の「RESTが正義」という原則は、GraphQLやgRPCによって相対化されました。原則も陳腐化します。それでも原則を学ぶ価値があるとすれば、「不
How I built a modern website in 2021September 29th, 2021 — 34 min read For over half of 2021, I worked on a complete rewrite of kentcdodds.com. You're reading this on the rewrite of this site! Are you using dark mode or light mode? Have you signed in and selected your team yet? Have you tried to call into the Call Kent Podcast? This blog post isn't about these and other features of the new site, b
class: middle, center <img src="./assets/logo.svg" align="center" width="200" /> Deno の これまで と これから --- アジェンダ - Deno とは - Deno のこれまでのロードマップ - Deno のこれからのロードマップ --- # 話す人 <img src="./assets/hinosawa.jpg" align="right" width="300" /> 日野澤歓也 twitter @kt3k - GREE (2012 - 2013) - Recruit (2015 - 2019) - Deno Land (2021 -) <small>2018年から Deno にコントリビュートを開始。2020年作者に誘われ Deno Land に転職。現在はフルタイムで Deno と Deno D
2023 年 4 月 18 日にリリースされた Node.js v20 の主な変更点を紹介します。 nodejs.org ファイルのアクセスやプロセスの起動を制限する新しいパーミッションモデル (experimental) --allow-fs-read でファイルの読み込みを許可する --allow-fs-write によるファイル書き込み許可 --allow-child-process による child_process の許可 --allow-worker による Worker Threads の許可 Permission Model の参考記事 V8 11.3 による新しい JavaScript の機能 テストランナー（node:test）が stable に昇格 Single Executable Application JSON の config を使った Blob が必要に
Easily the biggest question we get asked is something like: How is Remix different from Next.js? It appears we have to answer this question! We'd like to address it directly and without drama. If you're a fan of Remix and want to start tweeting smug reactions to this article, we kindly ask that you drop the smugness before hitting the tweet button 🤗. A rising tide lifts all boats. We've been frie
技術部の笹田（ko1）と遠藤（mame）です。クックパッドで Ruby (MRI: Matz Ruby Implementation、いわゆる ruby コマンド) の開発をしています。お金をもらって Ruby を開発しているのでプロの Ruby コミッタです。 昨日 12/25 に、恒例のクリスマスリリースとして、Ruby 3.2.0 がリリースされました（Ruby 3.2.0 リリース）。今年も Ruby 3.2 の NEWS.md ファイルの解説をします。NEWS ファイルとは何か、は以前の記事を見てください。 プロと読み解く Ruby 2.6 NEWS ファイル - クックパッド開発者ブログ プロと読み解くRuby 2.7 NEWS - クックパッド開発者ブログ プロと読み解くRuby 3.0 NEWS - クックパッド開発者ブログ プロと読み解く Ruby 3.1 NEWS -
REST API Design Best Practices Handbook – How to Build a REST API with JavaScript, Node.js, and Express.js
By Jean-Marc Möckel I've created and consumed many API's over the past few years. During that time, I've come across good and bad practices and have experienced nasty situations when consuming and building API's. But there also have been great moments. There are helpful articles online which present many best practices, but many of them lack some practicality in my opinion. Knowing the theory with
Developers are increasingly relying on AI coding assistants to accelerate our daily workflows. These tools can autocomplete functions, suggest bug fixes, and even generate entire modules or MVPs. Yet, as many of us have learned, the quality of the AI’s output depends largely on the quality of the prompt you provide. In other words, prompt engineering has become an essential skill. A poorly phrased
Tauri 2.0 Stable Release
We are very proud to finally announce the stable release for the new major version of Tauri. Welcome to Tauri 2.0! What is Tauri? In a Tauri application the frontend is written in your favorite web frontend stack. This runs inside the operating system WebView and communicates with the application core written mostly in Rust. When Should I Use Tauri? If you check any of the boxes below, you should
AIエージェントのUXを進化させる「A2UI」でアプリを構築 - Taste of Tech Topics
こんにちは。大塚です。 普段は生成AIを利用したサービスの開発に携わっているのですが、直近の仕事では、数理最適化の手法を扱う機会がありました。 実は、大学の授業で数理最適化をかじっており、教授がよく 「人の勘や経験に頼っている部分を数式に落とし込むことで、合理的な意思決定ができるようにしよう。」 といっていたことを思い出しました。 まさに今、それを仕事で実践できており、面白くなってきました。 さて、今回のテーマは最適化ではなく、AIエージェントのUIについてです。 現在の生成AIアプリのUIといえば、「チャット」形式がほとんどの状況だと思いますが、チャットのアプリは汎用的な一方で、入力を制御したり、画像や動画の表示が難しかったりと、限界を感じることってありませんか？ そんな悩みを解決してくれる、「A2UI（Agent to UI）」という、AIエージェント用のUIプロトコルが、2025年1
Today we are excited to announce the availability of TypeScript 6.0! If you are not familiar with TypeScript, it’s a language that builds on JavaScript by adding syntax for types, which enables type-checking to catch errors, and provide rich editor tooling. You can learn more about TypeScript and how to get started on the TypeScript website. But if you’re already familiar with the language, you ca
Announcing TypeScript Native Previews - TypeScript
This past March we unveiled our efforts to port the TypeScript compiler and toolset to native code. This port has achieved a 10x speed-up on most projects – not just by using a natively-compiled language (Go), but also through using shared memory parallelism and concurrency where we can benefit. Since then, we have made several strides towards running on large complex real-world projects. Today, w
IDEs CLion DataGrip DataSpell Fleet GoLand IntelliJ IDEA PhpStorm PyCharm RustRover Rider RubyMine WebStorm Plugins & Services Big Data Tools Code With Me JetBrains Platform Scala Toolbox App Writerside JetBrains AI Grazie Junie JetBrains for Data Kineto Team Tools Datalore Space TeamCity Upsource YouTrack Hub Qodana CodeCanvas Matter .NET & Visual Studio .NET Tools ReSharper C++ Languages & Frame
How modern browsers work
Note: For those eager to dive deep into how browsers work, an excellent resource is Browser Engineering by Pavel Panchekha and Chris Harrelson (available at browser.engineering). Please do check it out. This article is an overview of how browsers work. Web developers often treat the browser as a black box that magically transforms HTML, CSS, and JavaScript into interactive web applications. In tru
One of the most impactful things we’ve worked on in TypeScript 5.0 isn’t a feature, a bug fix, or a data structure optimization. Instead, it’s an infrastructure change. In TypeScript 5.0, we restructured our entire codebase to use ECMAScript modules, and switched to a newer emit target. What to Know Now, before we dive in, we want to set expectations. It’s good to know what this does and doesn’t m
Deno の話 @ Nikkei
class: middle, center <img src="./assets/logo.svg" align="center" width="200" /> Deno の話 --- アジェンダ - Deno とは - Deno の各種特徴 - Deno Deploy の紹介 --- # 話す人 <img src="./assets/hinosawa.jpg" align="right" width="300" /> 日野澤歓也 twitter @kt3k - GREE (2012 - 2013) - Recruit (2015 - 2019) - Deno Land (2021 -) <small>2018年から Deno にコントリビュートを開始。2020年作者に誘われ Deno Land に転職。現在はフルタイムで Deno と Deno Deploy を開発中。</small>
Modern Emacs Typescript Web (React) Config with lsp-mode, treesitter, tailwind, TSX & more - Ovi Stoica
Table of Contents Introduction Part 1: Treesitter for Typescript & TSX LSP Support Completion setup Linter setup LSP Setup Eslint (Optional) Tailwind LSP Server LSP Performance Emacs LSP Booster Structural editing Formatting buffers with Prettier Other resources Conclusion Introduction I've worked within the JS ecosystem for the past 8 years using editors like Webstorm and VSCode, I started using
Today we’re excited to announce the release of TypeScript 5.1! If you’re not yet familiar with TypeScript, it’s a language that builds on JavaScript by adding constructs called types. These types can describe some details about our program, and can be checked by TypeScript before they’re compiled away in order to catch possible typos, logic bugs and more. TypeScript also uses these types to provid
We are excited to announce the release of the new version of Prettier! We have made the migration to using ECMAScript Modules for all our source code. This change has significantly improved the development experience for the Prettier team. Please rest assured that when using Prettier as a library, you can still use it as CommonJS as well. This update comes with several breaking changes. One notabl
OWASP ZAP の finding を Rust/Axum の handler に戻して直す - じゃあ、おうちで学べる
はじめに vulnerable-app に ZAP の full scan を回すと、High finding が並びます。XSS、SQL Injection、Path Traversal。alert 名を眺めて、ふと気づく。これは「危険です」の一覧ではない。handler への差し戻し指示書だ。 OWASP ZAP を実行すると、HTML、Markdown、JSON のレポートが出ます。そこには Cross Site Scripting (Reflected)、SQL Injection、Path Traversal のような名前が並びます。ただ、alert 名だけを見ても修正は始まりません。必要なのは、finding を実際の handler、SQL、HTML 出力、ヘッダ設定に戻すことです。 OWASP ZAP は OSS の web 脆弱性スキャナで、HTTP リクエストを投げて挙
Actions
The app provides 180+ powerful extra actions for the Shortcuts app on macOS, iOS, and visionOS. These actions make it significantly easier to create shortcuts. IMPORTANT Restart your device if the actions do not show up in the Shortcuts app. Learn more › If you have any questions about how to use the different actions or for what, try asking the Actions GPT bot. And if you want to feed your own AI
The Fediverse has been a hot topic of discussion lately, with thousands, if not millions, of new users creating accounts on platforms like Mastodon to either move entirely to "the other side" or experiment and learn about this new social network. Today we're introducing Wildebeest, an open-source, easy-to-deploy ActivityPub and Mastodon-compatible server built entirely on top of Cloudflare's Super
Our strategy for making updates and maintaining stability while we migrate languages. In a previous blog post, we talked about why we are porting Turborepo, the high-performance build system for JavaScript and TypeScript, from Go to Rust. Now, let's talk about how. Today, our porting effort is in full swing, moving more and more code to Rust. But when we were starting out, we had to make sure that
Programming is an iterative process. As much as we would like to come up with the perfect solution from the start, it rarely works that way. Good programs often begin as quick prototypes. While many experiments remain prototypes, the best programs can evolve into production code. Whether you’re writing games, CLI tools, or designing library APIs, prototyping helps tremendously in finding the best
缶つぶし機とソフトウェア移行技術 - Refactoring to Rust の読書感想文 - じゃあ、おうちで学べる
はじめに ——あるいは、「知っている」と「理解している」の間 Rustのことは、知っていた。学習もしていた。実務でも使っていた。 でも、それは知っているつもりだった。 知ってるつもり 無知の科学 (ハヤカワ文庫NF) 作者:スティーブン スローマン,フィリップ ファーンバック早川書房Amazon 日々Rustで開発し、BoxとRcとArcを使い分け、tokio::spawnでタスクを生成し、?演算子を当たり前のように書いている。FFI？PyO3使えばいいでしょ。WebAssembly？wasm-bindgenがあるじゃない。技術的には、確かに「使える」レベルにはあった。 でも、心のどこかで感じていた違和感があった。 オートバイのエンジンを分解できる人と、エンジンが動く原理を理解している人は違う。コードが動くことと、なぜそう書くべきかを理解することも違う。私は前者だった。メカニックではあった
Hypershell: A Type-Level DSL for Shell-Scripting in Rust | Context-Generic Programming
Discuss on Reddit, Lobsters, and Hacker News. Summary I am thrilled to introduce Hypershell, a modular, type-level domain-specific language (DSL) for writing shell-script-like programs in Rust. Hypershell is powered by context-generic programming (CGP), which makes it possible for users to extend or modify both the language syntax and semantics. Table of Contents Estimated reading time: 1~2 hours
Today we’re excited to announce the release of TypeScript 4.8! If you’re not yet familiar with TypeScript, it’s a language that builds on JavaScript and adds syntax for types. These types let you put your expectations and assumptions into your code, and those assumptions can then be checked by the TypeScript type-checker. This checking can help avoid typos, calling uninitialized values, mixing up
Biome v2—codename: Biotype
We are happy to announce that Biome v2 is officially out! 🍾 Biome v2—codename: Biotype, the first JavaScript and TypeScript linter that provides type-aware linting rules that doesn’t rely on the TypeScript compiler! This means that you can lint your project without necessarily installing the typescript package. With this release, the Core Contributors of the project want to show to the whole comm
Today we are announcing the beta release of TypeScript 6.0! To get started using the beta, you can get it through npm with the following command: npm install -D typescript@beta TypeScript 6.0 is a unique release in that we intend for it to be the last release based on the current JavaScript codebase. As announced last year (with recent updates here), we are working on a new codebase for the TypeSc
Today we excited to announce the availability of TypeScript 5.7! If you’re not familiar with TypeScript, it’s a language that builds on JavaScript by adding syntax for type declarations and annotations. This syntax can be used by the TypeScript compiler to type-check our code, and it can also be erased to emit clean, idiomatic JavaScript code. Type-checking is helpful because it can catch bugs in
Notable changes built-in .env file support Starting from Node.js 20.6.0, Node.js supports .env files for configuring environment variables. Your configuration file should follow the INI file format, with each line containing a key-value pair for an environment variable. To initialize your Node.js application with predefined configurations, use the following CLI command: node --env-file=config.env
We've added JavaScript-native RPC to Cloudflare Workers
We've added JavaScript-native RPC to Cloudflare Workers2024-04-05 Cloudflare Workers now features a built-in RPC (Remote Procedure Call) system enabling seamless Worker-to-Worker and Worker-to-Durable Object communication, with almost no boilerplate. You just define a class: export class MyService extends WorkerEntrypoint { sum(a, b) { return a + b; } } And then you call it: let three = await env.
On January 14, 2006, John Resig introduced a JavaScript library called jQuery at BarCamp in New York City. Now, 20 years later, the jQuery team is happy to announce the final release of jQuery 4.0.0. After a long development cycle and several pre-releases, jQuery 4.0.0 brings many improvements and modernizations. It is the first major version release in almost 10 years and includes some breaking c
All JavaScript and TypeScript Features of the last 3 years
TypeScript as envisioned by Stable DiffusionThis article goes through almost all of the changes of the last 3 years (and some from earlier) in JavaScript / ECMAScript and TypeScript . Not all of the following features will be relevant to you or even practical, but they should instead serve to show what’s possible and to deepen your understanding of these languages. There are a lot of TypeScript fe
We’re excited to announce the release of Biome 2.3, bringing several features that have been highly requested by the community. This release marks a significant milestone in our journey to support the broader web ecosystem. Once you have upgraded to Biome v2.3.0, migrate your Biome configuration to the new version by running the migrate command: biome migrate --write Full support for Vue, Svelte,
Biome v1.6
Update Biome using the following commands: npm install --save-dev --save-exact @biomejs/biome@latestnpx @biomejs/biome migrate Partial support for Astro, Svelte and Vue files In this release, we’re happy to provide partial support for Astro, Svelte and Vue files. What does partial support mean? While the team is working on a unified data structure for HTML-ish languages, we discovered that we coul
This article is inspired by the talk How the TypeScript compiler compiles◹, you should really check it out for a more in-depth understanding about the TypeScript’s compiling process. At a high level, the TypeScript compiler is a tool to help us analyze and compile the TypeScript code into JavaScript (*.js), as well as some type definition files (*.d.ts) or source maps (*.js.map). If the source fil
In the early 90’s, being a frisian kid obsessed with computers there weren’t a ton of ways to get access to new software or learn more about computers. The two main ways were exchanging 3.5” diskettes with friends, or go to the library. One book I remember more than others was “Windows for Kinderen” (“Windows for Kids”) by Addo Stuur. I must have been around 10 years old and was obsessed by this b
The product https://joinmastodon.org Mastodon is a free, open-source social network server based on ActivityPub where users can follow friends and discover new ones. On Mastodon, users can publish anything they want: links, pictures, text, and video. All Mastodon servers are interoperable as a federated network. Open source The project is open source at https://github.com/mastodon/mastodon License
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
【令和最新版】令和のWebスクレイピング(クロール)【ベストプラクティス】
How I built a modern website in 2021
Deno のこれまでとこれから JSConf JP 2021
Node.js v20 の主な変更点 - 別にしんどくないブログ
Remix vs Next.js
プロと読み解く Ruby 3.2 NEWS - クックパッド開発者ブログ
The Prompt Engineering Playbook for Programmers
Announcing TypeScript 6.0 - TypeScript
JavaScript Best Practices | The WebStorm Blog
TypeScript's Migration to Modules - TypeScript
Announcing TypeScript 5.1 - TypeScript
Prettier 3.0: Hello, ECMAScript Modules! · Prettier
Welcome to Wildebeest: the Fediverse on Cloudflare
How Turborepo is porting from Go to Rust - Vercel – Vercel
Prototyping in Rust | corrode Rust Consulting
Announcing TypeScript 4.8 - TypeScript
Announcing TypeScript 6.0 Beta - TypeScript
Announcing TypeScript 5.7 - TypeScript
Node.js — Node.js 20.6.0 (Current)
jQuery 4.0.0 | Official jQuery Blog
Biome v2.3—Let's bring the ecosystem closer
04.01.2022 - TypeScript/How the compiler compiles
Neko - A brief history and porting to Javascript
Mastodon: Ruby on Rails Open Source Web App