mirror of
https://github.com/jackyzha0/quartz.git
synced 2025-12-03 03:07:58 +01:00
ec26ebcc9e
Some checks are pending
* feat: improve search tokenization for CJK languages Enhance the encoder function to properly tokenize CJK (Chinese, Japanese, Korean) characters while maintaining English word tokenization. This fixes search issues where CJK text was not searchable due to whitespace-only splitting. Changes: - Tokenize CJK characters (Hiragana, Katakana, Kanji, Hangul) individually - Preserve whitespace-based tokenization for non-CJK text - Support mixed CJK/English content in search queries This addresses the CJK search issues reported in #2109 where Japanese text like "て以来" was not searchable because the encoder only split on whitespace. Tested with Japanese, Chinese, and Korean content to verify character-level tokenization works correctly while maintaining English search functionality. * perf: optimize CJK search encoder with manual buffer tracking Replace regex-based tokenization with index-based buffer management. This improves performance by ~2.93x according to benchmark results. - Use explicit buffer start/end indices instead of string concatenation - Replace split(/\s+/) with direct whitespace code point checks - Remove redundant filter() operations - Add CJK Extension A support (U+20000-U+2A6DF) Performance: ~878ms → ~300ms (100 iterations, mixed CJK/English text) * test: add comprehensive unit tests for CJK search encoder Add 21 unit tests covering: - English word tokenization - CJK character-level tokenization (Japanese, Korean, Chinese) - Mixed CJK/English content - Edge cases All tests pass, confirming the encoder correctly handles CJK text. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
Quartz v4
“[One] who works with the door open gets all kinds of interruptions, but [they] also occasionally gets clues as to what the world is and what might be important.” — Richard Hamming
Quartz is a set of tools that helps you publish your digital garden and notes as a website for free. Quartz v4 features a from-the-ground rewrite focusing on end-user extensibility and ease-of-use.
🔗 Read the documentation and get started: https://quartz.jzhao.xyz/
Sponsors
