Back to projects
NLPML Infrastructure

Custom BPE Tokenizer

Advanced tokenizer engineering project implementing BPE, byte-level BPE, WordPiece, and SentencePiece-style unigram tokenization, with benchmarking against Hugging Face tokenizers and API/demo tooling.

GitHub repository

Views

1

Stars

0

Forks

0

Updated

Jun 22, 2026

Case study

A tokenizer engineering project showing low-level NLP infrastructure knowledge and practical benchmarking.

Problem solved

Explains and implements the core tokenization algorithms behind modern language models.

Related projects

Retrieval

Custom Retrieval Pipeline with Reranker

Featured

End-to-end local retrieval pipeline with BM25, FAISS dense search, hybrid ranking, a custom scikit-learn feature reranker, evaluation metrics, CLI tooling, and Streamlit demo.

Improves document retrieval quality by combining sparse search, dense search, and learned reranking.

PythonBM25FAISSHybrid SearchReranking
0 0 0
Case study Repo

NLP

Custom Embedding Model

Featured

Fine-tuning domain-adaptive Sentence-BERT embedding models for semantic search, retrieval, product search, support ticket matching, and job/CV matching.

Improves matching and retrieval quality by fine-tuning embeddings for specific domains and similarity tasks.

PythonSentence-BERTEmbeddingsSemantic SearchFine-tuning
1 0 0
Case study Repo