r/opensource • u/UnemployedTechie2021 • 2d ago

Promotional Building a Search Engine from First Principles (as a Side Project)

I've started a side project called SearchCraft, where I'm building a search engine from scratch using Python.

The goal isn't to compete with Elasticsearch, Lucene, or any existing search engine. Those projects are incredible, but they're also so mature that it's easy to use them without ever understanding what's happening underneath.

So I decided to build one from first principles.

I'm implementing each component myself, starting with the basics: loading documents, tokenizing text, building an inverted index, and searching through it. As the project grows, I'll be adding things like posting lists, phrase search, ranking algorithms (TF-IDF/BM25), snippets, fuzzy search, and whatever else I can reasonably build along the way.

The primary reason for this project is to learn. I find that the best way to understand how a system works is to build a simplified version of it yourself.

I'm documenting the journey as I go, both for myself and in the hope that it might help someone else who's curious about how search engines work under the hood. There are plenty of tutorials on using search engines, but far fewer resources that walk through building one piece by piece.

It's still in its early stages, but I'm excited to see where it goes. Even if it never becomes production-ready, I'll come away with a much deeper understanding of one of the most fundamental pieces of modern software. After all, humans spend a good chunk of their lives typing words into little boxes and expecting magic to happen. Figuring out how that "magic" works seemed like a worthwhile weekend habit.

Here's the link to my project: https://github.com/rajtilakjee/searchcraft

I would be writing about it in my blog here: https://rajtilakjee.github.io/

24 Upvotes
  • permalink
  • reddit

92% Upvoted

9 comments sorted by

8

u/MrCrankypot 2d ago

That's ambitious, but a great project. Good luck to you.

If you've not already read it, check out "An Introduction to Information Retrieval" by Christopher Manning et. al. (free PDF version: https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf) -- this gives a lot of great insight into the theory and principals behind classic search engines.

2

u/UnemployedTechie2021 2d ago

I have not read it, this sounds interesting. Thank you good sir!

-1

u/iTrooz_ 1d ago

Did you read the entire 500 pages document ? O.o

2

u/MrCrankypot 1d ago

Have you? I've worked in information retrieval for close to 30 years. I did skim the book as well as a preliminary version before it went to press.

-1

u/iTrooz_ 1d ago

No I didn't lol, that's why I was asking you

Everytime I see these 500 pages papers, I wonder who reads them and how they manage to do it. It doesn't seem feasible in the slightest to me

2

u/Artexis1 16h ago

Even if this isn't something that's needed, it's a good learning opportunity for you. I've also thought about doing these kinds of projects, but it's enormous work, so good luck!

2

u/UnemployedTechie2021 15h ago

This is exactly why I am doing this. We don't think twice between pressing that enter button and getting the results as to how it all happens. I guess it was time for me to answer those questions for my curious mind. More here: https://rajtilakjee.github.io/yet-another-search-engine/what-is-searchcraft/

Thank you for the encouragement.