We recently added a "search everywhere" feature on Screener. It allows us to do a full-text search on all the concall transcripts, announcements and key-insights of all the listed companies.

This was our second attempt on it.

There are various ways to implement full-text search:
- Using full-text index in relational database such as MySQL / PostgreSQL
- Using services such as ElasticSearch, Algolia, TypeSense or Meilisearch
- Using text search databases such as Lucene or Tantivy

We tried full-text using MySQL in our first attempt. This time we shifted to Tantivy. These have been our learnings.

Yesterday, Screener.in stopped working in the evening. I logged in via remote shell, ran `top` and saw mysql using all the memory.

Next I did `tail -f /var/log/mysql/error.log` and saw this frightful error:

10:20:39 UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0xffed4075d9f0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = ffff804c1748 thread_stack 0x100000

Nothing else after that.

It was …

We have been of multitasking between tax returns, feature prototypes and code reviews for the last two weeks.

We discovered Section 54F this time. This section exempts capital gains (with no upper limit) if the proceeds are invested in buying or constructing a house. Very interesting. This is something that my Dad does. He creates a physical asset in every bull run.

Last year we discovered section 115BAA. It allows companies to opt for lower tax rates of 22%. Awesome!

We added a lot of things in bulk and block deals over the last …

We had some trivial engineering challenges over last 2 weeks.

Reducing diff speed from 30+ seconds to 300ms
We have a wiki feature at Screener. It allows anyone to modify and add key-insights for companies. In the backend, we use Google's diff-match-patch library. It generates diffs between multiple versions and merges them.

The Python version of the library had a long processing time for some edits. The moderation page was taking over 30 seconds to load.

We found a faster version of the library which is a wrapper around the C++ version. It had a few issue around installation. We …