Database corruption issue [MySQL]

Yesterday, Screener.in stopped working in the evening. I logged in via remote shell, ran `top` and saw mysql using all the memory.

Next I did `tail -f /var/log/mysql/error.log` and saw this frightful error:

10:20:39 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0xffed4075d9f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = ffff804c1748 thread_stack 0x100000

Nothing else after that.

It was …

Read full post

13th Aug. 2022

"INSERT...ON DUPLICATE KEY UPDATE" in Django 4.1

Django 4.1 release notes - update in Queryset.bulk_create()

I upgraded Screener to Django 4.1 last week. I was surprised to see the support for conflicting updates in bulk_create method. This uses "insert...on duplicate key update" under the hood. [Django Docs].

INSERT...DUPLICATE UPDATE (MySQL docs) is one of my favourite SQL hacks. It allows us to create and update rows in bulk.

We pass the set of rows. The already existing rows are updated while the new ones are created. The "already existing" rows are detected on unique constraints. The good thing is that it is all done …

Read full post

31st July 2022

Reviewing each line of code with Git

"Every line of code, every comment, every line-break should have a purpose."

We usually use Github's review feature for code reviews. It is hard, however, to jump between files when the changes are large. An ideal way would be to manually mark each and every line as checked during the review.

I use a combination of git reset and git add -p to do just that. Sharing the recipe below 🤫.

Algorithm
- Switch to the branch we want to review.
- Merge main into it.
- Create a new review-branch from that branch. …

Read full post

31st July 2022

Weekly Notes - 10

We have been of multitasking between tax returns, feature prototypes and code reviews for the last two weeks.

We discovered Section 54F this time. This section exempts capital gains (with no upper limit) if the proceeds are invested in buying or constructing a house. Very interesting. This is something that my Dad does. He creates a physical asset in every bull run.

Last year we discovered section 115BAA. It allows companies to opt for lower tax rates of 22%. Awesome!

Bulk and Block Deals on Screener

We added a lot of things in bulk and block deals over the last …

Read full post

19th July 2022

Organisational Structures

I love reading how software companies work.

We are a very small tech team at Screener. Just 4-5 of us. Yet it becomes chaotic sometimes.

At the same time, there are companies which have 10s, 100s and 1000s of developers. They have fabulous products and keep rolling out new features. I admire them and try to learn a few tricks from them. About how they distribute work, how they keep the codebase sane and how they drive innovation.

These are few of the learnings and mental models from various books.

1. The E-Myth Revisited
This book introduced me to the …

Read full post

17th July 2022

Weekly Updates - 9

We had some trivial engineering challenges over last 2 weeks.

Reducing diff speed from 30+ seconds to 300ms
We have a wiki feature at Screener. It allows anyone to modify and add key-insights for companies. In the backend, we use Google's diff-match-patch library. It generates diffs between multiple versions and merges them.

The Python version of the library had a long processing time for some edits. The moderation page was taking over 30 seconds to load.

We found a faster version of the library which is a wrapper around the C++ version. It had a few issue around installation. We …

Read full post

2nd July 2022

Weekly Updates - 8

This was a busy week loaded with code-reviews, designs, meetings and coffees.

Investor Presentations on Screener

We added a section for investor presentations. It was built on the similar lines as concall transcripts.

The backend was developed by Mukesh and me. The frontend was trickier due to the paucity of space.

Design versions for investor presentation feature

Fun with Aeropress

While I have been pretty happy with the cappuccinos brewed on Bialetti Mukka, I wanted to give black coffee a try. Got an Aeropress this week.

Nothing much else this week.