Yesterday, Screener.in stopped working in the evening. I logged in via remote shell, ran `top` and saw mysql using all the memory.
Next I did `tail -f /var/log/mysql/error.log` and saw this frightful error:
10:20:39 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0xffed4075d9f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = ffff804c1748 thread_stack 0x100000
Nothing else after that.
It was a time to panic! This was the first time I was experiencing a MySQL corruption issue.
Luckily, the issue was resolved with `mysqlcheck --all-database` and `mysqlcheck -o db_name corrupted_table_name`. This StackOverflow answer helped.
This is the checklist we created for the next time:
# copy error log
scp -C screener_prod:/var/log/mysql/error.log ./
# put site on maintainance
# also disable crons
read deploy.sh
# clear mailq for error emails
sudo postsuper -d ALL
# reboot system
sudo reboot
# create a back-up
from AWS account
# get exact time of incidence
# check error log
vi /var/log/syslog
# run mysqlcheck
sudo mysqlcheck --all-databases
# run the above command again (and again) if it shows "MySQL" gone away
sudo mysqlcheck --all-databases
# hope the errors are only in indexes
sudo mysqlcheck -o db_name tbl_name
# the errors during mysqlcheck are again logged in error log
scp -C screener_prod:/var/log/mysql/error.log ./
# analyse the above
INSERT...DUPLICATE UPDATE (MySQL docs) is one of my favourite SQL hacks. It allows us to create and update rows in bulk.
We pass the set of rows. The already existing rows are updated while the new ones are created. The "already existing" rows are detected on unique constraints. The good thing is that it is all done at SQL level and is very fast. It is 1 query to rule them all.
We have been using a custom function of INSERT...DUPLICATE KEY UPDATE query from day 1. We use it while pulling updates from our data providers.
We pull the updates, read the CSVs and pass all the rows in this query. We don't need to worry or check whether there are conflicting rows. The query handles all of that.
"Every line of code, every comment, every line-break should have a purpose."
We usually use Github's review feature for code reviews. It is hard, however, to jump between files when the changes are large. An ideal way would be to manually mark each and every line as checked during the review.
I use a combination of git reset and git add -p to do just that. Sharing the recipe below 🤫.
Algorithm - Switch to the branch we want to review. - Merge main into it. - Create a new review-branch from that branch. - Git reset to unstage differences from main. - Use patch mode to break changes into small pieces. - Add and stage reviewed lines. This is where we manually mark each and every line we have reviewed. We can also make changes as we review. - Go back to the source branch we started the review with. - Checkout changes from our review-branch to this branch. - Push changes from our review.
Actual commands The whole process looks something like this.
# create a new review branch
git switch branch-to-review
git merge main
git checkout -b review-branch-to-review
# unstage changes from main
git reset main
# review each and every line using patch mode
git add . -p
# make edits during review
# work as we normally do
git commit -m "<message about changes>"
# go back to the branch-to-review
git checkout branch-to-review
# stage changes from our review branch
# the dot (.) tells to checkout the changes in current directory
# --no-overlay tells to include deleted files
git checkout branch-to-review . --no-overlay
git commit -m "<review message>"
Automating the above using Justfile These are still a lot of commands. It is easy to mess them up. Wouldn't it be great if we can do something like:
just review
# work on code as we normally do
# and then
just finish-review
Well, this is what this script does. I place it in ~/justfile. It uses Just for adding commands. Now we can use these commands in any project.
We have been of multitasking between tax returns, feature prototypes and code reviews for the last two weeks.
We discovered Section 54F this time. This section exempts capital gains (with no upper limit) if the proceeds are invested in buying or constructing a house. Very interesting. This is something that my Dad does. He creates a physical asset in every bull run.
Last year we discovered section 115BAA. It allows companies to opt for lower tax rates of 22%. Awesome!
Bulk and Block Deals on Screener
We added a lot of things in bulk and block deals over the last 2 weeks: - Shareholder pages display the bulk and block deals along with major holdings. - Company pages show bulk and block deals too now. - A "recent deals" indicator. - Reverse search for people from company pages. - A page to browse through latest deals (wip): https://www.screener.in/trades/
Just is a command runner. It allows us to create small recipes. We can use these recipes system-wide or for specific projects. I automated our code-review workflow using Just this week.
We are a very small tech team at Screener. Just 4-5 of us. Yet it becomes chaotic sometimes.
At the same time, there are companies which have 10s, 100s and 1000s of developers. They have fabulous products and keep rolling out new features. I admire them and try to learn a few tricks from them. About how they distribute work, how they keep the codebase sane and how they drive innovation.
These are few of the learnings and mental models from various books.
1. The E-Myth Revisited This book introduced me to the concept of creating franchisees. It prescribes methods for creating companies like McDonalds, Tanishq, HDFC, Hiltons etc. The fundamental concept is to "make ordinary people do extraordinary things." It does so by creating systems and having manuals for everything.
It recommends having a "functional prototype" where you do all the experiments. Then you observe and document all the processes, create an organisation chart and properly assign the responsibilities. This functional prototype is then distributed and scaled as franchisees.
2. Netflix The Netflix culture is totally opposite of The E-Myth. They believe that system driven companies fail in long run because they are unable to adapt to changes. They rather believe in hiring extraordinary people who can "use good judgement." They don't have any spending controls and very few signing controls.
This is in the lines of hiring the best, giving them independence and paying the highest.
3. Accidental Empires This is one of my favourite books. Though it is not a management book, it does have a few chapters around structures.
It recommends organising companies like Police, Army and Commandoes. A company needs a large team of police who manage existing products (existing cities). An army of strategists who plan new territories. And a team of commandoes who actually go and conquer those territories.
4. Creativity Inc Ed Catmull has worked all his way from being an animator to being the head of Pixar. He closely studied various working styles and shared them in this wonderful book.
Two of his major takeaways were: 1. Hiring people better than yourself 2. Putting quality first
Ed shares about multiple instances where their stories were not taking proper shape. Hiring the best story tellers, the best animators and the sharpest minds helped them achieve the best. As a manager, this required him putting his own insecurities away, and hire people more accompanied than himself.
Another insight that Ed shares is about the focus on quality. The Japanese manufacturing system allowed anyone in the assembly line to stop the conveyor belt. They encouraged (and expected) workers to halt production if they found any defect. This is an important part of Pixar culture. Though they have a hierarchy, everyone in the company is encouraged to take responsibility without requiring a permission.
5. The CEO Factory Saw this tweet about Unilever's organisation style.
They classify their team into Mavericks, Company Men and Rogues. The Mavericks are the creators. They are introvert and like to work alone. The Company Men are team players. The Rogues are high on energy but put their short-term self-interests first.
Unilever tries to encourage Mavericks and the Company Men. While Company Men climb to the very top, the Mavericks become the legends in the company's folklore. Rogues rarely make it past middle management.
This is a good way to think about the performances of individual employees.
6. Mythical Man Month I found this book most relatable. It focuses on product unity.
The book gives 2 analogies for organising and delegating work: 1. The Surgeon Way 2. The Architect Way
The surgeons way works great for small teams. We have a surgeon working on the core problem. There is a co-pilot, with whom the surgeon discusses everything. However, the final decisions rest with the surgeon. The rest of the team helps surgeon do their best work.
The architect way works great for larger teams where the work is divided. It is very similar to how an architect works on a building. They provide the blueprints and idealise the whole project. They define the interfaces that the user will ultimately use. They act on behalf of the user. At the same time, they restrain from dictating the implementation. The implementers use their creativity and create the given requirements. Of all the above books, I found Mythical Man Month the most actionable. It has covered all the aspects of software programming in a wonderful way.
I personally admire Warren Buffett's way of working. At 91 years of age, being richest person, his office still has less than 20 people. His calendars remain empty, and he gets to do what he enjoys the most.
The other structure that I admire is the open-source culture. Projects like Django have not only lasted but thrived without having any "leaders." They attract people who love the craft and have a common pain. This is the kind of culture I would like to have!
We had some trivial engineering challenges over last 2 weeks.
Reducing diff speed from 30+ seconds to 300ms We have a wiki feature at Screener. It allows anyone to modify and add key-insights for companies. In the backend, we use Google's diff-match-patch library. It generates diffs between multiple versions and merges them.
The Python version of the library had a long processing time for some edits. The moderation page was taking over 30 seconds to load.
While I did expect the loading times to improve, I didn't expect the C++ version to be 100x faster. The page loads in less than 500ms now. Yay!
Forgot Password not working for social logins - Django We recently discovered that users using social logins (login with Google) were not receiving emails for password reset. The issue was this code:
user = UserAccount.objects.create_user(
email=email,
password=None,
display_name=name,
)
Setting password as None disables password based logins. However, I didn't expect it to not send any email on clicking forgot password. Reading through Django's source code highlighted this issue. We fixed it with:
password = UserAccount.objects.make_random_password()
user = UserAccount.objects.create_user(
email=email,
password=password,
display_name=name,
)
Improved rendering on mobile We also refactored a lot of base templates on Screener. The text stuck to mobile walls sometimes due to the missing padding. We simplified the templates and fixed the rendering.
Tools and Books
I use MailBrew to generate personalised email digests. It sends me an email every morning of the Tweets of my favourite people. It also includes updates from HackerNews, Lobster, Pinboard network and any RSS feed I provide.
- Feed43comes helpful when the websites don't have an RSS feed. I use it to generate feeds for MySQL release notes and new feature phones by Nokia. The free version works pretty well for my use case.
- The Mythical Man-month is a book recommended by all the senior programmers, but hardly anyone reads it. I expected it to be dry, outdated and hard. Surprisingly, it is one of the most easy and useful books.
Each chapter starts with a photograph and a quote. Both of them picked very carefully. And they set the theme for the chapter. They also serve as the summary.
The book acknowledges the hard-problem of maintaining the code quality. And it provides some solutions. Solutions that are easy to adapt for a small team like us.
I posted a short summary about it in this blog post.
Thanks for reading. See you again soon :). Will hopefully add a comments feature on the blog in coming days.
