We saw a sudden increase in traffic on Screener over last few days. This made the whole website little slow.
Yet, all the core vitals looked fine: - Our 32 gb ram was at 40% utilisation - Our 8 core CPU showed the system load of 4-5. - Disk space utilisation was 70%.
I increased the number of web-workers from 8 to 12. But the website still appeared slow. I tried to look into how slow it was - and for how long. Google Analytics provided some details. The server response times had been increasing consistently There was a consistent and meaningful increase in the server-response time.
I reached out to Aniruddha for some help over screen sharing. Just sounding off the problem often provides some insights.
Aniruddha asked me to run htop command. It showed MySQL was using a lot of CPU. We ran SHOW PROCESSLIST on MySQL console to inspect the current query queue - nothing abnormal.
Aniruddha then guided me to check the innodb_buffer_pool_size value in MySQL. It turned out to be the exact bottleneck.
We used the default MySQL configuration. The defaults set the innodb_buffer_pool_size to 128 mb. I increased it to 16 gb - and wooo - the website was flying now. It was faster than ever before.
Technical details innodb_buffer_pool_size defines the cache size for tables and indexes. It is recommended to set it to 50% of the available RAM.
Epilogue This little fix reduced the page load times by 80%. It also reduced the runtime of various other scripts. Do let me know if there are other such magic settings :).
They suggest that instead of charging for "hosting" the open-source software, charge for "services" around it.
If you are a company that publishes open source tools, the question to ask yourself is, do you want to make money from your product (rent) or from services around it. Rent is exactly how proprietary products make money.
You can’t compete with your own free offering. There are billions of dollars flowing into proprietary platforms that should be the target.
This last part is pretty fabulous. Don't "compete with your own free offering". Your open source software replaces a proprietary software. Charge for services around "integration" and "installation". Places where companies are already spending huge money - be a replacement for that.
The trouble happens when someone else just wraps our free open source software as a paid service. Like AWS did with ElasticSearch. It feels "unfair". But the way they reframe it is amazing.
But as we became more mature, we started seeing how open source is truly different. We ourselves were users of many open source tools like CentOS, MariaDB, Python to name a few but were neither contributors nor did we buy any services from these companies.
Fabulous essay! Taught me a lot about how to think about going open-source.
A monologue I was having when someone asked, "how do I learn coding?”
——
I don't think coding can be learnt. It is about having a problem to solve and then solving it using a mix of tools. If we are able to solve it using any tool whatsoever, then it is coding.
There are over 100,000 (actually 800k+) computer engineers trained every year in India. Yet, the creators are handful. Most of these creators don't have a background in coding.
——
Learning coding
The first step to learn coding is to have a small problem. A tiny little problem that makes no sense otherwise:
- see all images on the page
- see all links on a page
- get that table from PDF to excel
These problems have a very specific output. Having that clearly defined output does half the work.
In other words, ask yourself: "What is the required output?"
Examples of what is the required output:
- get the mean of few numbers
- get the last trade price of any stock symbol
- get the text for any audio
- get the audio of any text
- get a picture for any word
The pattern is to have a clear defined output and a clearly defined input. These kinds of problems are the best problems to start learning coding.
——
Learning the language
Once you have a problem, it is then about finding the shortest way around it.
There are usually some tools available around your problem - but they require some setup. Most of the coding is learning that language of setup.
- Want to Tweet anything I post on Instagram: Setting up IFTTT hooks
- Want to apply 5 Photoshop filter each time: Writing Photoshop actions
- Avoiding writing same emails each time: Writing canned responses in Gmail
- Removing tons of emails from banks: Writing search filters in Gmail
- Avoiding writing same thing (url, command, text input) each time: Alfred scripts, AutoHotKey
- Fetching data of stocks: Google Sheet formula eg =Googlefinance
- Formatting Excel data each time: recording macros in VBA
- Doing some stuff in Excel each time: writing macros in VBA
- Getting stuff from a web-page: writing Javascript bookmarklets
- Generating powerful graphs: importing CSVs in Tablue
- Creating custom dashboards: writing SQL in Metabase
- Creating databases (complex Excel sheets): creating tables in Airflow
- Sending emails and notifications: setting up hooks in Zappier
You see? Learning the language is not about learning Python or Java. It is about learning the language of the software that gets it done.
Creating a solution for all
Extensive coding is only required when we want to distribute our creation with others. This stuff is hard.
We need to have an installer (app, website, exe). And we need to have a UI that is easy to use. These things often require more code than the code for problem itself.
But the good thing is that you will automatically start figuring these things out. How? Because now you know the art of solving the problems - the coding!
Thanks to Aniruddha for listening to this monologue.
A beautiful essay on the idea of friendship. How many friends do we have with whom we can "hang-out" with?
The average adult has roughly 16 people they would classify as friends. Of these, about three are “friends for life,” and five are people they really like. The other eight are not people they would hang out with one-on-one.
These other 8 are "deal-friends". Friends with whom we have kind of professional relationship.
You don’t need to have dozens of friends to be happy, and, in fact, people tend to get more selective about their friends as they age. But the number needs to be more than zero, and more than just your spouse or partner.
The essay also provides ways to ways to find friends who are not for any purpose. Friends who friends just because we care for each other.
It simply requires showing up in places that are unrelated to your worldly ambitions. Whether it is a house of worship, a bowling league, or a charitable cause unrelated to your work, these are the places where you meet people who might be capable of sharing your loves, but without advancing your career. When you meet someone you like, don’t overthink it: Invite them over.
