Goodness Ezeokafor

Had a battle last week where on a personal project, retrieving data from my database resulted in a 504 gateway error. Funny because just 24 hours ago everything seems to be fine. In the couple of weeks leading to that, I have been obsessed about getting all my response time to milliseconds, at least to triple digits like 100ms to 500ms. Got really into it and tried a few things.

1. Return only what is needed:- What I did was to dump everything to the client-side. This was quite ok for the first thousands transactions, but as our transactions started crossing ten thousands, we started experiencing a little delay in retrieving users' transactions, it got worse in the admin area where we had to get all transactions recorded in our database(mind you we had applied pagination). So instead of returning everything about a transaction when retrieving multiple records, I selected what needed to be captured in the UI.  
   A quick example: if the transaction’s table has 20 columns, and the UI just needed 7 columns displayed, my endpoints selects those 7 columns and returns to the UI to integrate. That way I dont load unnecessary data across the network that delays the response time.
   

2. Lazy Loading over Eager Loading:- Lazy loading and Eager loading is popular in a lot of ORMS. Basically in database designs there are a lot of relationships, especially when you are trying to model real world data. Take my transaction as an example. Users perform transactions, whether its deposits or withdrawals in almost every web application. When designing a database it's a good idea to have two different tables capture these scenarios. A table to store users details and another to store transactions. We then create some sort of relationship to tie a transaction to a user, known as a one-to-many relationship where a user has many transactions, and a transaction just has one user. When storing this transaction in our database, we store the userId. It's all good until the product designer wants the frontend developer to not show just the userId, but the first name, last name and any other information that we can get from the user.
   
   At first as the rookie that I am, I decided to make an extra call from the database when returning transaction records. So after I get the records, I then loop through the records, get the userId, then using the .findOne, I get the user’s record and return that. Trust me this worked fine until we started crossing ten thousand transactions and I started noticing our response time was getting a lot slower. I tried eager loading, observed the response time for a couple of weeks, and it did note get better. Not just that it dumped more data, when I retrieved transactions, it loaded everything about the user, and the most annoying thing about eager loading, is that if any other table is linked to the parent table, and you enable eager loading, it dumps all that too, what a mess of a solution.
   
   Lazy loading however proved to be more effective. The beauty of lazy loading is that the related entities are only fetched when they are accessed. It does not just load the related entities whenever you are retrieving a particular record, you have to explicitly access it. This really helped reduce our response time. There is a lot of material on the internet about when and when not to use lazy or eager loading. I suggest you do your research and tell me how you get to use it in your various projects.
   

3. Caching with Redis:-  I wager I could skip the first two and this would still be effective. Figuring out the right way to cache was not really straightforward. First thing I tried was to cache using typeORM. In typeORM you can add cache : true  to your config. What this does is to create a query-result-cache table, which stores responses for a particular time and returns that instead of hitting the table where the record is stored. Let me break this down, so when I first added this to my transactions table, typeORM stored my responses in the query-result-cache table, whenever i make another API call within 30 seconds, it goes to the query-result-cache table and retrieves the last data stores that matches the query. This proved to be the solution at first, at least for the first week. Then we started noticing retrieving user’s transactions was taking a lot of time. Whenever I make a transaction, before it gets updated on my list, 5 seconds would have gone. Did not make sense to me, I knew I haven't made any serious update in a week.
   
   After multiple complaints I decided to track the response time on staging, it was in milliseconds, I did on production and it was in seconds, hmmmmmmm. I logged into my admin dashboard and it was the worst, transactions were not loading(of course I blamed the frontend engineer for that). I got a reality check when transactions stopped loading, then my dms were flooded with complaints. I decided to check it on my local machine and it was the same thing, oh my night was not going well, I tested on staging, the same thing. I must be losing it, because I have not made a push. All my tests were returning a 504 gateway error, like what does that even mean???? With more tests and breaking a lot of things I finally figured that my solution was the problem, query-result-cache had gotten a lot of data stored in the table in such a short time, and instead of being the solution it became the problem. I had to figure out how to move these cache data from my main database, and also make them short lived.
   
   Well, Redis could do that. Redis is an in-memory database, the beauty of Redis is that it stores data using the key-value model. In my case I made my endpoints the key, while the response from my database my value. Whatever way the frontend developer decided to call my endpoint(filters included), Redis caches that and returns the stored value to the frontend. Be careful not to store a transaction for a long time, trust me. If a user makes a transaction you want the time you set as your ttl(time to live) to be short enough to cache the new transaction recorded.
   

Not to brag but our response time for getting transactions is currently averaging 207ms. I think it's not bad for an application built with a monolithic architecture and being hosted on a $7 DigitalOcean droplet. If your application is doing way better I would love to know on the comment sections, or you could email me Goodness Ezeokafor. May the code force be with you.

Laughing really hard writing this, I had a task a couple of weeks back to build a feature that helps this business add products to their marketplace. While working on it I got stuck trying to build a functionality to help them upload multiple images. Two hours gone and I could not find a fix, and the craziest thing is that I had just worked on the same functionality for another business just the previous  week. So what was the issue? I was using the @UploadedFile() to upload multiple images, what I should have known is that there are two decorators to use when uploading using NestJs. @UploadedFile() when you want to upload a single file, and @UploadedFiles() when you want to upload multiple files. So I suffered for two hours because of the letter s.

I learned a very important lesson that day, I need to pay more attention to details when coding. I had copied a file upload solution from Phind.Ai but I did not bother to read up on the explanation or even noticed that NestJs had two different decorators to handle file upload. I just went ahead and used the solution provided and it took me two frustrating hours to figure out why my uploads were not working as they should. 

If you find yourself in a similar situation, probably chill, relax, drink water, it might just be an s.

“Always think of the business”, my CTO keeps emphasizing this anytime we speak. It’s a very drastic shift from the way I've always approached software engineering. For the past 5 years building softwares for clients and companies, I've always put the code and technology first. If it's not challenging enough then to hell with the product or the client. If the client is not as excited about the technology as I am then I start doubting the client's ability to really utilize the product.

I joined Cutstruct  as a senior software engineer on contract in April, and I'm enjoying the job so far. I see myself putting the business first, does this problem really require codes? What is the best way to solve this particular issue faced by the marketing or finance department? Are there any already built solutions that can half the timeline of this task? I see myself and I'm surprised, this is not the coder or hacker in me. But I've come to realize this is what is gonna make me a better engineer or professional and I am here for it.

I had an incident just last month where I was accused of using the company's resources and time to build a rival product. Truth be told, I didn't start working on the product to challenge my current company. My friend just felt we needed a personal payment app, we built the first version and we felt this app could someday save the entire fintech industry in Nigeria and we kept adding more features and improving the entire product. It got leaked that I was building something similar and a rival product, the rest is history. Let us be very honest with ourselves, no one wants to admit but we have a shared playbook in this industry. What we think is a secret is not really a secret. The way certain things are done is shared, no one wakes up one day and thinks they have figured out how to build a fintech app. No way, you figure that out by interacting with others in the industry, from how to get a license, to building a very secure fintech app. Personally I feel we should start sharing more in the industry, there should be a playbook for any young mad fellow that thinks he has the solution to fintech in Nigeria. That young fellow should not get ass fucked by gamers because he could not implement the minimum basic security in his app. We need a Fintech 101 and it should be shared massively across the industry.

Suing a developer because you think he breached a non-compete is stupid, quote me anywhere. While you are at it, please sue ChatGPT, Stack Overflow, YouTube, and every single forum and blog post where he copied those codes to make your application work. It's like music, the fundamentals are the same, a whole lot of songs with the same chord progression, it's possible because the fundamentals are the same.

While we celebrate the OGs in this industry, let us not forget how their idea came to be and what made them who they are. Innovation and competition should be encouraged and not stifled.

Introduction

I have been obsessed about abstraction in software development since I became the Engineering Lead at Changera. In April 2022, I was promoted to lead Bitmama's sister company, Changera, now Changera Global. My first most important task was to rebuild the entire API. The api was initially written with PHP, then it was rewritten by the CTO and another developer to NodeJs. I came in and I immediately disliked the way things were done. I spent a year contributing to the API that powered Bitmama exchange, and from the lead developer at that time I understood how consistency and structure was important to the success of a working backend system. 

My first task was to rewrite the entire system to a microservices, oh boy a real challenge🤤🤤. I needed to prove myself, at Bitmama the lead developer set the terms and I followed, but now I get to set the terms. It was exciting, I did as much research I could do on microservices. Thats where I got to learn about clean-code architecture. I love the level of independence clean code architecture provides. The idea of separating your business logic from framework or dependencies was just so beautiful. I have worked on a lot of project where I had to change database drivers or ORM mid project, the toll it took to rewrite each functionality was heartbreaking. I started exploring clean code architecture in 2022 and I think I perfected it in 2024. I would be be breaking down some of my techniques in this post. 

My favourite development tool right now is NestJs, you can read up about NestJs here. NestJs is the best framework for building scalable NodeJs server side applications. Concepts like dependency injection, microservices are easier with NestJs. If you love playing around with weird concept relating to software development you would fall in love with NestJS. 

Technique

• You first need to to startup a Nest Js project. You can run git clone https://github.com/nestjs/typescript-starter.git project  on your terminal. You can familiarise yourself with NestJs  by going through their official docs.

• Create a framework folder in your src folder. The framework is where you store dependencies, tools that powers a functionality. For example I have a project and I don't know whether to use TypeORM or Sequelize. I can write both implementation and just inject it into my business logic using an abstract class. I can do the same with emails, redis and notifications functionalities.

• Create a core folder in your src folder. This folder contains your entities, dtos, enums, types, interfaces and abstract classes. Your abstract classes is the gateway to interact with your dependencies set up in your framework folder.
  
  
• Setup your controllers and services. I prefer the controller has its own folder, away from the services. I believe its more cleaner that way. Adhering to the principles of clean-code architecture we can create a user-case folder in our services folder. The user-case folder will contain the functionalities of our app. For example if I need to work on an authentication functionality, I can create an auth folder and write my business logic in it.
  

Deep Dive

• I did a code sample on this repository. You can check out the full source code here. 
  
  
• The generic repository for Sequelize and TypeORM contains the entire core database implementation, then injected into this abstract class to be used in our code logic. You have access to all the methods implemented in the generic repository can be used when you import the DatabaseServiceModule into your functionality module.
  
  
• Here is a sample of how to make crud operation after injecting the generic class to the abstract class
  
  
  
• If you focus on any of the db function, and ctrl + click, it should go into the abstract class definition. Our entire business logic is separated from the business logic defined in our use-case folders.
  

Going To Bed😴😴

My post might not be too technical but you can clone my repository and play around with my codebase to understand how clean code architecture works.  

Two month back, I was the type of dev that had to be passionate about an idea or product to contribute just a line of code. I could dump a project because the project owners or team where not obsessed with commit standards, PR standards, clean code architecture. If my idea which I always thought was better, was not picked I could get really pissed and decide to leave. So for the majority of 7 months I decided stay away from any other project, except my full time job and a little side gig of mine where I had at least 80% control of certain decisions. 

An incident occurred on April that really humbled me, an experience i will never forget for a very long time. My full time job delayed salary payment for about 3. During that period I had to crash at my close friend's place just to be able to eat, my family too were affected and some folks on my payroll. 

I wrote the most code i've ever within that 3 weeks all year, I had to source for side gigs and for the first time in a long time I was humble to accept things as the were. No pride or ego, I just wanted one thing, to get paid to write codes. I got onboarded into projects I disliked their architecture or the way they had done certain things, but I did not care. I was willing to learn it that way, write my codes that way just to survive. To be honest I had a mental shift, I learnt a valuable lesson "Passion and Obsession should never stop you from making a lot of money". I'm sure this might change, I might write another post saying not to pay attention to what I have written here and that is fine, but for now and in this moment, the money is important, the passion for doing it right can come later.