1. AWS Lambda

ဒါပေမယ့် Lambda သုံးတဲ့အခါမှာလဲ scale စာစဥ်းစားစရာမလိုဘူးဆိုတာလုံးဝတော့လဲမမှန်ပြန်ဘူး။​ AWS Lambda မှာလဲသူ့ limit သူရှိတယ်။​ Resource limit အပြင်တခြားစဥ်းစားရမယ့် limitation တွေရှိတယ်။ အဲ့တာတွေက

• Invocation payload limit
• Function timeout
• Concurrent execution limit (အဓိကပြောချင်တာဒီကောင်၊ ဒါမပြောခင်ပေါ်က၂ခုအရင်ပြောဦးမယ်)

Lambda Quota နဲ့ပတ်သက်ပြီးအဲ့မှာသွားကြည့်လို့ရတယ်။

1.1 Lambda invocation payload limit

Lambda မှာက Request/Response limitation တွေရှိတယ်။ Lambda က HTTP အပြင်တခြားလဲသုံးလို့ရတယ်ဆိုပေမယ့်ဒီမှာ API ဘက်ကိုအဓိကပြောမယ်။ သူက Synchronous ဆို 6 MB invocation limit ရှိတယ်၊​ Async ဆို 256 KB၊ ဆိုတော့ file upload တာမျိုးဆိုပုံမှန် web server ရေးသလို endpoint လဲပေးလိုက်တာမျိုးကအဆင်မပြေဘူး။ Serverless မှာဆို S3 လိုကောင်မျိုးနဲ့တွဲသုံးမှပိုအဆင်ပြေတယိ။ ဆိုတော့ File upload လိုလာရင် Lambda က presigned URL generate ပေးပြီး respond တာမျိုးလုပ်တာပိုအဆင်ပြေတယ်။ ပြီးမှ file upload ကအဲ့ presigned URL ကိုခေါ်တာမျိုး။ အလုပ်ရှုပ်တာကလွဲရင်အဲ့လောက်အဆင်မပြေတာ မျိုးတော့မဟုတ်ဘူးပေါ့။

API Response မှာလဲဒီ invocation payload limit ကိုထည့်စဥ်းစားဖို့လိုတယ်။ Payload limit exceed ဖြစ်ရင် executation fail သွားမှာပေါ့။

1.2 Function timeout

ဒါကတော့ရှင်းတယ်။ Lambda မှာက ၁၅မိနစ် timeout ရှိတယ်။ ပြောချင်တာက Lambda ကိုအဲ့ထက်ပို run ကို run လို့မရဘူး။​ Background task တွေဆို ၁၅မိနစ်ထက်ပိုမယ်ဆို AWS Batch Job လိုကောင်မျိုးသုံးပေါ့။ ဒီမှာလဲ API ဘက်ကနေပြောချင်တယ်။​ Lambda က 15 minutes timeout ရှိတယ်ဆိုတိုင်း API response time က 15 minutes ဖြစ်မှာမဟုတ်ဘူး။ ဘာလို့လဲဆို AWS Lambda ကို API အနေနဲ့တန်းမ run သင့်ဘူး။ ရှေ့မှာ API Gateway ရှိမယိ။ ဘာအတွက်လဲဆို

• rate limit
• caching

စတာတွေအတွက်ပေါ့။ REST API Gateway မှာဆို default က 29 seconds timeout ရှိတယ်​ (နောက်ပိုင်းတော့ increase လို့ရပြီ) ဆိုတော့ Lambda က 15 minutes run လို့ရရင်တောင်ရှေ့က API Gateway က 29 seconds ဆို response ကအဲ့အတွင်းမလုပ်ရင် API Gateway ကဖြတ်ချပြီး 504 Bad Gateway response ပဲပြန်လိမ့်မယ်။

1.3 Concurrent execution limit

Scaling ပိုင်းနဲ့ပတ်သက်ပြီးအဓိကပြောချင်တာကဒီအပိုင်းပဲ။​ Infrastructure scaling လုပ်ပြီးဆိုအကုန်လုံးတွေးရတော့တာပဲ။​ ဥပမာ Database ပါပြီဆိုရင်သူ့လဲသပ်သပ် scale ရတယ်။​ Read/Write cluster ခွဲတာမျိုးနောက် Read replica တောင်ထပ်ခွဲသင့်ခွံတာမျိုး၊ ပြောရရင် API Read နဲ့ကိုယ့် frontend server အတွက် Read replica သပ်သပ်မျိုးအဲ့တော့မှ API က load များလို့ကျသွားရင်တောင်ကိုယ့် frontend app ကကျမသွားအောင်။

အဲ့မှာကိုယ်က serverless သုံးရင် scale တဲ့အခါထည့်စဥ်းစားဖို့လိုလာတာက Lambda concurrent execution limit။ Original document ကဒီမှာသွားဖတ်ကြည့်လို့ရတယ်။ Lambda မှာက 1,000 concurrent executation limit ရှိတယ်သူက AWS region level။ ပြောရမယ်ဆိုကိုယ့် Lambda က Singapore ap-southeast-1 မှာဆိုဒီ executation limit က region တစ်ခုလုံးကိုထည့် count တာ။ ကိုယ့်အကောင်က Singapore region မှာ 1,000 ထက်ပို run လို့ရကိုမရဘူး။ ကျော်တာနဲ့ error တက်တာ။ Service Quotas Dashboard ကနေတော့ limit ကို increase လို့ရတယ်ဒါပေမယ့်အဲ့တာ scaling အတွက် right solution မဟုတ်ဘူး။​ ကိုယ့် app က spike traffic မှာ 100k - 200k ဆို limit ကိုသွားပြီး ၁သိန်း၂သိန်းတင်ခိုင်းနေလို့အဆင်မပြေဘူး။

ဒီတော့ Lambda သုံးရင် RDS scale ထားရင်တောင်ဒီကောင့်ကလာပြီး bottleneck ဖြစ်နိုင်တယ်။

2. Cloudflare

ဒီတော့ Scaling အပိုင်းပြောဖို့ Cloudflare ဘက်ကိုဆက်မယိ။ အဲ့တော့ Lambda API scale ဖို့ဆို Cloudflare နဲ့ကသင့်တော့တယ်။ အလုပ်မှာ service ကရှေ့မှာ Cloudflare ခံထားတာအဲ့တော့ Cloudflare ကသာ cache သေချာလုပ်ထားရင် AWS ကို request ကလာစရာကိုမလိုတော့ဘူး။ Cloudflare Edge Network ကနေပဲ response မယ်။ Scaling လဲအဆင်ပြေမယ်၊​ edge network ဖြစ်လို့ response time လဲနည်းမယ်။ ဒါပေမယ့်ဒီမှာလဲစဥ်းစားဖို့လိုတာတွေရှိတယ်။ Cloudflare Cache က silver bullet မဟုတ်ဘူး။

Cloudflare Caching Rule အလုပ်လုပ်ပုံက URL pattern ပေါ်မူတည်ပြီး Edge TTL cache ကို configure လို့ရတယ်။​ ဥပမာကိုယ့် API data ကတော်တော်ကြာကြာပြောင်းလေ့မရှိဘူးဆို Edge TTL ကို 1 year အထိထားလို့ရတယ်။ နားလည်လွယ်အောင် diagram နဲ့ဆွဲပြရမယ်ဆိုဒီလိုမျိုး။

ဒီမှာတယ်တော့သိပ်နားမလည်ကြတာတစ်ခုရှိတယ်။​ Edge TTL အလုပ်လုပ်ပုံက 1 year ထားတိုင်း Cloudflare Edge Network မှာ 1 year အထိထားပေးတာမဟုတ်ဘူး။ ဒါကိုကျွန်တော်လဲအစကသဘောမပေါက်ခဲ့ဘူး။ အလုပ်မှာ campaign တစ်ခုအတွက် service လုပ်နေတာအဲ့မှာ cache ကို pre-warm လုပ်ထားမယ်ပေါ့ all possibilites တွေအကုန်ကြို cache ထားချင်တာ Edge TTL configure ပြီး script နဲ့ API request combination အကုန်ထိုင်ခေါ်နေတာနောက် 1 နာရီလောက်နေတော့ပြန်ခေါ်ကြည့်တာ HTTP Response Header မှာဘာလို့ cf-cache-status: EXPIRED ဖြစ်လဲပေါ့ဆိုပြီးလိုက် debug ကြည့်တော့မှာဒါကိုသိတာ။

Discussion ကိုဒီမှာဖတ်ချင်ဖတ်ကြည့်လို့ရတယ်။ TLDR ပေးရမယ်ဆို Edge TTL cache duration အလုပ်လုပ်ပုံက 1 year ပြောထားရင် Cloudflare က Edge Servers တွေမှာ 1 year cache ခိုင်းတာထက် 1 year cache လို့ရတယ်လို့ပြောချင်တာ။​ ကိုယ်က 1 year ပြောထားလို့ 1 year အထိ cache ပေးမှာမဟုတ်ဘူး။ သူ့မှာ internal caching mechanisms တွေရှိတယ်ကိုယ်က 1 year ပြောထားရင်တောင်ဒီ resource ကသိပ်မသုံးဖြစ်ဘူးပြောရရင် Cache Hit သိပ်မဖြစ်ဘူးဆို auto edge servers တွေကနေဖျက်ပစ်တယ်။

ဆိုတော့ကိုယ်တော်ဥပမာပေးရမယ်ဆိုကျွန်တော့်မှာ campaign က peak concurrent traffic က 300k-400k (၃သိန်း၄သိန်း) HTTP requests ဝင်မယ်ဆို AWS Lambda ကိုတိုက်ရိုက်ထိဖို့ကအဆင်မပြေဘူး။ Edge TTL ကလဲ campaign မတိုင်ခင် pre-warm လုပ်ထားဖို့ကအဆင်မပြေဘူး Cloudflare က Cache Hit မဖြစ်တာကြာရင်ဖျက်ချမှာ ပြီးတော့ HTTP payload combination က 1 million, ၁၀သိန်းလောက်ရှိမယ်ထားပါတော့ဆိုတော့ Cache ကလဲလုပ်ကိုလုပ်ထားရမယ်။

2.1 Cloudflare Workers and Workers KV

Solution မပြောခင်ဒီမှာတချို့ကပြောမယ် Redis သုံးပေါ့ဘာညာ။ ဘာလို့အလုပ်မဖြစ်လဲပြောပြမယ်။ Redis ကဘယိနေရာမှာအဆင်ပြေမလဲဆို Lambda နဲ့ RDS ကြား cache ချင်ရင်။ ပြောရရင်အောက်ကလို

Lambda က Redis မှာအရင် check မယ်မရှိရင် RDS ကနေဖတ်ပြီးတော့ Redis ထဲမှာ save ဒီတော့နောက် request တွေက Redis ကတန်းဖတ်လို့ရအောင်။ ဒါပေမယ့်ဒီ solution ကအပေါ်က problem မှာအလုပ်မဖြစ်ဘူး။ ဘာလို့လဲဆို RDS က bottleneck မဟုတ်ဘူး။ Bottleneck က Lambda မှာ။ ဒီတော့ Redis ထည့်လဲ Lambda concurrent execution limit ကဖြစ်မှာပဲ။ Solution က Cache ကို client နဲ့အနီးဆုံး edge ဘက်မှာထားရမှာ။​ ပြောရရင် orange cloud aka Cloudflare မှာ AWS မှာမဟုတ်ဘူး။

Solution ကဘာလဲဆိုတော့ Edge TTL cache တိုက်ရိုက်လုပ်မယ့်အစားကြားထဲမှာ Cloudflare Worker ကို proxy အနေနဲ့သုံးပြီး Workers KV ကို cache အနေနဲ့သုံးတာ။

Cloudflare workers က proxy အနေနဲ့အလုပ်လုပ်မယ်ဆိုတာသူက Client/Frontend ကနေ request ကိုလက်ခံမယ်။ ပြီးရင် Workers KV မှာရှိမရှိ check မယ်မရှိရင် Lambda ကိုလှမိးခေါ်မယ်ပြီးရင် Workers KV မှာလာ save မယ်ပြီးရင်ပြန် response မယ် client ကို။ ပြောရရင်အောက်ကလိုမျိုး

Redis နဲ့တူတယ်ဒီမှာမတူတာက Workers KV ရှေ့က Edge servers ကဘာလို့ပါနေလဲပေါ့။ ပြီးမှသပ်သပ်ရှင်းပြမယ်။ အခုလို solution ဆိုအပေါ်က problem အတွက်အဆင်ပြေသွားပြီ။ ဥပမာကျွန်တော်က HTTP payload combination လံကြိုသိတယ် 1 million ဆိုအကုန်လုံးက campaign မစခင်မှာထဲက cache ကို warm ကြိုလုပ်ထားလို့ရတယ်။ အဲ့တော့ Lambda ကထိကိုမထိတော့ဘူး။​ Workers KV ကနေပဲဖတ်မယ်တန်း response မယ်။ Peak concurrent traffic က 300k-400k ဆိုလဲ RDS scaling တို့ Lambda concurrent execution limit တို့ကိုပူစရာမလိုတော့ဘူး။

အဲ့တော့ပြောဖို့ကျန်တာက KV ရှေ့က Edge Servers တွေ။ Cloudflare စီကပဲ image ကို copy လိုက်တော့မယ်။ Original document ကိုဒီ How KV works မှာသွားဖတ်ကြည့်လိုရတယ်။

KV မှာ hot and cold read ဆိ်ုတာရှိတယ်။​ Cloudflare worker က Lambda ကနေဖတ်ပြီး KV မှာ save ရင် Cloudflare က central storage တစ်နေရာမှာသွား save တာ။ KV ကနေ worker ကပြန်ဖတ်တဲ့အခါက user request location ပေါ်မူတည်ပြီး edge servers က client နဲ့အနီးဆုံး location တွေစီမှာ Cloudflare က KV data ကိုထပ် cache တယ်။ ပြောရမယ်ဆိုကျွန်တော် Lambda ကလာတဲ့ response ကို Cloudflare worker က India က central storage တစ်ခုမှာ save လိုက်တယ်။ နောက် user က Singapore ကနေခေါ်တဲ့အခါ Singapore Edge Server မှာရှိရငိ worker ကအဲ့ကနေဖတ်ပြီး client ကိုပြန် response မယ်။

Lower latency, faster response time ပေါ့။ မရှိရင် India က central storage ကနေဖတ်မယ်၊ Singapore က edge server မှာသိမ်းမယ် user ကို response မယ်။ ပြောရရင်ကျွန်တော် Lambda က Cloudflare KV central storage အလုပ်လုပ်သလိုပဲ။ ဒါကို Cloudflare က hot and cold read လို့ခေါ်တယ်။ Central server က read တာက cold read၊ Edge server က read တာက hot read။

ဒီ hot read အတွက် edge server မှာသိမ်းတာကိုလဲဘယ်လောက်ကြာကြာ cache လို့ရတယ်ဆိုတာကို configure လို့ရတယ်။ Default က 60 seconds ပဲ။ ဒီမှာဖတ်ကြည့်လို့ရတယ်။ ဒီ configuration ကကိုယ့်လိုအပ်ချက်ပေါ်မူတည်တယ်။ Cache TTL (time-to-live) များရင် Edge server မှာကြာကြာသိမ်းမယ်။​ Faster respones time ဒါပေမယ့် Cache discrepancy ပိုကြီးမယိ။ KV central storage မှာ save ပြီးအချိန်တစ်ခုကြာမှ Edge server မှာ reflect လာဖြစ်မယ် TTL ပိုကြာရင်။

ဒီ post ကတော့ဒီမှာပဲဆုံးပြီ။ ဒါကလက်ရှိအလုပ်မှာကြုံရတဲ့ case တစ်ခုနဲ့ပတ်သက်ပြီးပြန် share ချင်ရုံသက်သက်ပေါ့။ Serverless optimization နဲ့ပတ်သက်ပြီး share ချင်တာလဲတစ်ပိုင်းပေါ့။

With the popularity of Python in web server development, more and more people use frameworks like Flask. Flask is a web application framework, and it is not a web application server. Therefore, it needs a WSGI server to actually run the application.

Among those WSGI servers, Werkzeug is one of them. What is interesting about Werkzeug is that, it comes with a debugger. It is awesome during development, and with Debug=1 you can see what is going wrong with your web application.

However, when a developer forgets to disable debug mode in production, an attacker can make use of this to exploit the application.

Insecure Flask application

I will be using the following Flask application for the demo.

from flask import Flask, request

app = Flask(__name__)

@app.route('/')
def hello():
    return '<html><body><h2>My Notes</h2><li><a href="/notes?file=secret.txt"> secure notes </a></li></body></html>'

@app.route('/notes', methods=['GET'])
def notes():
    args = request.args
    filename = args["file"]
    with open(f'./notes/{filename}', 'r') as f:
        return f.read()
    return 'File not found'

Then I put a note under ./notes/secret.txt

Sup3rS3cr3t

Only 1 dependency which is Flask :) and you can run as follows.

FLASK_APP=main.py FLASK_DEBUG=1 flask run

It has a home page, that as a link to list of notes (1 for the demo), and /notes is GET request with query parameter for file name. However, there’s no input validation, and it is vulnerable to Local File Intrusion (LFI) vulnerability.

If you go to the homepage, you will see the following page.

If you click secure notes or curl the URL directory, you will see notes content.

However, since there is no input validation, LFI can be done as below.

LFI alone is dangerous, but it cannot be code execution yet. So this is where /console comes it. By default, Werkzeug console is PIN protected.

So no more code execution? NO

Because it can be cracked if we know some key parameters. We can know these information by first reading debug output. This can be done by crashing the application.

I read a non-existent file in order to crash the app and dump debug logs.

Exploiting the server

Using LFI vulnerability, I can read the debug logic code of werkzeug. This file also contains the code for generating the console PIN.

In order to crack the Werkzeug console PIN, I need the following 6 parameters.

1. username

can get from enumeration or, in my case, from logs /home/khant/…

2. modname

flask.app

3. app class name

getattr(app, '__name__', getattr (app .__ class__, '__name__')) → flask

4. flask app.py location

getattr(mod, '__file__', None) → from debug logs I can see where the package installed, so should be under /home/khant/Documents/work/security/demo/flask-hello-world/env/lib/python3.10/site-packages/flask/app.py

5. mac address

read /proc/net/arp from LFI to find devices then /sys/class/net/DEVICE_ID/address, and you will receive a mac address. Example 62:c8:fb:aa:85:39, and you need to convert the hexadecimal to decimal.

• https://www.rapidtables.com/convert/number/hex-to-decimal.html

6. machine ID

Read /etc/machine-id or /proc/sys/kernel/random/boot_i

Once you got everything, you can use the code from Hacktricks to generate a PIN.

• https://book.hacktricks.xyz/network-services-pentesting/pentesting-web/werkzeug

In my case, the pin is 298-808-092.

Now, when I enter this, I can log in to the Python console and execute system commands as below.

It means I have code execution access on the system with the rights of the user that is used to run the Flask application.

PS

In this article, I showed how I can use the LFI vulnerability to gather information for cracking /console password.

This is not always the case; sometimes you already have a foothold on the system, but you realize another user (possibly a privilege user) is running Flask server, therefore, you can use this attack to either escalate privileges or perform lateral movement to another user.

Before I talk about why I chose k6, let’s talk about why load testing is important.

1. Why load test?

Load testing enable you do measure response time, resource utilization levels, able to identify bottle-necks and the breaking point of the application.

There are 2 success criteria for load testing.

1. load forecasting
2. how the system handles usage spikes

Load forecasting

Based on the previous usage collected, we can predict the growth for the near future and put some rooms ahead for the system to be able to scale when the growth hits. With the forecasting result, we can add or remove resources accordingly.

In AWS, this strategy can analyze history for up to 14 days to forecast what demand is for the following two days. Updated every day, the data is created to reflect one-hour intervals.

Usage spike

In this case, the usage will grow by 4-5 times in a very short period.

E.g, one particular Reddit post gets popular, and the link is then shared on other social media. Everyone including ghost users (without Reddit accounts) started visiting the page. First, we need to see how the system scales automatically, and how it performs to the spike. By doing so, we will also learn what is the breaking point of the system.

Successful criteria should be

• Alerts are produced during the spike.
  • This can be achieved by setting up the resource monitoring and alert systems in place.
• System should auto-scale horizontally in general. Although scaling can be done manually, I still encourage the auto-scaling mechanism.
• System recovers by itself once the spike is over, aka self-healing.

2. Why k6? How about the alternatives?

So I spent few days finding out what is the most suitable tool out there for my need.

Locust.io

Python-based load testing library, and the oldest among the alternatives.

Pros

• Tests are written in Python.
• Docker support

Cons

• No official supports for result exports except CSV

Artillery

Another widely popular tool based on Javascript/Typescript.

Pros

• Artillery scripts are written in YAML with some Javascript injection for custom testing flow (this is also a con)
• Can easily integrate with CI/CD
• Has cloud version
• WebSockets supports other than HTTP
• Got plugins for additional functionalities such as exporting for Grafana
• Docker support

Cons

• DSL (YAML) is limited because YAML with Javascript injection makes the testing flow hard to read as the test gets larger. The use of YAML to control test flow feels unnatural. For example, you can look at the HTTP flow from the documentation.
• Artillery is slow compared to k6, and more resource hungry (this is debatable however, Artillery is running NodeJS at its core). You can look at the following benchmark by GitLab for comparison.

k6

Popular load testing framework made by Grafana Lab. Therefore, it has first-party support for Grafana.

Pros

• Has extensive examples for different test cases.
• Support several protocols (HTTP, WebSockets, gRPC) that can be useful in the future.
• Scriptable in JavaScript
• Official support for different visualization tools (Grafana dashboard, Prometheus metrics, Datadog, etc)
• Has cloud version
• Better documentation compared to artillery.io
• CI/CD support
• Has official Docker image

Cons

• No Typescript support yet, only Javascript for scripting

I chose k6 for these reasons

Documentation is one of the most important factors for choosing it because if it is not well documented, and lack of an extensive list of examples for different scenarios could slow down our progress at writing load testing. In addition to that, I look for export formats and support for several protocols so that we can use the same load testing library for our future projects. Also, CI/CD will be crucial when we start integrating all these testing into automated processes.

3. Plan/Pricing with k6

k6 is a great tool, however it is quite expensive compared to the rest.

• Developer plan costs only $99 a month however, with only up to 100 concurrent virtual users maximum, it is not feasible for most of the use case.
• Team Plan cost $500 a month, with 1000 concurrent virtual users maximum, this is the most suitable plan for most projects. However, this plan has a limitation of a 30-minute max duration per test. Therefore, if we need to run a soak test (I will explain below of each load tests), we will need to buy a larger plan instead.
• Pro Plan costs $1.5k a month with up to 3000 concurrent users, and 60 minutes max duration. This is quite expensive imo.

4. Privacy concerns

If your project is not accessible by public, and cannot be exposed to public/third party services, you can still use k6.

Tests can be run using our own local machine or using their cloud service.

The same applies to the visualization of the result. We can

• export the output in CSV
• send it to the self-hosted or cloud Prometheus instance and visualize it on the Grafana
• send the test result to the k6 cloud (not an option if you cannot expose to third party services)

If we do not want to share any information with the third-party services, we can use self-hosted option. However, the k6 cloud offers ease to set up, and its web UI shows a very details summary of the test, and it can also compare the result performance with previous tests.

With a pro plan, we can also generate a PDF summary of the test on k6 cloud.

5. Load test types

Before we talk about k6 any further, let me explain the different type of load tests and what are their purposes.

1. Load Test

• This is the generic testing strategy to determine how many concurrent users/requests per second the server can handle.
• It usually takes a reasonable amount of time, ~15 minutes.
• This is also the test we run to find the basic benchmark of the system, and how far we can push the system before it breaks.

2. Stress Test

• It is used to determine the limit of the system by sending the requests in different stages. It shows the stability and reliability of the system under extreme conditions.
• It starts with a slow load, and gradually increases the load to see what’s the limit.
• With multiple stages, it can also help us determine if the system can recover without manual intervention.
• Stress test usually lasts about 30-60 minutes, depending on how many stages are in the test, and how we gradually increase the load.

3. Spike Test

• It is very similar to a stress test, however, the key difference is unlike a stress test, it spikes the usage within a short period.
• This scenario is similar to cases where one Reddit post went viral and everyone starts visiting.
• The system will react in one of the following manners
• An excellent system will not degrade during the spike and manage to respond in a reasonable amount of time.
• Good system will respond slower than usual, however, no error is raised.
• Poor system may respond to some request or will raise an error, but manage to recover automatically after the surge.
• Bad system will crash, and not recover even after the surge is gone.
• Spike test does not take very long, as it only tests one scenario, having a large number of users in a short period.

4. Soak Test

• Soak test is the longest load test of all because its purpose is to find out errors that one occurs after running the system for a long time. This test helps us find an error that could not be produced easily, such as the limitation in the infrastructure, and the hardware the system is running on.
• Before we run the soak test, we need to determine the maximum number of users the system can handle, and we run the test with the large user but without crushing the system.
• Here are the cases where soak testing is useful
• Find out memory leaks, in which one starts showing obvious symptoms after the system is up for long period, and the allocated memory is not free properly
• Race conditions
• Database usage, whether the constant requests to the system by a large number of users exhaust the database resource
• Determine how many logs do we output by a certain number of users in a long run and whether it exhausts the disk space

5. Smoke Test

• A smoke test, aka confidence testing, is to run as a sanity check.
• The purpose of the test is to simply verify if the system gives out obvious errors under minimal load.
• Also unlike the other test, this can be run in CI as it finishes within ~1-2 minutes, and also it is only to see whether any small changes made to the underlying infra break the software.

6. k6 metrics

However, here are the key points we need to focus on in the metrics.

• http_req_duration
• http_req_connecting
• http_req_failed

http_req_duration

This metric is the combined duration of receiving, sending, tls_handshaking, and waiting. It helps us determine how long each HTTP request takes on average.

http_req_connecting

This metric shows how long it takes for the client to establish a connection with the server. This combined with the above metric basically shows how long it will take for the user to see the webpage for example.

http_req_failed

This shows the rate of failure, especially during the time of large requests coming into the system. We should be focusing more on this metric, especially in tests like stress and spike.

For a more detailed explanation of each metric, you can look at the following documentation.

7. How to do it k6 way

Write load testing script on local and configured the test to submit the result to the k6 cloud.

Configuration

First, you need to get the api-token from your k6 account.

Then follow the instructions from this page to set up the k6 cloud and k6 CLI.

Once it is done, go to the k6 cloud, create a new project, copy the project ID, and add it to the test script configuration as below.

Writing API code

When you need to interact with the API or application server, write it in api.js (using k6 HTTP modules) and import it back into different load tests (smoke, spike, load, etc). This way, you can reduce the duplication of the application flow.

Setup and teardown of the test

Sometimes, you may need to set up some stuff before running the test and clean up the data after the test. This can be done in k6 via setup and teardown methods.

setup()

If defined, k6 will run the code inside the function before running the actual load test.

teardown()

If defined, k6 will run the code after it finishes running the test.

Sharing data across different test phases

In k6, you can also send additional information from the previous phase to the next phase.

For example, after you have created a new user in setup() phase, by returning the values in the Javascript object, we can pass those values to the next phases (load testing default function, and teardown method).

8. Conclusion

k6 is pretty great tool if you can afford the price. I wanted to share how k6 cloud visualization looks like but for privacy reasons I cannot. Therefore, you can take a look at their k6 cloud promo page and how to visulization with their cloud for how their cloud service.

There are 3 candidates for my final image base. These are

• Alpine
• Scratch
• Distroless

Initially, I wanted to use Rust official images for building. However, there are a few issues with it

1. need to statically link my built to the musl libc because scratch and alpine are based on musl libc not glibc.
2. OpenSSL issue with cross-compilation, you can read more about it here.

glibc issue can be solved by simply installing the library and adding the target for cross-compilation as below. But the tough part comes with SSL.

RUN apt-get install 

RUN apt-get install -y musl-tools

RUN rustup target add x86_64-unknown-linux-musl

RUN cargo build --target x86_64-unknown-linux-musl --release

For both issues, someone already built a base image (rust-musl-builder) just for cross-compiling rust application. Therefore, I am using it to build my application.

# ==========
# Build step
# ==========
FROM ekidd/rust-musl-builder AS builder
WORKDIR /subtext

# copy files
COPY ./src ./src
COPY ./Cargo.toml .
COPY ./Rocket.toml .

# https://github.com/emk/rust-musl-builder#making-static-releases-with-travis-ci-and-github
ADD --chown=rust:rust . ./

# build release
RUN cargo build --release

# ===========
# Final image
# ===========
FROM scratch
WORKDIR /subtext

# copy build
COPY --from=builder /subtext/target/x86_64-unknown-linux-musl/release/subtext_api ./
COPY --from=builder /subtext/Rocket.toml ./

CMD ["/subtext/subtext_api"]

Built images

In Final image step, I used scratch, alpine, and distroless.

• Scratch has the lowest file size because it is barebone.
• Alpine is also another lightweight image, and it is also kind of a defacto standard.
• Distroless is a security-oriented image maintained by Google, and based on Debian packages therefore using this I will not need to do static linking musl.

So scratch is the best candidate to achieve the lowest image size, however, it comes with some costs. Especially with SSL. Scratch images cannot perform SSL cert certification because it does not come with SSL certs. You can read more about it here.

You can simply fix in scratch by installing it yourself or use alpine instead. With only ~6mb more, alpine comes with a ton of more utility tools as well.

Then for distroless, even though it is security oriented, there are a few issues with it. You can read the entire article from Red Hat here, but the key point is the size matters. A larger size also increases the attack surface.

Trivy for scanning vulnerability

This is shown by scanning all those 3 images using Trivy, ironically distroless comes with quite a few CVEs from libc. Trivy is an awesome tool to scan vulnerabilities for file systems and images.

Lower binary size (for Rust)

Until now, we are focusing on optimizing the image size. We can also optimize the Rust built as well to even lower the size. That however comes at the cost of performance. But if you are interested you can look up the following 2 links below.

• johnthagen/min-sized-rust
• arusahni.net/optimizing-rust-binary-size

It is a container engine very similar to and compatible with the current defacto tool in containerization space, Docker. Podman is from Red Hat and it’s sort of like a replacement for Docker because it uses the same syntax, same command, and even got tools like podman-docker to emulate docker CLI in podman.

However, there are a few keys underlying differences between Podman and Docker

1. Docker uses a daemon which is a program running in the background to handle services and processes. Unlike docker, podman is daemonless. You can read more about what is docker daemon here. Running daemonless means, it runs containers under the user starting the container.
2. Podman runs the container in rootless mode. This is considered to be safer than containers with root privileges. IIRC, docker also added rootless mode, you can check it here.

Podman is also more towards modularity compared to docker. For example, docker can build images on its own, however, in podman it uses another tool called Buildah to build images. Calling podman build will actually calls buildah bud behind the scene.

Since podman is a drop-in, you can use docker-compose with podman as well, or instead, you can use podman-compose which also works just like docker-compose.

Rootless containers in Arch

Running rootless in Arch is a bit tricky because in Arch subordinate user ids and subordinate group ids does not exist. These contain a list of users and user ids that the user is allowed to impersonate.

If you have not created those two files, you will receive an error as below.

Podman requires the user running the rootless container to have a range of UIDs listed in those files. Here is an explanation of how does Podman rootless works, and feel free to read about there for a better explanation of how it is working in underlying.

So first create files as below in root shell

touch /etc/subuid
touch /etc/subgid

run the following command to assign a range of subordinate users and groups to username

usermod --add-subuids 100000-150000 --add-subgids 100000-150000 username

Finalise the changes as below in user shell

podman system migrate

If you have not read about first part where I have talked about setting up the project for local deveopment. I talked about these points previously

1. SQL driver/ORM for database migration
2. Configuration management to handle different application environments (development, testing, staging, production)
3. Docker for local development and spwaning test database for DB tests
4. Pre-commits for linting, and code standardizing

In this article, I am going to talk about how I design Continuous Integration (CI) to setup linting, testing the software, and building docker images. For those, who wants a good explanation of Continuous Integration, Continuous Delivery and Deployment, check this article from Atlassian.

More than a year ago, few other developers and I decided to try Golang to build microservices for our new project. Previously, we used Python frameworks such as Django, Flask, and Starlette/FastAPI. They all have their ups and downs, especially FastAPI. It comes with Swagger UI and a data validation layer using Pydantic.

Python is okayish most of the time, and it is still one of my go-to languages. However, I always wanted to try Go on a large project, and the rest of the developers also had the same idea, so we gave it a try.

2. Intro

This article is about how I setup local development, design the project, setup CI/CD for one of the microservice application I developed in go. I am not going to explain to you what the microservice is. If you want to know, go and read Martin Fowler article. It is a good read, and even if you know what it is, I still recommend you to read it.

I will tell you what functionalities my service needs to provide, what tools/frameworks I used to implement those. Lastly, how I set up testing with CI, and dockerize it so that my service integrates nicely with the project overall. So let me break it down into several sections

1. Frameworks I used for the functionalities I need to provide
2. Local development setup, configuration management, and dockerization
3. CI/CD (also visualizing the test coverage)
4. How my service integrates with other services in the project overall

3. Frameworks

Web Service and Database

Since it is an API, I need a database (used Postgres) for storage, a web framework (Gin) with Pop as the ORM, and Soda CLI to handle database migration.

Previously, I used database driver (sqlx) instead of Pop. In most cases, I would recommend ORM over SQL drivers mainly for one reason. To not write SQL queries inside the application code. If you still want to use SQL driver, that is also not an issue because you can still use database migration tools like Soda CLI to handle the migrations separately. However, I would not suggest dropping the database migration tool a good idea because it makes your life so much easier. Trust me, doing database migration with .sql files is a nightmare.

It is very similar to Django with all those migration tools, etc, for those coming from Python. The only difference is, I am just taking out whatever frameworks I need for the requirement instead of using the whole ecosystem.

If you are into the complete package, you can look into the Buffalo ecosystem. However, I do not like frameworks like Django anymore because it is kinda overkilled. Even in Python, I prepare smaller frameworks like Starlette with no template engines out of the box but, you can install it only if you need it.

In addition to all those, my service also needs to provide Async Job/Queue for internal services communication. Therefore, I used Machinery with the Redis backend. Additionally, the service I am working on needs to schedule some tasks, and the Machinery is to trigger those cronjobs. I will explain further in detail later in part 2 of the article how my team and I handled the cronjob among our microservices.

Configuration Management

In every service, there will be four application environments in general. Developing, Testing, Staging, and Production. Your application will need to store some configurations like database URL string, debug flag, HTTP server port, etc. Your database URL string will be different between staging and production among the four stages I mentioned above, and the configuration management tool is quite helpful to handle it.

Traditionally, people handle it by the environment variables, and it works obviously. But using the config management tool is far cleaner. For my service, I used Viper to handle configs. Then Cobra is used to define the command line argument for my program.

Why do I need that? I want to build my API service into one standalone binary file that can provide several functionalities such as running the migration, running the webserver, etc. This makes it a lot cleaner than splitting multiple scripts and figuring out what script to run for different functionalities.

Example:

service run migration
service run api

4. Local Development Setup

I used to develop with virtual environments before, like PyEnv/VirtualEnv in Python. That is quite a good approach honestly. However, that is until I was introduced to docker 4 years ago. Initially, I just use docker to run the database. Along the way, I started to shift my workflow from running directly on the machine to containerization with Docker.

Docker

At work, I set up Dockerfile and docker-compose files together for local development as well. It takes some time to set up those however there’s an advantage when you work with a team. I do not need to worry about what operating system my colleagues are running. As long as docker is installed in their system, a simple compose-up command will work.

In addition to the simplicity, there’s another benefit, especially working with microservice architecture. Imagine the case where I am working on one service, and it depends on another service developed by my colleague. With the help of docker, I do not need to download his source code. All I need is the docker image built from CI/CD pipeline I will explain later.

In my opinion, docker is a really powerful tool that every web developer should get familiar with because it not only makes deployment simpler but also in developing programs.

Running Test

In most cases running a test is a pretty easy task. However, running a test with a database behind is not so simple. For testing service with a database, there are two approaches

1. Mock database queries: in Go, you can use go-sqlmock but after I worked with it for a few weeks, I decided to drop this approach for one main reason. Mocking database query calls is very time-consuming and makes the test case unnecessarily complicated.

2. Spawn test database with docker: compared to mocking database calls, this is much simpler. With the help of Viper I mentioned in Configuration Management section, you can simply define different database URL so that when the test runs it connects to the test database.

Spawning test database

At AcePointer, we have our own open-sourced shared library that is built on top of ory/dockertest to spawn test database easily. You can simply spawn the test database in docker as below

func DatabaseBootstrap(configPath string) func(t *testing.T) {
	return func(t *testing.T) {
		// load test config using viper
        ...

        dtHelper.RunTestPostgreSQL(
			t,
			viper.GetString("database.config.host"),
			viper.GetString("database.config.container_name"),
			viper.GetString("database.config.port"),
			viper.GetString("database.config.username"),
			viper.GetString("database.config.password"),
			viper.GetString("database.config.database_name"),
			viper.GetString("database.config.docker_tag"),
		)
	}
}

There’s a reason why you should spawn the database from go instead of docker because the spawning test database takes a few seconds. The program needs to wait until the database is ready before running the test. If not, the database connection will be broken and tests will fail.

Makefile

Docker is awesome, but you still need to type so many commands. For example, for running test, you spin up test database, run migration, then run go test. Once the test is finished, the database is still there, and you have to delete that so that it does not conflict with the next test when you run again. Using make you can create a proper workflow as below

go-test:
	go clean -testcache
	-go test -tags=unit,integration -p 1 ./apps/...
	docker stop test-db
	docker rm test-db

Live reload

Using docker has one disadvantage. That is you will need to rebuild the entire application to see changes. That is not so ideal in the development environment because the developer will make a lot of changes (obviously he is writing a code, duh), and rebuilding the image every time would be a very tedious job.

Using air and Docker volume mount, you can configure it such that when the code has changed it go will simply rebuild the app inside the container and reload it. Basically, using volume mount

• mount the source code folder to the container
• then using air look for the changes inside the container

A few months ago, I created this docker image and shared it on the docker hub called sparkling. It is an image pre-built with air and soda-cli inside. For local development, you can set up Dockerfile as below

FROM 1iquid/sparkling:latest
WORKDIR /app
CMD ["air"]

Linting

Linting is an important thing when it comes to coding. Every language has its standards when it comes to code quality. In Python, there’s PEP 8 and black is a defacto tool for standardizing. It is not surprising that Go also has her guidelines and tools. Usually, for linting, I set it up in 2 stages.

1. pre-commit hooks
2. CI stage

I will explain about CI stage in the second part of this article. For now, I will explain about setting up in pre-commit. Go standard library comes with a command called gofmt which is used to format your go code. Along with that, I use 3 other tools for linting go code

1. goimports for formatting package imports
2. errcheck to check unhandled errors in the program
3. govet is used to check the correctness of the go program

Of course, I will not be running each tool manually before every commit. So I usually set up those tools with pre-commit as below

# sh scripts/pre-commit-err-check.sh
for DIR in $(echo "$@"|xargs -n1 dirname|sort -u); do
    errcheck ./"$DIR"
done

# sh scripts/pre-commit-go-vet.sh
go vet --vettool=$(which shadow) ./...

repos:
  - repo: local
    hooks:
      - id: gofmt
        name: gofmt
        entry: gofmt -s -w .
        language: system
        types: [ go ]
        description: Format your Go code

      - id: goimports
        name: goimports
        entry: goimports -w .
        language: system
        types: [ go ]
        description: Format your Go imports

      - id: errcheck
        name: errcheck
        entry: sh scripts/pre-commit-err-check.sh
        files: '\.go$'
        language: system
        description: Check your Go source code with errcheck

      - id: govet
        name: govet
        entry: sh scripts/pre-commit-go-vet.sh
        files: '\.go$'
        language: system
        description: Analyze your Go code

I will stop this article here for part 1. If you are looking for how I setup the example gin api server, you can check the sample server here.

In next part, I will explain about how the project is configured at CI/CD stage. How docker images are built for each microservice, and how everything is put together for the project overall.

So the endpoint will have multiple responses as below

• 200 Ok returns data
• 400 Bad request when request data is invalid
• 404 Not found when data not found at the service layer
• 500 Internal server errors for cases like SQL driver failure and so on

Inside the route, it calls a method from the service layer to get data, perform JSON serialization on it, and then respond. In the case of failure, I want to return an error JSON response with an appropriate error message. So the code signature looks like below

#[get("/")]
pub fn get_records(state: &State<AppState>) -> Result<Json<Records>, Error> {
    ...
}

There’s an advantage of doing that, it makes error handling for service calls easier. Instead of having match error handling which is messy in my opinion, I can just use ? operator shortcut in route. Just see the difference between these two below.

pub fn get_records(...) -> Result<Json<Records>, Error> {
    let records = state.service.get_records()?;
    ...
}


pub fn get_records(...) -> Result<Json<Records>, Error> {
    let records = match state.service.get_records() {
        Ok(data) => data,
        Err(..) => ...,
    };
    
    if records.len() == 0 {
        return Err(Error::NotFound(...));
    }
    ...
}

returning record upon success, Json<Records> is simple, it will be 200 Ok response, easy. However, for Error part it can be any of the three (400, 404, 500) I mentioned above.

In rust, it is a little bit tricky when you need to return more than 2 types of responses with different HTTP statuses. Someone asked about this on GitHub before, he wanted to do HTTP redirection, template response, and failure.

As it is also stated by someone in the discussion, it would require something like Either type. Unfortunately, rust does not have it in the standard library. There is an external library but I do not want to go that route.

Enum saves the day

I created an enum, error to handle different errors as below. This error will be used across all layers, presentation, service, and repository. Then a struct is created along with an enum just for JSON serialization later.

// package declarations
...

#[derive(Serialize)]
pub struct ErrorResponse {
    message: String,
}

#[derive(Error, Debug, Clone)]
pub enum Error {
    #[error("{0}")]
    Internal(String),

    #[error("{0}")]
    NotFound(String),

    #[error("{0}")]
    BadRequest(String),
}

Then I will implement one method that will help me convert enum types into rocket HTTP status.

impl Error {
    fn get_http_status(&self) -> Status {
        match self {
            Error::Internal(_) => Status::InternalServerError,
            Error::NotFound(_) => Status::NotFound,
            _ => Status::BadRequest,
        }
    }
}

Now, I just need to use Responder trait on custom Error enum so that I can use it in routes for returning. For reference, you can check out official documentation of the Responder trait implementation.

impl<'r> Responder<'r, 'static> for Error {
    fn respond_to(self, _: &'r Request<'_>) -> Result<'static> {
        // serialize struct into json string
        let err_response = serde_json::to_string(&ErrorResponse{
            message: self.to_string()
        }).unwrap();

        Response::build()
            .status(self.get_http_status())
            .header(ContentType::JSON)
            .sized_body(err_response.len(), Cursor::new(err_response))
            .ok()
    }
}

The code is pretty simple, I use serde_json to serialize the JSON. Then using Response to build the response with different HTTP status and the response body.

If you look into the service layer, it uses the same Error enum for the return result. Same for routes.

use crate::{entities::Record, error::Error};

#[derive(Debug)]
pub struct Service {}

impl Service {
    pub fn new() -> Service {
        Service {}
    }

    pub fn get_records(&self) -> Result<Vec<Record>, Error> {
        let mut records = Vec::new();
        ...
        Ok(records)
    }
    
    ...
}

This may seem a bit complicated compared to languages like Python, Javascript, etc, where returning multiple data-type is much simpler. If you want to see the working example, I created one here.

Once you run the code, go to

• / for the success response
• /error for internal server error
• /bad for HTTP bad request error

PS: there may be a better way of handling an error in the rocket framework, as I am still new to rust programming, what I just showed may not be the best solution. But so far, this is what I find clean and manageable for handling different HTTP errors.
If there’s a better solution, feel free to create an issue in the repository I mentioned above.

🦀

Question explanation

ဒီ question ကမေးထားတာကရှင်းပါတယ်။ Grid ပေးထားတယ် (M x N grid ပေါ့)။ အဲ့မှာ land ဆိုရင် 1 လို့ယူပြီး 0 ဆို water လို့မှတ်တယ်။ Island ဆိုတာကဘယ်လိုသပ်မှတ်သလဲဆို land တစ်ခုကသူနဲ့ ဆက်စပ်နေတဲ့ land တွေကိုစုလိုက်မယ်။ စကားနဲ့ပြောရတာသိပ်မရှင်းရင်အောက်ကဆွဲထားတဲ့ပုံကိုကြည့်ရင်သဘောပေါက်မှာပါ။

အဲ့မှာဆက်စပ်တယ်ဆိုတာ left/right/up/down ဆက်နေတာကိုမှဆက်စပ်တယ်လို့ယူလို့ရပါတယ်။ Question ကရှင်းပါတယ် ပေးထားတဲ့ grid မှာ island ဘယ်နှခုရှိလဲတွက်ခိုင်းတာပါဒါပါပဲ။

ဒီ question ကို rust လုပ်ဖြစ်နေတော့ rust နဲ့ solve ထားပါတယ်။ ဒါပေမယ့် Python solution ကိုလဲအောက်မှာပေးထားပါတယ်။

Thought process

medium ဆိုပေမယ့်ဒီ question ကတကယ်တော့မခက်ပါဘူး။ လုပ်ရမှာက land တစ်ခုကို iterate လုပ်ပြီဆိုသူနဲ့ဆက်စပ်တဲ့ land တွေကိုပဲနောက်တစ်ခု visit မလုပ်အောင်လုပ်ရုံပါပဲ။ အပေါ်ကဥပမာကိုကြည့်ရင် grid[0][0] ရောက်ရင်သူ့ neighbors တွေကိုပါ visit လုပ်ပေးရမှာဒါပါပဲ။

ဆိုတော့သူ့ရဲ့ direction လေးမျက်နှာလုံးကိုကြည့်မယ်၊ ရေမဟုတ်ဘူးဆိုရင် visit လုပ်မယ်နောက် visit လုပ်သမျှကောင်တွေအကုန်လုံးကိုသိမ်းမယ်ဒါပါပဲ။ ဒါကိုကြည့်ခြင်းအားဖြင့် question က graph traversal question ဆိုတာသဘောပေါက်မိမှာပါ။ Graph traverse လုပ်ရင်၂မျိုးရှိမယ်။ DFS (depth first search) or BFS (breadth first search) ဒီမှာတော့၂ခုထဲကဘာသုံးသုံးအဆင်ပြေတယ်။

Regarding Graph Interview Questions
Graph question မြင်ရင်ခေါင်းထဲတန်း၀င်လာသင့်တာ၃ခုပဲရှိတယ်များသောအားဖြင့်က Breadth first search, Depth first search, and Topological sort. Interview မှာ question အများစုကဒါပဲ။
Niche ဖြစ်တဲ့ question တွေတော့ရှိသေးတယ်၊ ဒါပေမယ့်တော်ရုံတန်ရုံမေးလေ့မေးထမရှိဘူး။ ကိုယ်လုပ်ရမယ် job position က graph နဲ့တိုက်ရိုက်ဆက်စပ်တာဆိုရင်တော့မေးတတ်တယ်။ ဥပမာ self driving car company တစ်ခုမှာသူ့တို့ကားတွေအတွက် GPS data build လုပ်တဲ့ team မျိုး join မယ်ဆိုပါစို့အဲ့တာဆို Graph ကိုအပေါ်ကထက်နဲနဲပို niche ဖြစ်တဲ့ algorithm မျိုးတွေမေးကောင်းမေးလိမ့်မယ်။ ဥပမာ Dijkstra တို့ A* algorithm တို့။

Edge cases

edge case က traverse တဲ့အခါ out of bound check တာသတိထားဖို့ပဲလိုတယ်၊ တခြားတော့ဘာမှထွေထွေထူးထူးသတိထားစရာမရှိဘူး။

Solution

ပေးထားတာကတော့အောက်ကလိုပါပဲ။

impl Solution {
    pub fn num_islands(grid: Vec<Vec<char>>) -> i32 {
        
    }
}

အဲ့တော့ solution ကပေးတဲ့ graph ကိုကြည့်မယ်၊ island ဘယ်နှခုရှိလဲတွက်ပြီး interger return ပြန်ရမယ်။ ဒီ question ကိုတော့ DFS iterative နဲ့ပဲ solve မယ်။ အဲ့တော့ variable ကတကယ်လိုတာ၂ခုပဲလိုမယ်။

1. number of islands: ဒါကရှင်းတယ်၊ interger ပဲ။
2. visited cells: ဒါကိုက grid ကို flatten လုပ်လိုက်ပြီး array အနေနဲ့သိမ်းမယ်၊ visit လုပ်လိုက်တဲ့ cell ဆို boolean/int 1 လို့သပ်မှတ်မယ်မဟုတ်ရင် 0၊ ဘာလို့ visited သိမ်းဖို့လိုလဲဆိုသူ့ဆက်စပ်တာတွေကိုလိုက် visit ပြီးတိုင်းမှတ်ထားမှမဟုတ်ရင်ပြန် visit မိပြီး island count တွက်တဲ့အချိန်မှာမှားလိမ့်မယ်။

...
    pub fn num_islands(grid: Vec<Vec<char>>) -> i32 {
        let R = grid.len();
        let C = grid[0].len();

        let mut islands = 0;
        
        // create R * C length array with 0s
        let mut visited = vec![0; R * C];
        
        for i in 0..R {
            for j in 0..C {
                if grid[i][j] == '1' && visited[i * C + j] == 0 {
                    islands += 1;
                    
                    // the code below is incomplete for now
                    Solution::dfs();
                }
            }
        }
        
        /*
          In rust, you can just write `islands` instead of `return islands;`
           but I chose the latter for people who are not familiar in rust.
        */
        return islands;
    }
...

ရှင်းပါတယ်။ Cell တစ်ခုချင်းစီကို visit ဘယ်အဲ့မှာ island ဖြစ်ပြီး visit မလုပ်ရသေးတယ်၊ visited က 0 ဖြစ်မယ်အဲ့တာဆို island count ကို၁တိုးပြီး DFS လုပ်မယ်။ DFS ထဲမှာက island တစ်ခုလုံးကိုလိုက် visit လုပ်မယ်ပြီးတော့ visit လုပ်လိုက်တဲ့ cell တွေကို visited ဆိုတဲ့ variable ထဲမှာမှတ်မယ်။

ဆိုတော့ DFS function ကိုဆက်ရေးမယ်။ DFS လုပ်မယ်ဆိုအမြဲတမ်း end condition ကိုစစဥ်းစားတာကောင်းတယ်။ ဘယ်အချိန်မှာ traverse တာရပ်မလဲပေါ့။ အခု question မှာကရပ်မမယ့် condition ၃ခုရှိတယ်။

1. row ဖြစ်ဖြစ် column ဖြစ်ဖြစ်က out of bound ဖြစ်သွားရင်
2. visited လုပ်ပြီးသားဆိုရင်
3. land မဟုတ်တော့ရင်

နောက်ပြီး visit ရမယ့် condition ကလေးခုရှိတယ်။ up/down/left/right

ဒါတွေသဘောပေါက်ပြီးဆို code ကိုကြည့်မယ်။

...
    fn dfs(visited: &mut Vec<i32>, grid: &Vec<Vec<char>>, i: usize, j: usize) {
        let (R, C) = (grid.len(), grid[0].len());
        
        // check if out of bound or visited
        if i >= R || j >= C || visited[i * C + j] != 0 || grid[i][j] == '0' {
            return;
        }
        
        // set visited as 1
        visited[i * C + j] = 1;
        
        // visit top, down, left, right
        Solution::dfs(visited, grid, i + 1, j);
        Solution::dfs(visited, grid, i - 1, j);
        Solution::dfs(visited, grid, i, j + 1);
        Solution::dfs(visited, grid, i, j - 1);
    }
...

function မှာက parameter ၄ခုရှိမယ်။ Visited နောက် original grid ပြီးတော့ visit တဲ့ position (i, j). အဲ့မှာ out of bound အပေါ်ကပြောခဲ့တဲ့ end condition ၃ခုဖြစ်မဖြစ်ကြည့်မယ် မဖြစ်ရင် visit လုပ်ပြီးကြောင်း update လုပ်မယ်ပြီးရင်သူ့ဘေးက neighboring cells တွေကို visit လုပ်မယ်။

ဆိုတော့အပေါ်က incomplete ဖြစ်နေတဲ့ function call ကိုလဲပြန်သွားပြင်မယ်။

...
        for i in 0..R {
            for j in 0..C {
                if grid[i][j] == '1' && visited[i * C + j] == 0 {
                    islands += 1;
                    Solution::dfs(&mut visited, &grid, i, j);
                }
            }
        }
...

Time complexity က MxN ဖြစ်မယ်ဘာလို့လဲဆို cell အကုန် visit ရလို့နောက် Space complexity ကတော့ 2(MxN) ဒါက rust မှာမို့လို့၊ rust မှာသူက grid ကို immutable အနေနဲ့ pass ထားတော့ visited ကိုသပ်သပ်သိမ်းနေရတာမဟုတ်ရင် MxN ပဲ။ Python နဲ့ဆိုရင်တော့အောက်ကလိုပါပဲ။

class Solution:
    def numIslands(self, grid: List[List[str]]) -> int:
        self.grid = grid
        self.R = len(grid)
        self.C = len(grid[0])

        islands = 0
        
        for r in range(self.R):
            for c in range(self.C):
                if self.grid[r][c] == "1":
                    islands += 1
                    self.dfs(r, c)
        
        return islands
 
    def dfs(self, r, c):
        if r < 0 or r >= self.R:
            return
        
        if c < 0 or c >= self.C:
            return
        
        if self.grid[r][c] == "0":
            return
        
        self.grid[r][c] = "0"
        
        # go left right up down
        self.dfs(r + 1, c)
        self.dfs(r - 1, c)
        self.dfs(r, c + 1)
        self.dfs(r, c - 1)
        
        return

Requirement

မေးထားတာကတော့ရှင်းရှင်းလေးပါမယ်။ URL တွေကို visit လုပ်မယ်နောက် browser တွေလိုမျိုး forward/backward လုပ်လို့ရရမယ်ပေါ့ဒါပါပဲ။ အဲ့မှာဥပမာကိုယ်က

• ပထမ google.com ကိုသွားတယ်။ နောက် facebook.com ကိုသွားတယ်။ Backward ပြန်လုပ်ရင် google.com ကိုသွားရမယ်။ နောက်အဲ့ကနေမှ forward ပြန်လုပ်ရင် facebook.com ကိုပြန်ရောက်သွားမယ်။
• ပြီးတော့ twitter.com ကို visit ထပ်လုပ်မယ်။ အဲ့မှာ history က၃ခုဖြစ်သွားပြီ။ 1) google.com 2) facebook.com 3) twitter.com
• backward (facebook.com ကိုပြန်ရောက်သွားပြီ) လုပ်ပြီး yahoo.com ကိုသွားရင် twitter.com ကိုအရင် delete ရမယ်။ ဆိုလိုချင်တာက history ကအောက်ကလိုဖြစ်ရမယ်ပေါ့။
• ဆိုတော့ 1) google.com 2) facebook.com 3) yahoo.com

implement ရမယ့် requirement ကဒါပါပဲ။ ဆိုတော့အပေါ်က example မှာက backward/forward ကို၁ခါပဲလုပ်ပြသွားတာ။ requirement က N steps move နိုင်တယ်တကယ်က။

Thought Process

Leetcode problem တစ်ခုကို solve မယ်ဆိုအဖြေတန်းရေးတာကကောင်းတာမဟုတ်ဘူး။ Requirement ကိုသေချာနားလည်အောင်ဖတ်ရမယ်။ နောက် edge cases တွေစဥ်းစားရမယ်။ ဒါကို pseudocode မရေးခင်မှာထဲက တွေးသင့်တယ်။

Edge cases

• Backward/Forward ကို N steps နဲ့လုပ်နိုင်တယ်ဆိုတော့အကယ်လို့သာ N steps က history size ထက်ကြီးနေရင်သို့မဟုတ် 0 ထက်ငယ်နေရင် move မလုပ်သင့်ဘူးပေါ့။ ဥပမာ browser မှာ latest page ဆို forward button ကို disabled လုပ်ထားသလိုပေါ့။

Solution idea

ဒီ question က leetcode က medium question ဆိုတော့သိပ်တော့ခက်ခဲတာတော့မဟုတ်ပါဘူး။ ဒီ question အတွက် stack ကိုသုံးတာ suitable ဖြစ်တယ်။နောက် forward/backward လုပ်ပြီဆိုအခုက ဘယ် page ကိုရောက်နေတယ်ဆိုတာသိဖို့လိုတယ်။ ဒီအတွက်က cursor လိုမယ် current index ကိုသိမ်းဖို့။

ဆိုတော့ python နဲ့ solve မယ်။ ဒီမှာနဲနဲပြောစရာရှိတယ်။ Competitive programming တွေမှာ C++ ကအသုံးများတယ်နောက် interview တွေမှာတော့ Python ကအသုံးနဲနဲများတယ်။ Unless position specific, ပြောရရင် frontend engineer ခေါ်ရင်တော့ Javascript သုံးခိုင်းကောင်းသုံးခိုင်းလိမ့်မယ်။ C++ ကတော့ speed အတွက် Hackerrank လုပ်ဖူးတဲ့သူဆိုသိလိမ့်မယ်။ Interview တွေမှာက performance အတွက်ပါ score သပ်သပ်ရှိတယ်။ ဆိုတော့ brute-force solution သမားတွေဆို fail မှာပဲ။ Leetcode solve ရင်အကြံပေးချင်တာက bruteforce solution လောက်နဲ့ တော်ပြီလုပ်နေလို့မရဘူး။ Optimize ဖြစ်တဲ့ solution မျိုးလုပ်တတ်ဖို့လိုတယ်။ Optimize ဖြစ်မဖြစ်သိဖို့က time complexity တွက်တတ်ဖို့လိုတယ်။ ပြောရရင် computer science fundamental လေးတီးမိခေါက်မိ ရှိထားဖို့ပေါ့။

နောက် Facebook ဆို code ကိုပေးမ run တာ။ ဆိုလိုချင်တာ code ဘလိုအလုပ်လုပ်တယ်ဆိုတာကပါးစပ်နဲ့ thought process ပြောရတယ်။ Heap သုံးထားတယ်ဆို heap ကို in-deapth မေးတာ။ ဥပမာ heap insertion ဘယ်လိုအလုပ်လုပ်တယ်နောက် time-complexity ကဘယ်လောက်လဲစသဖြင့်။ Bruteforce နဲ့ solve ပြီး CS fundamental မသိရင် follow-up question မှာ fail မှာပဲ။

ဒီ post ကိုတော့ Python နဲ့ပဲ solve ထားတယ်။ လူအများစုလဲနားလည်တော့။

ကုဒ်ကအောက်ကလိုပေးထားတယ်။

class BrowserHistory:
    def __init__(self, homepage: str):

    def visit(self, url: str) -> None:

    def back(self, steps: int) -> str:

    def forward(self, steps: int) -> str:

### 1. init

အပေါ်ကပြောသလို stack နဲ့ cursor လိုမယ်ဆိုတော့ init မှာအဲ့တာတွေကို initialize အရင်လုပ်မယ်။ အဲ့မှာတခါထဲ homepage ကိုပါ stack ထဲထည့်မယ်။ current ကလက်ရှိရောက်နေတဲ့ page index ကိုသိမ်းဖို့။

 class BrowserHistory:
    def __init__(self, homepage: str):
        self.stack = [homepage]
        self.current = 0
...

### 2. visit

နောက်ဒီကောင်မှာက user က browser input မှာရေးရင် website သွားတဲ့ behavior လုပ်ရမယ်။ ဆိုတော့ဒီမှာ facebook.com သွားတယ်ဆို stack ထဲကို facebook.com ကိုထည့်မယ်၊ နောက် current ကိုလဲ အဲ့ facebook.com ရဲ့ index ကို point ရမယ်ပေါ့။ လက်ရှိရောက်နေတဲ့ page က facebook.com ဖြစ်ရမှာကို။

 ...
     def visit(self, url: str) -> None:
        self.stack.append(url)
        self.current += 1
...

### 3. back

ဒီကောင်က browser backward functionality ကို implement ရမှာ၊ ဆိုတော့လက်ရှိက facebook.com ကို visit ထားတယ် (index 1) အဲ့မှာ backward နိုပ်တယ်ဆို google.com ကိုပြန်သွားရမယ်ဆိုတော့ index ကို ၁ပြန်နုတ်ရမှာပေါ့။ ဒါပေမယ့် question မှာက N times move မှာဆို index ကို -N လုပ်ရမှာ။ နောက်အပေါ်ကပြောခဲ့သလိုမျိူး edge case ရှိမယ်နုတ်လိုက်လို့ 0 ထက်ငယ်ရင် 0 ယူရမှာ။ မဟုတ်ရင် index out of bound ဖြစ်သွားလိမ့်မယ်။

 ...
     def back(self, steps: int) -> str:
        self.current -= steps
        
        if self.current < 0:
            self.current = 0
        
        return self.stack[self.current]
...

ဒီ function ရဲ့ time complexity က O(1) ဘာလို့လဲဆို arithmetic operation က O(1) နောက် if condition ကလဲ O(1).

### 4. forward

သူလဲခုနကလိုပဲ။ edge condition ကတော့ length - 1 ထက်ကြီးရင် length - 1 ကိုယူရမှာ။

 ...
     def forward(self, steps: int) -> str:
        self.current += steps
        
        if self.current > len(self.stack) - 1:
            self.current = len(self.stack) - 1
        
        return self.stack[self.current]
 ...

သူကလဲ O(1) ပဲ။ len method ကို O(N) ထင်ကောင်းထင်လိမ့်မယ်။ Python documentation သွားဖတ်ကြည့်ရင်သိပါလိမ့်မယ်။ O(1) ဆိုတာဒီမှာပြောချင်တာရှိတာကကိုယ်သုံးတဲ့ language ကိုသေချာသိအောင်လုပ်ထားပါ။ ဥပမာ Java ထားပါတော့ string ၂ခုကို concatenation လုပ်ရင် O(N + M) ဘာလို့ဖြစ်တယ်ဆိုတာရှငိးပြတတ်ဖို့လိုပါတယ်။ မေးလဲမေးခံရဖူးပါတယ်။ ဘာလို့လဲဒီမှာသွားဖတ်လို့ရပါတယ်။

ကုဒ်ကအောက်ကလိုပါ။

class BrowserHistory:
    def __init__(self, homepage: str):
        self.stack = [homepage]
        self.current = 0

    def visit(self, url: str) -> None:
        self.stack.append(url)
        self.current += 1

    def back(self, steps: int) -> str:
        self.current -= steps
        
        if self.current < 0:
            self.current = 0
        
        return self.stack[self.current]

    def forward(self, steps: int) -> str:
        self.current += steps
        
        if self.current > len(self.stack) - 1:
            self.current = len(self.stack) - 1
        
        return self.stack[self.current]

ပြီးပြီလို့ထင်ရင်မပြီးသေးပါဘူး။ Case တစ်ခု handle ဖို့ကျန်ခဲ့တယ်။ အကယ်လို့ current page ကနောက်ဆုံး page မဟုတ်ဘူးဆိုရင်အသစ် visit တဲ့အခါသူ့နောက်က stack ထဲက pop ဖို့ပါပဲ။ အပေါ်မှာ facebook.com ကနေ yahoo.com ကိုသွားတဲ့ example ကိုပြောချင်တာ။ ဆိုတော့အောက်ကလို modify ဖို့လိုတယ်။

...
    def visit(self, url: str) -> None:
        while self.current < len(self.stack) - 1:
            self.stack.pop()
        
        self.stack.append(url)
        self.current += 1
...

Run time က O(N) ဖြစ်လိမ့်မယ်။ worst case ကရှေ့ဆုံးက page ကိုသွားပြီးတော့ visit တာပဲ။ အဲ့တာဆို N - 1 pages တွေ stack ထဲကနေ pop လုပ်ရလိမ့်မယ်။ Solution ကတော့ဒါပါပဲ။ Solution ရဲ့ space complexity ကတော့ N ပါ။ တစ်ခုမှ pop မလုပ်ပဲ visit ပဲလုပ်နေရင်အကုန်လုံးကို stack ထဲမှာသိမ်းရမှာပါ။

Coding interview preperation အတွက် Leetcode premium ကစျေးကြီးပေမယ့်အသုံးတည့်ပါတယ်။ မဟုတ်ရင် clone ဖြစ်တယ့် Lintcode ဆိုတာမှာ premium တချို့ကို free ပေးထားတာတွေရှိတယ်။ Coding interview preperation အတွက် data structure နဲ့ algorithm topic တွေသင်ပေးတာမှာ Grokking the coding interview ဆိုတဲ့ course ကမဆိုးဘူးပြောလို့ရတယ်။

နောက်ကျွန်တော်အရင်က Leetcode solve ထားတာတွေကိုလဲ repo တစ်ခုလုပ်ထားတာရှိတယ်။ အောက်မှာသွားကြည့်လို့ရတယ်။

https://github.com/the-robot/coding-challenges

နောက်ပိုင်းလဲအားရင်အားသလိုတော့ Leetcode ကစိတ်၀င်စားစရာကောင်းတဲ့ question လေးတွေကို solve ပြီး explain တာမျိုးလေးတွေရေးချင်တာတော့ရှိတယ်။