Introduction

This post will be about my recent entrepreneur project, it has been in development for some time so some parts will be left out and others will be shortened otherwise this post will be too long.

For easier navigation you can use the table of content above.

Gifs showing important parts

Product Idea

I saw a TV competition using sms for voting and I got curious how companies would implement sms solutions into their own product.

After some market research I came to a conclusion that the market mostly consists of SMS Gateways offering API’s.

💡I thought about. Why is there no easy way to create sms flows with competitions, customer clubs, statistics and be able to quickly edit such flow without having to hire programmers.

If I could create such service I could help companies save a lot of money and time usually spent on developing and updating their sms solutions.

Requirements

My customers is non-tech hence the product had to be very user-friendly but still versatile enough to create complex sms flows.

Frontend

For framework I had heard plenty of good things of Angular2. Its made by Google, its using Typescript and everything was separated into modules, components and views. I did a small project to get the hang of what I was getting myself into and I really liked how Angular2 worked.

Backend

As the sole developer I didn’t want to spend my time configuring load balancer’s or Linux VM’s I just wanted something that could automatically scale and work, this is where I found Firebase. Firebase at that time was in early stages having its Cloud Functions in beta and if I remember right its only database was a JSON schema RealtimeDB.

Using Firebase was a big risk because Firestore and Cloud Functions was in beta but from what I could gather at the time Google was serious about Firebase so I went with it.

In the beginning it was a wonky ride because Cloud Functions had long cold-start times which has gotten better and to this day I just deploy a ExpressJS API onto Cloud Functions which works flawlessly. Today both Cloud Functions and Firestore is out of beta🎉

I also implemented Stripe to handle payments.

Difficulties & Solutions

User-friendly but complex flows

My product is meant for the non-tech person so I had to find a way to boil down complex logic into something which anyone could use with ease, but on the backend I had to translate it back into complex logic.

While I was drawing diagrams for how the overall system should be put together I got an idea: These diagrams is pretty much logic visualized as boxes and connected with lines.

I liked the idea and began prototyping. I put the name of the action in the box, I made it possible to connect boxes with lines and boxes had inputs and outputs. Here are two images of early prototypes.

The prototypes was a success, very intuitive, quick and easy to use.

Translating visual blocks to code

This was very difficult. How do I translate visual blocks with connections into code.

The blocks consists of multiple elements:

• Input and output parameters. Input for supplying a block with necessary data for instance the block Send SMS requires which text to send. Outputs is because blocks can output data for use in later blocks via variables
• Connections between blocks
• Logic: The piece of code that runs when a block is executed.

Each block also got to have a name which the user can change to easily distinguish between blocks and a truly unique id so its easier to find/remove/edit specific blocks.

The flow have a global context for the current execution. The context contains data such as which MSISDN triggered this execution and the sms message.

I solved it (over a long time and a few times refactoring large parts) by creating a kinda graph structure but with a set of rules.

• Heavy usage of abstract classes. Example. I got a abstract class for a “block” which contains base properties and general block related logic.
• Each block has its own class which extends the base abstract block class. The block class also sets its inputs/outputs and parameters. These blocks and the abstract classes is shared between the backend and frontend to enforce a common contract.
• On the backend each block have a designated logic class. Example Send SMS would have a Send SMS Logic class which extends the Send SMS Class. The logic classes contains getting necessary parameters, preparing parameter variables, getting outputs and contain the actual logic regarding sending a SMS.

The logic class is using promises for the logic, it also evaluates which output it should move to and see if there is a valid connection to another logic class.

Storing a flow is done by translating the flow into JSON. By overriding the toJSON() method I only retrieve the properties I need. Translating the flow to JSON also makes it easier to validate and I can translate the schema back into visual blocks with data and connections and on the backend I can translate the JSON into logic blocks.

It’s important to notice that the user is not writing any code. The blocks is configurable and can be translated to and from JSON, the JSON is validated before accepted.

When an sms is received I’m checking which flow is associated with its lease then retrieves the JSON schema and translate it into logic blocks and run it.

Counters & Sharding

Firestore have a limitation of maximum one update per second per document. This is a problem if you want to have a sum of things and the sum is often updated.

In my situation I’m calculating the number of customers in a club. Firestore does not have a SUM() method. Another quick-fix could have been retrieving all customers and count the result but imagine having 20.000 in a club and my customer would look at these statistics regularly, that is 20.000 reads that quickly adds up to an expensive database usage.

What I do is whenever a customer is added to a club I’m also inserting a job in the database which triggers a cloud function.
The job contains information about the club, customer, number of retries, a lease lock and boolean tasks that needs completion before the job is successfully completed.

The cloud function runs as follows

1. Retrieve the job through a transaction and evaluates whether it is allowed to process by looking at the lease lock.
2. If allowed we increase the number of tries and update the lease lock so another cloud function can’t run this job for a time duration, this also eliminates duplicate trigger invocations.
3. Executes the job and set task booleans to true within an transaction.
4. If everything went fine I return from the trigger but if something went wrong I throw an error which will trigger another cloud function to try this job.

A counter consist of a few shards so when updating it we retrieve a random shard and try to update the number but if it fails we can either create a new shard or try updating another random shard.

To get the sum I retrieve the shards (to begin with I’ve set it to three shards) and sum the values. The result is only three reads or a few more if a few more shards was created.

Scaling with Google Pub/Sub

To handle large quantities of incoming and outgoing sms’s I’m doing something similar to the above but instead of database triggers I make use of Google Pub/Sub.

Having topics for different kind of jobs and then subscribe to these topics. If something goes wrong with a job I can retry it by deferring it’s processing to a later time.

In Counters & Sharding I made use of Firebase database triggers, I’m thinking I might refactor it to use Google Pub/Sub.

Conclusion

I learned a lot from this complex project and a few technologies was a first time try.

It was my first time using Angular2 for a big project and by now it just got updated to Angular8. I really like how works as it got everything included and it forces the developer to build a website in a very logical and modular way. The learning curve was steep but worth it.

It was also my first time trying serverless by using Firebase. I think serverless is great because the cloud provider takes care of the server instances and most of the setup while I the developer can spend more time on creating the app. Sometimes we certainly still need VM’s for some use-cases.

Serverless cost is cheap as you only pay for exactly what you use but it got a dark side because its auto scalable you can quickly receive a big bill.

With serverless you need to fully understand each service you use and how it works, let me give an example: If you add a onWrite trigger to your Firestore and if that trigger updates the value it will retrigger, if the only code path is keep updating the value it will retrigger in a loop forever until you either update or remove the function.