I learned these 3 lessons the hard way. I hope you don’t.

I learned the following lessons the hard way and I believe -despite being very obvious- most of teams tend to ignore these hints specially teams with low resources / capacity.
1. Always keep an alert on your cloud provider bill.

  • You will be one day senior enough that you will be trusted for your team’s / company’s cloud infrastructure bill. Please, don’t forget to put a budget alert.
  • Feature is available on azure, AWS and I’m not sure if it’s available for alibaba cloud and IBM cloud but there must be some sort of alternative for these platforms.

2. Always have a test environment that’s completely separate from production and avoid making the production environment / production accounts accessible in development.

Making production environment too accessible for your development team this may risk (top of my head):

  • Junk data in production
  • Loss of data in production
  • Loss of credit
  • Going out of API quota.

3. Don’t Exclude your Ops and DevOps from QA steps.

When your backend developers has some green tests locally that doesn’t necessarily mean that you made it. you have to actually get the card done by going to production with the correct config-maps, envs, service accounts and feature-flags and (whatever environment dependent change) that may make this code break) is in place then you may cross your fingers and celebrate.

In DevOps some teams (and I used to be part of these teams) usually tend to celebrate DevOps efforts very early be and exclude it from the traditional QA.

In short, you will regret it and I’m sure because I’ve been there. normal backend/front-end task usually can be recoverable but a back-up script that have an issue is not recoverable and won’t be noticeable until it’s too late.

I’m pretty sure that many of the readers of this small article may have more lessens that they learned the hard way; Please, Don’t hesitate to share them with me in the comments. you may save a life!

Thank you!

Designing Fine-grained systems – part1

Hi this is to start a new series of posts about microservices architecture, it’s going to be mainly in the format of Q/A.

In this post.

    • What’s the Microservices architecture ?
    • What are the patterns of Microservices architecture?
    • What are the benefits of Microservices architecture ?

Q:What’s the Microservices architecture ?

A: There’s no specific answer for this question however it’s generally accepted to say that it’s designing systems that are formed of many small (as in small responsibility list) and independently deployed services.

My definition: It’s a child for SOA that’s favoring distributed, single responsibility services over monolithic service(s).

This may trigger another question(s) like:

  1. How small is microservice ?
  2. How to ensure that services are independent in terms of responsibility and deployment ?

Please, check next question’s and next posts for answers.

Q:  What are the patterns of microservices architecture?

A: Following are the main patterns of what’s usually conceived to be a Microservices design:

  1. Services over libraries for components
    Instead of having systems that are formed of components that are libraries, system is now formed of small services that are often used to share code and logic between other services.
  2. Business layout over Technical layout
    For a normal eCommerce store, instead of having front-end and backend teams and components we will have:
    shopping-cart, search, collection services and teams.
  3. Automation
    “If it’s hurts, do it more often”   — Martin Fowler
    All the hurting stuff should be done more often or in other words (automated) hence things like (CI/CD, Automated Testing, Accept rewrite of components,etc..) are essentialNote: If you are resisting the fact that services should be completely rewritten in no time then services are either not small enough or it’s designed somehow to be tightly coupled to another service(s) which indicates a bad design.Hint: A good practice and well-known pattern is to have the ability to rewrite and roll-out a rewritten service in 2 weeks for example.
  4. Choreography vs imestration *
    choreography
    If you have many services this triggers a question how should you conduct some command, do you create a service that’s dedicated for orchestrating what other services should deliver ?
    Same for your team do you create a team that’s dedicated for management ?In Microservices it’s more like a ballet dance. Despite how accurate and synced are the dancers you don’t see some manager who guide dancers in execution time. This is called choreography.
    In this metaphor every dancer is a service and music is system inputs and services should react to this independently without external governance.
    Same for teams each team should be able to code, test and deploy his service independently.
    Which comes hand to hand with next pattern.
  5. Automation and monitoring.
    Infrastructure automation CI/CD are essential in microservices.
    In a system with 10s and may be 100s of services  rate of change and deployment is huge and it can’t be only Ops team responsibility in such architecture DevOps team has a main responsibility to enable automation in infrastructure.
    One of the benefits is being fault tolerance(on system level) but the failing service still need to know how did this happen, what’s the current response time what are the consumed Memory, etc..
    I can safely say that Monitoring and Automation are essential to any successful Microservices architecture.
  6. Failure Is Inevitable. What Matters Is How You Deal With It.
    Miniature-Circuit-Breaker-Application-Guide
    “If a short-circuit happened in your kitchen, your laptop in living room should survive.“– Me trying to explain electrical circuit breaker to one of my friends.The lesson learnt from great electrical engineers is that you should be able to handle failure because it will happen no matter you try to avoid it and I strongly recommend reading how netflix are fault tolerant.
    Caching, retrying, rerouting are brilliant ideas for mitigating failure in runtime and this article has lots of ideas.

Q:What are the xkeybenefits of Microservices architecture ?

A:  Microservices architecture isn’t a silver bullet yet it has many advantages and to name a few:

  1. Fault Tolerance (in other words failure isolation) unlike monolithic apps there’s no single point of failure.
    Example: In e-Commerce store:  if searching service is down this doesn’t mean that the store is down, it just means you can’t search now but you still can buy, add to shopping cart, etc..
  2. No vendor lock-in or technology stack lock-in (every service can be using different stack) microservices favor hetrogenity loose coupling generally hence stack and vendor unlocking is by design.
  3. Horizontal Scalability (well, it’s the  fact that you can split your system into small services and put every service in one machine/vm/container makes it better to scale wisely)
    Let’s say that your eCommerce system is composed of shoping-cart, search-engine and catalog
    With microservices architecture every component can be a separate service and each service deployment can be scaled (up/down) based on load applied on this specific part of the system.
    On the other monolithic hand, scaling up and down is per system deployment.
    spartplug
  4. Maintainability, test-ability (inherited from being small separable services you can verify if every service can work alone and hence test it, monitor it and rewrite if you want.) What if a spark plug wasn’t removable ?
  5. Resource utilization (think of it as space utilizing problem for a jar where filling is Sand vs  small ball vs big ball.)
    the smaller the better.

*1. I’ll do another post about event-driven asynchronous programming to support this point.

Sources

      • Building Microservices: Designing Fine-Grained Systems Kindle Edition by Sam Newman
      • Microservices a definition of this new architectural term www.martinfowler.com/articles/microservices.html
      • Micro services, what even are they?
        http://rea.tech/micro-services-what-even-are-they/
      • FrequencyReducesDifficulty
        https://www.martinfowler.com/bliki/FrequencyReducesDifficulty.html

13 months of MC

It has been 13 months since I first joined MC.

I started as 13 months ago and I’m still as excited as I first joined I’m doing different things here and there and I’m now responsible for a team of 5 besides my work.

Very recently I managed to pull of a face lift for couple of our main money generating components and managed to reduce performance of both apps dramatically, unfortunately I won’t be able to share details more than having this  Selection_002change on some major app for 3 weeks in a row and that was around 300% decrease in page load time.

I was happy 1 year ago and I’m still. this is what matters after all.

Technically the major updates was taking ECMA Script and Design more seriously (Weird mix) but this is reality.

Since I’m taking architecture more serious now (both application and system architecture), I’m willing to post and talk more frequently about design in the following posts.

See you soon.

LinkedIn vs. XING [linkout gem]

I have to admit. I always didn’t like linkedIn, very bad UX and UI and features aren’t aligned enough to convince me that I should stay more than 5 minutes browsing it not to mention the absolute clutter at it’s news feed and how non-sense is it to follow and watch completely irrelevant posts and shares.

Anyway, since it’s the hub of all companies and jobs and leading(so-called) professional network I had to make a profile there and keep it updated as much as I can.

In May 2015 LinkedIn announced a change in API  they commented on this API change by saying

For many developers, we understand that today’s changes may be disappointing and disruptive, but we believe these changes will provide further clarity and focus on which types of integrations will be supported by LinkedIn.

And this year with high resolution image of Microsoft’s CEO and linkedIn’s Microsoft has acquired linkedIn

“The LinkedIn team has grown a fantastic business centered on connecting the world’s professionals,” Nadella said. “Together we can accelerate the growth of LinkedIn, as well as Microsoft Office 365 and Dynamics as we seek to empower every person and organization on the planet.”

With that said I’m expecting no good from LinkedIn anymore except for helping Microsoft be a better in monopoly.

Hence I’ve developed  linkout ruby gem to allow scraping LinkedIn profile without the need of their API.

And I’m also developing a new app that uses this gem to port your profile to XING.

I’ll update this post when this app is ready so keep an eye on it.

Note: use linkout gem only to get your profile data not to scrap the whole linkedIn or else you’ll be violating ToS of linkedIn.

 

Lifelong Learning

Life isn’t always fair

It’s been very interesting quarter in MC and I believe I was doing good I received a raise and it feels great when I get a raise specially when it’s related to being high performance, I also (very recently) started learning “Machine Learning” and studying “Scala” more regularly.

On my personal life side, It’s been generally a happy quarter specially the end, since I’m getting to gym more regular and I visited  Sharm AlSheikh and Dahab(for the first time with my small family), did snorkeling in the most dangerous diving spots in the world (The Blue Hole).

However said, I believe things in my house isn’t as organised as in my work, my dear wife Reem is a new mother and she is new to feeding and raising Yahia while keeping up with house keeping tasks; Yahia is very demanding baby and keeping up with him is very hard task. I believe she won’t be able to get it right anytime soon. Good news is that she is constantly searching, acquiring help from her available resources, Learning and she is making some progress.

I’m constantly trying to develop a language using which she will get my comments and advices on how she should do her job right without feeling any criticism from my side. (Reality is I’m a critical person) but I’m constantly working on this.

The only disappointing part is that we are having a lot of fights on:
– Which source of information is valid enough ?(Should we seek help only from social media, medical websites and videos or should we see a doctor/physician/family mentor?).
– She is constantly feeling that I’m blaming her or underestimating he work every time I’m trying to advise her.

Reem is very knowledgable person who has her own beliefs and sources of truth and reaching a solid decision on how we should raise Yahia must be  a couple decision.

I believe the only way to do it is to believe that we have to keep learning for our lifelong.

Hopefully, we -one day- will get it right for the love of Yahia and our small home.

 

This part of my life is called POC

It has been very long time since I posted anything here and here we go a quick list of updates:
1- Still enjoying my time in MC.
2- I’m doing huge changes in my life I’m learning new technologies, adding new habits to my life and applying a very agile methodology on my life development.
3- Yahia is consuming a huge part of my resources yet I can’t love him more (A completely new type of LOVE).

POC (proof of concept), If I can have a slogan for this period of my life I’d have the slogan POC everything.

It turned out that I’m talented when it comes to developing POCs and POCs aren’t only technical small software to prove some concept, it turned out that I can apply this on my personal life.

Recently, I was concerned about these questions How can I earn money while sleeping ?,  What skills and technologies can I add to my Arsenal ? and Is there a silver bullet for my life problems?

I believe in the sliver bullet concept (not in the silver bullet itself), That’s why I’m thinking of giving course(s) online on udemy.
Courses will be related to topics like: python, django, rails, node-js, angular-js and git.

I’ll make sure that my understanding for these topics isn’t only about using them but sharp enough to teach them.(I’ve done that before with PHP and MySQL), I’ll also make some money while sleeping, And yes, this is supposed to be a sliver bullet.

Guess what? It can have a POC I’ll start with very small course about ruby on rails and later on I’ll evaluate based on feedback.

Stay tuned.

3 Months of MC

It has been three months of MC.

It has been very exciting time that I almost contributed to all projects, made my own and even practiced the role of scrum master in a start up that strongly believe in Agile development. In the following lines I will briefly describe my journey.

1- Moustafa our CEO, the very first thing to notice about MC is Moustafat.

Moustafa is innovative, passionate Engineer who is taking a business very serious. He inspires everyone around him, motivates them and simple get the best out of everyone/everything.

He understands the business problem and the techenical challenges and this exactly what makes him a perfect CEO msA.

2- The team, MC has a great team of young yet talented group of people who share a lot of interest and background yet have different approaches to solving our problems. What I really like is how truly (code wins arguments) is happening in MC we regularly benchmark and POC the tools we use and that’s how we use to resolve conflicts that’s why even if we landed a bad choice it’s nothing attached to anyone it’s all a technical decision.

3-Advisors, MC has set of business and technical advisors all have very long experience in business and technology adding up their experience MC becomes a very old wise man as fast as Cristiano Ronaldo.

4- Environment, this one requires a separate post but briefly I can say that I just feel home  @ office that I go home (the actual one) late so many times. The other good thing is it’s constantly improving.

New office is ~ 6 miles from home and in a very cool yet active place I Maadi St 9.

That’s it for now.

Updates: left vision, joining MC and more..

life += Challenge.toOpportunity()

After business and ownership problems in Vision advanced systems the whole development team was aggressively and suddenly laid off and being the head of development I got laid off too.

A group of talented people managed to see that challenge as an opportunity to a new thing. This is how a new start-up called mussder (Arabic word مصدر means source) is something big in the making isA.

Last week only we managed to get our legal shit together, our stamp on our very first direct customer invoice paid and an almost full furnished office.

Career.append(“MENA Commerce”)

Starting Nov. 2015 I’ll be isA part of the great start-up MENA Commerce (MC).

Before landing this job I went through a very harsh selection process for different employees and even harsher problem selecting my next step I had the chance to get offers from almost all my favorite corporates and start-ups in Egypt and I had to develop and update a selection function based on which I had to go with MC offer.

The selection function isn’t new to me however I made huge change to it based on my experiment in vision and I’m very excited to see if it works or not.

Going down the chain

I decided to go down the chain in order to enhance my development skills and my next position is Senior Software Engineer.

Training is learning

After the rails course I released that I want to give courses more to sharpen my skills.