Microservices Fad or Trend?

When talking to clients and the architecture community the term microservices is gaining traction and attention. So, what are microservices?

“[…] self-contained piece of business functionality with clear interfaces, and may, through its own internal components, implement a layered architecture” (https://en.wikipedia.org/wiki/Microservices)

The definition above is theoretical and leaves the field open for interpretation on how develop an architecture based on microservices. However, this kind of approach is not new if you have been working with IAF (Integrated Architecture Framework).

Many architecture frameworks take a top down approach when creating the architecture. They start with a high level approach and decompose down to the level required to document what the architecture consists of, much like in the figure below.

The main drawbacks of this approach are:

1. It’s not that easy to define and analyse solution alternatives, as thinking is guided and influenced by the hierarchical structure that is being created and the political arena in the company.

2. The structure makes it harder to think out of the box, since each process leads to stow pipes and no room for shared services.

3. The architect is implicitly combining the ‘what’ and the ‘how’ question, because the definition of the structure along with the definition of the requirements. Making it difficult to isolate the service without running into a redundancy issue with multiple services that do the same thing but are separated by the process. For example Purchasing has the service “review claim” and Sales process also has a “review claim” service. Here there is a risk of creating redundant services in a large and complex organisation.

IAF takes a different approach.

• Firstly, we define the services at the level of detail required without explicitly putting them into a hierarchical structure.

• Then we define grouping criteria that are based on the architecture principles.

• After that we create components by grouping the services into components based on the grouping criteria. The figure below visualizes the IAF approach.

OK – Claudio this sound theoretical can you give me a practical example? Sure: read Casimirs blog http://disruptivearchitecture.info/blog/2019/12/16/ea-case-study-top-down-or-bottom-up that describes a hands-on example of the topic discussed above.

EA case study - top-down or bottom-up?

If we follow the traditional way of software development we do requirements (waterfall or agile) then development in iterations.

With architecture, using IAF, after contextual layer, you start to define IS-services, then logical components and finally select physical applications.

But what about existing systems, standard applications and software as a service in the cloud?

Main application landscape for the film production company with both SaaS and applications run locally

We already have a lot of legacy even if we re-started everything from scratch a few years back.

Good luck as a small conpany with trying to influence Apple, Avid or Adobe on their roadmap for their NLE’s.

Finacials and HR are run as SaaS in the cloud (Visma eEkonomi and Tid), with regular updates and customized to a Swedish environment. The benefit for a detailed EA is not very high. However, as this application is the master for project id’s, we may need to integrate to other services.

The question is therefore why Enterprise Application architecture today?

Based on the situation as today for the film produktion company, these are the reasons why we need a lightweight application architecture.

• Integrations between SaaS and applications to increase throughput, e.g. faster workflow.

• Identify reasons for new SaaS or new applications and support product evaluation

• Support development of a very few number of business unique software services, e.g. micro-services architecture.

After Christmas break will I come back to describe how this lightweight application architecture will look. Hint, neither strictly top- down or bottom-up. Instead, little of both.

For you who visited the updated homepage for the film production company, you will se that we doesn’t talk about EA all.

Why is very simple. The audience for this site doesn’t care about EA, they care about film production.

EA case study - Scenarios for infrastructure

Logical scenarios is a core part of IAF and used to evaluate possible ways of solving several needs together.

Film production and post-processing process is a bit special when it comes to demands on infrastructure, thus this article.

Possible locations for post-production that the process and infrastructure need to support

We have these loosely defined requirements from the business side.

• One production creates, as average, between 100 Gb to 1000 Gb of media assets per day

• Bandwidth per editor for disk I/O is 250-500 MB/s. Using proxies 100 MB/s.

• Relatively few number of users as post-production is highly computer intensive.

• Professional NLE’s are not cloud ready yet. This is a very conservative business.

• Several editors need access to same media at the same time.

• With the type of productions we do, only one editor at a time per timeline / sequence.

• The media is either classified as confidential or secret.

• The value of one days media is between € 2500 to € 25000 for small to medium size. productions and repeatable shots. Value in larger productions and events can be much higher.

• Cost is a factor as nobody want to pay extra for infrastructure.

• All this media need to be managed as quick as possible so that the director can adjust next scene/shot/take.

• We have several offices around the county and we also need to be able to edit in a mobile production office whetever we are.

Logical scenarios

The logical scenarios we identified are:

1. Cloud only

2. Cloud plus local storage

3. On-Premise plus local storage

4. Local storage only

Based on this we start doing a high level evaluation. Normally as a SWOT, but in this case only in text.

Cloud only sounds good and is according to our strategy, but.

Storage cost with Amazon AWS, € 250 / TB / year is reasonable, but when talking about 100 TB of storage in a very near future will the cost be € 25,000 instead.

In addition to storage, you pay for bandwidth to download content, e.g. € 100 per TB. For 250 TB per month will this be an additional € 25,000 per year.

You need at least 1 Gbps internet connection to edit proxis and the cost is € 1000 per year. For other edits you need 10 Gbps connection if they are available. In addition, you need a reliable VPN that adds to the cost.

The limitations in bandwidth and cost for cloud storage makes cloud solutions to expensive right now. In five years time, it may be a different story.

On-premise storage plus local storage

A NAS for storing media costs with RAID6 solution and 18 disks of 10 TB each is a cost of € 12,500. With this, you get 10 GB/s local connection. A fast dedicated media server is twice this cost.

The cloud solution is 12x the cost during three years compared to on-premise storage if you don’t measure installation and maintenance cost. This is € 150,000 for cloud compared to € 12,500 for a local NAS.

Local storage is then € 1000 per seat. We also need a reliable VPN with a bandwidth of 1 Gb so editors in local offices can copy the material to their workstations.

Local storage works only as long as the the editors doesn’t share media on a regular basis or as remaining option if bandwidth is too low at a mobile production office.

Informations System Services

Production repository is where we store the timeline for the edit, before NLE’s this was the EDL, edit decision list. Not huge file sizes, updated often and shared between editors. Cloud is the preferred option.

Review repository is where review copies are shared between editors and others, including customers. This is short time storage and not huge volumes. This is why cloud solution is preferred.

Master repository are the delivered versions of the film from post-production. The size of them is much smaller than the actual size of the full media production. The masters will be shared to more parties than the actual editors and therefore a cloud solution is possible, even preferred.

Technical Infrastructure Services

VPN for assuring private communication.

Shared filesystem with SMB and/or NFS is needed for all professional NLE’s.

Backup is mandatory but the question is how to setup this in the best way.

• Local backup

• Local backup and cloud backup

• Cloud backup

Local backup only is not recommended due to risks with theft, fire etc. The benefit with local backup is time to recover, compared to cloud, plus less dependency of external service provider.

Archive Unless implementing an long term archive solution, we need to increase our storage capacity by 50-100 TB per year. This is not sustainable in the long run.

EA case study - Tell me a secret

All three examples are more or less real and good examples of why we need to manage information security and privacy regulations.

• XXXXXX describes his/her mental illness in an interview for a documentary

• XXXXXX and YYYYYY are fully nude in some shots done for a non X-rated feature film

• XXXXXX, a Hollywood star, participates in a commercial for a new product line.

Information classification

We start to define different levels of security within the company and with partners, customers, contractors etc. Those four levels are:

• Public

• Internal

• Confidential

• Secret

The next type of levels are about privacy and derived from GDPR.

• Non personal information

• Personal, but not identifiable information

• Personal identifiable information

• Personal sensitive identifiable information

In addition to these generic classifications for security do we have NDA’s and other terms in agreements to manage.

Ownership and responsibility

The ownership of the classification of the information objects belongs to the process owner for each capability.

The more difficult question is which level of classification each information object should have and if there are cases where we need more stringent classification.

Public is simple. Just assure that you can track who publish what and when externally.

Internal is information shared internally in the company and with key partners, contractors and clients.

Confidential is information shared within a specific group for a specific purpose. Need to be shared both internally and externally.

Secret is information shared with specific persons and is more sensitive than confidential. Need to be shared both internally and externally.

Implementation of information policy

How enforce security and privacy is another matter and we continue with the example.

Different types of media assets (video, photos, sound etc) belongs to the capability Production and the ownership and responsibility is therefore very clear.

Media assets should not be classified as public or internal as it could belong to different clients or key partners. This is why we need to have a higher classification.

The higher security classification the media assets have, the more time consuming and more expensive will it be to manage them.

Normally, confidential will be enough, but in these three cases do we have more sensitive information as it includes sensitive personal information or information under NDA’s with client.

1. Health information for an individual is sensitive personal identifiable information according to GDPR.

2. Nudity is very private, and even if nothing is shown in the released feature in cinemas, we still have revealing clips stored in the footage.

3. There are NDA’s in place and we are not allowed to reveal the participants until client allows this. I.e. after publishing and not before.

This is the reason why I recommend all three cases to be classified as secret information instead of confidential.

The next question is now how to implement this with state of art technology, while still be usable in real life and with a reasonable cost. It’s now time to have a look on this from a enterprise content management perspective.

EA case study - Architecture or not?

What is within the scope of Enterprise Architecture? Let us continue with an example from our project list.

Lack of needed contracts is paramount in majority of productions and we need to manage this area to be on the safe side.

Binder with copyright agreements

The situation now are as:

• We have legal obligation to keep copyright agreements for at least 70 years.

• We have a few architecture principles that guides development of our solutions.

• We have a defined business process in our Production Handbook that require those agreements to be in place before start shooting.

• The agreements includes personal identifiable information as defined in GDPR.

• Agreements are classified as company confidential. If we one day get a Star Wars contract will it be classified as “company secret”.

• The different types of agreements are implemented as Word templates. Unsigned contracts are stored in teams for each production. Each individual contract is printed in two copies and signed by both parties. We also have blank contract on paper with us as a backup if there are late changes in plans.

• The signed document is scanned and stored in Microsoft Teams per production. The signed original is kept in a paper archive, one tab per production, until copyright ends.

• The production code and name, e.g. “10117 Saving Mimosa”, is managed in the financial systems.

• The production code and name, e.g. “10117 Saving Mimosa”, is managed in the media management systems.

• The list of which person is part of which production need to solved by another project as part of pre-production process.

What of above parts are Enterprise Architecture for you? Please explain why and why not.