Author Archives: slior

ADR Flow: A Tool for Managing Architecture Decisions

With all the agile software development movement, the role of the architect has changed to some degree as well.

While exploring the full breadth of changes is an interesting (some might say tedious) discussion, one of the most prominent challenges I often face is finding an efficient way to preserve knowledge around design/architecture decisions that are made in a project.
It’s a challenge I faced when I joined a team already working on several different products, and me being the new guy (and the one who’s supposed to make design decisions!) – I had to spend considerable time researching, asking around, digging into code and old presentations, trying to figure out why the system was built the way it is. In some cases there was actually no clear answer.
It’s also a situation I expect people to face with any long-lived project/product development. Unless you maintain very good documentation of your design decisions,  it will be hard to trace back why things are built the way they are. I know I’ve faced similar situations in past projects.

Documenting decisions is tedious. Moreover, it’s often perceived as non-productive. It doesn’t add product value. But even that – convincing people to document design decisions – is the easy part in my experience. The harder task is to actually write something that is coherent, describes the context and the decision succinctly but completely.

One methodology I found, liked, and currently experimenting with, is that of Architecture Decision Records.
Refer to the link above for a complete overview, but in a nutshell: describe your architecture decision using a simple template, keep one file per decision, keep them with the source code, and setup a minimal process around these decisions, to be reflected in the files. The coupling to the code, and focus on specific decisions, is what I think will make this a more productive way to document our design decisions.

For me, at this point in time, this is still in “POC mode”. I’ve introduced it to my team, and currently pushing it, but we still need to see more value.

One thing I missed from the beginning is a way to to easily setup and manage ADRs. I found this open source project, which seems useful. But it’s based solely on linux shell scripts. Being mostly a windows user in my day-to-day work, I missed a good windows integration.

And thus the ADR Flow project was born.

The ADR Flow tool is meant to be a simple command line tool to aid in the process of creating and managing the ADR lifecycle. It’s meant to be simple enough so that it could be easily integrated into any workflow and/or standard tool.
It is not meant to hide the fact that ADRs are text files, or prevent editing them in whatever tools the user sees fit. Rather it tries to leverage some of the maintenance overhead, and make managing ADRs easy.

It’s a fun side project, also giving me the option to explore more possibilities for development with Node.js.

Comments, questions, feedback and of course contributions are more than welcome.

Code Writing from a UX Perspective

We often talk and discuss coding style and guidelines. Some of these discussions grow to be (some would say useless) flame wars, while others invoke thoughts and sometimes even reach a consensus, e.g. “GOTO considered harmful” (PDF). These discussions tend to be motivated by good intentions – making software development better; delivering software faster, with better quality and precision.

One area that keeps coming up is the need to have code written clearly; to make code easily understandable and changeable. It is generally agreed that code readability is important. And yet, the discussions are sometimes endless. We don’t seem to be able to agree what constitutes code that is written well enough, one that is easily understood. Claims made are often seen as very subjective and “a matter of taste”, as readability and code quality deals with human interpretation and judgement of said code.

At the same time, another field that deals with human-computer interaction seems to achieve better success in establishing at least some level of standard or agreed understanding of quality – software user experience (UX). Naturally, taste also plays a role in this case, but practitioners seem to agree more on some underlying practical principles to follow; similar to how coders have established some common understanding of good code design.

Which then begs the question – what if we approach code writing as a user experience problem, where the users are coders (incl. the code writer)?

Designers of applications’ user experience often deal with how the application is to be used. Characterizing its users and their main use cases. The answers to most of these questions are already known to us in the case of coders – we are the users. We write code, read it, test it, debug it. We know how we navigate code, skim through it, or how we approach refactorings.

Given that profiling the users is mostly out of the way, can we use this knowledge by taking a page from the UX designers’ playbook and applying it to code writing?
Can we design the code itself in such a way that will make our experience with it more productive and enjoyable, or dare I say delightful?

A naive approach would be to look at UI patterns and trying to apply them to code. While some may be somewhat useful (e.g. “inline hints”), it appears that the context of graphical UI design and its associated patterns does not immediately map to textual code writing. So while some insights can be had, this kind of mapping doesn’t seem to offer great value.

UX Principles in Code

But what if we go deeper?
Can we apply established principles of software-human interaction to code writing?

Take for example clarity. Good user interface designs are clear – at any point in the application they provide the user with an immediate understanding of what just happened, where is the user, what are the options available and their implications and so on. Think breadcrumbs, messages displayed, contextual disabling/enabling of buttons and so on.
What does this mean to code? Saying that “the code needs to be clear” is stating the obvious. Thinking about it from a user (=programmer) experience point of view we should be paying attention to the exact same things:

  • Is it clear to the user where he is from looking at the code?
  • Is the code conducive to understanding the options available at any given point?
  • Are implications of changes communicated clearly?

For example, consider a long file containing a lot of functionality, however related, a file with a lot of definitions, a class with a lot of methods and internal classes, nested callbacks, etc. These are examples of code that does not lend itself immediately to the structure and location in the code. It is harder to follow when reading and especially when debugging it; as debugging often involves jumping from file to file.
Similarly, having too many options to do one thing, e.g. overloaded functions, might introduce some confusion on what is the best way to do things. So there’s a clear tradeoff here between convenience (having the same method name, but with different parameters), and creating a confusing API.

Instead, keeping code listings shorter, tying code artifacts more closely to their location in the code base – e.g. one or two type(s) per file, can help with guiding the user of the code to understand what is the piece of code he is looking at. On the same note, providing one or two ways to do something, e.g. reading from a file or from stream, could actually make it easier for the caller of an API to understand what’s expected and how to work with that API.

Another principle to think about is familiarity and consistency. User interfaces, especially graphical ones, tend to prefer conventional solutions (hence patterns) for common tasks. Icons, links, color coding (red vs. green) are all examples of reusing familiar visual cues to convey meaning and being useful.

Familiarity can apply also to code writing. We can think of several places where this principle is applied:

  • Using common verbs, e.g. names of well known code design patterns
  • Naming getters and setters
  • Consistent naming for classes with similar roles
  • Well used parameter names – i for index, callback for callback functions, onXXX for event handlers
  • Packages following a similar scheme

Using well known patterns for organizing code, naming artifacts, etc. gives the programmer a better chance of understanding and navigating the code. At the project level it might make refactoring easier as it already suggests some code/module structure that can be followed, and may lead to more thoughtfulness with changes that break these familiar patterns.

Efficiency, of course, is also something to look into. Good user interfaces often emphasize the main scenarios supported by making them very efficient for the user. We count clicks, measure how much scrolling is needed, carefully design menus, etc. only to make sure that the common usage scenarios are easy to follow and accessible. In code this may translate to looking at the prevalent scenarios when coding. Readability is of course important, since we mostly read code. But another thing to consider is, for example, debugging and monitoring the running code. Do we insist, for example, on writing one operation per line, with variables defined for intermediate values; or do we cram together several access function calls making it harder to debug and trace the code?

Discouraging Bad Practices with Programmer’s User Experience

Another angle to look into the programmer’s user experience is to try and come with ways where the user experience actually encourages us to write better code.

An example for this is fibonacci indentation where each level of text indentation in the code is indented by a number of spaces matching the fibonacci number at that level. So deeper nesting levels are indented further away. At first, this may seem weird, but when you consider this, you quickly see how this discourages too much nesting – thus improving code readability and the overall experience when dealing with the code.

Another example might be to mandate no empty lines between a variable declaration and where the variable is used. This encourages declaring variables closer to where they are used, keeping related expressions closer visually; otherwise you end up with a big block of code – making it harder to follow.

How is this Useful?

These examples, and some of the conclusions presented here are nothing new to anyone considering the readability and clarity of code. Proper naming, for example, has always been in focus when considering code quality and readability. Still, it might be useful to frame the code readability/usability question as a user experience question, and borrow some practices from the field of user experience design. It helps to reason about better practices for structuring code.

Practically, this could be part of our code reviews, and configured/programmed into our already established set of coding tools – IDEs, linters, static code analysis tools, style guides, etc. The amount of tools available and polyglot nature of today’s software systems, where it’s not uncommon for a system to be coded in several languages, may in fact highlight the value of having a set of overarching principles and insights to guide us to better code writing. We should be able to define and follow principles that can be applied consistently and successfully in several languages at the same time.

One might argue, and justifiably so, that the job of making a programmer’s life easier is handled pretty well with the plethora of tools available, namely modern IDEs. But code can be made better (=easier to work with) by not relying on such mechanisms alone. Besides, not all editors are created equal, and code is viewed in tools other than the original IDE used to write it; e.g. code review tools, diff utilities. And of course, different people use different editors. So editors are very helpful, but I believe we can write clearer code if we don’t necessarily rely on that crutch.

Similarly, some languages also tend to lend themselves better to making code more readable. Type inference and lambda expressions help us make code more succinct and often more readable (though some would argue that explicit typing actually delivers a better experience). The user experience view of code writing is still useful in choosing how to use such language features, where choices are available.


To conclude, considering code writing from a user experience perspective might not be the most natural approach when programming. But if we agree that code readability and clarity is important – directly influencing quality and speed of development – then we should probably look at it from a proven perspective, using considerations and principles otherwise proven for software writing. A programmer’s “code experience” provides another perspective on how code is written and read and should probably be examined when making choices on how to write and structure code. Instead of debating on why a given code is written better this way or that, we can apply user experience guidelines to show us the path to a better code writing.

LiterateJS: Literate Programming with Javascript and HTML

Ever since I started to work on programming languages and models, I was intrigued by the problem of developer productivity. Always looking for effective ways to write and maintain code over time. One of the core issues I constantly struggle with when writing/looking at code is its readability. I generally maintain that the readability of the code has direct impact on its quality, but I digress.

There are all sorts of methodologies and tools to increase code readability, but one of the most interesting ones I came across is the idea of Literate Programming. The idea itself isn’t very new, and was conceived first by Donald Knuth in his book about that same subject.

I don’t claim to explain the idea better than the man who invented it (and I urge the reader to visit the links above to get a better understanding), but the rough idea is to write programs in the form of human-readable text, interspersed with program code. The program code is broken into snippets that are explained by the text. The resulting document is a fully functional (=”compilable”) program, using literate programming tools. It can be executed and it provides results, often embedded in the same document itself.

Being a fan of literature as well as programming, this struck me as a wonderful idea – you get to write your thoughts about what you’re trying to do, in a human readable way, and the code is right there, and it’s executable. There’s no separate documentation or anything – you explain the calculation, along with the code that actually executes it. Often times, the resulting document also contains the result of running the code, maybe even with graphs, images, etc.

There are of course limitations to this. And while I’ve no intention to start a complete debate on the validity of this approach, I will only say that I have my doubts on how this might work for developing a complex system, with several thousands/millions lines of code and a team of developers on it. While it’s theoretically possible to write such systems, I doubt any real benefit would be realized in such a scenario.

Still, for some scenarios, I believe this is a terrific method of writing software. Specifically, when trying to explain complex calculations, and making the calculations repeatable. This is why I think this kind of tools found its sweet spot in academia, where this use case is more prevalent. See iPython notebooks (python) and knitr (for R) as examples.


This is all fun and nice, but I’m not a data scientist, and I don’t delve too often into writing reports.

But then I came across a project at work, which required me to write a small tool to make some calculations, given a set of data. Besides the usual requirements for code quality, one of the prominent requirements was for users of this tool to easily understand the underlying code – the computation leading to the end results, along with intermediate results. And this struck a chord with me.

My first idea was to use R to write the code, both for its applicability as a language to the task and the availability of the knitr package mentioned above. But this was ruled out for reasons of skill set of the intended audience, and other developers who might to base their work on this tool (and also not wanting to install R/shiny to use it).

It was at this point that it came to me that even simple and readily available tools, namely a modern web browser with its javascript engine, are a great fit for the task. After all, today’s browser renders HTML pages with Javascript code running inside it all the time. So I have a presentation language (HTML), with a programming language (Javascript) and the engine to run it, all in one everyday tool – a browser. All that was left was to make it possible to write the tool that will marry both in a way that allows for the executable code to be part of the read document.

So I came up with LiterateJS.

LiterateJS was conceived originally as a proof of concept. I now evolved it to slightly more than that, but it’s no way near a mature project.

Still, it provides a demonstration and the basic facilities to write literate programming document using standard HTML and JS. ‘Standard’ being a key idea here – I intentionally try to avoid doing too much “magic”; instead trying to strike a balance between productivity and usefulness and remaining open to other tools and standards. The examples currently in the project use standard css (bootstrap) with another library for highlighting the code (highlight.js). Resulting in a standard HTML document that can be easily rendered, edited and debugged in standard tools. In this respect this isn’t a classic literate programming tool – there’s no explicit weaving step involved. The javascript code is simply extracted and with some help “injected” into the document, while preserving its highlighting and formatting. The resulting document is standard dynamic HTML document, with the code running in it.

LiterateJS is then a small framework that allows a developer to write literate programs in Javascript and HTML, and distribute them in any way he sees fit. It has a dependency on JQuery but that’s pretty much it (to be honest, the current code, at the time of writing, also depends on bootstrap css classes, but this dependency is planned to be removed).

LiterateJS is still very early in development, and I hope to have the time to continue developing it moving forward (any helpers?). Yet, I don’t expect it to grow too much, and aim to keep it a relatively small tool, with the possibility to add separate extensions later.

I encourage you to try it out, look through the examples and the code, suggest new ideas, features and of course bug fixes. Let me know if you find it useful, and program away!

The Understated Architect Role


So what’s in a software architect?

This is actually a question I’ve seen addressed a few times, and actually discussed during my career quite a bit. It’s a question I need to address if and when I search for new job as a software development architect. And it’s also a question that seems to have quite a few possible answers.

But eventually, there does some to be a common ground when trying to discuss the work of a software development architect. One could discuss the soft skills, the technical skills or the leadership skills required from an architect. It’s all true, at least to some degree. The exact combination of skills required to be a good software development architect varies between different organizations, operating contexts (the organization structure, company culture and size, history with the team, etc.) and your definition of “good”; so I won’t waste your time in trying to define an exact grocery list of skills.

I do believe, however, that one important role of the architect is often overlooked or not emphasized enough. And it’s probably not something you’ll learn in any course or training. It is something that I often felt during my work, but realized fully only when another (more experienced) architect has put it into words that resonated strongly with me.

Probably the most important job of an architect is to create and maintain a consistent understanding of the software system being developed. Or more precisely: creating a coherent and consistent picture of the developed system across all parties involved in the development project. To create an understanding between the development team and its stakeholders. To create an understanding within the team about the technical vision and direction for development. To create an understanding across teams on what is being developed and how systems interact.

This is probably the single most important job an architect has that is also unique to this role. Technical expertise is important, as is various design decisions and being able to weigh trade-offs properly. But the thing that truly separates the architect’s role from that of an expert software developer is the ability to convey a system structure and technical design, its capabilities, constraints and decisions taken when building it.

It’s really more than simply knowing the right words and using the right terminology. It’s about framing thoughts in a way that is consistent and understandable by the target audience. It’s about striking the balance between formalism and human comprehension. It’s about choosing the right methods to convey an idea. It’s about being precise and succinct, yet clear and understandable. It’s about being able to translate between different terminologies or “domains of thoughts”.

And herein lies the real challenge in being a software development architect, in my opinion. Technical mastery is important. But being able to convey an idea efficiently, to plant the right idea into the minds of people you’re communicating with is one thing you can’t learn on stackoverflow.com (which is a wonderful site, by the way).

And it’s a subtle act, often very delicate and hard to balance. Simply because people come from different schools of thoughts and experiences (and often with agendas). But choosing the right terminology, explaining it properly, making assumptions explicit or drawing the right diagram goes a long way towards aligning all involved parties on a single vision and a coherent mental picture of a system. Creating this mental picture in everyone’s mind is no easy task.

I had a boss who once told me that whenever there’s a disagreement on a technical direction to take, you should prefer to be the one drawing the diagrams. This simple fact already gives you an advantage. If you’re the one holding the marker pen, you already have an edge.

This is true from an organizational politics point of view, but also an important point to keep in mind when you want to reach an agreement and create this common understanding and consistency – if you wield the pen, you wield the power as well as the responsibility. And when you’re an architect, your job is to wield the pen.

When people talk about being an architect, they usually talk about what to do with the pen – what kind of diagrams to draw, what documents to write, how much code, etc. But there’s an overarching, implicit goal that is assigned to the one holding the pen – to make sure everyone is aligned and have a consistent mental picture in their heads, when approaching their individual jobs. This is especially harder in the software business, where there’s no shortage of abstractions and intangible concepts to keep in mind. Where the different levels of abstraction and complexity a developer has to keep in his head are often mind boggling. The ability to synthesize the important ideas and communicate them at the right place, at the right time, in a manner that will be adopted and accepted is quite a fit. Take someone who is good at doing that, add a decent technical and analytical abilities into the mix and you’ve got yourself a good software architect right there.

 

Effective System Modeling

This post is a rough “transcript” (with some changes and creative freedom) of a session I gave in the Citi Innovation Lab, TLV about how to effectively model a system.

A Communication Breakdown?

Building complex software systems is not an easy task, for a lot of reasons. All kinds of solutions have been invented to tackle the different issues. We have higher level programming languages, DB tools, agile project management methodologies and quite a bit more. One could argue that these problems still exist, and no complete solution has been found so far. That may be true, but in this post, I’d like to discuss a different problem in this context: communicating our designs.

One problem that seems to be overlooked or not addressed well enough, is the issue of communicating our designs and system architecture. By ourselves, experienced engineers are (usually) quite capable of coming up with often elegant solutions to complex problems. But the realities and dynamics of a software development organization, especially a geographically distributed one, often require us to communicate and reason about systems developed by others.

We – software engineers – tend to focus on solving the technical issues or designing the systems we’re building. This often leads to forgetting that software development, especially in the enterprise, is often, if not always, a team effort. Communicating our designs is therefore critical to our success, but is often viewed as a negligible activity at best, if not a complete waste of time.

The agile development movement, in all its variants, has done some good to bring the issues of cooperation and communication into the limelight. Still, I often find that communication of technical details – structure and behavior of systems, is poorly done.

Why is that?

“Doing” Architecture

A common interpretation of agile development methods I often encounter tends to spill the baby with the water. I hear about people/teams refusing to do “big up-front design”. That in itself is actually a good thing in my opinion. The problem starts when this translates to no design at all, and this immediately translates into not wanting to spend time on documenting your architecture properly, or how it’s communicated.

But as anyone who’s been in this industry for more than a day knows – there’s no replacement for thinking about your design and your system, and agile doesn’t mean we shouldn’t design our system. So I claim that the problem isn’t really with designing per-se, but rather in the motivation and methodology we use for “doing” our architecture – how we go about designing the system and conveying our thoughts. Most of us acknowledge the importance of thinking about a system, but we do not invest the time in preserving that knowledge and discussion. Communicating a design or system architecture, especially in written form, is often viewed as superfluous, given the working code and its accompanying tests. From my experience this is often the case because the actual communication and documentation of a design are done ineffectively.

This was also strengthened after hearing Simon Brown talk about a similar subject, one which resonated with me. An architecture document/artifact should have “just enough” up front design to understand the system and create a shared vision. An architecture document should augment the code, not repeat it; it should describe what the code doesn’t already describe. In other words – don’t document the code, but rather look for the added value. A good architecture/design document adds value to the project team by articulating the vision on which all team members need to align on. Of course, this is less apparent in small teams than in large ones, especially teams that need to cooperate on a larger project.

As a side note I would like to suggest that besides creating a shared understanding and vision, an architecture document also helps in preserving the knowledge and ramping-up people onto the team. I believe that anyone who has tried learning a new system just by looking at its code will empathize with this.

Since I believe the motivation to actually design the system and solve the problem is definitely there, I’m left with the feeling that people often view the task of documenting it and communicating it as unnecessary “bureaucracy”.
We therefore need a way to communicate and document our system’s architecture effectively. A way that will allow us to transfer knowledge, over time and space (geographies), but still do it efficiently – both for the writer and readers.
It needs to be a way that captures the essence of the system, without drowning the reader in details, or burden the writer with work that will prove to be a waste of time. Looking at it from a system analysis point of view, then reading the document is quite possibly the more prominent use case, compared to writing it; i.e. the document is going to be read a lot more than written/modified.

When we come to the question of modeling a system, with the purpose of the end result being readable by humans, we need to balance the amount of formalism we apply to the model. A rigorous modeling technique will probably result in a more accurate model, but not necessarily an easily understandable one. Rigorous documents tend to be complete and accurate, but exhausting to read and follow; thereby beating the purpose we’re trying to achieve. At the other end of the scale are free text documents, often in English and sometimes with some scribbled diagrams, which explain the structure or behavior of system, often inconsistently. These are hard to follow for different reasons: inaccurate language, inconsistent terminology and/or ad-hoc (=unfamiliar) modeling technique used.

Providing an easy to follow system description, and doing so efficiently, requires us to balance these two ends. We need to have a “just enough” formalism that provides a common language. It needs to be intuitive to write and read, with enough freedom to provide any details needed to get a complete picture, but without burdening the writers and readers with unnecessary details.
In this post, I try to give an overview and pointers to a method I found useful in the past (not my invention), and that I believe answers the criteria mentioned above. It is definitely not the only way and may not suit everyone’s taste (e.g. Simon Brown suggests something similar but slightly different); but regardless of the method used, creating a shared vision, and putting it to writing is something useful, when done effectively.

System != Software

Before going into the technicalities of describing a system effectively, I believe we need to make the distinction between a system and its software.

For the purposes of our discussion, we’ll define software as a computer-understandable description of a dynamic system; i.e. one way to code the structure and behavior of a system in a way that’s understandable by computers.
A (dynamic) system on the other hand is what emerges from the execution of software.

To understand the distinction, an analogy might help: consider the task of understanding the issue of global warming (the system) vs. understanding the structure of a book about global warming (the software).

  • Understanding the book structure does not imply understanding global warming. Similarly, understanding the software structure doesn’t imply understanding the system.
  • The book can be written in different languages, but it’s still describing global warming. Similarly, software can be implemented using different languages and tools/technologies, but it doesn’t (shouldn’t) change the emergent behavior of the system.
  • Reading the content of the book implies understanding global warming. Similarly, the system is what emerges from execution of the software.

One point we need to keep in mind, and where this analogy breaks, is that understanding a book’s structure is considerably easier than understanding the software written for a given system.
So usually, when confronted with the need to document our system, we tend to focus on documenting the software, not the system. This leads to ineffective documentation/modeling (we’re documenting the wrong thing), eventually leading to frustration and missing knowledge.
This is further compounded by the fact that existing tools and frameworks for documentation of software (e.g. UML) tend to be complex and detailed, and with the tools emphasizing code generation, and not human communication; this is especially true for UML.

Modeling a System

When we model an existing system, or design a new one, we find several methods and tools that help us. A lot of these methods define all sorts of views of the system – describing different facets of its implementation. Most practitioners have surely met one or more different “types” of system views: logical, conceptual, deployment, implementation, high level, behavior, etc. These all provide some kind of information as to how the system is built, but there’s not a lot of clarity on the differences or roles of each such view. These are essentially different abstractions or facets of the given system being modeled. While any such abstraction can be justified in itself, it is the combination of these that produces an often unreadable end result.

So, as with any other type of technical document you write, the first rule of thumb is:

Rule of thumb #1: Tailor the content to the reader(s), and be explicit about it.

In other words – set expectations. Set the expectation early on – what you’re describing and what is the expected knowledge (and usually technical competency) of the reader.

Generally, in my experience, 3 main facets are the most important ones: the structure of the system – how it’s built, the behavior of the system – how the different component interact on given inputs/events, and the domain model used in the system. Each of these facets can be described in more or less detail, at different abstraction levels, and using different techniques, depending on the case. But these are usually the most important facets for a reader to understand the system and approach the code design itself, or reading the code.

Technical Architecture Modeling

One method I often find useful is that of Technical Architecture Modeling (TAM), itself a derivative of Fundamental Modeling Concepts (FMC). It is a formal method, but one which focuses on human comprehension. As such, it borrows from UML and FMC, to provide a level of formalism which seems to strike a good balance between readability and modeling efficiency. TAM uses a few diagram types, where the most useful are the component/block diagram used to depict a system’s structure or composition; the activity and sequence diagrams used to model a system/component’s behavior and the class diagram used to model a domain (value) model. In addition, other diagram types are also included, e.g. state charts and deployment
diagrams; but these are less useful in my experience. In addition, TAM also has some tool support in the form of Visio stencils that make it easier to integrate this into other documentation methods.

I briefly discuss how the most important facets of a system can be modeled with TAM, but the reader is encouraged to follow the links given above (or ask me) for further information and details.

Block Diagram: System Structure

A system’s structure, or composition, is described using a simple block diagram. At its simplest form, this diagram describes the different components that make up the system.
For example, describing a simple travel agency system, with a reservation and information system can look something like this (example taken from the FMC introduction):

Sample: Travel Agency System

This in itself already tells us some of the story: there’s a travel agency system, accessed by customers and other interested parties, with two subsystems: a reservation system and an information help desk system. The information is read and written to two separate data stores holding the customer data and reservations in one store, and the travel information (e.g. flight and hotel information) in the other. This data is fed into the system by external travel-related organizations (e.g. airlines, hotel chains), and reservations are forwarded to the same external systems.

This description is usually enough to provide at least a contextual high level information of the system. But the diagram above already tells us a bit more. It provides us some information about the access points to the data; about the different kinds of data flowing in the system, and what component is interacting with what other component (who knows who). Note that there is little to no technical information at this point.

The modeling language itself is pretty straightforward and simple as well: we have two main “entities”: actors and data stores.
Actors, designated by square rectangles, are any components that do something in the system (also humans). They are they active components of the system. Actors communicate with other actors through channels (lines with small circles on them), and the read/write from/to data stores (simple lines with arrow heads). Examples include services, functions and human operators of the system.
Data store, designated by round rectangles (/circles), are passive components. These are “places” where data is stored. Examples include database systems, files, and even memory arrays (or generally any data structure).

Armed with these definitions, we can already identify some useful patterns, and how to model them:

Read only access – actor A can only read from data store S:
Read only access

 

Write only access – actor A can only write to data store S:
Write only access

 

Read/Write access:
Read/Write access

 

Two actors communicating on a request/response channel have their own unique symbol:
effective-system-modeling-004
In this case, actor ‘B’ requests something from actor ‘A’ (the arrow on the ‘R’ symbol points to  ‘A’), and ‘A’ answers back with data. So data flow actually happens in both ways. A classical example of this is a client browser asking for a web page from a web server.

 

A simple communication over a shared storage:
effective-system-modeling-005
actors ‘A’ and ‘B’ both read and write from/to data store ‘S’. Effectively communicating over it.

 

There’s a bit more to this formalism, which you can explore in FMC/TAM website. But not really much more than what’s shown here. These simple primitives already provide a powerful expression mechanism to convey most of the ideas we need to communicate over our system on a daily basis.

Usually, when providing such a diagram, it’s good practice to accompany it with some text that provides some explanation on the different components and their roles. This shouldn’t be more than 1-2 paragraphs, but actually depends on the level of detail and system size.

This would generally help with two things: identifying redundant components, and describing the responsibility of each component clearly. Think of this text explanation as a way to validate your modeling, as displayed in the diagram.

Rule of thumb #2: If your explanation doesn’t include all the actors/stores depicted in the
diagram – you probably have redundant components.

Behavior Modeling

The dynamic behavior of a system is of course no less important than its structure. The cooperation, interaction and data flow between components allow us to identify failure points, bottlenecks, decoupling problems etc. In this case, TAM adopts largely the UML practice of using sequence diagrams or activity diagrams, whose description is beyond the scope of this post.

One thing to keep in mind though, is that when modeling behavior in this case, you’re usually not modeling interaction between classes, but rather between components. So the formalism of “messages” sent between objects need not couple itself to code structure and class/method names. Remember: you generally don’t model the software (code), but rather system components. So you don’t need to model the exact method calls and object instances, as is generally the case with UML models.

One good way to validate the model at this point is to verify that the components mentioned in the activity diagram are mentioned in the system’s structure (in the block diagram); and that components that interact in the behavioral model actually have this interaction expressed in the structural model. A missing interaction (e.g. channel) in the structural view may mean that these two components have an interface that wasn’t expressed in the structural model, i.e. the structure diagram should be fixed; or it could mean that these two components shouldn’t interact, i.e. the behavioral model needs to be fixed.

This is the exact thought process that this modeling helps to achieve – modeling two different facets of the system and validating one with the other in iterations allows us to reason and validate our understanding of the system. The explicit diagrams are simply the visual method that helps us to visualize and capture those ideas efficiently. Of course, keep in mind that you validate the model at the appropriate level of abstraction – don’t validate a high level system structure with a sequence diagram describing implementation classes.

Rule of thumb #3: Every interaction modeled in the behavioral model (activity/sequence
diagrams) should be reflected in the structural model (block diagram), and vice versa.

Domain Modeling

Another often useful aspect of modeling a system is modeling the data processed by the system. It helps to reason about the algorithms, expected load and eventually the structure of the code. This is often the part that’s not covered by well known patterns and needs to be carefully tuned per application. It also helps in creating a shared vocabulary and terminology when discussing different aspects of the developed software.

A useful method in the case of domain modeling is UML class diagrams, which TAM also adopts. In this case as well, I often find a more scaled-down version the most useful, usually focused on the main entities, and their relationships (including cardinality). The useful notation of class diagrams can be leveraged to express these relationships quite succinctly.

Explicit modeling of the code itself is rarely useful in my opinion – the code will probably be refactored way faster than a model will be updated, and a reader who is able to read a detailed class diagram can also read the code it describes. One exception to this rule might be when your application deals with code constructs, in which case the code constructs themselves (e.g. interfaces) serve as the API to your system, and clients will need to write code that integrates with it, as a primary usage pattern of the system. An example for this is an extensible library of any sort (eclipse plugins are one prominent example, but there are more).

Another useful modeling facet in this context is to model the main concepts handled in the system. This is especially useful in very technical systems (oriented at developers), that introduce several new concepts, e.g. frameworks. In this case, a conceptual model can prove to be useful for establishing a shared understanding and terminology for anyone discussing the system.

Iterative Refinement

Of course, at the end of the day, we need to remember that modeling a system in fact reflects a thought process we have when designing the system. The end product, in the form a document (or set of documents) represents our understanding of the system – its structure and behavior. But this is never a one-way process. It is almost always an iterative process that reflects our evolving understanding of the system.

So modeling a specific facet of the system should not be seen as a one-off activity. We often follow a dynamic where we model the structure of the system, but then try to model its behavior, only to realize the structure isn’t sufficient or leads to a suboptimal flow. This back and forth is actually a good thing – it helps us to solidify our understanding and converge on a widely understood and accepted picture of how the system should look, and how it should be constructed.

Refinements also happen on the axis of abstractions. Moving from a high level to a lower level of abstraction, we can provide more details on the system. We can refine as much as we find useful, up to the level of modeling the code (which, as stated above, is rarely useful in my opinion). Also when working on the details of a given view, it’s common to find improvement points and issues in the higher level description. So iterations can happen here as well.

As an example, consider the imaginary travel agency example quoted above. One possible refinement of the structural view could be something like this (also taken from the site above):

Example: travel agency system refined

In this case, more detail is provided on the implementation of the information help subsystem and the ‘Travel Information’ data store. Although providing some more (useful) technical details, this is still a block diagram, describing the structure of the system. This level of detail refines the high level view shown earlier, and already provides more information and insight into how the system is built. For example, how the data stores are implemented and accessed, the way data is adapted and propagated in the system. The acute reader will note that the ‘Reservation System’ subsystem now interacts with the ‘HTTP Server’ component in the ‘Information help desk’ subsystem. This makes sense from a logical point of view – the reservation system accesses the travel information through the same channels used to provide information to other actors, but this information was missing from the first diagram (no channel between the two components).
One important rule of thumb is that as you go down the levels of abstraction, keep the names of actors presented in the higher level of abstraction. This allows readers to correlate the views more easily, identify the different actors, and reason about their place in the system. It provides a context for the more fine granular details. As the example above shows, the more detailed diagram still includes the actor and store names from the higher level diagram (‘Travel Information’, ‘Information help desk’, ‘Travel Agency’).

Rule of thumb #4: Be consistent about names when moving between different levels of abstraction. Enable correlations between the different views.

Communicating w/ Humans – Visualization is Key

With all this modeling activity going on, we have to keep in mind that our main goal, besides good design, is communicating this design to other humans, not machines. This is why, reluctant as we are to admit it (engineers…) – aesthetics matter.

In the context of enterprise systems, communicating the design effectively is as important to the quality of the resulting software as designing it properly. In some cases, it might be even more important – just consider the amount of time you sometime spend on integration of system vs. how much time you spend writing the software itself. So a good looking diagram is important, and we should be mindful about how we present it to the intended audience.

Following are some tips and pointers on what to look for when considering this aspect of communicating our designs. This is by no means an exhaustive list, but more based on experience (and some common sense). More pointers can be found in the links above, specifically in the visualization guide.

First, keep in mind node and visual arrangement of nodes and edges in your diagram immediately lends itself to how clear the diagram is to readers. Try to minimize intersection of edges, and align edges on horizontal and vertical axes.
Compare these two examples:

Aligning vertices

The arrangement on the left is definitely clearer than the one on the right. Note that generally speaking, the size of a node does not imply any specific meaning; it is just a visual convenience.

Similarly, this example:

Visual alignment

shows how the re-arrangement of nodes allows for less intersection, without losing any meaning.

Colors can also be very useful in this case. One can use colors to help distinguish between different levels of containment:

Using colors

In this case, the usage of colors helps to distinguish an otherwise confusing structure. Keep in mind that readers might want to print the document you create on a black and white printer (and color blind) – so use high contrast colors where possible.

Label styles are generally not very useful to convey meaning. Try to stick to a very specific font and be consistent with it. An exception might be a label that pertains to a different aspect, e.g. configuration files or code locations, which might be more easily distinguished when using a different font style.

Visuals have Semantics

One useful way to leverage colors and layout of a diagram is to stress specific semantics you might want to convey in your diagram. One might leverage colors to distinguish a set of components from other components, e.g. highlighting team responsibilities, or highlight specific implementation details. Note that when you use this kind of technique that it is not standard, so remember to include an explanation – a legend – of what the different colors mean. Also, too many colors might cause more clutter, eventually beating the purpose of clarity.

Another useful technique is to use layout of the nodes in the graph for conveying an understanding. For example, depicting the main data flow might be hinted in the block diagram by layouting the nodes from left to right, or top to down. This is not required, nor carries any specific meaning. But it is often useful to use, and provides hints as to how the system actually works.

Summary

As we’ve seen, “doing” architecture, while often perceived as a cumbersome and unnecessary activity isn’t hard to do when done effectively. We need to keep in mind the focus of this activity: communicating our designs and reasoning about them over longer periods of time.

Easing the collaboration around design is not just an issue of knowledge sharing (though that’s important as well), but it is a necessity when trying to build software across global teams, over long periods of time. How effectively we communicate our designs directly impacts how we collaborate, the quality of produced software, how we evolve it over time, and eventually the bottom line of deliveries.

I hope this (rather long) post has served to shed some light on the subject, and provide some insight, useful tips and encouraged people to invest some efforts into learning further.


Credit: The example and images presented in this post are taken from the FMC website and examples.

Pattern Matching in Java?

Writing some code at home, in Java, I came across the problem of trying to compute a value based on the data of another value. This isn’t nothing exciting in on itself, but this whole project I was writing was about trying to write in functional style, as much as possible, in Java 8.
Also, being a fan of fluent interfaces, or readable code in general, this was one of the cases I really missed Scala’s pattern matching capabilities.

So I set to the task of writing a small utility that will allow me to emulate Scala’s pattern matching syntax and behavior, but in Java, at least to some extent.

The end goal I wanted to achieve was to write something similar to:

(you can probably guess the project was something about chess)

In this case p was the parameter passed to a function, which consisted of this call alone. An equivalent piece of code would be to test for the class of p – a series of if-elses, or store this mapping in a map. These are all valid solutions, but I wanted something that would be both concise, and didn’t require extra data preparation. Also something that would be more generally usable than in this case. Besides, I wanted to invent some new way to do this stuff, have some fun doing it.

So I ended up with writing this small utility, which allows for this kind of code:

This code implements a “micro-dsl” of sorts for pattern matching on the data.
The usage of the builder pattern, with 3 internal classes, is to allow for the fluent interface, which was ultimately my goal: enable easy to follow syntax for this kind of operation.

Note that the code makes use of Java 8’s functional interfaces of Predicate and Function, and of course the lambda expression syntax. This allows both for flexibility (you can write any expression you want) as well as concise code.

You can even use it recursively, as the following simple implementation of factorial shows:

It’s not perfect, as it does still tend to be verbose, and not nearly as neat as Scala’s pattern matching, but I believe it does allow for cleaner code, and improves the situation quite a bit for this type of tasks. Note also that this code is pretty simple and self contained – it doesn’t rely on any external library, besides the standard JDK (I did use some trivial utility functions there, e.g. requireNotNull but that’s really beside the point).

Some points where I see this can improve are:

  1. The ability to provide a default match (an “otherwise” case) that will handle all unmatched cases
  2. The ability to provide the data to match as a lazily evaluated function, to allow for optimization (parallel evaluation of the cases?)
  3. A “vectorized” version of this, allowing to specify a pattern match over a complete stream of values, resulting in another stream of values

I will probably do these when I have some more time and the need arises. In the meantime, keeping it simple.

Full source, with updates, is available here.