Author Archives: slior

Building a Simple Game – Part 2

So after a short introduction to the project, let’s dive into some commits.

Our story starts with this commit, where we set the basis for drawing the board.

A lot of the files there are boiler plate (webpack config, package.json). The interesting parts are the index.js and drawing.js files:

	function initCanvas(canvasEl)
	{
	let ret = canvasEl && (new fabric.Canvas(canvasEl));
	if (ret)
	{
	ret.line = (x1,y1,x2,y2,c) => drawLineOn(ret,x1,y1,x2,y2,c);
	}
	return ret;
	}

	function drawBoard(cnvs)
	{
	let CELL_SIZE = 50;
	let CELL_COUNT = 14;

	let TOP_LEFT = { x : 50, y : 50};

	let playerCellCount = CELL_COUNT/2-1;
	let boardWidthInCells = playerCellCount+2 //for the side cells (player homes)
	let boardHeightInCells = 3;

	//frame
	horizLine(TOP_LEFT.x,TOP_LEFT.y,boardWidthInCells * CELL_SIZE); //top left to top right
	verticalLine(TOP_LEFT.x,TOP_LEFT.y,CELL_SIZE*boardHeightInCells); //top left to bottom left
	horizLine(TOP_LEFT.x,TOP_LEFT.y + CELL_SIZEboardHeightInCells,boardWidthInCells CELL_SIZE); // bottom left to bottom right
	verticalLine(TOP_LEFT.x + boardWidthInCells * CELL_SIZE,TOP_LEFT.y,CELL_SIZE * boardHeightInCells); //top right to bottom right

	//home cells
	verticalLine(TOP_LEFT.x + CELL_SIZE,TOP_LEFT.y,CELL_SIZE*boardHeightInCells)
	verticalLine(TOP_LEFT.x + CELL_SIZE(boardWidthInCells-1),TOP_LEFT.y,CELL_SIZEboardHeightInCells)

	//cell horizontal lines
	let upperCellY = TOP_LEFT.y + CELL_SIZE;
	let lowerCellY = TOP_LEFT.y + CELL_SIZE*2;
	let lineLen = (boardWidthInCells-2)*CELL_SIZE;

	horizLine(TOP_LEFT.x + CELL_SIZE,upperCellY,lineLen)
	horizLine(TOP_LEFT.x + CELL_SIZE,lowerCellY,lineLen)

	//cell borders
	range(2,CELL_COUNT/2).map(cellNum => {
	verticalLine(TOP_LEFT.x + cellNum*CELL_SIZE,TOP_LEFT.y,CELL_SIZE)
	verticalLine(TOP_LEFT.x + cellNumCELL_SIZE,TOP_LEFT.y+CELL_SIZE(boardHeightInCells-1),CELL_SIZE)
	} )

	function verticalLine(x,y,len) { cnvs.line(x,y,x,y+len); }

	function horizLine(x,y,len) { cnvs.line(x,y,x+len,y); }

view raw drawing.js hosted with ❤ by GitHub

	function initGame(cnvsELID)
	{

	drawBoard(initCanvas(cnvsELID));
	}

view raw index.js hosted with ❤ by GitHub

index.js and drawing.js – the interesting parts.

The index.js is the start of the MancalaGame I mentioned previously. Similarly, the drawing.js is the start of the BoardUI module mentioned as well. Both are written as node modules that export the necessary API as functions to be used by other modules.

At this point, not a lot is going on. We expose a single function from index.js which basically only asks the drawing module to initialize the canvas and draw the board.

Initializing the canvas is simply initializing a fabric.js object with our given HTML canvas element. We’re adding there another function for easier writing (and reading) later, but not much more. We’re doing this so we can leverage the fabric.js API for drawing on the canvas. We’ll be using this object from now on to manipulate the canvas.

Drawing the board itself is simply a matter of calculating line lengths and locations, nothing more. Note how the cell size, number and location of the board are all hard-coded into the function. We’re avoiding magic numbers, which is good. But this is clearly not very extendible or configurable code.

At the end of this step we can launch the index.html file which simply calls the game initGame function we mentioned above when the page loads.

Our next commit is a simple refactoring – renaming the index.js file to game.js. This is a simple enough refactoring, but I wouldn’t dismiss it so easily. It’s important to maintain a mental model of what each module does and how it’s connected to other modules in the code. I described that model in the previous post, and this is where it is manifested. It starts with the file names; designing the API and how it’s used is easier when we have some idea of where different responsibilities lie. It’s easier to track this when we know what module we’re working on by simply looking at the file name¹.

Drawing Board State

So far, we only drew the board itself, but no stones in it. If we only do minimal changes, we’d probably only be drawing circles and numbers at this point (to represent how much stones are in each cell). But there’s a more fundamental idea here. We’re in fact drawing the state of the game onto the board. At the start of the game there’s obviously an initial state of the board setup. But the more prominent idea here is that of drawing the state of the board onto the canvas, in correct places. In other words, we need some representation of the board state.

Enter board.js:

	class Board
	{
	constructor(_cellCount)
	{
	this.cellCount = _cellCount;
	this.board = [];
	range(1,this.cellCount).forEach(_ => this.board.push(0))
	this.forAllPlayer1Cells(cellInd => this.setCellStoneCount(cellInd,INITIAL_STONE_COUNT))
	this.forAllPlayer2Cells(cellInd => this.setCellStoneCount(cellInd,INITIAL_STONE_COUNT))
	dbg("Board initialized")
	}

	forAllPlayer2Cells(f) { … }

	forAllPlayer1Cells(f) { … }

	setCellStoneCount(boardCell,cnt) { this.board[boardCell] = cnt }
	stonesIn(boardCell) { return this.board[boardCell]}

	forAllCells(f) { … }

	isPlayer1Cell(boardCell) { return boardCell >= 1 && boardCell <= this.cellCount/2-1; }
	isPlayer2Cell(boardCell) { return boardCell >= this.cellCount/2+1 && boardCell <= this.cellCount-1; }

	isPlayer1Home(boardCell) { return boardCell == 0; }
	isPlayer2Home(boardCell) { return boardCell == this.cellCount/2; }

	totalCellCount() { return this.cellCount; }
	}

view raw board.js hosted with ❤ by GitHub

Board.js, initial version

This module exports a single class – Board, which maintains the data structure for maintaining the board state.

The state itself is maintained as a simple array of numbers – one number per board cell + the board’s cell count (the game’s “size”). But the array itself is not directly exposed. All API of the class does some query/manipulation over that array.

Note that there’s only one mutating method – setCellCount, which is the only method (for now) that changes the state. All other methods are simply querying the board’s state, or iterating over it.

One other noteworthy choice at this point: the API encodes the fact that there are only 2 possible players, e.g. by defining methods such as isPlayer1Cell, isPlayer2Cell. This is a deliberate choice, working under the assumption that the game, and its accompanying state, would only have 2 players playing. Accounting for a possibility of more players doesn’t seem useful, and more importantly worth the complication it could cause in the code. We’re “hard-coding” this choice (the number of players), but benefit by having a simpler and explicit API to the game’s state. This will also translate into easier expression of the game rules as we’ll see later.

Drawing the board state requires then a new function – drawBoardState in the drawing module:

	function drawBoardState(cnvs,board)
	{
	board.forAllCells(boardCell => {
	let stonesInCell = board.stonesIn(boardCell);
	switch (true)
	{
	case board.isPlayer1Home(boardCell) : drawPlayer1Home(stonesInCell); break;
	case board.isPlayer2Home(boardCell) : drawPlayer2Home(stonesInCell); break;
	case board.isPlayer1Cell(boardCell) \|\| board.isPlayer2Cell(boardCell): drawCell(boardCell,stonesInCell); break;
	default : ERR ("Invalid board cell when drawing state: " + boardCell); break;
	}
	})

	function drawPlayer1Home(stoneCount)
	{
	drawText(stoneCount,TOP_LEFT.x + CELL_SIZE / 2 – 10,TOP_LEFT.y + CELL_SIZE * 1.5 – 10)
	}

	function drawPlayer2Home(stoneCount)
	{
	drawText(stoneCount,TOP_LEFT.x + boardWidthInCells(board.totalCellCount()) * CELL_SIZE – CELL_SIZE/2 – 10,TOP_LEFT.y + CELL_SIZE*1.5-10)
	}

	function drawText(txt,left,top)
	{
	cnvs.add(new fabric.Text(txt+'',{fontSize : 20, left : left, top : top}))
	}
	}

view raw drawing.js hosted with ❤ by GitHub

Querying and drawing the board state

This function is now exposed outside, as another API function from drawing.js. There’s not a lot going on here except calculating positions and sizes for text elements. We can already see the meaning of choosing to be explicit in our API about player 1 and 2 – the function simply dispatches the correct calculation based on the characterization of the cell.

Tying It Together

To make sure all this drawing actually takes place, we make sure to tie it together. This happens by calling the new function from our controller module, `game.js`:

	const CELL_COUNT = 14;

	let board = new Board(CELL_COUNT)

	function initGame(cnvsELID)
	{

	drawBoard(initCanvas(cnvsELID));
	let cnvs = initCanvas(cnvsELID)
	drawBoard(cnvs,CELL_COUNT);
	drawBoardState(cnvs,board);
	}

view raw game.js hosted with ❤ by GitHub

Making sure we actually draw the state

At this point, we’re only adding the call to draw the state from the game initialization function. Note how at this point we already need to extract the CELL_COUNT constant. It was previously in the drawBoard state, in drawing.js; now it’s extracted to the game module, and used both to initialize the board data structure itself, as well as the drawing the board (lines 3,10 in the above snippet).

Drawing the Cell

Our next commit takes care of drawing the actual cell count in the cells:

	function drawCell(boardCell,stoneCount)
	{
	var left = 0;
	var top = 0;
	switch (true)
	{
	case board.isPlayer1Cell(boardCell) :
	top = CELL_SIZE /2 – FONT_SIZE/2;
	left = boardCell * CELL_SIZE + CELL_SIZE/2 – FONT_SIZE/2;
	break;
	case board.isPlayer2Cell(boardCell) :
	top = CELL_SIZE * 2.5 – FONT_SIZE/2;
	left = (board.totalCellCount() – boardCell) * CELL_SIZE + CELL_SIZE/2 – FONT_SIZE/2;
	break;
	default : ERR("Invalid board cell: must be either player 1 or player 2 cell");
	}
	drawText(stoneCount,TOP_LEFT.x + left,TOP_LEFT.y + top);
	}

view raw drawing.js hosted with ❤ by GitHub

Drawing the cell

The function itself is simple enough, simply calculating the position of the drawn elements based on whether we’re drawing for player 1 (top row) or 2 (bottom row). The text itself is the stone count.

Note also how we eliminate a magic number – the font number.

But a Cell Can Be Empty…

An acute reader should note that at this point we’re only drawing the text figures in cells. But updating the state of the board will require us to also remove the text, in case no stones are actually in the cell. This is a design choice – not showing a circle with a text (number of stones) in an empty cell; but it’s one we’re making for the sake of a nicer UI. We’d like an empty cell to be really empty, and not show ‘0’.

The next commit takes care of that. We add the code to essentially remove an existing drawing object. This leads us to the need to remember which UI objects belong to which cell. This is done by maintaining an array of canvas elements² that correspond to each cell, when drawn:

	let stoneUIElement = [];
	…

	function initDrawingElements(cellCount)
	{
	range(1,cellCount).forEach( _ => stoneUIElement.push(None));
	}

	function rememberUIObj(boardCell,el) { stoneUIElement[boardCell] = maybe(el); }
	function forgetUIObj(boardCell) { stoneUIElement[boardCell] = None; }
	function uiObjAt(boardCell) { return stoneUIElement[boardCell]; }

view raw drawing.js hosted with ❤ by GitHub

Maintaining an array of UI canvas objects

Each element in the cell is an Option object ³ allowing to write more easily read code.

All that’s left to do is to keep track (“remember” and “forget”) the drawn elements, whenever we draw them. Of course, when the stone count is 0, we remove the canvas object, and “forget” it:

	function drawBoardState(cnvs,board)
	{
	…
	board.forAllCells(boardCell => {
	let stonesInCell = board.stonesIn(boardCell);
	switch (true)
	{
	case board.isPlayer1Home(boardCell) : drawOrRemove(boardCell,stonesInCell,(_,stoneCount) => { drawPlayer1Home(stoneCount); }); break;
	case board.isPlayer2Home(boardCell) : drawOrRemove(boardCell,stonesInCell,(_,stoneCount) => { drawPlayer2Home(stoneCount); }); break;
	case board.isPlayer1Cell(boardCell) \|\| board.isPlayer2Cell(boardCell):
	drawOrRemove(boardCell,stonesInCell,(boardCell,stoneCount) => { drawCell(boardCell,stoneCount); });
	break;
	default : ERR ("Invalid board cell when drawing state: " + boardCell); break;
	}
	})

	function drawPlayer1Home(stoneCount)
	{
	rememberUIObj(board.player1Home(),
	drawStones(stoneCount,
	TOP_LEFT.x + CELL_SIZE / 2 – FONT_SIZE/2-MARGIN,
	TOP_LEFT.y + CELL_SIZE * 1.5 – FONT_SIZE/2-MARGIN));
	}

	function drawPlayer2Home(stoneCount)
	{
	rememberUIObj(board.player2Home(),
	drawStones(stoneCount,
	/* left = /TOP_LEFT.x + boardWidthInCells(board.totalCellCount()) CELL_SIZE – CELL_SIZE/2 – FONT_SIZE/2-MARGIN,
	/* top = /TOP_LEFT.y + CELL_SIZE1.5-FONT_SIZE/2-MARGIN));

	}

	function drawCell(boardCell,stoneCount)
	{
	switch (true)
	{
	case board.isPlayer1Cell(boardCell) :
	… //calculating top,left
	break;
	case board.isPlayer2Cell(boardCell) :
	… //calculating top,left
	break;
	default : ERR("Invalid board cell: must be either player 1 or player 2 cell");
	}
	rememberUIObj(boardCell,drawStones(stoneCount,TOP_LEFT.x + left,TOP_LEFT.y + top));
	}

	function drawOrRemove(boardCell,stoneCount,drawFunc)
	{
	if (stoneCount > 0)
	{
	drawFunc(boardCell,stoneCount);
	uiObjAt(boardCell).ifPresent(uiObj => {uiObj.on('mousedown', _ => {dbg('Clicked cell: ' + boardCell + '!')})})

	}
	else removeDrawingAt(boardCell);
	}

	function removeDrawingAt(boardCell)
	{
	uiObjAt(boardCell).ifPresent(uiObj => {
	cnvs.remove(uiObj);
	forgetUIObj(boardCell);
	});
	}

	function drawStones(stoneCount,left,top)
	{
	… //create canvas objects and add them to the canvas
	//return the created canvas object
	}

view raw drawing.js hosted with ❤ by GitHub

Drawing and keeping track of canvas objects

Note that the drawStones function does the actual “drawing” – creating the canvas objects and adding it to the canvas, but it also returns the created object (a Group). This allows calling code to “remember” the object. I intentionally did not add the code to remember the objects in this function, since it would’ve made the function’s code more complicated – mixing separate concerns ⁴. This way, there’s place in the code that does the drawing, and a place that does the accounting of the objects, each place operates at a different level of abstraction – the game board vs. the canvas objects.

Another thing to note is that regardless of the cell being drawn, the pattern of either drawing it or removing it is the same. The difference lies in how we calculate the coordinates of where to put it. We therefore create a single function – drawOrRemove – that encodes this repeating pattern. The function accepts another function – the drawFunc parameter – that does the actual drawing (calls drawCell or drawPlayer1Home, drawPlayer2Home). We could’ve chosen to abstract it differently – pass functions that simply calculate the coordinates. I eventually leaned towards this solution – a slightly higher abstraction (a function that draws the cell and not a function that calculates coordinates) thinking that I might want to alter the look and feel of different cells in the future.

One last thing to note in this part: when drawing the cell canvas objects, we’re already adding the mouse listener to respond to events (line 54 in the above snippet), at this point doing nothing except issuing a log message. We do this here since it’s the low level operation of attaching an event listener to the object. I’d like to maintain the rest of the code oblivious as much as possible to the details of the drawing objects.

Next, we’re going to look how the drawing will start responding to events, and what does that entail.

Coding for Fun – Building a Simple Game

Leave a reply

I know this may come as a shock to some readers, but I love writing computer programs. Alas, my code-writing time is somewhat limited these days. So I try to find some side projects which won’t consume too much time, with little overhead and will still be interesting enough to be considered fun.

So I turned to games, of the computerized kind. It still requires quite some coding, but fairly isolated from other dependencies, and not too complicated. If the game is simple enough, you can end up with a working prototype in one or two evenings. Plus, you end up with something you can play with and manual testing (shiver… ) is more fun.

Aside from having a fun end result, games can be a great practice for a lot of subjects in programming. User interfaces is an obvious one; but think also how you model data structure for a game, use AI, persist state (or not), deploy it, etc.

In this post (with follow ups) I’d like to go through the process of one such game I created, completely in JS. I’ll go through the actual evolution and design choices and try to show the interesting evolution points in the code. This is not intended as an ultimate guide to how write such a (simple) game. Nor do I claim to show the best way to write it, only one specific way. The objective is to show how the design decisions are made, and how they’re reflected in actual code.

The Game: Mancala

In this case, I chose a fairly simple 2-player game called Mancala. I’m far from an expert on the game itself, but after playing it a few times with my kids, I can relate to how simple and yet not trivial game it can be. The simplicity of the rules and user interface lends it self perfectly for a simple project that can be expressed in code fairly easily.

I won’t dive into the details and rules of the game itself, only where it’s needed. If you need a short description of the game rules, this 1-pager does a decent job of it. Otherwise, google is your friend.

Where relevant, I will refer to the rules of the game, and specify where and if I took some liberty for the programmed version.

Before Coding – Choosing a Path

My main purpose here is to provide a step by step explanation of how the game was created and why. But before we dive into that, I believe it’s worth having some kind of an idea of what we want to achieve, and why. This step is often referred to as “Design”⁵.

Besides having fun coding, my goal is to create a game that is easily deployed and played, with minimal dependencies on libraries, and can be played by as many people as possible, on standard modern technical infrastructure.

Enter Javascript and a Browser.

So as a first choice, I choose to have this deployed as a purely Javascript-based application (in a web page), with no server component. Running the game is as simple as pointing the browser to a web page and loading it. This of course means that there’s no persisted game state (we could use local storage, but I don’t see the need so far), no user identity that’s relevant, etc. It’s just a game. Also, running the entire game in a single JS file, makes it easier to run in a different scenario, i.e. not just in a browser, but more on that later.

I use npm for package management, with webpack to package it all into a single JS file. This results in a simple deployment – an HTML + single JS file that contains all the game.

Naturally, some libraries are used to make life easier when coding, but I try to minimize the use of libraries as well.

The Code

The code for the game itself can be found in this github repo (ignore other folders in that repo). I’ll be going through the different commits to show how it was built, and link back to this repo.

Before diving into actual code, I believe some high level explanation on how the code is structured can help a lot. Note that the code structure changes and evolves during development, but the basic simple pattern is stable enough and should be kept in mind⁶.

Mancala Game High Level Components

At a very high level, the code is centered around 3 main components:

The Board module is essentially a data structure maintaining a board state – how many stones are in each cell, etc.
The BoardUI module is what draws the board into an HTML canvas. It queries the board to understand what to draw and then translates that into shapes to be drawn on the canvas. The drawing itself timed by the MancalaGame module.
The MancalaGame module is the glue connecting all the parts. It gets UI events, translates them into changes in the board state, and times the redrawing of the board. This is where all the game rules are encoded.

The actual manifestation of this design changed a bit as the code evolved, and we’ll see that when we look into the code; but the core thinking is pretty much the same throughout.

Next, we’ll start looking at the code, specifically how we draw the board.

Managing Access Control – A Crash Course

Leave a reply

During my career, especially when involved in development of some kind of platform, I’ve run into the need to explore and make decisions on how to manage access control in all kinds of situations. I’ve done my research, but I still haven’t found a simple and clear explanation of access control models, specifically how to manage access control in a platform.

So my purpose here is to try and give a solid yet simple enough explanation of how access control models work. This is not intended necessarily to be a comprehensive view, nor a very precise and rigorous view. My aim here is to provide a structured conceptual model and facilitate further discussions and decisions on how to model the access control in applications/frameworks. I cover here mainly the declarative part of managing access control, implementation of an access control mechanism is beyond the scope of this post.

Let’s dive right in.

The Access Control Question

At the fundamental level, all access control models try to answer a simple question: for any given user, what actions can he/she perform on what objects, at the time of the query.

To put it slightly more formally, given a set of users U, a set of objects O and a set of actions A (associated with the objects), what is the relation P of tuples <u,a,o> (u∈U, a∈A, o∈O)?

If a tuple exists in the relation, then we say that user u is allowed to do a with object o.

Some examples:

<Joe,READ, Clients_Table> means that Joe is allowed to READ the Clients table.
<Alice,VIEW, Tenant Mgmt_Screen> means that the user named Alice is allowed to view the tenant management screen.
<Alice, EXECUTE, DeleteUsers> means that the user named Alice is allowed to execute the function called ‘DeleteUsers’.
<Bob, READ, Device[id=123]> means that the user named Bob can read the record with id 123 from the Device table.
<*,VIEW, Login_Screen> means that any user can view the login screen (adopting ‘*’ as a notation for wildcard)

The only question⁷ an authorization service aims to answer is the existence of such tuples at the time of querying – whether the user in question can execute a given action on a given object.

Note that the definition of users, actions and objects is anything we choose to be. Objects can be of course data objects (tables, records, documents), with common actions such as CREATE, READ, UPDATE, DELETE. But objects are also functions to execute (stored procedures, RPCs, actions accessed over URLs, … ) and it can also be screens to view.

Terminology-wise, objects are sometimes referred to as resources and users are more generally referred to as security principals (so it’s not just individual persons). A tuple of <action, object> is referred to as a permission or privilege.

Managing Permissions

Now that we’ve established the main question that is answered, the question becomes how to manage permissions effectively and efficiently. This is more than a question of convenience. An overly complicated permission management model might lead to confusion and assignment of permissions where they should not be assigned to. It’s important to be able to reason about a given user’s permissions so we can make sure a user (usually a group of users) doesn’t get permissions they shouldn’t have.

A straightforward optimization is to assign a permission (action, object tuples) to a set of users at once. So access control systems usually let us define arbitrary groups of users to allow this convenience. Still, the definition and assignment of a large set of permissions is cumbersome.

Roles

Another common and useful way to organize permission assignment is to group permissions and assign (groups of) users to these sets of permissions.

Remember that a permission is simply a tuple of <action,object>. A role is simply a set of such tuples.

For example, an Account Manager role (AccountManagerRole) might be defined as:

  <View, Account Configuration Screen>,
  <Execute, UpdateAccount>,
  <View, Account Dashboard>
  …

Assuming we have such a role defined in the system, we can now assign users to it.

So <Bob, AccountManagerRole> is simply a shortcut for:

  <Bob, View, Account Configuration Screen>,
  <Bob, Execute, UpdateAccount>,
  <Bob, View, Account Dashboard>
  …

Naturally, role definitions usually correlate with business-related roles. So a user that is assigned to a role is expected to fulfill some function in a business process, which the given permissions allow him to perform.

This is also true for operational roles over management applications/resources. We define a role of “administrator” or “operator” with the necessary privileges to accomplish the role. It’s important to have even operational functionality access-controlled with the same mechanism so definitions are assigned effectively. A common example is a “user administrator” role, which is allowed to create/update/delete user records (other principles) and assign them with privileges. This role definition is not different than any other, it’s simply defined over the “Roles” and “Users” objects.

Assignment of users/principles to roles, whether mechanized or manual, is usually a human-driven process. Assignment of (groups of) users to roles happens with some utility or GUI-based tool, but essentially human driven and performed as needed, e.g. as part of some onboarding process.

A word of caution: I’ve seen in some discussions how roles get mixed up with user groups. In other words, roles are conceptualized as sets of users. While technically you can view a role as designating a set of users (the principles assigned to the role at a given point in time), this is actually a byproduct of the access control system, and not the original purpose and motivation for defining a role. A role should ideally represent a business function, with a set of privileges.

Data-Level Permissions

A common and useful extension to this permission management model is that of data-level permissions, often referred to as row-level permissions, because of its root in RDBMSs. In this extension we usually define permissions at a lower granularity of objects – individual data objects (rows/documents) defined in our application.

Note that this is not a different model then what we presented as the basic permissions model. We’re still assigning actions, usually READ/UPDATE/DELETE, to individual objects. The difference is in how the assignment happens – how we connect a user to the object in question.

Up until now, the user was assigned arbitrarily/manually to a set of permissions, probably through a role. With this extension we’re also configuring the authorization service⁸ to match a specific property of the user to a property of the accessed object.

For example, we might want to limit Charlie’s access only to Clients from his region. We can define the permission as:

<READ,Clients[region=user.region]>

Where user here refers to the “current” user being matched (the principal trying to access the Clients table). This assumes of course that the user object in the system has such a property (region) defined.

This allows us to assign this permission to a group of users, but still having each user seeing only the appropriate set of data he should be allowed to access, without defining explicit roles for each such subset. It’s both easier to manage and more robust since new property values (e.g new regions) are automatically accessed only by allowed users, without changing role assignment.

The choice of properties to match can of course be anything that can be matched between the data object and the user. A common use case for such a mechanism is to separate the data between different business units or tenants, in the same data store. So a “BU”/”Tenant” property is defined for each data object (row), and users are assigned to business units/tenants, allowing this match to happen when examined.

Note that there’s nothing that prevents this definition from being part of a role definition. It is, however, often impractical/inefficient to enforce this kind of restriction at the same place where more general access control is enforced. This is simply because enforcement of data-level permissions requires intimate knowledge of the underlying data model to be queried, which usually doesn’t exist in a generic access control service. It’s also more efficient to “push” the data-level permissions to underlying query engines to reduce retrieved data volumes.

Extending the Accessible Object Set

It’s often useful/desired to extend the set of permitted data objects to a user, either individually or through a role. We do this by extending the set of matched values on the user’s side. In other words, the user will be able to access data objects (rows) that match a larger set of values.

The most straightforward way to achieve this is to explicitly specify more values in the permission definition. But this is often impractical from an operational perspective.

Another way is to rely on some other structure that exists between (sets of) users and leverage its definition to extend the set of matched property values for that user.

A common example of such a mechanism is the definition and assignment of users to business units, which often form a hierarchy derived from its business/organization hierarchy.

For example, assume ACME Corporation is structured as follows:

ACME Global

⤷ ACME US

⤷ ACME West Coast

⤷ ACME East Coast

⤷ ACME EMEA

⤷ ACME Middle East

⤷ ACME Africa

⤷ ACME EU

⤷ ACME APAC

And assuming Charlie is assigned to ACME US, i.e. has a property that says that his business unit is ACME US. Then we can easily extend Charlie’s set of accessible objects by matching also on descendant business units (ACME West Coast, ACME East Coast), so objects that have these business units set to them are also viewable.

It’s important to note that this is simply leveraging the assignment of users to business units and their hierarchy in order to extend the sets of accessible objects. But there’s nothing that prevents us from using any other existing relationship between users. For example, consider users that are mapped to an organization hierarchy (a Manager-Employees relationship). We can define that a user has access to all data objects where their “category” matches the user’s “expertise” value (whatever that means); plus all the ‘expertise’ values of the users managed by this manager. In this manner we allowed the manager to automatically view his employees’ data, e.g. assigned tickets. Changes in the organizational structure, or ticket categories will automatically change the permissions assigned to said users.

To summarize, we defined a simple (semi) formal model for thinking about the access control problem, and specifically how to manage it. Different systems and paradigms exist to ease the management of this kind of access control.

When considering how to manage access control, especially when creating platforms, it’s usually helpful to think of it in terms how principles are assigned to <action,object> tuples effectively.

Hope this helps.

ADR Flow: A Tool for Managing Architecture Decisions

With all the agile software development movement, the role of the architect has changed to some degree as well.

While exploring the full breadth of changes is an interesting (some might say tedious) discussion, one of the most prominent challenges I often face is finding an efficient way to preserve knowledge around design/architecture decisions that are made in a project.
It’s a challenge I faced when I joined a team already working on several different products, and me being the new guy (and the one who’s supposed to make design decisions!) – I had to spend considerable time researching, asking around, digging into code and old presentations, trying to figure out why the system was built the way it is. In some cases there was actually no clear answer.
It’s also a situation I expect people to face with any long-lived project/product development. Unless you maintain very good documentation of your design decisions, it will be hard to trace back why things are built the way they are. I know I’ve faced similar situations in past projects.

Documenting decisions is tedious. Moreover, it’s often perceived as non-productive. It doesn’t add product value. But even that – convincing people to document design decisions – is the easy part in my experience. The harder task is to actually write something that is coherent, describes the context and the decision succinctly but completely.

One methodology I found, liked, and currently experimenting with, is that of Architecture Decision Records.
Refer to the link above for a complete overview, but in a nutshell: describe your architecture decision using a simple template, keep one file per decision, keep them with the source code, and setup a minimal process around these decisions, to be reflected in the files. The coupling to the code, and focus on specific decisions, is what I think will make this a more productive way to document our design decisions.

For me, at this point in time, this is still in “POC mode”. I’ve introduced it to my team, and currently pushing it, but we still need to see more value.

One thing I missed from the beginning is a way to to easily setup and manage ADRs. I found this open source project, which seems useful. But it’s based solely on linux shell scripts. Being mostly a windows user in my day-to-day work, I missed a good windows integration.

And thus the ADR Flow project was born.

The ADR Flow tool is meant to be a simple command line tool to aid in the process of creating and managing the ADR lifecycle. It’s meant to be simple enough so that it could be easily integrated into any workflow and/or standard tool.
It is not meant to hide the fact that ADRs are text files, or prevent editing them in whatever tools the user sees fit. Rather it tries to leverage some of the maintenance overhead, and make managing ADRs easy.

It’s a fun side project, also giving me the option to explore more possibilities for development with Node.js.

Comments, questions, feedback and of course contributions are more than welcome.

Code Writing from a UX Perspective

We often talk and discuss coding style and guidelines. Some of these discussions grow to be (some would say useless) flame wars, while others invoke thoughts and sometimes even reach a consensus, e.g. “GOTO considered harmful” (PDF). These discussions tend to be motivated by good intentions – making software development better; delivering software faster, with better quality and precision.

One area that keeps coming up is the need to have code written clearly; to make code easily understandable and changeable. It is generally agreed that code readability is important. And yet, the discussions are sometimes endless. We don’t seem to be able to agree what constitutes code that is written well enough, one that is easily understood. Claims made are often seen as very subjective and “a matter of taste”, as readability and code quality deals with human interpretation and judgement of said code.

At the same time, another field that deals with human-computer interaction seems to achieve better success in establishing at least some level of standard or agreed understanding of quality – software user experience (UX). Naturally, taste also plays a role in this case, but practitioners seem to agree more on some underlying practical principles to follow; similar to how coders have established some common understanding of good code design.

Which then begs the question – what if we approach code writing as a user experience problem, where the users are coders (incl. the code writer)?

Designers of applications’ user experience often deal with how the application is to be used. Characterizing its users and their main use cases. The answers to most of these questions are already known to us in the case of coders – we are the users. We write code, read it, test it, debug it. We know how we navigate code, skim through it, or how we approach refactorings.

Given that profiling the users is mostly out of the way, can we use this knowledge by taking a page from the UX designers’ playbook and applying it to code writing?
Can we design the code itself in such a way that will make our experience with it more productive and enjoyable, or dare I say delightful?

A naive approach would be to look at UI patterns and trying to apply them to code. While some may be somewhat useful (e.g. “inline hints”), it appears that the context of graphical UI design and its associated patterns does not immediately map to textual code writing. So while some insights can be had, this kind of mapping doesn’t seem to offer great value.

UX Principles in Code

But what if we go deeper?
Can we apply established principles of software-human interaction to code writing?

Take for example clarity. Good user interface designs are clear – at any point in the application they provide the user with an immediate understanding of what just happened, where is the user, what are the options available and their implications and so on. Think breadcrumbs, messages displayed, contextual disabling/enabling of buttons and so on.
What does this mean to code? Saying that “the code needs to be clear” is stating the obvious. Thinking about it from a user (=programmer) experience point of view we should be paying attention to the exact same things:

Is it clear to the user where he is from looking at the code?
Is the code conducive to understanding the options available at any given point?
Are implications of changes communicated clearly?

For example, consider a long file containing a lot of functionality, however related, a file with a lot of definitions, a class with a lot of methods and internal classes, nested callbacks, etc. These are examples of code that does not lend itself immediately to the structure and location in the code. It is harder to follow when reading and especially when debugging it; as debugging often involves jumping from file to file.
Similarly, having too many options to do one thing, e.g. overloaded functions, might introduce some confusion on what is the best way to do things. So there’s a clear tradeoff here between convenience (having the same method name, but with different parameters), and creating a confusing API.

Instead, keeping code listings shorter, tying code artifacts more closely to their location in the code base – e.g. one or two type(s) per file, can help with guiding the user of the code to understand what is the piece of code he is looking at. On the same note, providing one or two ways to do something, e.g. reading from a file or from stream, could actually make it easier for the caller of an API to understand what’s expected and how to work with that API.

Another principle to think about is familiarity and consistency. User interfaces, especially graphical ones, tend to prefer conventional solutions (hence patterns) for common tasks. Icons, links, color coding (red vs. green) are all examples of reusing familiar visual cues to convey meaning and being useful.

Familiarity can apply also to code writing. We can think of several places where this principle is applied:

Using common verbs, e.g. names of well known code design patterns
Naming getters and setters
Consistent naming for classes with similar roles
Well used parameter names – i for index, callback for callback functions, onXXX for event handlers
Packages following a similar scheme

Using well known patterns for organizing code, naming artifacts, etc. gives the programmer a better chance of understanding and navigating the code. At the project level it might make refactoring easier as it already suggests some code/module structure that can be followed, and may lead to more thoughtfulness with changes that break these familiar patterns.

Efficiency, of course, is also something to look into. Good user interfaces often emphasize the main scenarios supported by making them very efficient for the user. We count clicks, measure how much scrolling is needed, carefully design menus, etc. only to make sure that the common usage scenarios are easy to follow and accessible. In code this may translate to looking at the prevalent scenarios when coding. Readability is of course important, since we mostly read code. But another thing to consider is, for example, debugging and monitoring the running code. Do we insist, for example, on writing one operation per line, with variables defined for intermediate values; or do we cram together several access function calls making it harder to debug and trace the code?

Discouraging Bad Practices with Programmer’s User Experience

Another angle to look into the programmer’s user experience is to try and come with ways where the user experience actually encourages us to write better code.

An example for this is fibonacci indentation where each level of text indentation in the code is indented by a number of spaces matching the fibonacci number at that level. So deeper nesting levels are indented further away. At first, this may seem weird, but when you consider this, you quickly see how this discourages too much nesting – thus improving code readability and the overall experience when dealing with the code.

Another example might be to mandate no empty lines between a variable declaration and where the variable is used. This encourages declaring variables closer to where they are used, keeping related expressions closer visually; otherwise you end up with a big block of code – making it harder to follow.

How is this Useful?

These examples, and some of the conclusions presented here are nothing new to anyone considering the readability and clarity of code. Proper naming, for example, has always been in focus when considering code quality and readability. Still, it might be useful to frame the code readability/usability question as a user experience question, and borrow some practices from the field of user experience design. It helps to reason about better practices for structuring code.

Practically, this could be part of our code reviews, and configured/programmed into our already established set of coding tools – IDEs, linters, static code analysis tools, style guides, etc. The amount of tools available and polyglot nature of today’s software systems, where it’s not uncommon for a system to be coded in several languages, may in fact highlight the value of having a set of overarching principles and insights to guide us to better code writing. We should be able to define and follow principles that can be applied consistently and successfully in several languages at the same time.

One might argue, and justifiably so, that the job of making a programmer’s life easier is handled pretty well with the plethora of tools available, namely modern IDEs. But code can be made better (=easier to work with) by not relying on such mechanisms alone. Besides, not all editors are created equal, and code is viewed in tools other than the original IDE used to write it; e.g. code review tools, diff utilities. And of course, different people use different editors. So editors are very helpful, but I believe we can write clearer code if we don’t necessarily rely on that crutch.

Similarly, some languages also tend to lend themselves better to making code more readable. Type inference and lambda expressions help us make code more succinct and often more readable (though some would argue that explicit typing actually delivers a better experience). The user experience view of code writing is still useful in choosing how to use such language features, where choices are available.

To conclude, considering code writing from a user experience perspective might not be the most natural approach when programming. But if we agree that code readability and clarity is important – directly influencing quality and speed of development – then we should probably look at it from a proven perspective, using considerations and principles otherwise proven for software writing. A programmer’s “code experience” provides another perspective on how code is written and read and should probably be examined when making choices on how to write and structure code. Instead of debating on why a given code is written better this way or that, we can apply user experience guidelines to show us the path to a better code writing.

LiterateJS: Literate Programming with Javascript and HTML

Ever since I started to work on programming languages and models, I was intrigued by the problem of developer productivity. Always looking for effective ways to write and maintain code over time. One of the core issues I constantly struggle with when writing/looking at code is its readability. I generally maintain that the readability of the code has direct impact on its quality, but I digress.

There are all sorts of methodologies and tools to increase code readability, but one of the most interesting ones I came across is the idea of Literate Programming. The idea itself isn’t very new, and was conceived first by Donald Knuth in his book about that same subject.

I don’t claim to explain the idea better than the man who invented it (and I urge the reader to visit the links above to get a better understanding), but the rough idea is to write programs in the form of human-readable text, interspersed with program code. The program code is broken into snippets that are explained by the text. The resulting document is a fully functional (=”compilable”) program, using literate programming tools. It can be executed and it provides results, often embedded in the same document itself.

Being a fan of literature as well as programming, this struck me as a wonderful idea – you get to write your thoughts about what you’re trying to do, in a human readable way, and the code is right there, and it’s executable. There’s no separate documentation or anything – you explain the calculation, along with the code that actually executes it. Often times, the resulting document also contains the result of running the code, maybe even with graphs, images, etc.

There are of course limitations to this. And while I’ve no intention to start a complete debate on the validity of this approach, I will only say that I have my doubts on how this might work for developing a complex system, with several thousands/millions lines of code and a team of developers on it. While it’s theoretically possible to write such systems, I doubt any real benefit would be realized in such a scenario.

Still, for some scenarios, I believe this is a terrific method of writing software. Specifically, when trying to explain complex calculations, and making the calculations repeatable. This is why I think this kind of tools found its sweet spot in academia, where this use case is more prevalent. See iPython notebooks (python) and knitr (for R) as examples.

This is all fun and nice, but I’m not a data scientist, and I don’t delve too often into writing reports.

But then I came across a project at work, which required me to write a small tool to make some calculations, given a set of data. Besides the usual requirements for code quality, one of the prominent requirements was for users of this tool to easily understand the underlying code – the computation leading to the end results, along with intermediate results. And this struck a chord with me.

My first idea was to use R to write the code, both for its applicability as a language to the task and the availability of the knitr package mentioned above. But this was ruled out for reasons of skill set of the intended audience, and other developers who might to base their work on this tool (and also not wanting to install R/shiny to use it).

It was at this point that it came to me that even simple and readily available tools, namely a modern web browser with its javascript engine, are a great fit for the task. After all, today’s browser renders HTML pages with Javascript code running inside it all the time. So I have a presentation language (HTML), with a programming language (Javascript) and the engine to run it, all in one everyday tool – a browser. All that was left was to make it possible to write the tool that will marry both in a way that allows for the executable code to be part of the read document.

So I came up with LiterateJS.

LiterateJS was conceived originally as a proof of concept. I now evolved it to slightly more than that, but it’s no way near a mature project.

Still, it provides a demonstration and the basic facilities to write literate programming document using standard HTML and JS. ‘Standard’ being a key idea here – I intentionally try to avoid doing too much “magic”; instead trying to strike a balance between productivity and usefulness and remaining open to other tools and standards. The examples currently in the project use standard css (bootstrap) with another library for highlighting the code (highlight.js). Resulting in a standard HTML document that can be easily rendered, edited and debugged in standard tools. In this respect this isn’t a classic literate programming tool – there’s no explicit weaving step involved. The javascript code is simply extracted and with some help “injected” into the document, while preserving its highlighting and formatting. The resulting document is standard dynamic HTML document, with the code running in it.

LiterateJS is then a small framework that allows a developer to write literate programs in Javascript and HTML, and distribute them in any way he sees fit. It has a dependency on JQuery but that’s pretty much it (to be honest, the current code, at the time of writing, also depends on bootstrap css classes, but this dependency is planned to be removed).

LiterateJS is still very early in development, and I hope to have the time to continue developing it moving forward (any helpers?). Yet, I don’t expect it to grow too much, and aim to keep it a relatively small tool, with the possibility to add separate extensions later.

I encourage you to try it out, look through the examples and the code, suggest new ideas, features and of course bug fixes. Let me know if you find it useful, and program away!

The Understated Architect Role

So what’s in a software architect?

This is actually a question I’ve seen addressed a few times, and actually discussed during my career quite a bit. It’s a question I need to address if and when I search for new job as a software development architect. And it’s also a question that seems to have quite a few possible answers.

But eventually, there does some to be a common ground when trying to discuss the work of a software development architect. One could discuss the soft skills, the technical skills or the leadership skills required from an architect. It’s all true, at least to some degree. The exact combination of skills required to be a good software development architect varies between different organizations, operating contexts (the organization structure, company culture and size, history with the team, etc.) and your definition of “good”; so I won’t waste your time in trying to define an exact grocery list of skills.

I do believe, however, that one important role of the architect is often overlooked or not emphasized enough. And it’s probably not something you’ll learn in any course or training. It is something that I often felt during my work, but realized fully only when another (more experienced) architect has put it into words that resonated strongly with me.

Probably the most important job of an architect is to create and maintain a consistent understanding of the software system being developed. Or more precisely: creating a coherent and consistent picture of the developed system across all parties involved in the development project. To create an understanding between the development team and its stakeholders. To create an understanding within the team about the technical vision and direction for development. To create an understanding across teams on what is being developed and how systems interact.

This is probably the single most important job an architect has that is also unique to this role. Technical expertise is important, as is various design decisions and being able to weigh trade-offs properly. But the thing that truly separates the architect’s role from that of an expert software developer is the ability to convey a system structure and technical design, its capabilities, constraints and decisions taken when building it.

It’s really more than simply knowing the right words and using the right terminology. It’s about framing thoughts in a way that is consistent and understandable by the target audience. It’s about striking the balance between formalism and human comprehension. It’s about choosing the right methods to convey an idea. It’s about being precise and succinct, yet clear and understandable. It’s about being able to translate between different terminologies or “domains of thoughts”.

And herein lies the real challenge in being a software development architect, in my opinion. Technical mastery is important. But being able to convey an idea efficiently, to plant the right idea into the minds of people you’re communicating with is one thing you can’t learn on stackoverflow.com (which is a wonderful site, by the way).

And it’s a subtle act, often very delicate and hard to balance. Simply because people come from different schools of thoughts and experiences (and often with agendas). But choosing the right terminology, explaining it properly, making assumptions explicit or drawing the right diagram goes a long way towards aligning all involved parties on a single vision and a coherent mental picture of a system. Creating this mental picture in everyone’s mind is no easy task.

I had a boss who once told me that whenever there’s a disagreement on a technical direction to take, you should prefer to be the one drawing the diagrams. This simple fact already gives you an advantage. If you’re the one holding the marker pen, you already have an edge.

This is true from an organizational politics point of view, but also an important point to keep in mind when you want to reach an agreement and create this common understanding and consistency – if you wield the pen, you wield the power as well as the responsibility. And when you’re an architect, your job is to wield the pen.

When people talk about being an architect, they usually talk about what to do with the pen – what kind of diagrams to draw, what documents to write, how much code, etc. But there’s an overarching, implicit goal that is assigned to the one holding the pen – to make sure everyone is aligned and have a consistent mental picture in their heads, when approaching their individual jobs. This is especially harder in the software business, where there’s no shortage of abstractions and intangible concepts to keep in mind. Where the different levels of abstraction and complexity a developer has to keep in his head are often mind boggling. The ability to synthesize the important ideas and communicate them at the right place, at the right time, in a manner that will be adopted and accepted is quite a fit. Take someone who is good at doing that, add a decent technical and analytical abilities into the mix and you’ve got yourself a good software architect right there.

Effective System Modeling

3 Replies

This post is a rough “transcript” (with some changes and creative freedom) of a session I gave in the Citi Innovation Lab, TLV about how to effectively model a system.

A Communication Breakdown?

Building complex software systems is not an easy task, for a lot of reasons. All kinds of solutions have been invented to tackle the different issues. We have higher level programming languages, DB tools, agile project management methodologies and quite a bit more. One could argue that these problems still exist, and no complete solution has been found so far. That may be true, but in this post, I’d like to discuss a different problem in this context: communicating our designs.

One problem that seems to be overlooked or not addressed well enough, is the issue of communicating our designs and system architecture. By ourselves, experienced engineers are (usually) quite capable of coming up with often elegant solutions to complex problems. But the realities and dynamics of a software development organization, especially a geographically distributed one, often require us to communicate and reason about systems developed by others.

We – software engineers – tend to focus on solving the technical issues or designing the systems we’re building. This often leads to forgetting that software development, especially in the enterprise, is often, if not always, a team effort. Communicating our designs is therefore critical to our success, but is often viewed as a negligible activity at best, if not a complete waste of time.

The agile development movement, in all its variants, has done some good to bring the issues of cooperation and communication into the limelight. Still, I often find that communication of technical details – structure and behavior of systems, is poorly done.

Why is that?

“Doing” Architecture

A common interpretation of agile development methods I often encounter tends to spill the baby with the water. I hear about people/teams refusing to do “big up-front design”. That in itself is actually a good thing in my opinion. The problem starts when this translates to no design at all, and this immediately translates into not wanting to spend time on documenting your architecture properly, or how it’s communicated.

But as anyone who’s been in this industry for more than a day knows – there’s no replacement for thinking about your design and your system, and agile doesn’t mean we shouldn’t design our system. So I claim that the problem isn’t really with designing per-se, but rather in the motivation and methodology we use for “doing” our architecture – how we go about designing the system and conveying our thoughts. Most of us acknowledge the importance of thinking about a system, but we do not invest the time in preserving that knowledge and discussion. Communicating a design or system architecture, especially in written form, is often viewed as superfluous, given the working code and its accompanying tests. From my experience this is often the case because the actual communication and documentation of a design are done ineffectively.

This was also strengthened after hearing Simon Brown talk about a similar subject, one which resonated with me. An architecture document/artifact should have “just enough” up front design to understand the system and create a shared vision. An architecture document should augment the code, not repeat it; it should describe what the code doesn’t already describe. In other words – don’t document the code, but rather look for the added value. A good architecture/design document adds value to the project team by articulating the vision on which all team members need to align on. Of course, this is less apparent in small teams than in large ones, especially teams that need to cooperate on a larger project.

As a side note I would like to suggest that besides creating a shared understanding and vision, an architecture document also helps in preserving the knowledge and ramping-up people onto the team. I believe that anyone who has tried learning a new system just by looking at its code will empathize with this.

Since I believe the motivation to actually design the system and solve the problem is definitely there, I’m left with the feeling that people often view the task of documenting it and communicating it as unnecessary “bureaucracy”.
We therefore need a way to communicate and document our system’s architecture effectively. A way that will allow us to transfer knowledge, over time and space (geographies), but still do it efficiently – both for the writer and readers.
It needs to be a way that captures the essence of the system, without drowning the reader in details, or burden the writer with work that will prove to be a waste of time. Looking at it from a system analysis point of view, then reading the document is quite possibly the more prominent use case, compared to writing it; i.e. the document is going to be read a lot more than written/modified.

When we come to the question of modeling a system, with the purpose of the end result being readable by humans, we need to balance the amount of formalism we apply to the model. A rigorous modeling technique will probably result in a more accurate model, but not necessarily an easily understandable one. Rigorous documents tend to be complete and accurate, but exhausting to read and follow; thereby beating the purpose we’re trying to achieve. At the other end of the scale are free text documents, often in English and sometimes with some scribbled diagrams, which explain the structure or behavior of system, often inconsistently. These are hard to follow for different reasons: inaccurate language, inconsistent terminology and/or ad-hoc (=unfamiliar) modeling technique used.

Providing an easy to follow system description, and doing so efficiently, requires us to balance these two ends. We need to have a “just enough” formalism that provides a common language. It needs to be intuitive to write and read, with enough freedom to provide any details needed to get a complete picture, but without burdening the writers and readers with unnecessary details.
In this post, I try to give an overview and pointers to a method I found useful in the past (not my invention), and that I believe answers the criteria mentioned above. It is definitely not the only way and may not suit everyone’s taste (e.g. Simon Brown suggests something similar but slightly different); but regardless of the method used, creating a shared vision, and putting it to writing is something useful, when done effectively.

System != Software

Before going into the technicalities of describing a system effectively, I believe we need to make the distinction between a system and its software.

For the purposes of our discussion, we’ll define software as a computer-understandable description of a dynamic system; i.e. one way to code the structure and behavior of a system in a way that’s understandable by computers.
A (dynamic) system on the other hand is what emerges from the execution of software.

To understand the distinction, an analogy might help: consider the task of understanding the issue of global warming (the system) vs. understanding the structure of a book about global warming (the software).

Understanding the book structure does not imply understanding global warming. Similarly, understanding the software structure doesn’t imply understanding the system.
The book can be written in different languages, but it’s still describing global warming. Similarly, software can be implemented using different languages and tools/technologies, but it doesn’t (shouldn’t) change the emergent behavior of the system.
Reading the content of the book implies understanding global warming. Similarly, the system is what emerges from execution of the software.

One point we need to keep in mind, and where this analogy breaks, is that understanding a book’s structure is considerably easier than understanding the software written for a given system.
So usually, when confronted with the need to document our system, we tend to focus on documenting the software, not the system. This leads to ineffective documentation/modeling (we’re documenting the wrong thing), eventually leading to frustration and missing knowledge.
This is further compounded by the fact that existing tools and frameworks for documentation of software (e.g. UML) tend to be complex and detailed, and with the tools emphasizing code generation, and not human communication; this is especially true for UML.

Modeling a System

When we model an existing system, or design a new one, we find several methods and tools that help us. A lot of these methods define all sorts of views of the system – describing different facets of its implementation. Most practitioners have surely met one or more different “types” of system views: logical, conceptual, deployment, implementation, high level, behavior, etc. These all provide some kind of information as to how the system is built, but there’s not a lot of clarity on the differences or roles of each such view. These are essentially different abstractions or facets of the given system being modeled. While any such abstraction can be justified in itself, it is the combination of these that produces an often unreadable end result.

So, as with any other type of technical document you write, the first rule of thumb is:

Rule of thumb #1: Tailor the content to the reader(s), and be explicit about it.

In other words – set expectations. Set the expectation early on – what you’re describing and what is the expected knowledge (and usually technical competency) of the reader.
—

Generally, in my experience, 3 main facets are the most important ones: the structure of the system – how it’s built, the behavior of the system – how the different component interact on given inputs/events, and the domain model used in the system. Each of these facets can be described in more or less detail, at different abstraction levels, and using different techniques, depending on the case. But these are usually the most important facets for a reader to understand the system and approach the code design itself, or reading the code.

Technical Architecture Modeling

One method I often find useful is that of Technical Architecture Modeling (TAM), itself a derivative of Fundamental Modeling Concepts (FMC). It is a formal method, but one which focuses on human comprehension. As such, it borrows from UML and FMC, to provide a level of formalism which seems to strike a good balance between readability and modeling efficiency. TAM uses a few diagram types, where the most useful are the component/block diagram used to depict a system’s structure or composition; the activity and sequence diagrams used to model a system/component’s behavior and the class diagram used to model a domain (value) model. In addition, other diagram types are also included, e.g. state charts and deployment
diagrams; but these are less useful in my experience. In addition, TAM also has some tool support in the form of Visio stencils that make it easier to integrate this into other documentation methods.

I briefly discuss how the most important facets of a system can be modeled with TAM, but the reader is encouraged to follow the links given above (or ask me) for further information and details.

Block Diagram: System Structure

A system’s structure, or composition, is described using a simple block diagram. At its simplest form, this diagram describes the different components that make up the system.
For example, describing a simple travel agency system, with a reservation and information system can look something like this (example taken from the FMC introduction):

This in itself already tells us some of the story: there’s a travel agency system, accessed by customers and other interested parties, with two subsystems: a reservation system and an information help desk system. The information is read and written to two separate data stores holding the customer data and reservations in one store, and the travel information (e.g. flight and hotel information) in the other. This data is fed into the system by external travel-related organizations (e.g. airlines, hotel chains), and reservations are forwarded to the same external systems.

This description is usually enough to provide at least a contextual high level information of the system. But the diagram above already tells us a bit more. It provides us some information about the access points to the data; about the different kinds of data flowing in the system, and what component is interacting with what other component (who knows who). Note that there is little to no technical information at this point.

The modeling language itself is pretty straightforward and simple as well: we have two main “entities”: actors and data stores.
Actors, designated by square rectangles, are any components that do something in the system (also humans). They are they active components of the system. Actors communicate with other actors through channels (lines with small circles on them), and the read/write from/to data stores (simple lines with arrow heads). Examples include services, functions and human operators of the system.
Data store, designated by round rectangles (/circles), are passive components. These are “places” where data is stored. Examples include database systems, files, and even memory arrays (or generally any data structure).

Armed with these definitions, we can already identify some useful patterns, and how to model them:

Read only access – actor A can only read from data store S:

Write only access – actor A can only write to data store S:

Read/Write access:

Two actors communicating on a request/response channel have their own unique symbol:

In this case, actor ‘B’ requests something from actor ‘A’ (the arrow on the ‘R’ symbol points to ‘A’), and ‘A’ answers back with data. So data flow actually happens in both ways. A classical example of this is a client browser asking for a web page from a web server.

A simple communication over a shared storage:

actors ‘A’ and ‘B’ both read and write from/to data store ‘S’. Effectively communicating over it.

There’s a bit more to this formalism, which you can explore in FMC/TAM website. But not really much more than what’s shown here. These simple primitives already provide a powerful expression mechanism to convey most of the ideas we need to communicate over our system on a daily basis.

Usually, when providing such a diagram, it’s good practice to accompany it with some text that provides some explanation on the different components and their roles. This shouldn’t be more than 1-2 paragraphs, but actually depends on the level of detail and system size.

This would generally help with two things: identifying redundant components, and describing the responsibility of each component clearly. Think of this text explanation as a way to validate your modeling, as displayed in the diagram.

Rule of thumb #2: If your explanation doesn’t include all the actors/stores depicted in the
diagram – you probably have redundant components.

Behavior Modeling

The dynamic behavior of a system is of course no less important than its structure. The cooperation, interaction and data flow between components allow us to identify failure points, bottlenecks, decoupling problems etc. In this case, TAM adopts largely the UML practice of using sequence diagrams or activity diagrams, whose description is beyond the scope of this post.

One thing to keep in mind though, is that when modeling behavior in this case, you’re usually not modeling interaction between classes, but rather between components. So the formalism of “messages” sent between objects need not couple itself to code structure and class/method names. Remember: you generally don’t model the software (code), but rather system components. So you don’t need to model the exact method calls and object instances, as is generally the case with UML models.

One good way to validate the model at this point is to verify that the components mentioned in the activity diagram are mentioned in the system’s structure (in the block diagram); and that components that interact in the behavioral model actually have this interaction expressed in the structural model. A missing interaction (e.g. channel) in the structural view may mean that these two components have an interface that wasn’t expressed in the structural model, i.e. the structure diagram should be fixed; or it could mean that these two components shouldn’t interact, i.e. the behavioral model needs to be fixed.

This is the exact thought process that this modeling helps to achieve – modeling two different facets of the system and validating one with the other in iterations allows us to reason and validate our understanding of the system. The explicit diagrams are simply the visual method that helps us to visualize and capture those ideas efficiently. Of course, keep in mind that you validate the model at the appropriate level of abstraction – don’t validate a high level system structure with a sequence diagram describing implementation classes.

Rule of thumb #3: Every interaction modeled in the behavioral model (activity/sequence
diagrams) should be reflected in the structural model (block diagram), and vice versa.

Domain Modeling

Another often useful aspect of modeling a system is modeling the data processed by the system. It helps to reason about the algorithms, expected load and eventually the structure of the code. This is often the part that’s not covered by well known patterns and needs to be carefully tuned per application. It also helps in creating a shared vocabulary and terminology when discussing different aspects of the developed software.

A useful method in the case of domain modeling is UML class diagrams, which TAM also adopts. In this case as well, I often find a more scaled-down version the most useful, usually focused on the main entities, and their relationships (including cardinality). The useful notation of class diagrams can be leveraged to express these relationships quite succinctly.

Explicit modeling of the code itself is rarely useful in my opinion – the code will probably be refactored way faster than a model will be updated, and a reader who is able to read a detailed class diagram can also read the code it describes. One exception to this rule might be when your application deals with code constructs, in which case the code constructs themselves (e.g. interfaces) serve as the API to your system, and clients will need to write code that integrates with it, as a primary usage pattern of the system. An example for this is an extensible library of any sort (eclipse plugins are one prominent example, but there are more).

Another useful modeling facet in this context is to model the main concepts handled in the system. This is especially useful in very technical systems (oriented at developers), that introduce several new concepts, e.g. frameworks. In this case, a conceptual model can prove to be useful for establishing a shared understanding and terminology for anyone discussing the system.

Iterative Refinement

Of course, at the end of the day, we need to remember that modeling a system in fact reflects a thought process we have when designing the system. The end product, in the form a document (or set of documents) represents our understanding of the system – its structure and behavior. But this is never a one-way process. It is almost always an iterative process that reflects our evolving understanding of the system.

So modeling a specific facet of the system should not be seen as a one-off activity. We often follow a dynamic where we model the structure of the system, but then try to model its behavior, only to realize the structure isn’t sufficient or leads to a suboptimal flow. This back and forth is actually a good thing – it helps us to solidify our understanding and converge on a widely understood and accepted picture of how the system should look, and how it should be constructed.

Refinements also happen on the axis of abstractions. Moving from a high level to a lower level of abstraction, we can provide more details on the system. We can refine as much as we find useful, up to the level of modeling the code (which, as stated above, is rarely useful in my opinion). Also when working on the details of a given view, it’s common to find improvement points and issues in the higher level description. So iterations can happen here as well.

As an example, consider the imaginary travel agency example quoted above. One possible refinement of the structural view could be something like this (also taken from the site above):

In this case, more detail is provided on the implementation of the information help subsystem and the ‘Travel Information’ data store. Although providing some more (useful) technical details, this is still a block diagram, describing the structure of the system. This level of detail refines the high level view shown earlier, and already provides more information and insight into how the system is built. For example, how the data stores are implemented and accessed, the way data is adapted and propagated in the system. The acute reader will note that the ‘Reservation System’ subsystem now interacts with the ‘HTTP Server’ component in the ‘Information help desk’ subsystem. This makes sense from a logical point of view – the reservation system accesses the travel information through the same channels used to provide information to other actors, but this information was missing from the first diagram (no channel between the two components).
One important rule of thumb is that as you go down the levels of abstraction, keep the names of actors presented in the higher level of abstraction. This allows readers to correlate the views more easily, identify the different actors, and reason about their place in the system. It provides a context for the more fine granular details. As the example above shows, the more detailed diagram still includes the actor and store names from the higher level diagram (‘Travel Information’, ‘Information help desk’, ‘Travel Agency’).

Rule of thumb #4: Be consistent about names when moving between different levels of abstraction. Enable correlations between the different views.

Communicating w/ Humans – Visualization is Key

With all this modeling activity going on, we have to keep in mind that our main goal, besides good design, is communicating this design to other humans, not machines. This is why, reluctant as we are to admit it (engineers…) – aesthetics matter.

In the context of enterprise systems, communicating the design effectively is as important to the quality of the resulting software as designing it properly. In some cases, it might be even more important – just consider the amount of time you sometime spend on integration of system vs. how much time you spend writing the software itself. So a good looking diagram is important, and we should be mindful about how we present it to the intended audience.

Following are some tips and pointers on what to look for when considering this aspect of communicating our designs. This is by no means an exhaustive list, but more based on experience (and some common sense). More pointers can be found in the links above, specifically in the visualization guide.

First, keep in mind node and visual arrangement of nodes and edges in your diagram immediately lends itself to how clear the diagram is to readers. Try to minimize intersection of edges, and align edges on horizontal and vertical axes.
Compare these two examples:

The arrangement on the left is definitely clearer than the one on the right. Note that generally speaking, the size of a node does not imply any specific meaning; it is just a visual convenience.

Similarly, this example:

shows how the re-arrangement of nodes allows for less intersection, without losing any meaning.

—

Colors can also be very useful in this case. One can use colors to help distinguish between different levels of containment:

In this case, the usage of colors helps to distinguish an otherwise confusing structure. Keep in mind that readers might want to print the document you create on a black and white printer (and color blind) – so use high contrast colors where possible.

—

Label styles are generally not very useful to convey meaning. Try to stick to a very specific font and be consistent with it. An exception might be a label that pertains to a different aspect, e.g. configuration files or code locations, which might be more easily distinguished when using a different font style.

Visuals have Semantics

One useful way to leverage colors and layout of a diagram is to stress specific semantics you might want to convey in your diagram. One might leverage colors to distinguish a set of components from other components, e.g. highlighting team responsibilities, or highlight specific implementation details. Note that when you use this kind of technique that it is not standard, so remember to include an explanation – a legend – of what the different colors mean. Also, too many colors might cause more clutter, eventually beating the purpose of clarity.

Another useful technique is to use layout of the nodes in the graph for conveying an understanding. For example, depicting the main data flow might be hinted in the block diagram by layouting the nodes from left to right, or top to down. This is not required, nor carries any specific meaning. But it is often useful to use, and provides hints as to how the system actually works.

Summary

As we’ve seen, “doing” architecture, while often perceived as a cumbersome and unnecessary activity isn’t hard to do when done effectively. We need to keep in mind the focus of this activity: communicating our designs and reasoning about them over longer periods of time.

Easing the collaboration around design is not just an issue of knowledge sharing (though that’s important as well), but it is a necessity when trying to build software across global teams, over long periods of time. How effectively we communicate our designs directly impacts how we collaborate, the quality of produced software, how we evolve it over time, and eventually the bottom line of deliveries.

I hope this (rather long) post has served to shed some light on the subject, and provide some insight, useful tips and encouraged people to invest some efforts into learning further.

Credit: The example and images presented in this post are taken from the FMC website and examples.

Pattern Matching in Java?

Leave a reply

Writing some code at home, in Java, I came across the problem of trying to compute a value based on the data of another value. This isn’t nothing exciting in on itself, but this whole project I was writing was about trying to write in functional style, as much as possible, in Java 8.
Also, being a fan of fluent interfaces, or readable code in general, this was one of the cases I really missed Scala’s pattern matching capabilities.

So I set to the task of writing a small utility that will allow me to emulate Scala’s pattern matching syntax and behavior, but in Java, at least to some extent.

The end goal I wanted to achieve was to write something similar to:

(you can probably guess the project was something about chess)

In this case p was the parameter passed to a function, which consisted of this call alone. An equivalent piece of code would be to test for the class of p – a series of if-elses, or store this mapping in a map. These are all valid solutions, but I wanted something that would be both concise, and didn’t require extra data preparation. Also something that would be more generally usable than in this case. Besides, I wanted to invent some new way to do this stuff, have some fun doing it.

So I ended up with writing this small utility, which allows for this kind of code:

This code implements a “micro-dsl” of sorts for pattern matching on the data.
The usage of the builder pattern, with 3 internal classes, is to allow for the fluent interface, which was ultimately my goal: enable easy to follow syntax for this kind of operation.

Note that the code makes use of Java 8’s functional interfaces of Predicate and Function, and of course the lambda expression syntax. This allows both for flexibility (you can write any expression you want) as well as concise code.

You can even use it recursively, as the following simple implementation of factorial shows:

It’s not perfect, as it does still tend to be verbose, and not nearly as neat as Scala’s pattern matching, but I believe it does allow for cleaner code, and improves the situation quite a bit for this type of tasks. Note also that this code is pretty simple and self contained – it doesn’t rely on any external library, besides the standard JDK (I did use some trivial utility functions there, e.g. requireNotNull but that’s really beside the point).

Some points where I see this can improve are:

The ability to provide a default match (an “otherwise” case) that will handle all unmatched cases
The ability to provide the data to match as a lazily evaluated function, to allow for optimization (parallel evaluation of the cases?)
A “vectorized” version of this, allowing to specify a pattern match over a complete stream of values, resulting in another stream of values

I will probably do these when I have some more time and the need arises. In the meantime, keeping it simple.

Full source, with updates, is available here.

Schejter

Lior Schejter et al

Author Archives: slior

Building a Simple Game – Part 2

Drawing Board State

Tying It Together

Drawing the Cell

But a Cell Can Be Empty…

Coding for Fun – Building a Simple Game

The Game: Mancala

Before Coding – Choosing a Path

The Code

Managing Access Control – A Crash Course

The Access Control Question

Managing Permissions

Roles

Data-Level Permissions

Extending the Accessible Object Set

ADR Flow: A Tool for Managing Architecture Decisions

Code Writing from a UX Perspective

UX Principles in Code

Discouraging Bad Practices with Programmer’s User Experience

How is this Useful?

LiterateJS: Literate Programming with Javascript and HTML

The Understated Architect Role

Effective System Modeling

A Communication Breakdown?

“Doing” Architecture

System != Software

Modeling a System

Technical Architecture Modeling

Block Diagram: System Structure

Behavior Modeling

Domain Modeling

Iterative Refinement

Communicating w/ Humans – Visualization is Key

Visuals have Semantics

Summary

Pattern Matching in Java?