Tag: frameworks

Categories
Front End Development Haxe Haxe Log Small Universe

Writing a framework: web application architectures I’m inspired by

  1. I’m going to write a web app framework (again) (maybe)
  2. Writing a framework: web application architectures I’m inspired by

A look at recurring architectural patterns I see in both front end and back end, that have potential to tie together into a full stack framework.

In my last post I said part of my reason for wanting to write another web framework was that I’ve been exposed to similar ideas in both the front end and back end, and wanted to experiment with an architecture that ties it all together.

In this post, I’m going to explore those: Unidirectional data flow, the elm architecture, CQRS and event sourcing. And then I’ll look at the common themes I see tying them together.

State, views, and one-directional data flow

Almost every popular front end architecture I’ve encountered recently shares an idea: you have data representing the current state of your page, and you use that state to render the view that the user can see and interact with.

If you want to change something on the page, you don’t update the page directly, you update the state, and that causes the view to update.

You might have heard this described in a few ways:

  • Data Down, Actions Up: the data flows down from the state to the view. And the actions come up from the view to modify the state.
  • Model, View, Controller: Your have a data “model” layer that holds information about the current state, and a “view” layer that knows how to update it, and a “controller” that does the communication in the middle.
  • Unidirectional data flow: You often hear this term in the React ecosystem. Data flows from the top of your view down into it the nested components. So a component only knows about the data passed into it, and nothing else. The data always flows downwards. How do you change the data then? As well as passing down the data from the state, we also pass down a function that can be used to update the state.
An illustration showing a circular flow of data and actions. There is an arrow from “State” to “View” labelled “Data down” and from “View” back to “State” labelled “Actions Up”.
Actions Up, Data Down. Every application has “State” (data for the application) and “View” (what the user sees). The view is always rendered and updated from the current state. (We call this “Data down”). And the view can trigger actions to update the state. (We call this “Actions up”).

This pattern is used in frameworks as diverse as React and Ember and Elm. Why is it so common? Here’s some of the advantages it provides:

  • Each function in your code has one job: turn the state into a view, or to update the state in response to an action. This makes it nice and easy to wrap your head around an individual piece of code.
  • The functions don’t need to know about each other. If you have a “to-do list” app, and an action that adds a new to-do item – you don’t need to know the 3 different places it’ll show up in the UI and change them all – you just update the state. Likewise, if you want to add a new view, you don’t have to touch all the places that edit the “to-do” list, you can just look at the current state, however it came to be, and use that data to render your page.
  • It becomes easy to debug. If there’s a bug, you can check if the state data is correct. If the state data looks correct, then the bug will be in one of your view functions. If that state data looks wrong, then the bug is probably in one of your action functions that change the state.
  • It makes it easier to write tests. Your action functions test simple things: if our state looks like this, and we perform this action, then the state should now look like that. That’s easy to write a unit test for. And for your view functions, you can write tests that use mocked state data to test all the different ways your view might be rendered. You can even create “stories” with Storybook, or take visual snapshots, for quick visual tests of many different ways the UI looks.

The Elm architecture

Elm takes this concept to the extreme. By making a language and a framework that are tightly integrated, they can force you to follow the good advice from “data down, action up”.

The Elm architecture has four parts:

  • Model – the state of your application,
  • View – a way to turn your state into HTML,
  • Messages – a message triggered by the UI (like clicking a button) that has information about an action the user is requesting (like attempting to mark an item as complete)
  • Update – a way to update your state based on “messages”
An illustration titled "The Elm Architecture" showing a circular flow diagram between State, View, Html and Update. There is some psuedocode in each of "State" "View" "Html" and "Update" which is described in the caption.

The Elm Architecture.

State. The page has some current state that is used to render the view. We use the type system to make sure the structure of this is consistent. In this example, it might have a title “My work” and a list of tasks like “Plan week” and “Check Slack”

View. Render HTML based on the current state. Following on the example above, we might have a view function that receives our state, and calls functions to render a <h1> element and a <ul> with the list of tasks, and a form for adding new to-do items.

Html. We never manually edit HTML or the DOM, we just update state, which updates the view, and the framework checks what HTML needs changing. In the example above, this would be the rendered h1, ul and form HTML / DOM produced by the framework.

Update. The only type of events we trigger from the HTML view are strictly typed and exactly what our update function is expecting. When an action happens, the framework uses the update function to consider what the new state should be based on the previous state and the action that occurred. Following on the example, an “AddTodo” action might append a new item to the list. While the “CompleteTask” action might remove an item from the list. The only way to update the state is to use the update function, one action at a time. This makes state management easy to unit test and debug!

There is also the opportunity to interact with the outside world – things like a Backend API. Elm provides a way to do this from the Update function (which can in turn trigger new actions) but it doesn’t have strong opinions on what happens in the Backend API.

And Elm will enforce this. You can only update the view of the page via your “view” function. Your “view” function only has access to the model to decide what to render. The only actions your view can trigger are the list of messages you define. And you can only update the state in the model using your update function to change things as messages come through.

And Elm has a fantastic type system and compiler to make this all work really nicely together. To show how it works, imagine you have a “to-do list” and you have a button to mark an item as complete:

  • In your “view” function where you render the button, you can set an “onClick” event.
  • The “onClick” event will trigger a message that something has happened. You decide the names of the possible messages that can happen, so we might chose a name like “MarkToDoAsComplete”.
  • Because we have told Elm we have a message called “MarkToDoAsComplete”, it will force us to handle this code in our “update” function.
  • In our “update” function we write the code that updates the model, setting the current to-do as complete.
  • When a user views the page they see the button. When they click the button, the message is triggered, the update function is run, the model is updated with the to-do item now being marked as complete, and the view will update in response. By the time we did all the things the compiler asks, it all just worked.

The great thing about this is that Elm knows exactly what code is needed for all the pieces of your application to work. If you’re missing anything, it will give you a nice error message showing what to fix. This means that you never get runtime errors in your Elm code.

But even nicer than that, it means you have a great workflow:

  • You add your button, and a “MarkToDoAsComplete” message
  • The compiler tells you that message needs to be added to your list of app messages. You do that.
  • The compiler tells you that your “update” function needs to handle the message. You do that.
  • It now all works.

This “chase the compiler” workflow is what originally got me excited about the Elm language, not just the architecture – you can see Kevin Yank’s talk “Developer Happiness on the Front End with Elm” for a more detailed overview.

(As a bonus, if you do spot anything wrong, the strict framework for updating state based on actions, one at a time, allows powerful debugging tools like “time travel debugging”, where you can replay events one at a time to see their effect.)

Command Query Responsibility Separation (CQRS)

On the back end, we sometimes find a similar pattern to “data down, actions up”. It’s called “Command Query Responsibility Separation”. You separate the queries (data) and the commands (actions) into separate code paths, separate API endpoints, or even separate services.

If your back end uses an SQL database, you can think of the “queries” using SELECT statements, and the commands using INSERT or UPDATE statements.

An illustration titled "Command / Query Responsibility Separation CQRS" showing a circular flow diagram. A Database is at the top, with an arrow labelled "SELECT FROM..." pointing to a section named "Query (Data Down)", which has an arrow to "Client UI", which has an arrow to "Command (Action Up)", which has an arrow labelled "INSERT INTO" pointing back to the database.
Command Query Responsibility Separation (CQRS) encourages splitting up the “queries” (ways of reading the state) from the “commands” (ways of modifying the state). This ends up with many of the same benefits as “data down, actions up”, but for back end endpoints or services.

We use separate code paths for Commands (writes) and Queries (reads). Rather than having an API return new data after a command, we have the client repeat the full query.

And you end up with similar advantages:

  • Each endpoint has one job.
  • The endpoints don’t need to know about each other.
  • It becomes easier to debug.
  • The command endpoints and the query endpoints can adopt different scaling strategies. For example caching can be applied to the “query” endpoints.

Event Sourcing

When we talked about the Elm architecture, we saw a front end framework with a strict way to update the current state: by processing one message at a time. Event sourcing brings a similar concept to our back end, and crucially, to our data and our “source of truth”.

It’s normal for the “source of truth” in a web application to be a database that represents the current state of all of your data. For a todo list, you might have a row for each todo item, and columns to set the text of the item, whether it is complete or not, and the order it appears in the list.

That table would be your source of truth.

A diagram titled "event sourcing" with two parts. One part is labelled "Source of truth: current state" and shows a traditional database table view with a row for each to-do task, and columns for "id", "task" and "complete". The other part is labelled "Source of truth: events'. And shows a sequence of squares representing events: "Create Task", "Create Task", "Complete Task", "Create Task". Each event has associated data like an id or task text.

Event Sourcing – changing the source of truth. Traditionally in a web application the “source of truth” might be a database table that represents the current state of the application. In event sourcing, the source of truth is an event log: all of the actions that occurred, one at a time. We can use this event log to build up a view or the current state.

Event sourcing is about changing the “source of truth” to be the events that occurred. Rather than caring exactly which todo items currently exist, and if they are complete, we care about when a user created a task, or marked a task as complete, or changed the order of the tasks in a list. These are the “events”, and they are our “source” of truth.

And we can process them, one event at a time, to build up a view of the data (in event sourcing these are often called “projections” of the data). Some projections might look very similar to what we had before – a database table with a row for each todo item, a column for the item text, whether it is complete, and the order in the list.

The power of event sourcing is that we can also create other views of the data. Perhaps we want to create a trend line graph showing how many open tasks we’ve had over time. If our source of truth was the current state, we wouldn’t be able to tell you how many tasks you had open last week (or this week last year!) With event sourcing, we can go back over all of the events, and build new views of the data.

A diagram showing the flow of data in an event sourced application, from "Client Actions" to "Command Handler" to "Event Log" to "Projections" (we show two examples) to a view of the data (again with two examples).
Event Sourcing allows us to create different “projections” for different ways of viewing the data.

Client actions. We can start with actions that are triggered by the user. Event sourcing as a pattern has no strong opinions on how this is handled.

Command handler. A service takes the action coming in from the client (like ticking off a task) and decides if it is allowed. If it is, it adds it to the event log.

Event log. A log of all the events that have occurred. Often this is in a database table. In the diagram above, we show 4 events: “Create Task”, with id “1” and task “Draw Diagram”. Then “Create Task” with id “2” and task “Write post”. Then “Complete Task” with id “1”. (Meaning “Draw Diagram” is complete). Then “Create Task” with id “3” and task “Clean dishes”.

Here the diagram splits into two, because rather than just having the task list, we can process the events and generate other interesting data. These “projections” are different data to derived from the same events.

We have a “Your Tasks projection” that is a database table with “id”, “task” and “complete” columns. Based on the events, Task 1 “Draw diagram” is complete, while 2 “Write post” and 3 “Clean dishes” are not. This could be used to render a traditional to-do list UI.

And we have an “Open Tasks Graph projection” with two columns, “date” and “open tasks’. It shows how for any given date, we can see how many tasks were opened. We could use this t draw a trend line graph.

Similar to the front end with actions and state, this also opens up some powerful debugging options. We can replay the events and “time travel” to see exactly how our system responded, and look out for points where things may have gone astray.

Bringing it together: the common concepts I want my “Small Universe” framework to draw on

You probably spotted some similar themes running through the above architectures:

  • Keeping code to fetch data and code to process actions separate
  • Having a way to get the “current state” for a page from our data
  • Always rendering the pages based on that current state
  • Allowing the views to trigger actions or “events”
  • These events being tracked, and considered our source of truth
  • Responding to one event at a time to update our application data

Following these allows us to write code which is simpler – each function is focused on either fetching data for the current view, displaying the current view, or processing an action. That makes code easier to understand, easier to test, and simpler to debug.

Making sure our data is updated one action at a time also opens up potential for time travel debugging, which is an incredibly powerful feature for you as a developer when you’re investigating how something went wrong. It also leaves the door open for new features that you can build, being able to take full advantage of all the past user actions.

Finally, by strictly defining the shape of your state, and the set of actions (or “events” or “messages”) that are possible, you can have the framework and compiler do a lot of work for you, ensuring that if a button exists, it has an action, and the action updates the state, and the view reflects the updated state.

So you can find a very productive workflow where you start adding one new line of code for your feature, and the compiler will guide you all the way to completing the feature as valid code, and you can be relatively confident it’ll work.

So, for this “Small Universe” framework I’m starting, I am taking inspiration from these architectures to try build something that leads you to write code that is easy to write, understand, test and debug. Something that uses events as a source of truth to make it easy to build new features that process previous actions into features or views we hadn’t imagined up front. And something that leads to a happy and productive workflow, with the compiler able to provide ample assistance as you build out new features.

I’ll come back to describe the architecture I’m aiming for in a future post. But I hope this helps you understand the direction from which I’m approaching this project.

Have any questions? Things that weren’t clear? Ideas you want to share? I’d love to hear from you in the comments!

Categories
Haxe Small Universe

I’m going to write a web app framework (again) (maybe)

  1. I’m going to write a web app framework (again) (maybe)
  2. Writing a framework: web application architectures I’m inspired by
A photo from a children’s book. Text: But then I realized, what do they really know? This is my idea, I thought. No one knows it like I do. And it’s okay if it’s different, and weird, and maybe a little crazy.​ I decided to protect it, to care for it. I fed it good food. I worked with it, I played with it. But most of all, I gave it my attention.

This is my idea, I thought. No one knows it like I do. And it’s okay if it’s different, and weird, and maybe a little crazy. I decided to protect it, to care for it. I fed it good food. I worked with it, I played with it. But most of all, I gave it my attention.

“What do you do with an idea” by Kobi Yamaha and Mae Besom.

I’m thinking about writing a web framework. This wouldn’t be my first time.

Why?

  • Primarily, because I’ve got an idea, and want to explore it. The quote and photo above from “What Do You Do With An Idea” reminded me that creativity and inspiration are muscles we can train – the more we explore our ideas with curiosity, the more the ideas will keep coming.
  • I want to play with code more, try out things for the sake of curiosity and experimentation and be okay with it not necessarily building toward something as part of my day-job.
  • I’m a software engineer that enjoys building the thing to build the thing. At Culture Amp, I’m on the “Foundations” team that builds things like the design system and our common tooling, rather than working on the main products.
  • Every now and then I want to build a little side project, but get paralyzed – what should I build it with? None of the existing web frameworks I look at appeal to me – I want a single framework for front end and back-end, I want a language with a good compiler, and I want something I can grasp from front-to-back, if there is magic I want it to be magic I understand.
  • I’ve learned a lot since I last tried this. In particular, at Culture Amp I’ve been exposed to languages like Elm on the front end and concepts like Event Sourcing / CQRS on the back end. And they’re similar and interconnected and I want to see what it looks like to build a framework that builds on patterns from all of these.
  • I’ve enjoyed this kind of project in the past!

(Side note: I’m aware that being in a position where I have the time and money to indulge in this is a sign of incredible privilege… I’m still learning what it means to actively tear down the unfair systems that contribute to that privilege. There are more significant things I can invest my time in for certain. But I’ll save that for another blog post 😅)

What have I tried in the past?

  • I once was the main developer (including doing a ground-up rewrite) of a framework called Ufront. It loosely copied MVC.net on the backend, but could compile to several backend languages (thanks to Haxe). The killer feature was that you could re-use code – routing, controllers, views, models, validators – on the front end, and have seemless compiler help when calling backend APIs from the Front End.

    To this day I haven’t seen another framework attempt that level of back-end front-end integration, with the possible exception of Meteor. (Admittedly, I haven’t been looking for a while). I feel this tried to be too many things – being a generally useful backend, as well as a front-end integration layer, as well as an ORM, and Auth system, and more. At the end of the day, with the majority of the development coming from me, it didn’t have momentum for such an ambitious scope. I did build two useful education apps with it though!
  • I also attempted a more tightly scopoed project that never got off the ground, called “Small Universe“. I started this around the time I was interviewing for Culture Amp, and it was heavily influenced by Kevin Yank’s talk “Developer Happiness on the Front End With Elm“. (I now work with Kevin at Culture Amp). The idea was to have a clear data flow for a small “Universal Web App”. The page can trigger actions. Actions get processed on the back end. The back end can generate props for a page. Then the props are used to render a view. Basically, the Elm architecture but with an integrated back end / API layer.

    I liked this a lot, and built out quite a prototype, but haven’t touched it in over a year. I like this scope of “small, opinionated framework to give structure to a universal web app”.

    The first prototype I built integrated with React on the client, which I think I’d skip this time. I also think I’d go further with the data flow and push for event-sourcing (I didn’t know the terminology at the time, but I’d implemented CQRS without Event Sourcing).

    I also liked the name, and think I’ll reuse it! “Small Universe” spoke to it being a framework for “universal web apps”, where code is shared seamlessly between client and server. And also it being “small” – giving you all the building blocks for an entire app in a tight, coherent framework that is easy to build a mental model for.

So am I going to do this?

I think so! I’m interested in having a project, and “working out loud” with blog posts alongside PRs to explore the problems I’m trying to solve and the approaches I’m experimenting with.

I don’t think it’ll necessarily become anything – and there’s a good chance I’ll not follow through because life gets busy (my wife and I are expecting a second child next month 😅) but I’m interested in sharing my initial thoughts and seeing where it goes from there.

What am I optimizing for?

Scribbles from my iPad as I explored what I’m optimising for. (The section below is derived from this).
  • My own learning, curiosity and practice. (See “Why” above)
  • An Elm like workflow, but full stack. Elm has this great workflow where you can create a new button in your view. The compiler will ask you to define an action for the button. Once you define an action (like `onClick save`) the compiler will ask you to write an “update” handler for when that action occurs. When you do that you’ll write the code to handle the save. You start with the UI, and then the compiler guides you through every step needed to see the feature working. By the time the compiler has run out of things to say, your front end probably works. I want the compiler to provide that experience, with guidance, hints and safety to build features across the front and back end stacks.
  • Clear flow of data and logic. Every piece of logic in the app should have its place in the architecture, and it should be easy to find where something belongs. On the back end this looks like CQRS (Command Query Responsibility Separation) – having the code paths that fetch data for a page (Queries) be completely separate to the code paths that change data (Commands). On the front end this looks like separating out state management from the views – the page state should be parsed into a view function. There’s lots to dive into here, but I’ll save it for a future post.
  • Start small but keep options open. I want this for myself and side projects. I want something that can start small, and where the entire mental model can fit in my head. But which makes it easy to migrate to a more traditional framework, or a more advanced architecture, if the thing ever grows.
  • Keeping open the possibility for a host of technical nice-to-haves:
    • Event sourcing.
    • Offline client-side usage.
    • Multiple projections.
    • Server side rendering.
    • GDPR “right to be forgotten”.
    • Using Web Sockets for speedy interface updates.
    • Ability to have “branch previews” so you can see what a PR will look like.
  • And to call out some trade offs:
    • I’m not aiming for compatibility with React or other view layers. I think the idea I’m chasing handles data flow differently enough that it’s not worth trying to shoe-horn existing components.
    • I’m not aiming for micro services. For side projects I think a “marvelous monolith” is more sane. If the data is event sourced a future transition to micro services will be easier.
    • Not aiming to support multiple back end targets, or front ends. I’ll probably pick just one back end stack to focus on, and focus on the web (not native / mobile).
    • Not aiming for optimum performance. If I can write an API signature in a way that can be optimized and parallelized in future, I will, but I’ll probably do some naive implementations up front (such as updating all projections synchronously in the main thread).
    • Not aiming to be a general HTTP framework that can handle arbitrary request and response types. There are plenty of good tools for that job.

Let’s begin

So to start off, here’s where I’ll do my work:

Categories
Haxe

Trying to understand dependency injection

Coming closer to the end of writing my first really complex app, I’m beginning to wish I’d taken the time at the start to properly learn and grasp some of the concepts that I see other programmers using.  My client-side code uses a basic MVC pattern, but a bunch of Flash Developers use something that to me comes across as much more complex – as seen in RobotLegs or some of the Haxe ports, such as MMVC.

These frameworks are all about Inversion Of Control (and because we love acronyms: IOC).  And they use Dependency Injection (DI) to do this.  You can see already why the learning curve can be steep, especially if you haven’t come across this pattern before.

So today I googled “I don’t understand dependency injection” and got this fantastic explanation by Kevin William Pang.  I found that really helpful, and I’m sold.  I’d been wondering how the hell I was supposed to unit-test some of my more complex client code anyway… If you’re one that, like me, hasn’t understood this concept in much detail: his article is well worth a read, I won’t bother to try and explain it here.

But, I’m still not sure where the fancy dependency injection tools and frameworks come in.  I see the value in it, and I think “injecting” it into the constructor by using arguments given to the constructor seems just fine.  Meanwhile, the tools which are supposed to help, such as SwiftSuspenders or Minject seem to me to be really verbose, and not offer a lot more than the constructor method.

Maybe that style is more relevant for gaming.  Or maybe it’s benefits show up with more complex projects.  Or maybe I just haven’t been enlightened.

Either way, I’ll try to get enlightened… I think it’s worth spending time learning these patterns.  They’ve obviously gained traction for a reason, and a lot of better developers than me swear by them, so I’ll try wrap my head around it.  Maybe one day it’ll seem as clear to me as MVC, or even OOP do now (both of which took me a while at first).

If you have any suggestions / comments / explanations – I’d love to hear them in the comments section.