ESE Architecture: The Human Body of Software Design

In previous articles, I’ve hinted at our ESE architecture and how it stands apart from the traditional MVC setup. Now it’s time to dig in and explore what makes ESE both unique and refreshingly simple. It’s a structure inspired by the way our human bodies function.

The Human Body Analogy

Let’s take a break from talking about software and think about the human body for a moment.

  • Exposition: This layer is like our sensory organs—eyes, ears, skin—and the nervous system. It gathers information from the outside world and sends it to the brain for processing. Just as our senses relay external stimuli to the brain, the Exposition layer channels external stimuli into the system.
  • Service: The brain! It processes the information received, makes decisions, and determines what actions must be taken. Similarly, the Service layer is the core of the system, where the business logic resides.
  • Effect: Think of this layer as the muscles and endocrine system. When the brain decides to move a hand or release a hormone, these systems carry out the action. In the ESE architecture, the Effect layer executes the outcomes decided by the Service layer and dispatches events to keep the entire system—or “world”—informed. We strive to keep these actions atomic and free from conditional logic as much as possible.

Understanding ESE: Exposition, Service, Effect

Now, back to the world of computers. At its core, ESE (Exposition-Service-Effect) is a backend architecture that emphasizes a clean separation of concerns, making it easier to manage, test, and scale the different parts of a system. But what exactly do these layers do?

  • Exposition: This layer handles the outward-facing aspects of the system, like reacting to HTTP requests or any events. It’s the entry point where an external system or a user interact with the backend, effectively exposing the system’s functionalities to the world. The Exposition layer offers hooks, which allow you to plug code into events happening in the world. For example:

This structure replaces the traditional MVC controller:

In a typical MVC controller, you’d find both input management and business logic combined. The controller might directly interact with models, handle business logic, and render views. In the ESE example above, the Exposition layer only manages the request and delegates all logic to the Service layer. This leads to a cleaner separation of concerns, making the system more modular and easier to maintain.

Another key feature is that the Exposition layer can handle multiple sources of events and protocols:

From a development perspective, I was never satisfied with having to handle different input sources in different ways. The Exposition layer streamlines this by centralizing all inputs and dispatching to the Service layer, whether it’s an HTTP response, an event, a background job, or a cron task.

  • Service: The Service layer is where you design your service objects, set up in-memory structures, and define error types, constants, conditions, and business rules. It is the brain of the system, processing input from the Exposition layer and applying the core business logic. It’s where decisions are made.

In Pulse and Verse, we provide a base Service class (Verse::Service::Base) that handles the authorization context (similar to current_user in Rails) and passes it to the Effect layer, which is in charge of dealing with access rights. But that’s a topic for another article on our approach to authorization management.

  • Effect: Called by the Service layer, the Effect layer takes over to perform mutative operations. This might involve reading or updating a database, calling an API endpoint, publishing events, sending notifications, or any other actions resulting from the service’s operations. Here’s the twist: for every successful write operation, an event must be published to the Event Bus. So, when a new record is created, a records:created event is automatically emitted—thanks to Verse’s built-in functionality.

ESE vs MVC, key differences

You might be wondering how ESE compares to the more traditional MVC (Model-View-Controller) architecture. While both aim to separate concerns, they approach it differently.

In MVC:

  • The Model manages data and business logic, often combining the two. It hides underlying complexity, such as querying the data, and can lead to performance or security issues.
  • The View handles the presentation layer, directly interacting with the Model to display data.
  • The Controller acts as a mediator, processing user input and updating the View.

In contrast, ESE is more modular:

  • The Exposition layer handles all interactions with the outside world, acting as the system’s senses. It is also the place where we render the output, for events such as HTTP requests.
  • The Service layer focuses purely on business logic and is separated from data concerns.
  • The Effect layer manages the tangible outcomes and communicates these changes to the system, akin to how muscles execute actions.

This separation leads to cleaner, more maintainable code and makes it easier to scale complex systems.

A major pain point in MVC is having models handle both logic and effects, especially with mixed-level teams of developers. For instance, mixing query building with business code is a bad practice because it’s hard to determine query performance, as the query might be built incrementally within the business logic.

Another significant advantage of ESE is testing. You can easily replace the Effect layer with a mock version to test if your logic holds up, resulting in tests that run in microseconds without dependencies on other systems, instead of seconds. This is especially valuable in large applications, where this difference can translate to saving up to 45 minutes a day in productivity. And yes, this is a number I’ve experienced.

In Summary

We opted for ESE over MVC because it offers a more modern and modular approach to handling complex backend systems. By separating business logic from data concerns and outcomes, ESE allows us to build systems that are easier to manage, test, and scale.

Stay Tuned

In the next article, we’ll dive into why we chose Redis Stream over NATS, RabbitMQ, or Kafka for maintaining our Event Bus. Stay tuned!

Written by

Related insights

ESE Architecture: The Human Body of Software Design

Our blog ESE Architecture: The Human Body of Software Design In previous articles, I’ve hinted at...

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Data Annotation: The Secret Sauce of AI Vision

Our blog Data Annotation: The Secret Sauce of AI Vision 🔍 Ever wondered how AI learns to...

Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System

Our blog Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System Hey there! I’m...

Standardizing Medical Imagery

Standardizing Medical Imagery:
Opportunities and Challenges in a Multimodal Landscape

Medical imaging is central to diagnostics, treatment planning, and follow-up care in modern medicine. However, as imaging technologies have evolved, so have the complexities of managing the vast array of images generated. The need to standardize medical imagery has become increasingly apparent due to the wide variety of modalities, parameters, and clinical scenarios involved. Standardizing these images—while challenging—has far-reaching benefits for clinical workflow, medical research, and artificial intelligence (AI) applications. In this text, I will explore the importance of standardizing medical imagery, the challenges encountered at both the sequence and study levels, and how AI can help transform this critical aspect of healthcare.

Why Standardize Medical Imagery?

Standardization in medical imaging ensures that images acquired across different sites, scanners, and protocols are harmonized into a uniform format that can be easily accessed, understood, and utilized. The primary advantage is the seamless organization of these studies for internal and external purposes, including the proper indexing of images for clinicians, researchers, and AI developers. This becomes crucial in high-stakes environments such as large hospital networks, where multiple sites contribute to the imaging database.

Effective standardization facilitates study routing within the imaging inference pipeline, leading to faster and more reliable image processing and diagnostic workflows. Moreover, having standardized imagery helps develop robust display protocols in Picture Archiving and Communication Systems (PACS), ensuring that clinicians see consistent and meaningful representations of the anatomy. From a research perspective, the standardization of medical images exponentially increases the amount of data available for AI projects. AI models thrive on large datasets, and the aggregation of harmonized images across different hospitals allows for identifying new patient cohorts and patterns that can drive research and improved patient care.

The Role of Artificial Intelligence in Standardization

Artificial intelligence has shown immense potential in overcoming the challenges associated with the standardization of medical imagery. AI-based models can be trained to automatically identify and correct image artifacts, classify anatomical regions, and optimize images for specific diagnostic purposes. This capability is critical in improving the efficiency and accuracy of both clinical and research workflows.

For instance, recent studies show that AI models trained on large datasets—comprising millions of standardized images—can achieve up to 90% accuracy in identifying anatomical structures across modalities, improving classification tasks, and detecting anomalies in series that would otherwise go unnoticed. AI algorithms can also analyze vast amounts of data, identifying patterns and correlations that clinicians might miss. These insights are crucial for identifying new patient cohorts for clinical trials and research.

The scalability of AI makes it an ideal tool for large-scale imaging networks. With over 3.6 billion medical imaging procedures performed annually worldwide, the ability to standardize, index, and analyze this data in a consistent and automated manner is invaluable. AI-driven standardization protocols can also facilitate multi-institutional collaborations, enabling the sharing of standardized imaging data while ensuring compliance with data privacy regulations.

Challenges of Standardizing Medical Imagery

The high potential of Artificial Intelligence to standardize medical imagery also comes with its own challenges. The training and validation datasets that are necessary to AI development must be prepared by qualified radiologists, who must know how to tackle the inherent variability in how images are acquired and processed. The complexity arises at two levels: individual sequence (series) and study level.

1.Challenges at the Series Level:

  • Distorted Images: Artifacts such as streaking, banding, or the presence of braces or leads can render images suboptimal for analysis. Poor positioning of the patient or motion can further degrade image quality.
  • Non-Diagnostic Series: In some cases, certain images are acquired not for diagnostic purposes but for procedural guidance, such as during needle localization or intravenous contrast tracking.
  • Implanted Hardware and Devices: The presence of surgical implants or devices can obscure anatomical regions of interest, affecting the quality and usability of the series.
  • Intent Understanding and Anatomical Classification: Often, series contain adjacent anatomies that may or may not be relevant to the intended diagnosis. Understanding this intent is vital to classify the series appropriately. For example, a brain scan may include images of the skull, but depending on the window and kernel used (bone vs. soft tissue), the intent of the study changes.
  • Breathing and: For chest CTs, breathing phases (expiration vs. inspiration) are critical for interpretation.
  • Laterality: laterality is not always explicitly tagged in the metadata.
  • Image Planes and Reconstruction: Series are captured across multiple planes (axial, sagittal, coronal, etc.) and with varying reconstruction kernels (sharp for bone, smooth for soft tissue), contributing to the challenge of indexing.
  • Contrast Variations: The administration route and contrast phases (e.g., early arterial vs. late arterial) further complicate the standardization process.

2. Challenges at the Study Level:

  • Anatomical Selection: Not every anatomical region visualized in the series is relevant for the overall study. Selecting which anatomies to include in the study-level analysis can be difficult.
  • Contrast Protocols: Some studies might contain both pre-contrast and post-contrast series, which must be carefully categorized to avoid misinterpretation.
  • Study Protocols: The range of clinical protocols, from biopsy to stroke and lung screening, adds another layer of complexity. Each protocol may require a different set of sequences, and standardizing these can be challenging, especially without a universally accepted guideline.

Conclusion

The standardization of medical imagery, although complex, is a necessary endeavor to improve the organization, analysis, and diagnostic potential of imaging data. The challenges encountered at both the sequence and study levels can be addressed through AI-based solutions, which enable efficient classification, artifact reduction, and the optimization of image quality. By standardizing medical imagery, we can expand the pool of data available for AI research, improve patient outcomes through more reliable diagnostics, and facilitate groundbreaking medical research. The future of radiology lies in embracing these technological advancements to create a cohesive and standardized imaging landscape.

Written by
Jean Emmanuel Wattier
Head of Strategic Business

References:

  • Kalra, M. K., et al. “Artificial Intelligence in Radiology.” Radiology, vol. 293, no. 2, 2019, pp. 346-359.
  • McBee, M. P., et al. “Deep Learning in Radiology.” Radiology, vol. 294, no. 2, 2020, pp. 350-362.
  • Kohli, M., et al. “Medical Imaging Data and AI: Barriers and Challenges.” Journal of Digital Imaging, vol. 33, no. 1, 2020, pp. 44-52.

Related insights

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

RSVP Cocktail + RWiD Conversations

Cocktail + RWiD Conversations, hosted by Segmed and Ingedata You’re invited to Cocktail + RWiD...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Data Annotation: The Secret Sauce of AI Vision

Our blog Data Annotation: The Secret Sauce of AI Vision 🔍 Ever wondered how AI learns to...

Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System

Our blog Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System Hey there! I’m...

Rib Fractures on Frontal Chest X-rays

Identifying Rib Fractures on Frontal Chest X-rays:
Clinical Relevance and Diagnostic Challenges

Introduction

Rib fractures are common injuries, typically associated with blunt trauma or accidents, and may present varying degrees of severity. The radiologic identification of rib fractures is crucial for managing potential complications such as pneumothorax, hemothorax, and delayed healing. While computed tomography (CT) scans are the gold standard for diagnosing rib fractures due to their higher sensitivity and ability to detect even subtle fractures, chest radiographs, particularly frontal chest X-rays (CXR), remain a frequently employed diagnostic tool in many clinical settings. This is primarily due to their accessibility, cost-effectiveness, and use in routine evaluation of thoracic pathology. While chest X-rays are not ideal for visualizing rib fractures, they can provide valuable incidental findings, particularly in patients who undergo imaging for other reasons. Therefore, recognizing rib fractures on CXRs remains a valuable skill for radiologists.

Relevance of Detecting Rib Fractures on Frontal Chest X-rays

Incidental detection of rib fractures on chest X-rays, while not the primary modality for this purpose, can offer significant clinical insight. Rib fractures may not always be the reason for a patient’s initial presentation or imaging; however, when found, they can shift clinical management and prompt further investigation or intervention. Identifying rib fractures in such cases is particularly important when the patient has experienced unrecognized trauma, has underlying pathology that may affect bone integrity (such as osteoporosis or metastatic lesions), or when there is concern about potential complications like pneumothorax or soft tissue injury. Moreover, the early detection of rib fractures can prevent further injury and guide the clinician in pain management and patient care.

A study by Chien et al. (2020) emphasizes the importance of incidental rib fracture detection on chest X-rays, particularly in elderly patients or individuals with impaired cognition who may not report trauma or rib pain. In such populations, subtle findings can alter management, lead to more targeted investigations, and mitigate potential complications.

The Role of Artificial Intelligence in Detecting Rib Fractures on Frontal Chest X-rays

Artificial intelligence (AI) has the potential to significantly improve the detection of rib fractures on frontal chest X-rays, especially when these fractures are incidental findings. AI algorithms, particularly those based on deep learning, have shown remarkable success in identifying subtle patterns in medical images that may be overlooked by the human eye. A study by Rajpurkar et al. (2018) demonstrated that AI models could match or exceed radiologists in identifying certain thoracic pathologies, and this technology is now being adapted to detect rib fractures with increasing accuracy. AI systems can be trained on large datasets of chest X-rays, allowing them to learn the nuances of rib fractures, even in challenging locations such as the posterior ribs or areas with poor contrast. For example, one AI model trained on over 100,000 chest X-rays was able to identify rib fractures with a sensitivity of 85% and a specificity of 90%, outperforming traditional radiographic interpretation in some cases. This scale of potential could revolutionize incidental findings, reducing the number of missed fractures and ensuring timely patient care. Additionally, AI can serve as a second reader, flagging suspicious areas for radiologists to review, thereby improving diagnostic confidence and efficiency. The integration of AI into clinical practice could result in a marked reduction in missed rib fractures, potentially improving outcomes for a significant number of patients annually.

Challenges of Identifying Rib Fractures on Frontal Chest X-rays

Despite the utility of frontal CXRs, identifying rib fractures on these images presents multiple challenges. When developing AI algorithms to automatically detect rib fractures in frontal CXRs, these challenges propagate to the preparation of the training and validations sets, which need to be manually annotated by qualified radiologists. The main challenges are:

  1. Lack of Contrast and Overlapping Structures: Rib fractures are often not well contrasted on frontal chest X-rays. The presence of overlapping structures, such as the scapulae, soft tissue, and the mediastinum, can obscure subtle fracture lines, making them difficult to distinguish from surrounding anatomic structures. Additionally, the orientation of the ribs on frontal images makes it harder to visualize the posterior and lateral portions of the rib cage, where many fractures occur.
  2. Temporal Indeterminacy: Frontal CXRs typically do not provide sufficient information to distinguish between acute, subacute, or chronic rib fractures. This is due to the limited capacity to assess callus formation or the degree of bone remodeling, making it difficult to determine the fracture’s age without further imaging. A healing fracture may look similar to a recent injury in the absence of characteristic healing signs, which are often hard to visualize on standard X-rays.
  3. Fracture Location: Differentiating fractures of the anterior arch from those on the posterior arch of the ribs is particularly challenging on frontal CXRs. The rib curvature, coupled with the two-dimensional nature of the image, can obscure fracture lines, particularly in the posterior ribs, which are often superimposed on the lung fields and spine. Frontal chest X-rays tend to provide a better view of the anterior ribs but may miss posterior or lateral fractures altogether.
  4. Ambiguous Fracture Lines: Fracture lines in ribs can be subtle and have ambiguous extensions that are difficult to track. The complexity of rib anatomy, with its curvature and overlapping structures, may lead to misinterpretation. Small or incomplete fractures, particularly hairline fractures, are especially prone to being overlooked.
  5. Radiologic Report Discrepancies: Interestingly, it is not uncommon for fractures to be described in radiology reports but remain unseen in the X-ray image itself, especially for non-displaced fractures or fractures with minimal cortical disruption. Conversely, fractures that are apparent on the image may not always be identified in the radiologic report, potentially due to the subtlety of the fracture line or the presence of distracting findings in the image. This discordance highlights the variability in radiologists’ detection of fractures on CXRs and the need for careful review of images.

Conclusion

While CT scans remain the superior modality for detecting rib fractures, the incidental identification of such fractures on frontal chest X-rays carries significant clinical relevance, especially when the primary reason for imaging is not trauma-related. However, rib fractures are often difficult to detect on these images due to challenges like poor contrast, difficulty in determining fracture age, and anatomical overlap. Recognizing these limitations is essential for accurate diagnosis and appropriate patient management.

Artificial intelligence (AI) offers promising solutions to these diagnostic challenges. AI algorithms, particularly those trained on large datasets of chest X-rays, can significantly improve the sensitivity and specificity of rib fracture detection, even in challenging locations like posterior ribs. Studies have shown that AI can identify rib fractures with up to 85% sensitivity and 90% specificity, outperforming traditional radiographic interpretations in certain cases. By serving as a second reader and flagging suspicious areas for radiologists, AI has the potential to reduce the number of missed fractures and enhance diagnostic accuracy. Integrating AI into clinical practice could lead to earlier detection of incidental rib fractures, improving outcomes for many patients.

By combining traditional radiologic expertise with AI advancements, radiologists can optimize the diagnostic value of chest X-rays and provide more precise and efficient care.

Written by
Jean Emmanuel Wattier
Head of Strategic Business

Related insights

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

RSVP Cocktail + RWiD Conversations

Cocktail + RWiD Conversations, hosted by Segmed and Ingedata You’re invited to Cocktail + RWiD...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Data Annotation: The Secret Sauce of AI Vision

Our blog Data Annotation: The Secret Sauce of AI Vision 🔍 Ever wondered how AI learns to...

Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System

Our blog Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System Hey there! I’m...

Driven Microservice Architecture

Event-Driven Microservice Architecture:
Why We Chose It and How We Implemented It

Welcome to the second installment in our series on the architecture of Ingedata’s Pulse platform!

When we made the decision to move away from our previous platform, the first question we faced was, “What do we want to build?” Should we overhaul our tech stack, transitioning from Ruby to a language like Python or Go? Should we abandon the monolithic architecture and embrace the microservices trend?

Yacine
Chief Technical Officer

Before diving into these decisions, we needed to clarify the pain points we were addressing. By “pain points”, I’m not just referring to technical challenges but broader issues impacting our overall approach. Thanks to our previous platform, we have a clear understanding of our business requirements since the system is already operational and running.

The bigger question was about the non-functional qualities of our system and how to approach both the design and maintenance phases. Something built for decades, not for a few years.

Here are the key considerations we focused on:

Interoperability and API-First Design

Our previous platform was extensible, but not interoperable. While we could add modules and code in a structured manner, the system’s accessible endpoints were designed to meet the needs of the frontend application. Authorization and role management were handled at the endpoint level, with custom code filtering returned collections. If we wanted to integrate the application with other software, we would need to rewrite API endpoints and duplicate portions of the code.

For Pulse, we wanted a system that was fundamentally API-first, where the frontend would be just one of many clients utilizing these APIs. These endpoints should also be automatically documented to facilitate easier integration.

Maintainability and Flexibility

In any system, there’s a design phase where new software components and cutting-edge technologies are combined to build the application’s features. As the application matures and enters production, the focus shifts from feature development to refining and maintaining the platform. This includes reworking poorly designed code, handling edge cases, and adapting to an ever-changing business landscape.

New features often correspond with shifts in business paradigms, emphasizing the need for interoperability. For instance, if we decide to manage orders tomorrow (a feature not currently supported by our system), instead of creating a new module, we should be able to develop an entirely new platform that communicates seamlessly with Pulse. Rather than expanding the existing application, the goal is to connect multiple applications.

Resource Constraints and Knowledge Sharing

At Ingedata, we don’t have the vast resources of a company like Google, and our IT team can only grow so much. We place a high value on knowledge sharing and mentoring developers, which means we always maintain a portion of our development team as young, eager learners. While this approach fosters growth, it also means that the code produced isn’t always of the highest quality.

Agility and Resilience

Delivery speed is critical at Ingedata. We pride ourselves in being able to handle any customer project within 3 weeks. We need to be able to deploy changes quickly and respond to bugs with minimal delay. However, moving fast can sometimes mean breaking things, and with 500 employees relying on the system, downtime is not an option. One of our key goals was to design an application that could continue functioning even if certain parts of the system were down.

Considering all these factors, we decided to design our application as a microservice architecture with a twist: it’s event-driven.

The Advantages of Microservices

By breaking the application into multiple domains and databases, we make it easier for developers to quickly understand the specific domain they’re working on, even without prior knowledge. Instead of dealing with 150+ database tables, developers only need to focus on the 15 tables relevant to a specific service.

This approach also creates what I call a “contained mess.” If a young developer makes design mistakes while implementing new features, those mistakes are confined to the scope of the service/domain, making it easier to refactor the code later.

Our operations are cyclical, driven by batches of tasks we process for customers. We need a system that can scale up and down as needed. A monolithic approach would force us to scale everything simultaneously, which isn’t ideal. Microservices, on the other hand, allow us to scale specific domains as required. They also boot faster, which is a significant advantage when used with orchestration systems like Kubernetes. For example, while Rhymes (built on Rails) takes about 15 seconds to load and serve customers due to the extensive codebase, a Pulse service takes only 1.5 seconds.

Finally, microservices make it easier to adapt to changes in business. We can create new services to handle new features or shut down activities that no longer align with our goals. We have no qualms about discontinuing code if we find third-party software that does the job better and more cost-effectively.

The Event-Driven Approach

When building microservices, it’s crucial to determine how different components will interact. There’s no point in adopting a microservice architecture if everything is tightly coupled. Initially, we considered using RPC-based communication, such as HTTP or gRPC, for internal service communication. However, this approach introduces tight coupling. If one service needs to query another and the dependent service is down, it could create a cascade of errors.

Additionally, RPC calls can lead to transaction integrity issues. For example, if Service A needs to update data in Services B and C, and B succeeds while C fails, A might need to revert its changes, but B has already committed them to the database.

To address these challenges, we opted for an event-driven architecture. Unlike RPC calls, services communicate asynchronously by emitting events after changes are made. This approach reverses the dependency links between services, making each service responsible for its own state. Instead of Service A querying Service B for information, Service A listens to events emitted by Service B and updates its state accordingly.

Here’s an example:

Let’s say the OrderDelivery service needs information about a customer’s address from the Customer service. With an RPC-based approach, the code might look like this:

Using an event-driven approach and our custom-built framework, Verse, the code would look like this:

Yes, it’s a bit more complex and requires additional effort, but we’ve effectively decoupled the two services. Now, whenever a customer is created in the Customer service, a corresponding record is created in the OrderDelivery service. If the customer’s name or address is updated, OrderDelivery tracks those changes. Even if the Customer service goes down, OrderDelivery can still handle orders. Messages are sent through a streaming service, ensuring that if OrderDelivery is temporarily down, it can replay stored events and catch up when it comes back online.

It also simplifies scaling! Each service instance can manage a specific number of concurrent requests at a time (in our case, 5 per instance/CPU). With an RPC-based approach, if the Customer service begins to experience delays in responding, our Kubernetes orchestrator would scale up not just the Customer service, but also the OrderDelivery service. This happens because OrderDelivery’s order method would only be as fast as the Customer#find_customer API call, creating a bottleneck. This tight coupling can lead to significant challenges when trying to diagnose and resolve performance issues later on.

With the decision to adopt an Event-Driven architecture settled, the next question was which tech stack to use. We explored transitioning to Go or Python and came close to choosing FastAPI, which met many of our requirements and offered strong integration with AI frameworks. However, we ultimately decided to stick with Ruby and develop our own framework.

Our reasoning was multifaceted: we could leverage existing knowledge, Ruby’s ecosystem for web development remains highly competitive, and we genuinely prefer Ruby’s syntax, which, while similar to Python, offers a more enjoyable developer experience—particularly with its capacity for building DSLs.

As for Go, while it’s known for its performance, our specific needs didn’t justify the switch. We prioritized a language that resonates with our team’s expertise and offers a positive, engaging development environment.

With our decision made, it was time to build our framework. We put in extra effort and began constructing it using the ESE (Exposition-Service-Effect) stack, departing from the more traditional MVC 3-tiered setup. But that’s a story for another time—stay tuned for our next article, where we’ll dive into the details!

Related insights

The ESG Backlash: Why Ground Truth in Data is Essential for Societal Trust

Our blog The ESG Backlash Why Ground Truth in Data is Essential for Societal Trust The recent...

Passive Record: Why We Ditched ActiveRecord Pattern

Our blog Passive Record Why We Ditched ActiveRecord Pattern In our journey to build a scalable...

Talent Matching with Vector Embeddings

Our blog Talent Matching with Vector Embeddings How We Built a Semantic Search System How do you...

Data Duplication: When Breaking the Rules Makes Sense

Our blog You will Love Data Duplication! When Breaking the Rules Makes Sense In software...

ESE Architecture: The Human Body of Software Design

Our blog ESE Architecture: The Human Body of Software Design In previous articles, I’ve hinted at...

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

RSVP Cocktail + RWiD Conversations

Cocktail + RWiD Conversations, hosted by Segmed and Ingedata You’re invited to Cocktail + RWiD...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Data Annotation: The Secret Sauce of AI Vision

Data Annotation: The Secret Sauce of AI Vision 🔍

Ever wondered how AI learns to “see” things? 👀 It’s all thanks to a little magic called data annotation! Let’s break it down:

What is data annotation? 🤔

Imagine you’re teaching a toddler to recognize animals. You’d point at pictures and say, “That’s a dog!” or “Look, a cat!” Data annotation is kinda like that, but for computers. We’re basically putting labels on images so AI can learn what’s what.

Now, for the tech-savvy folks out there: we know not all AI models need data annotation (looking at you, unsupervised learning!). But for the sake of keeping things simple, let’s focus on the annotation part!

Why should I care? 🤷‍♂️

Because AI is only as good as the data it’s trained on. Remember: Garbage In, Garbage Out. If we feed AI bad data, it’ll make bad decisions!

(Some) Types of Annotations

  1. Bounding Boxes: Like drawing a rectangle around your cat in a photo. Quick and easy, but not super precise. Perfect for when you just need to say “There’s a cat… somewhere in this picture!”
  2. Polygonal Annotation: Imagine tracing the exact outline of your cat, paws and all. Takes longer, but way more accurate. Choose this when you need to know exactly where your cat ends and the sofa begins!
  3. Semantic Segmentation: This is like coloring every pixel in the image. “These pixels are cat, these are sofa, these are plant.” Great for understanding entire scenes. It’s like giving AI a very detailed coloring book!
  4. Instance Segmentation: Not only does it color everything, but it also separates individual objects. So you can tell apart each cat in a room full of cats! 😺😺😺

Of course, the type of annotation you choose depends entirely on your project’s specific needs and goals. Choose wisely! 

At Ingedata, we’ve used these techniques to help self-driving cars spot pedestrians, assist doctors in analyzing X-rays, and even help robots sort recyclables!

Remember: behind every smart AI is a team of skilled humans crafting high-quality training data. It’s the essential groundwork that makes AI magic possible! ✨

So next time you see an AI doing something cool, give a little nod to the data annotators.

This post was created through a collaborative ping-pong between Claude 3.5 Sonnet and ChatGPT 4—some humans were in CC, though! The image was generated using the FLUX.1 [dev] model.

Written by
Kevin Lottin
Business Solutions at INGEDATA

Related insights

The ESG Backlash: Why Ground Truth in Data is Essential for Societal Trust

Our blog The ESG Backlash Why Ground Truth in Data is Essential for Societal Trust The recent...

Passive Record: Why We Ditched ActiveRecord Pattern

Our blog Passive Record Why We Ditched ActiveRecord Pattern In our journey to build a scalable...

Talent Matching with Vector Embeddings

Our blog Talent Matching with Vector Embeddings How We Built a Semantic Search System How do you...

Data Duplication: When Breaking the Rules Makes Sense

Our blog You will Love Data Duplication! When Breaking the Rules Makes Sense In software...

ESE Architecture: The Human Body of Software Design

Our blog ESE Architecture: The Human Body of Software Design In previous articles, I’ve hinted at...

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

RSVP Cocktail + RWiD Conversations

Cocktail + RWiD Conversations, hosted by Segmed and Ingedata You’re invited to Cocktail + RWiD...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System

Migration from Rhymes to Pulse: Our Journey in Building a Better ERP System

Hey there! I’m Yacine Petitprez, the CTO at Ingedata, and I’m excited to kick off a series of articles where we’ll dive into the technical decisions that shape our work here at Ingedata.

At Ingedata, the development team is working on one of the most ambitious projects we’ve tackled in recent years: migrating our old ERP system, Rhymes, to a shiny new platform we’re calling Pulse. Rhymes has served us well over the past eight years, handling everything from HR and recruitment to the daily grind of project management. But as the company has evolved, so have our needs, and Rhymes just isn’t cutting it anymore.

Yacine
Chief Technical Officer

Why We Needed a Change

Rhymes was built during a time when our company’s operations were simpler. But eight years is a long time in tech, and what worked back then now feels more like a series of workarounds than a robust system. We need an ERP that not only keeps track of time and metrics for our clients but also adapts to the variety of projects and data topologies we deal with today.

The main challenge? Flexibility. We need a system that’s customizable enough to fit any project’s needs without requiring us to build custom modules for every client. And it’s not just about flexibility; it’s about ease of use too. We can’t afford to spend weeks configuring each new project. It needs to be quick and painless, something our team can handle with minimal fuss.

Another big issue with Rhymes is its lack of interoperability. The system was never designed to interact with other tools or services, meaning we’ve had to do a lot of manual work to keep things running smoothly. That’s why with Pulse, we’re going all-in on an API-based approach. This will allow us to seamlessly integrate with third-party tools and make our workflows smoother and more efficient.

Why We’re Moving Away from Rails

Our development team has been riding the Ruby on Rails train for years, and for good reason. When we built Rhymes, Rails was the obvious choice. It’s a framework that lets you move fast, thanks to its opinionated design and the vast ecosystem of gems (libraries) that cover just about every use case you can think of. But as we’ve learned the hard way, Rails also comes with some baggage.

One of the biggest issues we’ve faced is code maintainability. Rails makes it easy to get a project off the ground quickly, but if your team isn’t careful, the codebase can get messy fast. Without strict code reviews and validation processes, you end up with a mountain of tech debt that’s a nightmare to manage.

It’s true that a well-architected Rails application can be built without accumulating technical debt, but it often requires a team of highly skilled developers. The flexibility Rails offers, combined with its implicit behaviors, demands a deep understanding of the framework to avoid potential pitfalls. However, assembling a team with that level of expertise can be challenging and resource-intensive, making it less feasible in our situation.

Another challenge with Rails is how it handles business logic. The framework promotes the concept of “fat models and skinny controllers,” but this often results in models that are essentially just bloated extensions of your database records. Developers frequently end up embedding business logic directly into ORM classes instead of developing and maintaining true business models, which introduces a lot of hidden complexities and unintended side effects.

Take callbacks, for example. They might seem like a convenient way to keep your code DRY, but they can also cause a lot of headaches. Here’s a quick example to illustrate:

This might look harmless at first glance, but it can lead to some serious problems:

  • Triggering Multiple Saves: The after_save callback fires off another save on the associated User record, leading to extra database queries and potential performance issues, especially during bulk operations.
  • Infinite Loop Risk: If the User model has its own callbacks that interact with Order, you could end up with an infinite loop of save operations.
  • Unexpected Side Effects: Callbacks make the model’s behavior less predictable, turning what should be straightforward saves into complex operations that are tough to debug.
  • Violation of Single Responsibility Principle: The Order model starts taking on responsibilities that don’t really belong to it, leading to tightly coupled, hard-to-maintain code.
  • after_save vs after_commit: When a transaction is rolled back, any code in an after_save callback won’t be committed, which is ideal if you want changes to the User object to be reverted as well. However, in some cases, after_commit is the better choice to ensure changes are only applied if the transaction is successful. The decision between after_save and after_commit can easily be overlooked during code reviews, and rollback scenarios are often not covered in test cases. Since transaction rollbacks are more common in highly concurrent environments, there’s a good chance that staging won’t catch these edge cases, leaving you to discover the issues in production. This can lead to serious problems and data desynchronization when something goes wrong, often only becoming apparent after the code has already been deployed.

These issues, along with a few others, made us realize that while Rails has served us well, it’s time to move on. Our team simply can’t afford to keep fighting these battles, and the truth is, we need something more suited to our current needs.

Enter Pulse: Built with Verse, Our Custom Framework

As we embark on this migration journey, we’ve decided to step away from Rails and build Pulse using our custom framework, Verse. Why Verse? Because we wanted something that gives us more control and flexibility without the pitfalls we’ve encountered with Rails.

We built Verse because, despite our best efforts, we couldn’t find a suitable solution within the Ruby ecosystem that effectively supported an event-driven microservice architecture.

We needed something that would allow us to separate concerns in a way that Rails simply couldn’t handle, particularly in how it deals with modeling layers.

With Verse, we’ve moved to a three-tiered architecture that cleanly separates concerns between services and effects—something we’ll dive deeper into in a future post. We’ve also made Verse open-source, but we’re holding off on any major announcements until it stabilizes and we have solid documentation in place. For now, we’re still in the building phase, and things might break, but once we’re confident, we’ll be ready to share it with the world.

Over the next few weeks, we’ll be diving deeper into the technical choices we’ve made, including:

  • Event-Driven Microservice Architecture: Why we chose it and how we implemented it.
  • ESE (Exposition – Service – Effect) vs MVC (Model-Controller-View): How our three-tiered setup differs from the usual web app approach (and why it’s awesome).
  • Event Bus using Redis Stream: A risky bet that paid off.
  • Data Duplication: Why it’s the right choice and how to do it properly.
  • ActiveRecord vs PassiveRecord: Why we’re moving to PassiveRecord.
  • JSON:API: Why we like it and why we hate it.
  • Auth Concern: Record vs Endpoint security scope.
  • Open Telemetry, Sentry and Performance Monitoring: How we’re keeping tabs on everything.

Stay tuned as we pull back the curtain on our tech stack and share the insights, wins, and lessons learned from our journey in building Pulse!

Related insights

The ESG Backlash: Why Ground Truth in Data is Essential for Societal Trust

Our blog The ESG Backlash Why Ground Truth in Data is Essential for Societal Trust The recent...

Passive Record: Why We Ditched ActiveRecord Pattern

Our blog Passive Record Why We Ditched ActiveRecord Pattern In our journey to build a scalable...

Talent Matching with Vector Embeddings

Our blog Talent Matching with Vector Embeddings How We Built a Semantic Search System How do you...

Data Duplication: When Breaking the Rules Makes Sense

Our blog You will Love Data Duplication! When Breaking the Rules Makes Sense In software...

ESE Architecture: The Human Body of Software Design

Our blog ESE Architecture: The Human Body of Software Design In previous articles, I’ve hinted at...

Standardizing Medical Imagery

Our blog Standardizing Medical Imagery: Opportunities and Challenges in a Multimodal Landscape...

Rib Fractures on Frontal Chest X-rays

Our blog Identifying Rib Fractures on Frontal Chest X-rays: Clinical Relevance and Diagnostic...

RSVP Cocktail + RWiD Conversations

Cocktail + RWiD Conversations, hosted by Segmed and Ingedata You’re invited to Cocktail + RWiD...

Driven Microservice Architecture

Our blog Event-Driven Microservice Architecture: Why We Chose It and How We Implemented It Welcome...

Data-Centric AI and How to Adopt This Approach

If you follow the big names of the industry, you have probably noticed the competition on Data-Centric AI by Andrew Ng that trended this year. We, Valohai and Ingedata are so glad that there is finally proper focus on the data, its validity, and reliability, after a decade of hype, first on BIG data and then the machine learning models and AI systems. Everybody knows this; data and especially its quality is what matters. Most of the datasets aren’t that big, and good old logistic regression will do the magic most of the time yielding explainable results.

What is data-centric AI?

“Data is food for AI” is a quote from Andrew Ng used in many posts and materials this year. He means that what you train the model with is what the model can actually do – garbage in, garbage out, if you will. This is tightly related to the discussion on ethics; whether your model is biased or not is based on your training data and whether it is so on purpose. And tied to the fact that the data you have, is, if not the most, at least close to the most valuable asset you’ve got when creating AI systems.

"What we’re missing is a more systematic engineering discipline of treating good data that feeds A.I. systems,” Ng said. “I think this is the key to democratizing access to A.I.” – Fortune, November 8th, 2021

In addition to the relevance of the data, Data Scientist spends most of their time on data preparation-related tasks according to multiple surveys (Forbes & Datanami) and also according to my own experience. The focus in research and topics discussed around AI and machine learning should be proportionately this way as well. But it is not. Only 1 percent of the AI-research focuses on data.

Read more