Manufacturing relies not just on data, but on documentation as well. Equipment manuals, maintenance procedures, quality protocols, work instructions, training materials, change orders. This documentation is essential, but its sheer volume creates a practical problem: the information you need is there, but finding it quickly enough to matter is the challenge.

With Critical Manufacturing MES v11.2, we’re introducing two AI-powered capabilities designed specifically to address this reality: Ask AI and View Summary. These capabilities exist because there is a real problem to solve. Manufacturing teams need rapid access to information within dense technical documentation. AI happens to be the right tool for this job.

Documentation in Manufacturing

Equipment maintenance manuals can span hundreds of pages with detailed technical specifications, troubleshooting matrices, calibration procedures, and parts diagrams. Quality procedures reference other procedures. Work instructions evolve through revisions and changes. Training documentation accumulates as processes mature.

The knowledge is documented. The challenge is retrieval.

Traditional search mechanisms have tried to ameliorate the proliferation and scattering of information. Nevertheless, they are fundamentally bound to the user knowing the exact term they are looking for, but manufacturing problems don’t always present themselves with searchable keywords. An equipment error might require understanding the relationship between several subsystems. A quality issue might need context from multiple procedures. These situations require understanding context and meaning, not just keyword matching. They also, like a doctor making a diagnosis, have a decision tree of most likely issues and solutions, that require a context and an iteration process.

Two Capabilities, One Goal: Faster Access to Better Answers

Ask AI: Your Documentation Intelligence Layer

Ask AI allows users to interact directly with a large language model that has access to documents and attachments within Critical Manufacturing MES. Rather than searching for keywords, users ask questions and interact with documentation in natural language. The same way they would ask a colleague who knew the documentation thoroughly.

How it works in practice:

A maintenance technician encounters a recurring quality issue with a reflow oven. Instead of manually searching through the equipment manual, they open Ask AI and type: “What causes PCB discoloration?”

The AI analyzes the relevant maintenance documentation and provides a focused answer drawn from the manual, including the likely causes, the step-by-step troubleshooting procedure, and any safety precautions specific to that error condition.

The technician can follow up: “What are the recommended steps to correct the issue?”

The expertise and decision making remains with the technician. What changes is how quickly they can access the documented procedures that inform their decisions.

View Summary: Rapid Document Comprehension

View Summary generates concise summaries of selected documents or attachments, with the ability to ask follow-up questions about specific aspects of the content.

Typical use case:

A quality engineer receives a 45-page deviation report that requires review and response. Rather than reading the entire document linearly to understand its scope, they generate a summary that provides:

• The core issue being addressed
• Key findings and root causes identified
• Recommended corrective actions
• Affected products or processes

With this context established in minutes rather than hours, the engineer can then ask targeted questions: “What validation testing was performed?” or “Which suppliers are mentioned in this report?”

The summary provides situational awareness. The follow-up questions allow focused deep-dives into areas that matter for the engineer’s specific responsibility.

Why These Capabilities, Why Now

We evaluate every potential AI integration against clear criteria:

Does it solve a real, recurring problem? Documentation access and comprehension are consistent pain points across manufacturing operations. Time spent searching is time not spent solving problems.

Does it integrate naturally into existing workflows? These capabilities live within the MES interface, accessible where users already work with documents and attachments. There’s no separate system to learn, no context switching required.

Does it enhance rather than replace human expertise? The AI provides information and context. The maintenance technician still diagnoses the equipment. The quality engineer still makes the compliance judgment. The production supervisor still decides on the corrective action. We’re accelerating access to documented knowledge, not attempting to automate technical decision-making.

Is it built on a foundation we can extend? These capabilities use AWS Bedrock, providing enterprise-grade security and a platform we can build on as AI technology matures and as we identify additional high-value applications in manufacturing operations.

This release represents one deliberate step in a longer strategic direction: progressively augmenting Critical Manufacturing MES with intelligence capabilities that align with how manufacturing actually works.

Technical Foundation: Built for Manufacturing Environments

The implementation uses AWS Bedrock, which means:

• Enterprise-grade security and compliance standards apply
• Flexibility in model selection as AI capabilities evolve

The AI works with documents and attachments already managed within Critical Manufacturing MES: maintenance manuals, work instructions, SOPs, quality procedures, change orders, training materials. If it’s documented in your MES, Ask AI can help users find relevant information within it.

Using AI Responsibly in Manufacturing

We are clear-eyed about what AI does well and what it does not. Large language models excel at understanding natural language, identifying relevant information across large documents, and synthesizing that information into coherent answers. They’re powerful tools for information retrieval and comprehension.

They are not infallible. AI-generated responses should be verified against source documentation for critical decisions, just as you would verify information from any source. The AI cites its sources, allowing users to validate responses against the original documentation. This is crucial, the user is able to traverse, validate and deepen his understanding by leveraging all the original references the AI provides.

This is AI as an assistant, knowledgeable, fast, helpful, but not as a replacement for human judgment in manufacturing operations.

Getting Started

These capabilities are available now in Critical Manufacturing MES v11.2. There’s no complex setup required. If you are managing documents and attachments in the MES, you can start using Ask AI and View Summary immediately.

This is an evolving capability, and your real-world experience will help shape how we extend it.

Looking Ahead

We don’t add technology because it’s trending. We add capabilities that deliver measurable operational value, fewer errors, faster decisions, better outcomes on the shop floor. All of this leveraging a robust shop floor control system, that has years of context and experience, providing a standard and guarded way for AIs to operate.

Document intelligence solves a specific, recurring problem: rapid access to critical information buried in complex technical documentation. It’s a natural fit for AI, and it delivers tangible results. But it’s one capability among many we’re developing.

Our broader vision is clear: we are transforming Critical Manufacturing MES from a transaction system into an intelligent partner. Not by bolting on chat bots or AI assistants, but by using AI to build the actual features and solutions manufacturing operations need.

This means:

Frontline empowerment - AI that augments operators, technicians, and supervisors with the right information and insights when they need them, helping people work smarter.

Operational intelligence - AI that provides clarity and foresight across processes, equipment, and materials, identifying issues before they become problems.

Accelerated value - AI that shortens implementation timelines and reduces complexity, getting you to measurable business results faster.

The document capabilities in v11.2 are another keystone in our foundation. We are building deliberately, learning from real-world usage, and advancing where it genuinely matters.

The MES should progressively understand more about your operations and provide increasingly sophisticated support to the people running them. Not to replace their expertise, but to amplify it.

That’s the direction we’re committed to. V11.2 is the first step.

Author

Hi! My name is Ricardo Magalhães. 🤘

You can check me on LinkedIn

When we set out to migrate our Operational Data Store (ODS) from SQL Server to ClickHouse, we knew the ripple effects would extend far beyond the database layer. Our entire reporting infrastructure needed a rethink. What started as a database migration evolved into a comprehensive modernization of how we deliver information to manufacturing operations.

Why We Had to Move

Two driving forces converged to make migration inevitable:

The ClickHouse shift. As we detailed in our previous post on the ClickHouse migration, moving to ClickHouse brought performance improvements for analytical workloads. But Microsoft Report Builder and SQL Server Reporting Services (SSRS) were built for the SQL Server ecosystem. While technically possible to connect SSRS to other data sources, it’s not where the platform shines.

The Kubernetes reality. Our infrastructure is largely containerized and orchestrated by Kubernetes, though we still run our SQL Server databases in VMs. SSRS, however, comes from a different time, one of Windows Servers and IIS. Forcing SSRS into our cloud-native architecture would have been fighting against the current. We needed a reporting solution that embraced modern deployment patterns: containers, horizontal scaling, and cloud-agnostic infrastructure.

The Evaluation Process

We didn’t rush into selecting a replacement. Manufacturing systems demand reliability, and our reports are mission-critical for operations, quality control, and compliance. Our evaluation criteria included:

• Data source flexibility: Native or robust support for ClickHouse, plus the ability to consume OData feeds. Our data access strategy centers on OData exposed through our DataManager component and DataSet system entity type, so seamless OData integration was essential

• Kubernetes-friendly deployment: Docker containers, stateless architecture, and straightforward orchestration

• Report design capability: A designer that wouldn’t feel like a significant step backward from Report Builder

• Export formats: PDF, Excel, and other formats commonly needed in manufacturing

• Migration path: How difficult would it be to convert existing reports?

• Licensing and cost structure: Predictable costs that scale with our needs

After evaluating several options (including open-source alternatives and commercial platforms), Stimulsoft emerged as the best fit. It checked all our technical boxes: native ClickHouse support, robust OData connectivity, runs in containers, and offers a report designer that our team could adopt without a steep learning curve. The JS-based architecture also aligned well with our existing technology stack. An additional advantage was our existing familiarity with Stimulsoft. Our PrintableDocuments module had already been implemented using the same tool, which meant we had proven experience with its capabilities and understood its strengths and limitations in a production environment.

The reality of the market. Let’s be honest: printable reports aren’t sexy. The developer community has moved on to dashboards, real-time visualizations, and interactive analytics. Open a tech conference schedule or browse GitHub trending projects, and you’ll find dozens of dashboard frameworks but very few modern reporting tools. The market for traditional report designers has shrunk considerably. Most innovation and investment flows toward tools like Grafana, Tableau, and Power BI, not toward PDF generation engines. This made our evaluation more challenging than expected. The options were limited, and many tools felt dated or were poorly maintained. Finding a modern, actively developed reporting solution that fit our cloud-native architecture required looking harder than we anticipated.

Why Printed Reports Still Matter in Manufacturing

With real-time dashboards and interactive analytics becoming standard, you might wonder: why invest in traditional reports at all? Why not just build everything as a Grafana dashboard?

The reality of manufacturing operations tells a different story. Here’s why printable reports remain essential:

Compliance and audit trails. Regulated industries require signed, dated documentation. A PDF report with formal approval signatures is often legally required. You cannot simply attach a compliance officer’s signature to a dashboard.

Material documentation. When a material or production batch is complete, you need a comprehensive, frozen-in-time record of what happened. That batch report needs to travel with the material, be archived for years, and be reproducible exactly as it was generated. Dashboards are dynamic; reports are immutable.

Offline accessibility. Shop floor environments don’t always have reliable network access. A printed report or a PDF on a tablet works anywhere, anytime. When a quality inspector is in a warehouse or on a production line, they need information at their fingertips, not behind a login screen.

Formal communication. Some information needs to be packaged formally: reports sent to customers, presented to management, or submitted to regulatory bodies. A well-formatted PDF report conveys professionalism and authority in ways a dashboard screenshot simply cannot.

Complex, detailed layouts. Some manufacturing reports contain dense tables, detailed specifications, and precisely formatted information that spans multiple pages. These are better suited to the document paradigm than dashboard tiles.

The key is: dashboards and reports serve different purposes. They’re complementary, not competing approaches to data visualization.

Implementation: Migrating the Report Portfolio

With Stimulsoft selected, the real work began: migrating existing reports from Report Builder’s RDL format to Stimulsoft’s MRT format.

No magic conversion button. The hard truth is that there’s no automated way to convert RDL files to MRT. Each report required manual recreation. However, this wasn’t entirely bad news; it forced us to examine each report and consider whether it still served its intended purpose. For example, our Material Yield Information report, which was paginated in SSRS, evolved into a Yield Loss Information dashboard in Grafana, better suited to real-time monitoring than static reports.

Leveraging familiar concepts. Stimulsoft’s designer feels familiar to anyone who’s used Report Builder. Concepts like datasets, parameters, groups, and expressions translate cleanly. Our team could open an existing RDL report in Report Builder, have Stimulsoft designer open alongside, and reconstruct the report with reasonable efficiency.

Data source adaptation. Every query needed review. Moving from SQL Server to ClickHouse meant adapting SQL dialects, rethinking some join strategies, and occasionally restructuring queries to leverage ClickHouse’s columnar strengths. In some cases, we discovered opportunities to dramatically improve performance by rewriting queries with ClickHouse’s capabilities in mind.

Template standardization. The migration presented an opportunity to create standardized templates (headers, footers, color schemes, and typography) that could be reused across reports. Consistency in our previous Report Builder environment was not our biggest strength, with each report often having its own visual style and branding approach.

Design Improvement

If you’re going to recreate reports, you might as well make them look better.

Our old reports had accumulated over years, created by different people with varying skill levels and design sensibilities. Let’s just say that some reports looked like they were designed by someone who thought Times New Roman was a perfectly reasonable font choice, while others appeared to have been assembled by developers who believed that if gray text on a gray background was good, then slightly different gray text on a slightly different gray background was even better. And don’t get us started on the creative interpretations of our corporate blue. Some of our reports featured what can only be described as “fifty shades of blue,” ranging from navy to cyan, with each developer using their own personal definition of what “blue” meant.

Some looked professional; others were purely functional. The migration gave us a chance to apply consistent design principles:

Visual hierarchy and white space. Many legacy reports crammed information into every available pixel. We introduced breathing room, clearer section breaks, and better use of typography to guide the reader’s eye.

Consistent branding. Color schemes, logos, and font choices now align with our corporate identity across all reports. This sounds superficial, but it matters when reports are customer-facing or presented to executives.

Data visualization improvements. Where we previously relied on tables alone, we introduced charts and sparklines to make trends more obvious at a glance. Not every report needs to be a wall of numbers.

Better parameter interfaces. The UI for selecting report parameters (date ranges, facilities, product lines) received attention. Clearer labels, better defaults, and logical grouping make reports easier to run for operations staff.

Before and After: A Visual Comparison

The transformation is best seen rather than described. Below is a side-by-side comparison of one of our production reports:

The old version is functional but dense, with inconsistent spacing, color saturation, and weak visual hierarchy. The new version breathes, guides the eye naturally through the information, and presents the same data in more readable and organized layout.

Credit where credit is due: our design team worked wonders throughout this migration. They took reports that had been cobbled together over years by different developers and transformed them into polished, consistent documents. In some cases, the underlying data structure or business logic made a design challenging. Sometimes the team had to put lipstick on a pig, and they did it well. Not every report can be a masterpiece when the data itself is complex and unglamorous, but they managed to make even the most utilitarian reports look readable, functional, and visually attractive.

Dashboard vs. Report: Making the Cut

The migration forced an important strategic question: which of our existing reports should actually remain as reports? We conducted a systematic review of our entire report portfolio, asking:

• Is this information needed in real-time? If yes, it probably belongs in Grafana.

• Does this need to be printed, signed, or archived? If yes, it stays as a report.

• Is this primarily for exploration and investigation? Dashboards excel here.

• Is this a formal, structured document? Reports are the right choice.

• How often is this actually used? Some reports existed simply because they always had.

The result was a rationalization of our information delivery strategy:

To Grafana: Real-time production monitoring, live equipment status, ongoing quality metrics, and anything operators need to watch continuously. These became interactive dashboards where users can drill down, filter, and explore.

To Stimulsoft: Batch documentation, compliance reports, shift summaries, monthly production reports, and anything requiring formal structure or offline access. These remained as traditional reports.

To retirement: Quite a few reports that nobody had run in years. Migration is a great time to kill zombie reports that exist only out of inertia.

This clarity (knowing what belongs where) has made both our dashboards and our reports more focused and effective.

Lessons Learned

Looking back on the migration, a few key insights stand out:

Forced migrations create opportunities. Yes, we had to migrate because of technical necessity. But framing it as an opportunity to improve, not just replicate, made the effort worthwhile.

Don’t underestimate design time. Recreating reports takes longer than you think, especially if you’re improving them along the way. Budget accordingly.

Involve end users early. We previewed redesigned reports with the people who actually use them. Their feedback was invaluable and sometimes surprising.

Containerization pays dividends. Deploying Stimulsoft server in Kubernetes has been smooth. Scaling, updating, and managing the infrastructure is straightforward in ways SSRS never was.

SQL dialect matters. If you’re changing databases as part of your migration, don’t underestimate the query rewriting effort. ClickHouse and SQL Server speak similar but not identical languages.

The best report is the one that gets used. If nobody runs it, it doesn’t matter how well-designed it is. Be ruthless about eliminating unused reports.

Moving Forward

The migration continues, and our manufacturing operations are transitioning to a modern reporting infrastructure: ClickHouse for data storage, Stimulsoft for formatted reports, and Grafana for real-time dashboards. Each tool plays to its strengths. We are currently running both old and new reports in parallel as we methodically work through the portfolio, ensuring each migrated report meets our quality standards before we retire its predecessor.

The transition from Report Builder to Stimulsoft is giving us the opportunity to rethink how we deliver information in a manufacturing context. The reports we’ve completed so far demonstrate the benefits: faster performance, clearer visual design, and an infrastructure that fits our cloud-native architecture. The work continues, but the direction is clear and the results are promising.

For teams facing similar migrations, our advice is simple: embrace the opportunity and don’t just recreate what you had. Build what you need for the future. And be realistic about the timeline; this kind of transformation takes time, but it’s time well spent.

The Problem: When Data Becomes a Barrier

Picture this scenario: Your production manager walks into Monday morning’s operations meeting with a critical question: “Why did Line 3 underperform last week, and how can we prevent it from happening again?”

It is a straightforward business question that should have a straightforward answer. But, in reality, answering this question requires navigating tens of interconnected database tables, each with its own relationships and technical nuances. Your production manager needs deep technical knowledge of table relationships, foreign keys, and complex joins, knowledge that has nothing to do with actually running a production line.

What should be a five-minute query becomes a three-day project requiring database expertise, business context, and significant back-and-forth between IT and operations teams. Meanwhile, opportunities for improvement slip away. Decisions get made based on intuition rather than data.

This complexity doesn’t just slow down analysis; it creates a fundamental barrier between the people who understand the manufacturing process and the data that could help them optimize it. We realized that, while our MES software was technically sound, it was inadvertently creating a data democracy problem.

The Solution: A Universal Language for Manufacturing Data

To bridge this gap, we developed a Canonical Data Model (CDM) that serves as a universal translator between complex technical systems and business needs. Rather than creating another proprietary approach, we anchored our CDM in the ISA-95 standard.

Why ISA-95? It provides a robust foundation for manufacturing data models by defining clear hierarchical relationships between business processes, manufacturing operations, and control systems. The standard establishes common terminology and structures that are already familiar to manufacturing professionals worldwide, making it an ideal backbone for our democratization effort.

Our CDM leverages ISA-95’s hierarchical model of enterprise, site, area, work center, and work unit levels while transforming the traditional table-based approach into an event-driven model. This combination provides the industry-standard rigor that technical teams require, while presenting manufacturing data as a series of meaningful events that naturally align with how production teams think about their processes.

From Tables to Events: Reimagining Manufacturing Data

In manufacturing, everything happens as events. A batch starts, materials get consumed, a quality check occurs, equipment goes down, production completes. These are the natural building blocks of how manufacturing operations actually unfold. Yet traditional data models force us to fragment this natural flow across separate tables for materials, resources, production orders, etc.

This event-centric approach mirrors how manufacturing professionals naturally think about their operations. When a production supervisor investigates an issue, they are not thinking about table relationships. They are thinking about the sequence of events that led to the current situation.

The Core Innovation: Unified Events

Our Canonical Data Model (CDM) restructures data around manufacturing events rather than technical entities, presenting unified events that capture complete stories:

Material Events capture the complete story of what happened to a specific material at a specific time: when we hold or release a lot, issue quality-related losses or bonuses, track defects, handle genealogy events (splits, merges, etc.), and perform all material operations.

Resource Events record when equipment, personnel, or facilities are allocated, utilized, or experienced downtime. These events maintain context about capacity, efficiency, and availability without requiring joins across multiple resource tables.

Production Events track the creation, modification, and completion of production orders with full traceability and context, eliminating the need to reconstruct production history from fragmented work order systems.

Maintenance Events track maintenance activities, results, outcomes, and checks as complete transactions that include the what, when, where, and outcome in a single, coherent record.

Calendar Events inform the CDM about manufacturing and fiscal calendars, and shifts.

Each event is self-contained and includes all the context you need: what happened, when it happened, where it happened, who was involved, and what the result was. No more hunting through multiple tables to piece together the story.

Solving the State Management Challenge

Traditional event-driven systems face a critical challenge: maintaining state across distributed processing pipelines introduces significant complexity. Processors must track entity states, handle out-of-order events, manage state persistence, and recover from failures, while ensuring consistency across multiple event streams. This complexity often becomes a bottleneck that undermines the very benefits the event model was designed to provide.

Our solution embeds previous state directly within applicable events, eliminating the need for downstream systems to maintain complex state machines or query external state stores. A Resource State Change Event, for instance, includes both the current SEMI-E10 Resource State (“Unscheduled Down”) and the previous state (“Productive”), allowing processors to immediately understand the transition without reconstructing historical context.

Practical Advantages

This approach delivers several key benefits:

Simplified Processing Logic: Downstream systems can make decisions based solely on the event payload, without maintaining session state or querying historical data.

Enhanced Resilience: Stateless processors can handle failures more gracefully, as they don’t risk losing accumulated state information.

Improved Scalability: Without state synchronization requirements, processing can be distributed more easily across multiple instances.

Faster Implementation: Teams can focus on business logic rather than state management infrastructure.

While events are inherently temporal, the processing systems don’t need to be temporally complex. By thoughtfully including state transitions within events, we have created a model that respects the natural flow of manufacturing operations while remaining practical for real-world implementation. This design decision reflects the practical realities of building robust data processing systems that actually work in manufacturing environments.

The Technical Foundation

From a technical perspective, our CDM serves as an abstraction layer that sits between the complex underlying MES database and the analytics tools that business users interact with. The system continuously processes data from hundreds of source tables and transforms it into standardized events that maintain referential integrity while dramatically simplifying access patterns.

The ISA-95 foundation ensures that our model maintains semantic consistency with industry standards. Each event in our CDM includes ISA-95 compliant hierarchical information, temporal data, contextual metadata, and standardized attributes that enable cross-plant comparisons and trend analysis. Because we are using ISA-95 as the foundation, the model automatically supports industry best practices for equipment hierarchies, material definitions, and production scheduling concepts.

The transformation process handles the complex logic of joining related tables, resolving data quality issues, and maintaining historical consistency, while ensuring that the resulting events conform to ISA-95 principles. This means that when a production manager queries for efficiency data, they’re getting information that’s been validated, contextualized, and structured according to internationally recognized manufacturing standards.

The ISA-95 foundation also provides significant advantages for system integration and interoperability. External systems that understand ISA-95 concepts can immediately work with our CDM without requiring custom mapping or translation layers.

Real-World Impact: From Hours to Minutes

The difference in user experience is dramatic while maintaining technical rigor. Previously, answering our Line 3 efficiency question required a data analyst to write complex SQL queries joining material consumption tables with resource allocation records, work order status updates, and quality inspection results, often with inconsistent terminology and relationships.

-- Traditional approach (simplified) 
SELECT ... FROM T_MaterialHistory  
  JOIN T_ServiceHistory ON ... 
  JOIN T_OperationHistory ON ... 
  JOIN T_Step ON ... 
  JOIN T_MaterialResourceHistory ON ... 
  JOIN T_ResourceHistory ON ... 
  JOIN T_StepArea ON ... 
  JOIN T_Area ON ... 
  JOIN T_ProductHistory ON ... 
  JOIN T_FlowHistory ON ... 
  JOIN T_Facility ON ... 
  JOIN T_Site ON ... 
  JOIN T_Enterprise ON ... 
 WHERE production_line = 'Line 3'  
   AND date_range = 'last_week' 
-- Plus dozens of additional conditions and joins 

With our CDM, the same question becomes a straightforward event query:

-- CDM approach 
SELECT * FROM Material_Operations  
WHERE resource_id = 'Line 3'  
AND event_date >= 'last_week' 
AND Operation IN ('TrackIn', 'TrackOut', 'MoveNext') 

But the real transformation isn’t just in query simplicity. It is in the semantic clarity that ISA-95 provides by getting everyone speaking the same language. Production managers can directly explore efficiency patterns using business intelligence tools with terminology they already understand from industry training and best practices. Quality engineers can correlate inspection results with production conditions using ISA-95’s standardized quality frameworks. Plant managers can compare performance across facilities using consistent metrics that align with industry benchmarks.

Business Value Beyond Technical Simplification

The CDM unlocks capabilities that extend far beyond query simplification. We enable real-time operational dashboards that update as events occur, providing immediate visibility into production performance. Cross-plant benchmarking becomes possible because the standardized event structure allows meaningful comparisons between facilities with different underlying systems.

Perhaps most importantly, we democratized data exploration while maintaining industry standard rigor. Teams that previously waited days for custom reports can now investigate issues, test hypotheses, and discover insights independently using familiar ISA-95 concepts. This accelerates continuous improvement cycles and increases engagement with data-driven decision making across the organization.

The ISA-95 foundation has also significantly simplified integration with external systems. Instead of exposing hundreds of proprietary tables to business intelligence tools, ERP systems, or analytics platforms, we provide a clean, documented interface through our event-based CDM that speaks the common language of manufacturing. This standardized approach also enables seamless connectivity to Unified Namespace (UNS) architectures, facilitating real-time data sharing across the entire industrial ecosystem.

Additionally, because the CDM is fully documented, integration with state-of-the-art LLM (Large Language Models) and MCP (Model Context Protocol) is straightforward, opening new possibilities for AI-driven manufacturing insights and automation. This reduces integration complexity, improves system maintainability, and makes vendor selection easier since many industrial software solutions already support ISA-95 concepts and UNS connectivity. We will explore UNS integration patterns and best practices, as well as LLM and MCP integration approaches, in more detail in future blog posts.

Implementation Considerations and Lessons Learned

Building an effective CDM requires careful balance between simplification and completeness. We learned that the most successful approach involves close collaboration between technical teams who understand the underlying systems and business users who understand the manufacturing context.

Key success factors include establishing clear governance for event definitions, maintaining data quality standards in the transformation process, and providing adequate training and support for teams transitioning from traditional reporting approaches.

Now, here is something crucial about Critical Manufacturing Canonical Data Model: the CDM is a living document.

We don’t treat the CDM as a static specification that gets locked down and forgotten. Manufacturing is dynamic. New processes emerge, quality standards evolve, critical metrics shift, and regulatory requirements change. Our data model must evolve alongside these industry developments.

We continuously expand our event types based on real-world manufacturing requirements. For instance, emerging Industry 4.0 technologies might necessitate new Machine Learning Model Events or IoT sensor integration patterns that we hadn’t anticipated.

Traditional database schemas make adding new event types a complex undertaking, requiring system downtime, data migration, and extensive testing cycles. Our CDM approach enables seamless integration of new events without disrupting existing operations.

Most importantly, we maintain complete backward compatibility. When we release updated CDM versions with new events, your existing queries, reports, and integrations continue functioning exactly as before. Historical data remains accessible in its original format. Dashboards stay operational. AI models don’t require retraining.

This seamless evolution is enabled by our strict versioning principles: we can introduce new events and enrich existing ones with optional fields, but we never remove events or modify existing field definitions.

We are not just transforming our database architecture. We are democratizing manufacturing intelligence across the entire industrial ecosystem.

Looking Forward: The Future of Manufacturing Analytics

Our Canonical Data Model represents more than a technical solution. It is a step toward truly democratized manufacturing analytics. By removing the barriers between manufacturing expertise and data insights, we are enabling faster decision-making, more effective problem-solving, and ultimately better manufacturing outcomes.

As we continue to refine and expand the CDM, we are exploring opportunities to incorporate machine learning, predictive analytics, and advanced visualization capabilities. The event-driven foundation provides an ideal structure for these advanced analytics approaches while maintaining the accessibility that makes the system valuable to everyday users.

The goal isn’t to replace technical expertise but to amplify it by ensuring that manufacturing knowledge and data insights can work together seamlessly within established industry frameworks. In a world where manufacturing competitiveness increasingly depends on data-driven decisions, creating systems that bridge the gap between complexity and usability, while maintaining standards compliance, isn’t just a technical nice-to-have. It is a business imperative.

Our experience demonstrates that with thoughtful design and implementation grounded in industry standards, it is possible to maintain technical sophistication while improving accessibility. The result is manufacturing analytics that truly serve the people who understand manufacturing best: the teams on the factory floor who turn data into action every day.

Author

Hi! My name is Ricardo Magalhães. 🤘

You can check me on LinkedIn

Introduction

For more than 14 years, our analytics stack, anchored by Microsoft SQL Server, was a dependable part of our MES infrastructure. It powered production reports, shift summaries, and dashboards that kept operations running smoothly. It was stable, well-understood, and tightly integrated into our processes.

But as our manufacturing footprint expanded and data volumes exploded, especially with the growth of data collected from equipment, the cracks started to show:

• Dashboards lagged behind real-time events.
• Historical queries took minutes or even hours to complete.
• Maintaining indexes, partitions, and aggregation jobs became a constant battle.

And, as user expectations shifted from static reports to instant insights, our legacy architecture began to hold us back.

Our system wasn’t broken. It was outgrown. What had once been a strength was now a bottleneck.

We needed an analytics layer built for the real-time, high-volume, event-driven reality of modern manufacturing. That’s when we began exploring ClickHouse.

1. The Challenge: MES at Scale

Critical Manufacturing MES is able to generate large volumes of time-series and event-based data: resource events, operator actions, data collections, and shift-level KPIs. With the standard relational approach, we faced several growing pains:

• Slow analytics on historical production data (shift reports, OEE, downtime analysis).
• High storage costs due to the need for large indexes and partitions.
• Painful query tuning to handle growing tables (millions to billions of records).
• Limited real-time visibility. Aggregations lagged or required overnight batch processing.

2. What We Needed from Our Next-Gen Operational Data Store

Our MES setup needed a database that could:

• Efficiently handle time-series and event-based data at high frequency.
• Enable real-time dashboards and shift reports with sub-second queries.
• Be cost-effective, scalable, and easy to maintain as our data volumes doubled.
• Support fast aggregations and filtering for KPIs like OEE, scrap rate, and throughput.

3. Why We Chose ClickHouse

During our evaluation of OLAP (Online Analytical Processing) and time-series database solutions, we considered Apache Pinot, Apache Druid, PostgreSQL with TimescaleDB, and other TSDBs. Each had strengths: Pinot and Druid offered low-latency ingestion; PostgreSQL brought familiarity; TSDBs were optimized for time-series data.

ClickHouse stood out early due to its remarkably straightforward installation and lack of heavyweight dependencies. A key feature that attracted us was its native Kafka table engine, allowing seamless integration with our existing data streams.

As our use case evolved, we implemented our own Kafka sinker, which gave us finer control over batching, error handling, and ingestion tuning.

We chose the ReplacingMergeTree engine over the standard MergeTree to enable efficient deduplication of records, critical in our environment, where late-arriving or duplicate events can occur.

By carefully designing our ORDER BY keys to include accurate identifiers, we maintain both State and History tables:

• State tables reflect the latest record per entity.
• History tables retain the full change log.

This setup ensures accurate analytics while simplifying data management and minimizing complex deduplication logic at query time.

Combined with ClickHouse’s columnar storage, vectorized execution, and high compression ratios, this architecture delivers consistently fast analytical performance at scale.

Another powerful feature we leveraged was ClickHouse’s TTL (Time-To-Live) functionality. By setting TTL rules at the table level, we can automatically move or delete older data without manual intervention. This simplifies our archiving strategy considerably. No more specific jobs to purge old records. Instead, we define TTL policies directly in the table schema, allowing us to keep hot data readily accessible while seamlessly expiring or relocating cold data to cheaper storage tiers.

Additionally, ClickHouse’s robust JSON support has proven invaluable in dealing with semi-structured MES data. In our migration, we chose to flatten many of our tables and embed formerly foreign-keyed records as JSON documents within a single column. This significantly simplified our queries. What used to require complex multi-table joins can now be expressed with straightforward access to structured fields inside the JSON. With functions like JSONExtract, JSONExtractKeysAndValues, and support for nested objects, we can parse, filter, and aggregate JSON content on the fly without preprocessing or ETL transformations.

4. How We Migrated

We took an incremental, phased approach:

• Identified key use cases: production reporting, machine state tracking, and quality metrics.
• Transformed historical MES data into ClickHouse-friendly schemas (denormalized, event-based).
• Implemented a process to sink old historical data from SQL Server to ClickHouse.
• Rewrote reporting queries and dashboards to use ClickHouse SQL.
• Validated results by running SQL Server and ClickHouse in parallel.

5. The Results

Since migrating, the improvements have been dramatic:

• Query performance improved by 50–100x, even on massive datasets.
• Shift and daily production dashboards now update in real-time.
• Storage footprint dropped by over 50%, thanks to compression.
• Maintenance overhead reduced. No more managing massive indexes or partition schemes manually.
• We can now analyze long-term trends (e.g., by line, product, operator) instantly instead of waiting hours.

6. Lessons Learned for MES Teams

• Model your data around events and time. ClickHouse thrives on flat, time-series-like tables.
• It’s not a replacement for OLTP. Keep transactional operations in the MES Online Database; use ClickHouse for analytics.
• Indexing works differently! Learn about primary keys, partitions, and data skipping indexes.
• Use materialized views for fast access to high-level metrics.
• Integrate with tools like Grafana for visualizations.

Conclusion

ClickHouse has fundamentally changed the way we manage and analyze manufacturing execution data. By embracing a modern, event-driven, and time-series-friendly architecture, we’ve unlocked real-time visibility into production, simplified data modeling, and dramatically improved query performance. What once took hours can now be done in seconds with less overhead and more flexibility.

This transformation has empowered our teams with faster insights, streamlined analytics, and a scalable platform built for the future of MES. For operations facing similar data challenges, ClickHouse is more than a powerful engine. It’s a game changer for smart manufacturing at scale.

Author

Hi! My name is Ricardo Magalhães. 🤘

You can check me on LinkedIn