Topics

1. Overview and Motivation
2. Problem Statement
3. Using what is Near
4. Making it a Part of the Whole

Overview and Motivation

“Sometimes you just need to stop and think.”

There are several reasons to automate processes, I’ll give you mine: “This is a waste of time (aka boring)”, “hum, what did I have to run to do this (*checks digital notes, *then analogue notes, *then under the table)” and “Ok… param 1 run ERROR:, ok now again param1 param2 ERROR: ah yeah right param2 param1 run SUCCESS: You just wasted 30mins on this.

The first part of automating processes is understanding what are your processes and workflows. As an engineer you want to keep at the creative and engaging tasks and try to offload what is deterministic and repetitive to a known procedure. This procedure may not even be possible to fully automate, but the more you document and catalog the known path to achieve success, the more you can tweak and improve it and of course automate it.

Scripts are a very dangerous tool, a script when wrongly used is a great way to obfuscate bad practices. A script is not a spell, it shouldn’t be a set of wires glued together in order to transform the frog into a princess. It’s important to understand the process and the logical steps that you are automating, and sometimes this exercise is great to reduce the noise that exists in our work process. We will show also how to mitigate the second problematic part of script, which I call the pandemic script. This is a script that was done by someone, sometime, somewhere and has been shared a million times and each time it has mutated. Sometimes it mutates into features, sometimes it just mutates into a mutant.

This drive to improve our work and be proactive is a great impulse and has bred a whole tribe in CM “Platform Engineering”. One of their jobs is to create and help us create amazing tooling that makes our life easier and allows us to focus on the creative work. This also ensures that a script is created in a moment in time to fix one issue, but has a development lifecycle where it is maintained and improved and becomes a proper work tool.

Problem Statement

The goal for this talk was not to be an abstract presentation on automation, but to show actual helpful examples of things we found that would be or were helpful to automate.

I focused on a use-case that is near to me and that matches three out of three of my criteria to automate. The problem is that in order to be able to launch separately and debug a Connect IoT component (monitor, controller or driver), I am first, required to manually boot each one.

If for example I have a manager with a controller and a driver file-raw:

• Go to the cache of the automation monitor and for each package that I wanted to reroute to my code source, create a symbolic link with the same name of the package the manager expected. Then I would have to start the components.

Start the Automation Monitor

node ./monitor.js --dev --config='../config.downloaded.json'  --mp=88

Start the Automation Controller

• In order to start the controllers, I need to see the monitor (or go to the MES) to see what are the Ids the Automation Monitor is expecting and what is the controller package it is expecting.

node index.js --id=AutomationControllerInstance/2403060210100000002 --monitorPort=88 --config=C:/Users/<user>/Downloads/<manager>/config.downloaded.json --dev=true

Start the Automation Driver for File-Raw

• In order to start the controllers, I need to see the monitor (or go to the MES) to see what are the Ids the Automation Monitor is expecting and what is the controller package it is expecting.

node index.js --id=AutomationDriverInstance/2403060210100000004 --monitorPort=88 --config=C:/Users/<user>/Downloads/<manager>/config.downloaded.json --dev=true

NOTE: If I had more drivers or more controllers, I would have to start them as well. Of course the one I wanted to test, I would have to launch it from the solution I wanted to debug.

As you can imagine this is a very cumbersome process.

Using what is Near

The easiest way to create scripts that easy to use in Windows is to use the scripting language for Windows Powershell. Powershell has the concept of profile, where you can declare a set of functionality that is loaded whenever you open a shell. For me this is a good starting point as it allows me to have very fast and lean iterations and it’s instantly available in the shell without having to install or use third party components.

My first try at it was kind of bad, but that is fine and is part of a proof of concept approach. This is key when trying to automate processes, sometimes it’s not clear if automating a process will be easy, and if actually the process that you think is repetitive is actually repetitive and deterministic. So having a first approach to prove that the script works and is usable is helpful for me.

So I added the following script to my profile:

Function StartAutomationManagerComponents {
    param(
        [Parameter(Mandatory = $true)][string] $ManagerLocation,
        [Parameter(Mandatory = $true)][hashtable] $Controller,
        [Parameter(Mandatory = $true)][hashtable] $Driver,
        [Parameter(Mandatory = $false)][string] $LinkDir
    )

		
    # Check if I am in a Manager Directory
    $path = (Get-Childitem -Path $ManagerLocation -Include package.json -Recurse | Where-Object { $_.FullName -notlike "*\node_modules\*" } | Select-Object -First 1).fullName
    $name = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "name"
		
    if ($name -ne "@criticalmanufacturing/connect-iot-manager") {
        throw "You are not in a Directory with an Automation Manager"
    }
		
    # Check what is my config
    # Order is config.downloaded.json -> config.full.json -> config.json
    $config = Get-Childitem -Path $ManagerLocation -Include config.downloaded.json  -Recurse
    if (!$config) { Get-Childitem -Path $ManagerLocation -Include config.full.json -Recurse }
    if (!$config) { Get-Childitem -Path $ManagerLocation -Include config.json -Recurse }
		
    if (!$config) { throw "Couldn't find a valid config file" }
		
    # Now we will check for the components in the cache
    $cache = (Get-Content $config.FullName) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "cache"

    # Validate for Temp
    if ($cache.Contains('${temp}')) {
        $cache = $cache.Replace('${temp}', $env:TEMP) 
    }
	
    # Make Links
    if ($LinkDir) {
        $folders = Get-ChildItem -Path $LinkDir
    }

    $Monitor = "$($ManagerLocation)/src/" 
    $execMonitor = "node monitor.js --dev --config=$config --mp=88"
    $exec = ""
    $execControllers = ""
    $execDrivers = ""

    $exec = 'wt --maximized --title "Monitor" -d $Monitor pwsh -c $execMonitor'

    $Controller.keys | ForEach-Object {
		
        $action = "split-pane"
        if (!$execControllers) {
            $action = 'new-tab --title "Automation Controllers"'
        }
        
        if (!$Controller[$_]) {
            if (!$managerVersion) {
                $managerVersion = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "version"
            }
            $Controller[$_] = "connect-iot-controller@$managerVersion"
        }
        elseif (!$Controller[$_].Contains("connect-iot-controller")) {
            $Controller[$_] = "connect-iot-controller@$_"
        }

        $value = $Controller[$_]
        # Write-Host "Using Controller: $value for instance $_"
    		
        $packageName = $Controller[$_].Split("@")[0]
        $controllerPath = Get-ChildItem -Path $cache -Include $value -Recurse

        if (-not $controllerPath) { throw -join ("Expected ", $value, " in ", $cache, " does not exist") }
    
        if ($folders) {
            Foreach ($folder in $folders) {
                if ($packageName.Replace("connect-iot-", "") -eq $folder.Name) {

                    if (Test-Path -Path $controllerPath) {
                        Remove-Item $controllerPath  -Force -Recurse
                    }
                    new-item -itemtype symboliclink -path $cache -name $Controller[$_] -value $folder.FullName
                    break;
                }
            }
        }

        $execControllers += " ``; $action -d $controllerPath pwsh -c node src/index.js --id=AutomationControllerInstance/$_ --monitorPort=88 --config=$config --dev=true" 
    }

    $Driver.keys | ForEach-Object {
        $action = "split-pane"
        if (!$execDrivers) {
            $action = 'new-tab --title "Automation Drivers"'
        }

        if (!$Driver[$_].Contains("connect-iot-")) {
            $Driver[$_] = -join ("connect-iot-", $Driver[$_])
        }
        if (!$Driver[$_].Contains("@")) {
            if (!$managerVersion) {
                $managerVersion = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "version"
            }
            $Driver[$_] = -join ($Driver[$_], "@", $managerVersion)
        }
        $value = $Driver[$_]
        # Write-Host "Using Driver: $value for instance $_"

        $packageName = $value.Split("@")[0]
        $driverPath = Get-Childitem -Path $cache -Include $value -Recurse
	
        if (-not $driverPath) { throw -join ("Expected ", $value, " in ", $cache, " does not exist") }
    
        if ($folders) {
            Foreach ($folder in $folders) {
                if ($packageName.Replace("connect-iot-", "") -eq $folder.Name) {

                    if (Test-Path -Path $driverPath) {
                        Remove-Item $driverPath -Force -Recurse
                    }
                    new-item -itemtype symboliclink -path $cache -name $value -value $folder.FullName
                    break;
                }
            }
        }

        $execDrivers += " ``; $action -d $driverPath pwsh -c node src/index.js --id=AutomationDriverInstance/$_ --monitorPort=88 --config=$config --dev=true `;" 
    }

    $exec = $exec + $execControllers + $execDrivers

    Invoke-Expression $exec
}

The code itself it’s not important. The important part is that right away we have a lot of repeated code blocks, we have a load of inputs and honestly this could would be very hard to maintain without knowing the use-case very well. But even if all that is true, this script is fast to write and proved that scripting this task was possible and helpful and, importantly, sparked ideas on how it could be changed and remade.

Notice that we start the automation manager, we download the default packages. We then kill the automation manager and execute our script. The script will receive the ids of the components and the directory to link the packages. Then it will create an automation workspace.

Making it a Part of the Whole

Critical Manufacturing already has a structure to incorporate tooling in a way that is simple to use, that is maintainable and easy to share with control throughout the company. This is using the CM CLI, the CLI is something that everyone knows how to use and that we can add our own plugins to. This command is linked to how the development process should be used, so instead of creating a plugin from scratch, which I could have done, I used the existing cmf dev plugin.

The source code for the plugins resides at Dev Ops Plugins.

Looking at the source code it’s easy to organize and orientate our script in this tool set mindset. I want to be able to just run cmf dev iot startComponents.

In order to do this in the dev project we will create a folder to hold our iot related dev commands and two cs files. Our parent command IoTCommand and our command StartComponentsCommand;

The IoTCommand is a parent command so it’s very simple, it serves just to aggregate commands. Notice the attributes in the method header, those will be important to map to the command console execution.

[CmfCommand("iot", Id = "dev_iot")]
public class IoTCommand : BaseCommand
{
    public IoTCommand() : this(new FileSystem())
    {
    }

    public IoTCommand(IFileSystem fileSystem) : base(fileSystem)
    {
        PluginExecutionContext.Initialize(this.fileSystem);
    }
(...)

Now, our actual command. A command is divided into two parts: the Configure, where we treat and prepare all the arguments that will be used by the command and the Execute.

[CmfCommand("startComponents", Id = "iot_startComponents", ParentId = "dev_iot")]
public class StartComponentsCommand : IoTCommand
{
    /// <summary>
    /// Configure command
    /// </summary>
    /// <param name="cmd"></param>
    public override void Configure(Command cmd)
    {
        cmd.AddArgument(new Argument<IDirectoryInfo>(
            name: "managerLocation",
            parse: argResult => Parse<IDirectoryInfo>(argResult)
        )
        {
            Description = "Location of your downloaded manager"
        });

        cmd.AddOption(new Option<IDirectoryInfo>(
            aliases: new string[] { "--linkDir", "-ld" },
            parseArgument: argResult => Parse<IDirectoryInfo>(argResult),
            description: $"This specifies the directory you wish to use to create links in order to debug or to use your code."));

        cmd.AddOption(new Option<string>(
            aliases: new string[] { "--linkVersion", "-lv" },
            description: $"Specify version that will be used for the link packages, to allow using a linked package with a different version that the one defined for the runtime (note: will be used if there are no packages in the cache)"));

        // Add the handler
        cmd.Handler = CommandHandler.Create<IDirectoryInfo, IDirectoryInfo, string>(Execute);
    }
(...)

Notice how the tooling already does most of the heavy-lifting, it already allows to differentiate between optional and mandatory arguments, allows to add helpful descriptions for the user and the use of aliases. We have one mandatory arguments that is the manager location and two optional arguments, the directory where we want to link our cache and in the case of an empty cache the link version we want to give to the symbolic links.

Let’s take a look at the execution phase. The first phase is validation, we will validate the correct operating system, then we validate that the manager location is actually a manager location and we will replace a very common token in the config, by a location the filesystem understands. Also, we will interpret the configs with the correct order of precedence.

(...)

    /// <summary>
    /// Executes the specified target
    /// </summary>
    /// <exception cref="PluginException"></exception>
    public void Execute(IDirectoryInfo managerLocation, IDirectoryInfo linkDir, string linkVersion)
    {
        using var activity = ExecutionContext.ServiceProvider?.GetService<ITelemetryService>()?.StartExtendedActivity(this.GetType().Name);
        if (!OperatingSystem.IsWindows())
        {
            throw new PluginException(Cmf.CLI.Plugin.Common.Messages.OSNotSupported);
        }
        IFileInfo configJson;
        try
        {
            // Check if directory is a Manager Directory
            dynamic packageJson = managerLocation.GetFile("package.json");

            if (packageJson == null || packageJson.name != "@criticalmanufacturing/connect-iot-manager")
            {
                throw new PluginException($"Manager Location either does not contain a package.json or the packagen.json does not match the expected package @criticalmanufacturing/connect-iot-manager");
            }

            // Check config to use:
            // Start with downloaded -> full -> config.json
            configJson = managerLocation.GetFiles("config.downloaded.json").FirstOrDefault();
            configJson ??= managerLocation.GetFiles("config.full.json").FirstOrDefault();
            configJson ??= managerLocation.GetFiles("config.json").FirstOrDefault();
        }
        catch
        {
            throw new PluginException($"This Directory {managerLocation} does not contain an Automation Manager");
        }

        // Read the config.json to see where the package dependencies are
        dynamic configJsonContent = JsonConvert.DeserializeObject(configJson.ReadToString());
        string cacheLocation = configJsonContent.cache.ToString();
        // Replace temp token with temp location
        if (cacheLocation.Contains("${temp}"))
        {
            cacheLocation = cacheLocation.Replace("${temp}", Environment.GetEnvironmentVariable("TEMP"));
        }
(...)

The next phase is preparing the links. If a link directory is passed, we must change the cached package names and create symbolic links to the packages in the directory link. If there are packages in the cache we will match those with the ones in the directory link and perform just the link for those. If the cache is empty, we will link all the packages, here we can either link them with their own version or use the linked version argument version.

(...)

        #region Prepare Links

        // Check if I have to do links
        if (linkDir != null)
        {
            IDirectoryInfo cache = this.fileSystem.DirectoryInfo.New(cacheLocation);

            var cachedPackages = cache.GetDirectories();

            // If there are packages we will link the ones that match, else we will link all
            Dictionary<string, string> packagesToLink = new();

            if (cachedPackages != null && cachedPackages.Any())
            {
                foreach (var cachedPackage in cachedPackages)
                {
                    // if the package as already processed or if it us a link we will skip it
                    if (cachedPackage.LinkTarget != null || cachedPackage.Name.Contains("_CACHE_ORIGINAL"))
                    {
                        Log.Debug($"This packaged is already a link, so it will not be relinked {cachedPackage.Name}");
                        continue;
                    }

                    // We will go to each cached package and see the name and the version of the package
                    IFileInfo cachedPackageJson = cachedPackage.GetFiles("package.json").FirstOrDefault();
                    dynamic content = JsonConvert.DeserializeObject(cachedPackageJson.ReadToString());
                    packagesToLink.Add(content.name.ToString(), content.version.ToString());
                }
            }

            // Check matches between cache and folders in the directory
            // Rename existing folders to _CACHE_ORIGINAL_
            // Make Links
            var linkedDirectories = linkDir.GetDirectories();
            List<IDirectoryInfo> createdLinkedDirectories = new();
            foreach (var linkedDirectory in linkedDirectories)
            {
                IFileInfo linkedPackageJson = linkedDirectory.GetFiles("package.json").FirstOrDefault();
                dynamic content = JsonConvert.DeserializeObject(linkedPackageJson.ReadToString());

                // We will check if the package name exists in the list of cached packages
                // Or if there are no cached packages in that case we link all
                if (!packagesToLink.Any() || packagesToLink.ContainsKey(content.name.ToString()))
                {
                    // Remove scope
                    var packageVersion = packagesToLink.Any() ? packagesToLink[content.name.ToString()] : linkVersion ?? content.version.ToString();
                    var packageName = $"{content.name.ToString().Split("/")[1]}@{packageVersion}";
                    var dirName = Path.Join(cache.FullName, packageName);

                    // If it's a downloaded package we will rename it
                    // Else we will just delete it
                    if (packagesToLink.Any()) { this.fileSystem.Directory.Move(dirName, dirName + "_CACHE_ORIGINAL"); }
                    var directory = this.fileSystem.DirectoryInfo.New(dirName);
                    if (directory.Exists) { directory.Delete(); }

                    // Create a Link between the cache and our package with the correct name <packageName>@<cachedPackageVersion>
                    this.fileSystem.Directory.CreateSymbolicLink(dirName, linkedDirectory.FullName);
                    createdLinkedDirectories.Add(this.fileSystem.DirectoryInfo.New(dirName));
                }
            }
        }
        #endregion
(...)

I really wanted to avoid the need for me to know the ids of the components and the arguments to start them. So I decided a different approach, I start the automation monitor for a dummy execution and sniff the output of the console for the parts where the monitor tells me what I need to start. Then I kill the monitor and perform a clean start. If you are curious for the messages being intercepted, try to start only the monitor and see what it logs, and you will see the messages we are intercepting. We wait a set amount of time for the monitor to finish and then kill the process.

(...)
        #region Dummy Execution Phase

        // Start Monitor
        // Intercept Messages regarding Drivers and Controllers
        // We will sniff the console output of the monitor and see what are the components that must be started and their Ids and packages
        // Then the monitor we will kill the monitor and start everything
        CommandLineRunner runMonitor = new(fileSystem.DirectoryInfo.New(Path.Join(managerLocation.FullName, "src")));

        List<ComponentExecution> componentExecutions = new List<ComponentExecution>();
        string idInForkPattern = "(?<=id=')[^']+";
        string pathInForkPattern = "(?<=path=')[^']+";
        string idInArgumentsPattern = "(?<=--id=)(.*?)(?= --managerId)";
        string argumentsInArgumentsPattern = "--id=(.*)";
        bool killedMonitor = false;
        Action<string> monitorConsoleHandler = output =>
        {
            if (output != null)
            {
                // In this Message we will know the Id of the component and the package it needs to boot
                if (output.Contains("Fork requested *NOT* executed"))
                {
                    var id = Regex.Match(output, idInForkPattern).Value;
                    var path = Regex.Match(output, pathInForkPattern).Value;

                    componentExecutions.Add(new ComponentExecution(id, path));
                }
                // In this Message we will know the arguments required to execute the component
                else if (output.Contains("Arguments: "))
                {
                    var id = Regex.Match(output, idInArgumentsPattern).Value;
                    var arguments = Regex.Match(output, idInArgumentsPattern).Value;
                    var comp = componentExecutions.FirstOrDefault(x => x.Id == id);
                    comp.Arguments = Regex.Match(output, argumentsInArgumentsPattern).Value;
                }
                else if (output.Contains("error"))
                {
                    runMonitor.KillProcess();
                    throw new PluginException($"Something went wrong when booting up the Monitor, please fix it before running the command again - {output}");
                }
                else if (output.Contains("Processes and packages processed"))
                {
                    // When all packages are processed we will kill the Monitor Process that we started
                    runMonitor.KillProcess();
                    killedMonitor = true;
                }
            }
        };

        // Grab a random port in the 81-90 range
        Random random = new();
        int randomPort = random.Next(81, 90);
        var monitorExec = $"node {Path.Join(managerLocation.FullName, "src", "monitor.js")} --dev --config={configJson.FullName} --mp={randomPort}";
        runMonitor.Run(monitorExec, waitForExit: true, outputHandler: monitorConsoleHandler, logOutputOnError: false, errorCodeToIgnore: -1);

        // We will wait a set amount of time for the monitor process to log all components
        try
        {
            WaitFor(60, () =>
            {
                if (killedMonitor)
                {
                    // Wait for the monitor to finish
                    return true;
                }
                return false;
            }, "No Fork was Requested");
        }
        catch (Exception)
        {
            runMonitor.KillProcess();
            throw new PluginException("No Fork was Requested from monitor in the allotted 60s period");
        }

        Log.Debug($"Number of Components found - {componentExecutions.Count}");

        #endregion
(...)

In the last phase we construct the windows terminal workspace, similar to what we were already doing in the script.

(...)
        #region Run All Components

        CommandLineRunner runAll = new(fileSystem.DirectoryInfo.New(Path.Join(managerLocation.FullName, "src")));
        var exec = $"wt --maximized --title \"Monitor\" pwsh -c {monitorExec}";

        // For each component type we will open a new tab and for each component of the same type we will spit the tab
        ComponentType? previousType = null;
        foreach (var component in componentExecutions.OrderBy(x => x.Type))
        {
            var action = "split-pane";
            if (previousType != component.Type)
            {
                action = $"new-tab --title \"Automation {component.Type}\"";
            }
            else
            {
            }
            exec += $" ``; {action} pwsh -c node {component.Path} {component.Arguments}";

            previousType = component.Type;
        };

        // Controller - Collect Ids and Package versions
        // Driver - Collect Ids, Package Names and Versions
        // Start Controllers and Drivers
        Log.Debug($"Executing Command - {exec}");
        runAll.Run(exec, waitForExit: false);

        #endregion
(...)

In order to execute our command

To just start the components:

cmf dev iot startComponents <managerlocation>

To start the components and make links, with an existing cache:

cmf dev iot startComponents <managerlocation> -ld <linkDirectory>

To start the components and make links, without an existing cache, but with a specific version:

cmf dev iot startComponents <managerlocation> -ld <linkDirectory> -lv <linkVersion>

With this we finish the loop, we started with an idea on how to improve a part of our workflow that was very cumbersome and prone to error and now we have a code block with head, torso and members, this way closing the anatomy of a script.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview and Motivation
2. Generic Package

Overview and Motivation

It is very common to want to deploy a customization package that is based not on a full customization projects, but holds only some prepacked artifacts. For example when we want to use pre-release versions of product packages but don’t want to include the product source code in our project.

In this example I wanted to generate a deployable package outside of a project scaffolding. This example is mostly valid for Connect IoT / Factory Automation packages, but it can be tweaked for other type of artifacts.

Generic Package

Let’s first create our root package. A root package requires only the existence of a cmfpackage.json that describes this as a root package. We will also add an entry to our package that will hold the tgz files.

{
  "packageId": "Cmf.Custom.DeployTGZ",
  "version": "1.0.0",
  "description": "Package root for deploying TGZ",
  "packageType": "Root",
  "isInstallable": true,
  "isUniqueInstall": false,
  "dependencies": [
    {
      "id": "Cmf.Environment",
      "version": "<%= $CLI_PARAM_MESVersion %>",
      "mandatory": false
    },
    {
      "id": "Cmf.Custom.DeployTGZ.Packages",
      "version": "<%= $CLI_PARAM_MESVersion %>",
      "mandatory": false
    }
  ]
}

Now lets create a folder to hold our generic package. Let’s create a new cmfpackage.json.

{
  "packageId": "Cmf.Custom.DeployTGZ.Packages",
  "version": "1.0.0",
  "description": "Package to deploy TGZ",
  "packageType": "Generic",
  "isInstallable": true,
  "isUniqueInstall": false,
  "targetDirectory": "UI/Html",
  "targetLayer": "ui",
  "contentToPack": [
    {
      "source": "projects/*.tgz",
      "target": "runtimePackages",
      "ignoreFiles": [
        ".npmignore"
      ]
    }
  ],
  "xmlInjection": [
	"steps.xml",
    "ui.xml"
  ]
}

The generic package allows us to configure the build steps and steps to install as we wish. In this case we will not have build steps, but we will inject our deployment steps, in order to deploy the tgz files to the connect iot repository. The content to pack will dictate that the tgz files will be in the projects/ directory and will be deployed to a folder called runtimePackages. Then we will add a set of injectable steps, these will be joined and parsed to create the deployment metadata.

The ui xml will emulate the devops center options and steps in the GUI.

ui.xml:

<ui>
<wizardStep id="Deployment.ConnectIoT" order="100" title="IoT Configuration" type="generic" requiresValidation="true">
  <groups>
	<group id="Deployment.Steps.ConnectIoT.Directory" order="0" title="DIRECTORY REPOSITORY" type="generic" requiresValidation="false">
	  <variables>
		<variable name="Local.Repository.Directory.Enabled" readOnly="false" isRequired="false" valueType="Boolean" label="Is Enabled" default="false" />
		<variable name="Local.Repository.Directory.Location" readOnly="false" isRequired="false" valueType="Text" label="Location" placeholder="Example: C:\repositoryLocation" default="C:\repositoryLocation" />
	  </variables>
	</group>
	<group id="Deployment.Steps.ConnectIoT.Server" order="1" title="NPM SERVER REPOSITORY" type="generic" requiresValidation="false">
	  <variables>
		<variable name="Local.Repository.Server.Enabled" readOnly="false" isRequired="false" valueType="Boolean" label="Is Enabled" default="false" />
		<variable name="Local.Repository.Server.Address" readOnly="false" isRequired="false" valueType="Text" label="Address" placeholder="Example: http://127.0.0.1:4873" default="http://127.0.0.1:4873" />
		<variable name="Publish.Tag" readOnly="false" isRequired="false" valueType="Text" label="Tag" placeholder="Example: latest" default="latest" />
		<variable name="Local.Repository.Server.UserName" readOnly="false" isRequired="false" valueType="Text" label="Registry User" placeholder="Example: admin" default="admin" />
		<variable name="Local.Repository.Server.UserPassword" readOnly="false" isRequired="false" valueType="Password" label="Registry User Password" placeholder="Example: password" default="password" />
		<variable name="Local.Repository.Server.UserEmail" readOnly="false" isRequired="false" valueType="Text" label="Registry User Email" placeholder="Example: admin@companyname.com" default="admin@companyname.com" />
	  </variables>
	</group>
  </groups>
</wizardStep>
</ui>

The steps xml will be responsible to determine the actions to be performed. In this case we want not only to deploy but also to generate the repository index.

steps.xml:

  <steps>
    <!-- ONLY VALID For VM - Will be ignored in a container installation
    <step type="DeployFiles" contentPath="runtimePackages/**" />
    <step type="DeployFiles" contentPath="*.ps1" />
    <step type="DeployFiles" contentPath="*.bat" />
    <step type="Generic" onExecute="$(Package[Cmf.Custom.DeployTGZ.Packages].TargetDirectory)/runtimePackages/ValidateIoTInstall.ps1" />
    <step type="Generic" onExecute="$(Package[Cmf.Custom.DeployTGZ.Packages].TargetDirectory)/runtimePackages/PublishToRepository.ps1" />
    <step type="Generic" onExecute="$(Package[Cmf.Custom.DeployTGZ.Packages].TargetDirectory)/runtimePackages/PublishToDirectory.ps1" /> 
    -->
    <step type="DeployRepositoryFiles" contentPath="runtimePackages/**" />
    <step type="GenerateRepositoryIndex" />
  </steps>

Download Example Package

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Presenter

We had an amazing talk by our very own David Sora. He is one of the most driven and focused members of Critical Manufacturing. So it was no surprise to find he was an avid reader. He was challenged with presenting a book that made him be better. He choose one of the most requested books in the Gutenberg library, Atomic Habits.

Book - Atomic Habits

Walkthrough

David gave us a vivid account of the impact of the book on his own life and at the same time walking us step by step in the main concepts and ideas of the book. The major impact of small and incremental changes is a big eye opener. Everyone that is trying to build good habits and kick off bad habits should definitely grab this book.

The idea of compound interest and that a 1% improvement generates a 37x improvement after just one year, is a unbeatable argument to nay sayers. The contrast between the proof / identity methods is also a great brain poke to everyone that wants to understand the construction of the habit.

The big lifecycle of Cue -> Desire -> Reaction -> Reward is very important to understand ourselves and our relation between what we do and why we do it.

It was not just a presentation we also had a big Give Away of four Atomic Habits books and a big lunch!

This was an awesome way to kick-off the Gutenberg Library. Stay tuned for more presentations.

Final Notes

This event was driven in the framework of the [Productivity Month] Boost your Productivity: Unlock your Potential initiative, created by the Learning Lab team.

Topics

1. The problem
2. What is node-gyp

The Problem

The other day, everything broke down… Depending on external systems makes us gain a lot of features, maturity and experience that was gathered by third parties, but you abdicate control. It’s not uncommon, for small updates to generate disruption in our workflow. Being mindful of these trade-offs should be at the forefront of the decision making process of every developer, questions like stability, updates, community and documentation are key factors when deciding what dependencies to chose and build on.

Knowing how to read is also an important feature of a software engineer, mainly reading logs. We wasted some minutes not doing that and jumping to conclusions, but when we stopped to read the stack trace, we saw the following message:

This was a strange issue, as it was working on our local machines. So we already knew that the problem was either in the pipeline or in the dependencies installed in the system that ran the pipeline.

Looking carefully, we can see that the problem was on a dependency called node-gyp. The dependency finds the vs tools to build C++, but it subsequently fails when it reaches the Python312 execution.

A quick search on the internet gave us some clues distutils is removed in Python 3.12 and in the official documentation. Looking further [DOCS] The latest version of python (3.12) doesn't support node-gyp, so we know that the node-gyp latest version supports Python312. Stupid question, what is the latest version and are we on it?

In the node-gyp github releases we know the latest version is 10.0.1. Let’s see what is ours by running in our package npm list node-gyp, we could also see globally by running npm list node-gyp -g

We see that we have version 9.4.1 being used in the angular/cli@15.1.0. In the github history we can see that the fix was applied to gyp-next and this was released in gyp-next releases v0.16.0, looking back at node-gyp we can see that the bump to gyp-next v0.16.0 only happens for node-gyp v10.x.

In order to fix this, we could update node-gyp but node-gyp is a sub-dependency of a product dependency, which may be impacted by a version change. The best approach is to follow the python official documentation and install the setup tools.

Run:

python3 -m pip install setuptools

In order to check your python version:

python3 --version

If your version is above 3.1.12 you will be impacted.

What is node-gyp?

Why does node which is a javascript engine, need python. Actually, the dependency that we’ve been talking about node-gyp is a cross-platform command-line tool for compiling native addon modules for Node.js. This allows developers to write code in C, C++, or Rust and integrate them with JavaScript running in Node.js. node-gyp is a wrapper around GYP (Generate Your Projects), a meta-build system that generates project files for use with various IDEs and build systems, such as Visual Studio, Xcode, and make 12.

Notice, how this matches what we saw with our original problem, where the fix was not only in node-gyp but was in gyp. So actually, it’s not node-gyp that needs python, but it’s the parent dependency gyp that is built on python.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. Example
3. Conclusion

Overview

Scripting is how you fine tune your workflow to your needs. Maybe you are always running the same commands, or maybe it’s a hassle to change between projects. In this example, I am going to use a real scenario that I had and how I addressed it with very little effort.

But I already have my own tools to optimize my flow and I love it how it is, then this is not for you. Keep up the great work!! The goal is just to expand what is possible in order for you to get more autonomous on improving your flow.

In this tutorial we will use powershell core and windows terminal, there are a million other tools, but these are what I am currently using in my work flow.

Important: Scripting is a double edged knife. It is there to improve your work flow and to abstract complexity, but there’s always a danger in abstracting complexity and losing control on the actions that you are doing. Also, be very mindful that you are using what exists and being more efficient, not trying to create a parallel solutions to problems everyone has. The goals is not create another “magic solution”.

Example

In my use case, I was tired of shifting between different local environments on my team. In my team, I have to boot the start the gui, the host, the message bus, a simulator application and an automation manager just to run the tests!!! 😩 Then, after all that I was always shifting between tabs on the console, because sometimes I was focusing on one component then on another. So I set about creating a workspace, that had to be predictable, easy to use and easy to switch between projects and I had very little time (no one gives you a sprint to improve your workflow, maybe they should 😜)

First I read the doc Windows Terminal. So, now I knew that splitting tabs and creating new tabs was super easy! Next step what do I actual care about, because that has to have a tab just for it. I work mostly with Connect IoT, so for sure the Automation Manager deserves a tab, then the host, the rest I don’t care and doesn’t give that great logging anyhow.

Powershell Profile

Profiles are a powerful tool to add things that you want to always exist in the context of your shell. Specific functions or variables that you always want accessible The sky is the limit.

Finding your profile, run a powershell console:

$PROFILE | Select-Object *

But I don’t have any:

if (!(Test-Path -Path $PROFILE)) {
  New-Item -ItemType File -Path $PROFILE -Force
}

In the text editor of your choice, i.e notepad ++ open the profile and let’s start cracking.

Customizing the Profiler

First thing I did was starting with one project (let’s call it project X). I declared the paths that I thought were more or less constant as constants on the top of profile:

$env:REPOS = "C:\cmf\repos\DS\__PROJECT__"

$env:AM = "C:\Users\jroque\Downloads\ActiveTestManager"
$env:VF = "C:\cmf\repos\DS\Library\src\VirtualFactory"

$env:BIZ_LOCAL = "\LocalEnvironment\BusinessTier"
$env:MB_LOCAL = "\LocalEnvironment\MessageBusGateway"
$env:HTML_LOCAL = "\Features\Cmf.Custom.__PROJECT__\Cmf.Custom.__PROJECT__.HTML"

Then I set about creating a function:

Function StartProject($Root, $Project, $DB) {
	
	wsl -- sudo service docker start
	If (docker ps --filter 'expose=1433' --format '{{.ID}}') { 
     docker stop (docker ps --filter 'expose=1433' --format '{{.ID}}')
	}

	docker start $DB 
	Write-Host  $env:REPOS
	Write-Host  $Root
	Write-Host  $env:REPOS.replace("__PROJECT__", $Root)
	$Biz = Join-Path -Path $env:REPOS.replace("__PROJECT__", $Root) -ChildPath $env:BIZ_LOCAL;
	$MB = Join-Path -Path $env:REPOS.replace("__PROJECT__", $Root) -ChildPath $env:MB_LOCAL;
	$Html = Join-Path -Path $env:REPOS.replace("__PROJECT__", $Root) -ChildPath $env:HTML_LOCAL.replace("__PROJECT__", $Project);
	$AM = Join-Path -Path $env:AM -ChildPath "\scripts";
	
	wt --maximized --title "Local Env" -d $Biz pwsh -c .\Cmf.Foundation.Services.HostService.exe `; split-pane -d $Html pwsh -c ng serve `; split-pane -H -d $MB pwsh -c .\Cmf.MessageBus.Gateway.exe `; new-tab  --title "VirtualFactory" -d $env:VF pwsh -c npm run start `; new-tab --title "Automation Manager" -d $AM pwsh -c .\StartConsole.bat 
}

A function is something that if registered on the profile can then be executed after the shell is opened. My function has three parts, one just to find the paths of all the components I need to boot up and another to use the windows terminal CLI.

First I check if there is any docker container running that is occupying the default DB port. I am doing this as my components require the database to be up.

In the windows terminal CLI I am saying that I want launch a new terminal with the path to business and to execute the command .\Cmf.Foundation.Services.HostService.exe. Then I want to split the pane in the path of Html and run ng serve, the split the splitted panel horizontally in the path of the MessageBus and run .\Cmf.MessageBus.Gateway.exe. Then I want a new tab for a simulator called Virtual Factory and lastly another new tab for the automation manager.

I created a small function that the only thing that it does, is inform the user of what exists and asks for a choice. This function will orchestrate the project to start:

Function StartLocalEnv {
	# Define menu options
	$options = @{
		'Active' = 'Active'
		'Baseline' = 'Baseline'
	}

	Write-Host "Select one of the supported Local Envs:"
	# Display menu to the user
	foreach ($key in $options.Keys) {
		Write-Host "$($options[$key])"
	}

	# Get user input
	$choice = Read-Host "Select one of the options"

	# Process user choice
	if ($options.ContainsKey($choice)) {
		Write-Host "You selected $($options[$choice])"
		switch ($choice) {
			"Active" {
				StartProject "Active" "Active" "ACTIVE_v1-DB"
			}
			"Baseline" {
				StartProject "Baseline" "Baseline" "Template_v2-DB"
			}
			default {
				Write-Host "Invalid choice. Please select a valid option."
			}
		}
	} else {
		Write-Host "Invalid choice. Please select a valid option."
	}
}

Now let’s see it work.

Notice also that it is resilient, I can kill the console and start it again and it will keep the same context.

Conclusion

I know what you’re thinking, but it’s still missing so much. The goal here is to show the sandbox, now you need to tweak it and do what works best for you.

Author

Hello, my name is Olivério Sousa ✌️

I am a TechLead at Deployment Services.

Skills: Tech Leading

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. Default Config
3. Reading the logs

Overview

Connect IoT is a back-end application, as such logging is one of the most crucial parts of understanding its execution. Logging is what brings visibility and transparency to what is happening in the application. It’s common that iot, processes a lot of information and that you need to perform some investigation in order to spot what caused an error or an unpredicted behavior. In this blog post, we will walkthrough how the log is organized and how you can read it.

Default Config

When an Automation Manager is downloaded the system comes with a default strategy and structure for logging. The template for all Automation Managers can be found in a Configuration /Cmf/System/Configuration/ConnectIoT/ConfigurationTemplate/ and can be changed for each Automation Manager in the Configuration button and editing the json.

Focusing in the subset for logging:

{
	"logging": [
		{
			"type": "Console",
			"options": {
				"level": "debug",
				"prettyPrint": true,
				"colorizeMessage": true
			},
			"applications": [
				"*"
			]
		},
		{
			"id": "ControllersAndDrivers",
			"type": "File",
			"options": {
				"filename": "${applicationName}_${date}.log",
				"dirname": "${temp}/ConnectIoT/Logs/Instances/${entityNameNormalized}/${componentId}",
				"level": "debug",
				"timestampFormat": "HH:mm:ss.SSSSS",
				"maxSize": "10m",
				"maxFiles": "30d"
			},
			"applications": [
				"AutomationController",
				"Driver*"
			]
		},
		{
			"id": "ManagerAndMonitor",
			"type": "File",
			"options": {
				"filename": "${applicationName}_${date}.log",
				"dirname": "${temp}/ConnectIoT/Logs/${applicationName}",
				"level": "debug",
				"timestampFormat": "HH:mm:ss.SSSSS",
				"maxSize": "10m",
				"maxFiles": "30d"
			},
			"applications": [
				"AutomationMonitor",
				"AutomationManager"
			]
		},
		{
			"id": "Jobs",
			"type": "File",
			"options": {
				"filename": "Job_${jobId}_${date}.log",
				"dirname": "${temp}/ConnectIoT/Jobs",
				"auditFile": "${temp}/ConnectIoT/Jobs/.audit.json",
				"level": "debug",
				"timestampFormat": "HH:mm:ss.SSSSS",
				"maxSize": "10m",
				"maxFiles": "30d"
			},
			"applications": [
				"AutomationJob"
			]
		}
	]
}

NOTE: All settings are explained in the documentation under Help Logging Structure Right away we can see that we have multiple entries that are logging different applications. Let’s go entry by entry.

{
	"logging": [
		{
			"type": "Console",
			"options": {
				"level": "debug",
				"prettyPrint": true,
				"colorizeMessage": true
			},
			"applications": [
				"*"
			]
		}
	]
}

The first entry is of type Console, this type controls what is shown when you run the Automation Manager as a console (i.e running StartConsole.bat which is included in the .zip installation package). By default, applications is * this means it will log all the existing components of ConnectIoT. In options, it is set as level debug (possible levels are debug, info, warn, error in order of verbosity), also we have prettyPrint and colorizeMessage, these are just to format the message into a more reader friendly output.

{
	"logging": [
		{
			"id": "ControllersAndDrivers",
			"type": "File",
			"options": {
				"filename": "${applicationName}_${date}.log",
				"dirname": "${temp}/ConnectIoT/Logs/Instances/${entityNameNormalized}/${componentId}",
				"level": "debug",
				"timestampFormat": "HH:mm:ss.SSSSS",
				"maxSize": "10m",
				"maxFiles": "30d"
			},
			"applications": [
				"AutomationController",
				"Driver*"
			]
		}
	]
}

We can see the second entry has more things going on. This log applies to type File, so we know this log will be persisted in a file. Now the applications are not set to everything, but this entry in the log configuration only applies to the Automation Controller and to all Driver* (notice the *, to signal all applications that start with Driver).

In the options, we now have some more interesting arguments, the dirname and filename will control the folder and file structure of the logs.

The timestampFormat is a value appended to the beginning of all log entries.

The maxSize and maxFiles are probably the most important configuration for the log. The log out of the box comes with a roll-over mechanism, in this case we can read the configuration as saying that every time a file reaches 10mb, it will generate a new file (i.e AutomationController_2023-12-06.log -> AutomationController_2023-12-06.log1), if a log is older than 30d it will be deleted as it is considered as stale. The maxFiles can define a time limit or a set number of files.

Note: Notice that we allow not only static text, but also some tokens that are replaced in runtime all tokens are documented in the help. The Normalized, can be added to perform some operations such as encoding and replacing special characters for _.

Important: The logs occupy disk space, if you are dealing with an integration that is very verbose and generates a lot of logs, keep in mind that the disk space will be proportional to what you are logging. Feel free to tweak the settings as needed.

{
	"logging": [
		{
			"id": "ManagerAndMonitor",
			"type": "File",
			"options": {
				"filename": "${applicationName}_${date}.log",
				"dirname": "${temp}/ConnectIoT/Logs/${applicationName}",
				"level": "debug",
				"timestampFormat": "HH:mm:ss.SSSSS",
				"maxSize": "10m",
				"maxFiles": "30d"
			},
			"applications": [
				"AutomationMonitor",
				"AutomationManager"
			]
		}
	]
}

The third entry is very similar to the second entry. Now we change the dirname, as the manager and monitor are not grouped by instances and we change the applications to monitor the Automation Monitor and the Automation Manager.

When generating the logs you will see a structure similar to this:

• ConnectIoT
  • Logs
    • Automation Manager
      • (i.e AutomationManager_2023-12-06.log)
    • Automation Monitor
      • (i.e AutomationMonitor_2023-12-06.log)
    • Instances
      • <Entity Name>
        • Controller
          • (i.e AutomationController_2023-12-06.log)
        • <Driver FriendlyName>
          • (i.e DriverOpcUA_2023-12-06.log)

Reading the logs

Automation Manager

Let’s take a look at the beginning. This is a small snippet of the first log of the Automation Manager:

11:23:11.49149 info: Starting Manager with pid "31468".
11:23:11.49349 debug: Command line: _=[], config='..\config.json', monitorPort=0, monitor-port=0, monitorHost='', monitor-host='', monitorApplication='', monitor-application='', $0='..\src\index.js'
11:23:11.51952 debug: Monitor applications persistency data update with data from the following path "C:\Users\jroque\Downloads\ActiveTestManager\monitorApplication.json"
11:23:11.51952 debug: Using file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json' for configurations
11:23:11.51952 debug: Registering configuration section 'logging' (Logging section)
11:23:11.51952 info: Parsing and validating configuration file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json'
11:23:11.52052 debug: Using the following configuration settings:
Logging:
  Console (Console):
    level='debug', specificLevelLock='false', label='AutomationManager, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='YYYY-MM-DD HH:mm:ss.SSS', maxLength='0'
    colorize='true', colorizeMessage='true'
    applications='<All>'
  ManagerAndMonitor (File):
    level='debug', specificLevelLock='false', label='AutomationManager, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='HH:mm:ss.SSSSS', maxLength='0'
    dirname='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Logs/AutomationManager', filename='AutomationManager_%DATE%.log', options='mode=511, flags='a''
    datePattern='YYYY-MM-DD', maxSize='10m', maxFiles='30d', zippedArchive='false'
    applications='[AutomationMonitor, AutomationManager]'
11:23:11.52252 debug: This application is not allowed to use this transport (ControllersAndDrivers)
11:23:11.54254 debug: This application is not allowed to use this transport (Jobs)
11:23:11.54254 warn: *** Minimum log entries set to 'debug' ***
11:23:11.54254 info:   Id='ActiveTestManager'.
11:23:11.54254 info:   Configuration file='C:\Users\jroque\Downloads\ActiveTestManager\config.json'.
11:23:11.54254 info:   Monitor Address='localhost:0'.
11:23:11.54354 info:   Monitor Application='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Cache/connect-iot-monitor@10.1.2-202309224/src/index.js'.
11:23:11.54655 debug: Starting Monitor 'C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Cache/connect-iot-monitor@10.1.2-202309224/src/index.js' with args: '[--config=C:\Users\jroque\Downloads\ActiveTestManager\config.json, --id=ActiveTestManager, --dev=false]'
11:23:11.55455 info: Monitor started with pid 29084.
11:23:11.55455 info: Manager process started with success

As you can see the logs always start with timestamp the verbosity and then the log message. Right away we can see that the Automation Manager was started with pid 31468 and all the arguments used for starting the Manager. It is also explicit that the Monitor was started. These are the kinds of logs that you can expect in the Automation Manager as its main function is starting and stopping processes. The Automation Monitor and the Automation Manager logs are very useful to validate configurations and check if the boot up sequence is correct.

Automation Monitor

Looking at the Monitor. Let’s split the logs in sequential steps, so we can understand what is happening. First off the booting up sequence:

11:23:13.40440 debug: Using file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json' for configurations
11:23:13.40541 debug: Registering configuration section 'logging' (Logging section)
11:23:13.40541 debug: Registering configuration section 'monitor' (Monitor Section)
11:23:13.40541 debug: Registering configuration section 'monitor/processCommunication' (Process communication for Monitor)
11:23:13.40541 debug: Registering configuration section 'repository' (Repository section)
11:23:13.40541 debug: Registering configuration section 'system' (System access section)
11:23:13.40641 info: Parsing and validating configuration file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json'
11:23:13.40741 debug: Using the following configuration settings:
Logging:
  Console (Console):
    level='debug', specificLevelLock='false', label='AutomationMonitor, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='YYYY-MM-DD HH:mm:ss.SSS', maxLength='0'
    colorize='true', colorizeMessage='true'
    applications='<All>'
  ManagerAndMonitor (File):
    level='debug', specificLevelLock='false', label='AutomationMonitor, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='HH:mm:ss.SSSSS', maxLength='0'
    dirname='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Logs/AutomationMonitor', filename='AutomationMonitor_%DATE%.log', options='mode=511, flags='a''
    datePattern='YYYY-MM-DD', maxSize='10m', maxFiles='30d', zippedArchive='false'
    applications='[AutomationMonitor, AutomationManager]'
Monitor:
  notifyBeforeKill='true', killNotificationTimeout='10000' ms
  retryAttempts='30', sleepBetweenAttempts='1000' ms
  processHeartbeat: isEnabled='true'
   interval='60000', timeout='10000'
   allowedFailures='3', action='Restart'
  processCommunication: useSsl='false'
Repository: type='System'
System:
  tenantName='BBRAUN', loadBalancing='false', urlSuffix='undefined'
  address='localhost', port='903', ssl='false', timeout='120000 ms'
  authentication: type='Password'
    domain='cmf', user='jroque'
11:23:13.40941 debug: This application is not allowed to use this transport (ControllersAndDrivers)
11:23:13.42743 debug: This application is not allowed to use this transport (Jobs)
11:23:13.42743 warn: *** Minimum log entries set to 'debug' ***
11:23:13.42843 info: Starting Monitor with pid "29084".
11:23:13.42843 debug: Command line: _=[], config='C:\Users\jroque\Downloads\ActiveTestManager\config.json', id='ActiveTestManager', dev='false', monitorPort=0, mp=0, monitor-port=0, monitorHost='localhost', mh='localhost', monitor-host='localhost', cache='', cd='', $0='..\ConnectIoT\Cache\connect-iot-monitor@10.1.2-202309224\src\index.js'
11:23:13.42943 info:   Id='ActiveTestManager'
11:23:13.42943 info:   ConfigurationFile='C:\Users\jroque\Downloads\ActiveTestManager\config.json'
11:23:13.42943 info:   HostName='localhost'
11:23:13.42943 info:   MonitorHost='localhost'
11:23:13.42943 info:   MonitorPort='0'
11:23:13.42943 info:   CacheLocation='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Cache'
11:23:13.43443 info: Monitor version '10.1.2-202309224' is starting...
11:23:13.43443 debug: Setting up MES system connection
11:23:13.43443 info: Setting up SystemAPI Rest communication
11:23:13.43443 debug: tenantName='BBRAUN', loadBalancing='false', address='localhost', port='903', ssl='false'
11:23:13.43443 info: Authenticating client...
11:23:13.43443 debug: Password Authenticate: domain='cmf', username='jroque'
11:23:13.52052 info: Authentication successful for user 'CMF\jroque'
11:23:13.52553 debug: Retrieving data identified with 'httpPort'
11:23:13.52753 info: Monitor is starting on 'localhost:0'
11:23:13.52853 info: Listening websocket http address ":::64211" for driver and controller messages
11:23:13.52953 debug: Storing data for 'httpPort'
11:23:13.53053 debug: Identifying Automation Manager 'ActiveTestManager' in System
11:23:13.53053 debug: Executing query 'IdentifyAutomationManager' in System, over entity 'AutomationManager'
11:23:13.53554 info: Connected with message bus!

Notice the node process is started info: Starting Monitor with pid "29084". Then the configurations for the monitor. All boot up phases of the components are similar. It’s also clear that the monitor communicates with the MES (System) and with the MessageBus.

The next phase, it retrieves the information from the MES regarding what processes it needs to start and request to the manager to start those processes (in this case we have a controller with two drivers).

11:23:13.56156 info: Executed query, and received '1' results
11:23:13.56156 debug: Identified Automation Manager Id='2310311623180000002', Name='ActiveTestManager', MonitorPackageVersion='10.1.2-202309224'
11:23:13.56356 info: Getting list of instances (controller and driver) to monitor ('ActiveTestManager')
11:23:13.56356 debug: Executing query 'ConnectIoTGetInstancesToRun' in System, over entity 'AutomationDriverInstance'
11:23:13.56356 info: Starting process heartbeat
11:23:13.56456 info: Monitor process started with success
11:23:13.73974 info: Executed query, and received '3' results
11:23:13.74074 info: Resolving EntityType name from the Id '1805111613350000045'
11:23:13.88789 info: Resolved EntityType name 'Resource' from the Id '1805111613350000045'
11:23:13.88789 debug: Getting ObjectById: Id='2310311623180000157', Type='Resource'
11:23:14.17217 info: Resolved Object Name='ControllerTest_FleetManager_CleanRoom' from Id='2310311623180000157'
11:23:14.17217 debug: Getting ObjectById: Id='2310311623180000158', Type='Resource'
11:23:14.18118 info: Resolved Object Name='FactoryAutomationWorkerController' from Id='2310311623180000158'
11:23:14.18118 info: Preparing package '@criticalmanufacturing/connect-iot-driver-opcua', version '10.1.2-202309224'
11:23:14.18218 info: Resolving package '@criticalmanufacturing/connect-iot-driver-opcua' with version '10.1.2-202309224' location.
11:23:14.20020 info: Found package '@criticalmanufacturing/connect-iot-driver-opcua' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224'
11:23:14.20020 info: Preparing package '@criticalmanufacturing/connect-iot-controller', version '10.1.2-202309224'
11:23:14.20120 info: Resolving package '@criticalmanufacturing/connect-iot-controller' with version '10.1.2-202309224' location.
11:23:14.20821 info: Found package '@criticalmanufacturing/connect-iot-controller' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224'
11:23:14.20821 info: Preparing package '@criticalmanufacturing/connect-iot-driver-opcua', version '10.1.2-202309224'
11:23:14.20821 info: Resolving package '@criticalmanufacturing/connect-iot-driver-opcua' with version '10.1.2-202309224' location.
11:23:14.20921 info: Found package '@criticalmanufacturing/connect-iot-driver-opcua' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224'
11:23:14.20921 info: Preparing package '@criticalmanufacturing/connect-iot-driver-factoryautomation', version '10.1.2-202309224'
11:23:14.20921 info: Resolving package '@criticalmanufacturing/connect-iot-driver-factoryautomation' with version '10.1.2-202309224' location.
11:23:14.21622 info: Found package '@criticalmanufacturing/connect-iot-driver-factoryautomation' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-factoryautomation@10.1.2-202309224'
11:23:14.21622 info: Preparing package '@criticalmanufacturing/connect-iot-controller', version '10.1.2-202309224'
11:23:14.21622 info: Resolving package '@criticalmanufacturing/connect-iot-controller' with version '10.1.2-202309224' location.
11:23:14.21722 info: Found package '@criticalmanufacturing/connect-iot-controller' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224'
11:23:14.21822 info: Processes to start:
11:23:14.21822 info: AutomationControllerInstance/2310311623180000011  @criticalmanufacturing/connect-iot-controller@10.1.2-202309224
11:23:14.21822 info:     AutomationDriverInstance/2310311623180000032  @criticalmanufacturing/connect-iot-driver-opcua@10.1.2-202309224 (Server)
11:23:14.21922 info:     AutomationDriverInstance/2310311623180000033  @criticalmanufacturing/connect-iot-driver-opcua@10.1.2-202309224 (Client)
11:23:14.21922 info: AutomationControllerInstance/2310311623180000012  @criticalmanufacturing/connect-iot-controller@10.1.2-202309224
11:23:14.21922 info:     AutomationDriverInstance/2310311623180000025  @criticalmanufacturing/connect-iot-driver-factoryautomation@10.1.2-202309224 (Handler)
11:23:14.21922 info: Starting process id='AutomationDriverInstance/2310311623180000032', name='@criticalmanufacturing/connect-iot-driver-opcua', version='10.1.2-202309224'
11:23:14.21922 info: Starting 'Server' main script 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js'
11:23:14.22022 info: Requesting fork Id='AutomationDriverInstance/2310311623180000032', Path='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js'
11:23:14.22022 debug:  Arguments=--id=AutomationDriverInstance/2310311623180000032 --managerId=ActiveTestManager --componentId=Server --serverHost=localhost --monitorPort=64211 --monitorHost=localhost --monitorToken=monitorSecurityToken --monitorSsl=false --config=C:\Users\jroque\Downloads\ActiveTestManager\config.json --dev=false --entityName=ControllerTest_FleetManager_CleanRoom
11:23:14.22723 info: Received notification of process 'AutomationDriverInstance/2310311623180000032' start with Pid '11724"
11:23:14.22723 info: Started process 'AutomationDriverInstance/2310311623180000032' not being handled by Monitor Pid='11724'. Message ignored?
11:23:14.22723 info: fork "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js" succeeded with pid "11724"
11:23:14.22823 info: Starting process id='AutomationControllerInstance/2310311623180000011', name='@criticalmanufacturing/connect-iot-controller', version='10.1.2-202309224'
11:23:14.22823 info: Starting 'Controller' main script 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js'
11:23:14.22823 info: Requesting fork Id='AutomationControllerInstance/2310311623180000011', Path='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js'
11:23:14.22823 debug:  Arguments=--id=AutomationControllerInstance/2310311623180000011 --managerId=ActiveTestManager --componentId=Controller --serverHost=localhost --monitorPort=64211 --monitorHost=localhost --monitorToken=monitorSecurityToken --monitorSsl=false --config=C:\Users\jroque\Downloads\ActiveTestManager\config.json --dev=false --entityName=ControllerTest_FleetManager_CleanRoom
11:23:14.23323 info: Received notification of process 'AutomationControllerInstance/2310311623180000011' start with Pid '30428"
11:23:14.23323 info: Started process 'AutomationControllerInstance/2310311623180000011' not being handled by Monitor Pid='30428'. Message ignored?
11:23:14.23423 info: fork "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js" succeeded with pid "30428"
11:23:14.23423 info: Starting process id='AutomationDriverInstance/2310311623180000033', name='@criticalmanufacturing/connect-iot-driver-opcua', version='10.1.2-202309224'
11:23:14.23423 info: Starting 'Client' main script 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js'
11:23:14.23423 info: Requesting fork Id='AutomationDriverInstance/2310311623180000033', Path='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js'
11:23:14.23423 debug:  Arguments=--id=AutomationDriverInstance/2310311623180000033 --managerId=ActiveTestManager --componentId=Client --serverHost=localhost --monitorPort=64211 --monitorHost=localhost --monitorToken=monitorSecurityToken --monitorSsl=false --config=C:\Users\jroque\Downloads\ActiveTestManager\config.json --dev=false --entityName=ControllerTest_FleetManager_CleanRoom
11:23:14.24224 info: Received notification of process 'AutomationDriverInstance/2310311623180000033' start with Pid '19324"
11:23:14.24224 info: Started process 'AutomationDriverInstance/2310311623180000033' not being handled by Monitor Pid='19324'. Message ignored?
11:23:14.24224 info: fork "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-opcua@10.1.2-202309224\src\index.js" succeeded with pid "19324"
11:23:14.24324 info: Starting process id='AutomationDriverInstance/2310311623180000025', name='@criticalmanufacturing/connect-iot-driver-factoryautomation', version='10.1.2-202309224'
11:23:14.24324 info: Starting 'Handler' main script 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-factoryautomation@10.1.2-202309224\src\index.js'
11:23:14.24324 info: Requesting fork Id='AutomationDriverInstance/2310311623180000025', Path='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-factoryautomation@10.1.2-202309224\src\index.js'
11:23:14.24324 debug:  Arguments=--id=AutomationDriverInstance/2310311623180000025 --managerId=ActiveTestManager --componentId=Handler --serverHost=localhost --monitorPort=64211 --monitorHost=localhost --monitorToken=monitorSecurityToken --monitorSsl=false --config=C:\Users\jroque\Downloads\ActiveTestManager\config.json --dev=false --entityName=FactoryAutomationWorkerController
11:23:14.24925 info: Received notification of process 'AutomationDriverInstance/2310311623180000025' start with Pid '22304"
11:23:14.24925 info: Started process 'AutomationDriverInstance/2310311623180000025' not being handled by Monitor Pid='22304'. Message ignored?
11:23:14.25025 info: fork "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-driver-factoryautomation@10.1.2-202309224\src\index.js" succeeded with pid "22304"
11:23:14.25025 info: Starting process id='AutomationControllerInstance/2310311623180000012', name='@criticalmanufacturing/connect-iot-controller', version='10.1.2-202309224'
11:23:14.25025 info: Starting 'Controller' main script 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js'
11:23:14.25025 info: Requesting fork Id='AutomationControllerInstance/2310311623180000012', Path='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js'
11:23:14.25025 debug:  Arguments=--id=AutomationControllerInstance/2310311623180000012 --managerId=ActiveTestManager --componentId=Controller --serverHost=localhost --monitorPort=64211 --monitorHost=localhost --monitorToken=monitorSecurityToken --monitorSsl=false --config=C:\Users\jroque\Downloads\ActiveTestManager\config.json --dev=false --entityName=FactoryAutomationWorkerController
11:23:14.25726 info: Received notification of process 'AutomationControllerInstance/2310311623180000012' start with Pid '4144"
11:23:14.25726 info: Started process 'AutomationControllerInstance/2310311623180000012' not being handled by Monitor Pid='4144'. Message ignored?
11:23:14.25726 info: fork "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller@10.1.2-202309224\src\index.js" succeeded with pid "4144"
11:23:14.25826 info: Processes and packages processed
11:23:15.47447 debug: Client Connected! from 64223. Have now 1 clients.
11:23:15.48348 debug: Client Connected! from 64226. Have now 2 clients.
11:23:16.59459 debug: Client Connected! from 64232. Have now 3 clients.
11:23:16.60260 debug: Client Connected! from 64233. Have now 4 clients.
11:23:16.62462 info: Component "AutomationDriverInstance/2310311623180000032" notified it just become available
11:23:16.62963 info: Notifying 'AutomationControllerInstance/2310311623180000011' with connection information to 'AutomationDriverInstance/2310311623180000032' (Server)
11:23:16.62963 debug: Data provided: type='websocket', host='localhost', port='64235', ssl='false'
11:23:16.63163 info: Component "AutomationDriverInstance/2310311623180000033" notified it just become available
11:23:16.63263 info: Notifying 'AutomationControllerInstance/2310311623180000011' with connection information to 'AutomationDriverInstance/2310311623180000033' (Client)
11:23:16.63363 debug: Data provided: type='websocket', host='localhost', port='64234', ssl='false'
11:23:16.80080 info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-core-tasks' with version '10.1.2-202309224' location.
11:23:16.80180 info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks' with version '10.1.2-202309224' location.
11:23:16.80881 info: Found package '@criticalmanufacturing/connect-iot-controller-engine-core-tasks' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-core-tasks@10.1.2-202309224'
11:23:16.81782 info: Found package '@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-factoryautomation-tasks@10.1.2-202309224'
11:23:17.01001 debug: Client Connected! from 64236. Have now 5 clients.
11:23:17.08509 info: Component "AutomationDriverInstance/2310311623180000025" notified it just become available
11:23:17.08609 info: Notifying 'AutomationControllerInstance/2310311623180000012' with connection information to 'AutomationDriverInstance/2310311623180000025' (Handler)
11:23:17.08609 debug: Data provided: type='websocket', host='localhost', port='64237', ssl='false'
11:23:17.55355 info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-core-tasks' with version '10.1.2-202309224' location.
11:23:17.55455 info: Found package '@criticalmanufacturing/connect-iot-controller-engine-core-tasks' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-core-tasks@10.1.2-202309224'
11:23:17.55656 info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-opcua-tasks' with version '10.1.2-202309224' location.
11:23:17.56657 info: Found package '@criticalmanufacturing/connect-iot-controller-engine-opcua-tasks' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-opcua-tasks@10.1.2-202309224'
11:23:18.42442 info: Component "AutomationControllerInstance/2310311623180000012" notified it just become available
11:23:18.42543 info: Notifying 'AutomationControllerInstance/2310311623180000012' with connection information to 'AutomationDriverInstance/2310311623180000025' (Handler)
11:23:18.42543 debug: Data provided: type='websocket', host='localhost', port='64237', ssl='false'
11:23:18.55055 info: Received order to 'setautomationstates' the Controller 'AutomationControllerInstance/2310311623180000012' for this AutomationManager
11:23:18.82482 info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks' with version '10.1.2-202309224' location.
11:23:18.82583 info: Found package '@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks' with version '10.1.2-202309224' in 'C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-factoryautomation-tasks@10.1.2-202309224'
11:23:19.18519 info: Component "AutomationControllerInstance/2310311623180000011" notified it just become available

The processes to start are found in:

11:23:14.21822 info: Processes to start:
11:23:14.21822 info: AutomationControllerInstance/2310311623180000011  @criticalmanufacturing/connect-iot-controller@10.1.2-202309224
11:23:14.21822 info:     AutomationDriverInstance/2310311623180000032  @criticalmanufacturing/connect-iot-driver-opcua@10.1.2-202309224 (Server)
11:23:14.21922 info:     AutomationDriverInstance/2310311623180000033  @criticalmanufacturing/connect-iot-driver-opcua@10.1.2-202309224 (Client)
11:23:14.21922 info: AutomationControllerInstance/2310311623180000012  @criticalmanufacturing/connect-iot-controller@10.1.2-202309224
11:23:14.21922 info:     AutomationDriverInstance/2310311623180000025  @criticalmanufacturing/connect-iot-driver-factoryautomation@10.1.2-202309224 (Handler)

Then the retrieval of all the cached packages, like for example info: Resolving package '@criticalmanufacturing/connect-iot-controller-engine-core-tasks' with version '10.1.2-202309224' location. As the components are started by the manager the monitor will log that they are now available (i.e 11:23:18.42442 info: Component "AutomationControllerInstance/2310311623180000012" notified it just become available).

Automation Controller

The Automation Controller logs all the logic in our workflows execution will be described. Everything that is orchestrated inside the workflows will be logged in this component.

11:23:15.34034 debug: Using file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json' for configurations
11:23:15.34334 debug: Registering configuration section 'logging' (Logging section)
11:23:15.34334 debug: Registering configuration section 'controller' (Controller Section)
11:23:15.34334 debug: Registering configuration section 'monitor/processCommunication' (Process communication with Monitor)
11:23:15.34334 debug: Registering configuration section 'driver/processCommunication' (Process communication with Drivers)
11:23:15.34334 debug: Registering configuration section 'system' (System access section)
11:23:15.34334 info: Parsing and validating configuration file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json'
11:23:15.34635 debug: Using the following configuration settings:
Logging:
  Console (Console):
    level='debug', specificLevelLock='false', label='AutomationController, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='YYYY-MM-DD HH:mm:ss.SSS', maxLength='0'
    colorize='true', colorizeMessage='true'
    applications='<All>'
  ControllersAndDrivers (File):
    level='debug', specificLevelLock='false', label='AutomationController, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='HH:mm:ss.SSSSS', maxLength='0'
    dirname='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Logs/Instances/ControllerTest_FleetManager_CleanRoom/Controller', filename='AutomationController_%DATE%.log', options='mode=511, flags='a''
    datePattern='YYYY-MM-DD', maxSize='10m', maxFiles='30d', zippedArchive='false'
    applications='[AutomationController, Driver*]'
Controller:
  retryAttempts='30', sleepBetweenAttempts='1000' ms
  profilerSessionsLocation='C:\Users\jroque\AppData\Local\Temp\ConnectIoT\ProfilerSessions'
  monitorProcessCommunication: useSsl='false'
  driverProcessCommunication: useSsl='false'
System:
  tenantName='BBRAUN', loadBalancing='false', urlSuffix='undefined'
  address='localhost', port='903', ssl='false', timeout='120000 ms'
  authentication: type='Password'
    domain='cmf', user='jroque'
11:23:15.36236 debug: This application is not allowed to use this transport (ManagerAndMonitor)
11:23:15.36236 debug: This application is not allowed to use this transport (Jobs)
11:23:15.36236 warn: *** Minimum log entries set to 'debug' ***
11:23:15.36637 info: Starting NodeJS Controller with pid "30428".
11:23:15.36637 debug: Command line: _=[], id='AutomationControllerInstance/2310311623180000011', managerId='ActiveTestManager', manager-id='ActiveTestManager', componentId='Controller', component-id='Controller', serverHost='localhost', monitorPort=64211, mp=64211, monitor-port=64211, monitorHost='localhost', mh='localhost', monitor-host='localhost', monitorToken='monitorSecurityToken', mt='monitorSecurityToken', monitor-token='monitorSecurityToken', monitorSsl='false', config='C:\Users\jroque\Downloads\ActiveTestManager\config.json', dev='false', entityName='ControllerTest_FleetManager_CleanRoom', entity-name='ControllerTest_FleetManager_CleanRoom', port=13000, p=13000, $0='index.js'
11:23:15.36637 info:   ConfigurationFile='C:\Users\jroque\Downloads\ActiveTestManager\config.json'
11:23:15.36737 info: Using 'C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Persistency' for directory persistency storage. Retention time is '2592000' seconds
11:23:15.36737 info: Controller "AutomationControllerInstance/2310311623180000011" is booting...
11:23:15.36737 debug: Setting up MES system connection
11:23:15.36737 info: Setting up SystemAPI Rest communication
11:23:15.36737 debug: tenantName='BBRAUN', loadBalancing='false', address='localhost', port='903', ssl='false'
11:23:15.36737 info: Authenticating client...
11:23:15.36837 debug: Password Authenticate: domain='cmf', username='jroque'
11:23:15.46747 info: Authentication successful for user 'CMF\jroque'
11:23:15.47147 info: Setting up Monitor connection
11:23:15.47147 debug: Establishing connection with Monitor, reconnectInterval=1000, reconnectMaxAttempts=0, host='localhost', port=64211, securityToken='monitorSecurityToken', requireBuffering=true, bufferingId='AutomationControllerInstance/2310311623180000011'
11:23:15.47748 info: Connected with message bus!
11:23:15.47948 info: Connection with monitor established
11:23:15.47948 info: Connection with Monitor successfully established!
11:23:15.48048 debug: Executing query 'ControllerFromInstance' in System, over entity 'AutomationControllerInstance'
11:23:15.50350 info: Executed query, and received '1' results
11:23:17.54655 info: Adding workflow: Name='Setup', Id='2310311623180000702', LastServiceHistoryId='2310311623180044467'
11:23:17.55355 info: Requesting package location. Package='@criticalmanufacturing/connect-iot-controller-engine-core-tasks', Version='10.1.2-202309224'
11:23:17.55656 info: Requesting package location. Package='@criticalmanufacturing/connect-iot-controller-engine-opcua-tasks', Version='10.1.2-202309224'
11:23:17.55656 info: Package location retrieved. Package='@criticalmanufacturing/connect-iot-controller-engine-core-tasks', Version='10.1.2-202309224', Location='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-core-tasks@10.1.2-202309224'
11:23:18.21822 info: Package location retrieved. Package='@criticalmanufacturing/connect-iot-controller-engine-opcua-tasks', Version='10.1.2-202309224', Location='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-opcua-tasks@10.1.2-202309224'
11:23:18.82382 info: Adding workflow: Name='RequestTransport_Command', Id='2310311623180000713', LastServiceHistoryId='2310311623180044467'
11:23:18.82482 info: Requesting package location. Package='@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks', Version='10.1.2-202309224'
11:23:18.82783 info: Package location retrieved. Package='@criticalmanufacturing/connect-iot-controller-engine-factoryautomation-tasks', Version='10.1.2-202309224', Location='C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Cache\connect-iot-controller-engine-factoryautomation-tasks@10.1.2-202309224'
11:23:19.14314 info: [<<root>>|RequestTransport_Command|task_2804|codeExecution] Initializing Virtual Machine Engine version 1.0.0
11:23:19.17217 info: Adding workflow: Name='RequestReleaseOfNextAction_Command', Id='2310311623180000709', LastServiceHistoryId='2310311623180044467'
11:23:19.17518 info: Unsubscribing in 'Server' a previously registered subject
11:23:19.17518 info: Subscribing in 'Server' for messages of type 'connect.iot.driver.opcua.OnCommand.RequestReleaseOfNextAction'
11:23:19.17518 info: Adding workflow: Name='JobStatusUpdate_Command', Id='2310311623180000707', LastServiceHistoryId='2310311623180044467'
11:23:19.17818 info: Unsubscribing in 'Server' a previously registered subject
11:23:19.17818 info: [<<root>>|JobStatusUpdate_Command|task_4493|codeExecution] Initializing Virtual Machine Engine version 1.0.0
11:23:19.18519 info: Subscribing in 'Server' for messages of type 'connect.iot.driver.opcua.OnCommand.JobStatusUpdate'
11:23:19.18519 info: Controller 'AutomationControllerInstance/2310311623180000011' notifying monitor that is available. Pid='30428'
11:23:19.18719 info: Received entity information assigned to this process. Name='ControllerTest_FleetManager_CleanRoom', Type='Cmf.Navigo.BusinessObjects.Resource, Cmf.Navigo.BusinessObjects'
11:23:19.18819 info: Received connection information for "Server"
11:23:19.18819 debug: Details: type='websocket', host='localhost', port=64235, useSsl=false, securityToken='309f3390c5bf38dcba9c4c1764353621a579eb5b0a6183e09e93a2fd4d506083eec04e06241a62b910d65b7584288fa122cb314c1ce629ee34478d611255a92ccedf59de45d82b89d9ccd875062910c41530e82244a621eaae8259ac5942825babdaf2fef28f55177b1c9eaf6f2ae0939f04390fb6b4eb12a47f5ecc548934b69993cb2aedaafcad738a61b026ecff0b93142131d00de833cce4d50957aaaadc93fca8e942c304fe47fdcffb7ae2c0f7c15e3e37ec016e33778bfe41d6684ae90eeaf8dee4a247c1fd1ce374260678a5463fac42691099583f24025b17830381ac4ea1f4dea47132e9057f03b8994d6818460c2837f1bd4d16ad64833ba12842', $id='4'
11:23:19.18819 info: Received connection information for "Client"
11:23:19.18819 debug: Details: type='websocket', host='localhost', port=64234, useSsl=false, securityToken='fef4e74349ebb1f03a4ea7f7e73fbc7f8cf770d37ec71f31b1fe3bd63e1fb9bcfb9cdf33fff7c914aaecd034d606e47369321e2275ea4e244657fe97952e09a05c92358dbcf959f22cbf5c1440a1f9f1edf685e5dc3d17ccd9e39a5dccd91a1fd085ab1895420d51884e226ba5e5ca464d2b0e6c071eaf235b76aa216c9cc9b370c7a72bd661ff98d3fc9c492bcb3ebca712ec235d68834ee815ccb1567d5a2a52a38c8884680e8c95e31308720cd36058e62f08c098a8d440923af8245ba18659db4afd715f7fb661aac00f8c7021b30b8746b889b692d3c7191f32349bab5a6e4e80616a38f96f316c2f1d295ac2d0ecd90c27681b730468421831d1230f58', $id='4'
11:23:19.18919 info: Controller available notification was accepted
11:23:19.18919 debug: Updating 'AutomationControllerInstance/2310311623180000011' instance to 'Running'
11:23:19.28128 info: Updated 'AutomationControllerInstance/2310311623180000011' instance to 'Running'
11:23:19.28128 info: Controller process started with success

This log is a mirror to the monitor as some of the packages that were retrieved in the monitor were retrieved because the controller requested that it needed them. We also see the normal boot up with all the process configurations. There is the cycle of adding workflows and if those workflows require them, to generate subscriptions for particular types of messages.

Let’s take a look at an example of receiving a command execution for an OPC-UA Server and then replying:

15:09:33.896 info: [25689036|JobStatusUpdate_Command|task_3478|onCommand] Received Command for OPC-UA Server with id:'e61d71f3-c82b-4b53-8588-44febd241a16' and values: '[{"name":"TransportStatus","valueType":{"type":4},"editable":false,"removable":false},{"name":"SLCID","valueType":{"type":4},"editable":false,"removable":false},{"name":"AMRID","valueType":{"type":4},"editable":false,"removable":false}]'
15:09:33.903 debug: [25689036|JobStatusUpdate_Command|anyToConstant] Converted to constant value="0"
15:09:33.903 debug: [25689036|JobStatusUpdate_Command|anyToConstant] Converted to constant value="No Error"
15:09:33.922 debug: [25689036|JobStatusUpdate_Command|task_3479|commandResult] Sending command result for request with id e61d71f3-c82b-4b53-8588-44febd241a16.
15:09:33.922 info: Sending to 'Server' a message of type 'connect.iot.driver.opcua.CommandResult'

All this transaction works in the zone 25689036 zones zone.js. This id is very helpful to follow the transaction context throughout the logs. When the context is <root> it means no context was generated. We can see the name of the workflow page JobStatusUpdate_Command and the id and name of task task_3478|onCommand. Looking at the messages it’s apparent we received a command with an id and a payload id:'e61d71f3-c82b-4b53-8588-44febd241a16' and values: '[{"name":"TransportStatus","valueType":{"type":4},"editable":false,"removable":false},{"name":"SLCID","valueType":{"type":4},"editable":false,"removable":false},{"name":"AMRID","valueType":{"type":4},"editable":false,"removable":false}]'. Then we do some conversions of values and we reply back the command execution.

Automation Driver

The Automation Driver logs will describe the interface with a particular protocol.

14:21:20.04004 debug: Using file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json' for configurations
14:21:20.04204 debug: Registering configuration section 'logging' (Logging section)
14:21:20.04204 debug: Registering configuration section 'driver' (Driver Section)
14:21:20.04204 debug: Registering configuration section 'driver/processCommunication' (Process communication with Controller (from driver))
14:21:20.04204 debug: Registering configuration section 'monitor/processCommunication' (Process communication with Monitor)
14:21:20.04204 info: Parsing and validating configuration file 'C:\Users\jroque\Downloads\ActiveTestManager\config.json'
14:21:20.04404 debug: Using the following configuration settings:
Logging:
  Console (Console):
    level='debug', specificLevelLock='false', label='DriverOpcUA, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='YYYY-MM-DD HH:mm:ss.SSS', maxLength='0'
    colorize='true', colorizeMessage='true'
    applications='<All>'
  ControllersAndDrivers (File):
    level='debug', specificLevelLock='false', label='DriverOpcUA, format='${log.timestamp} ${log.level}: ${log.message}'
    timestamp='true', timestampFormat='HH:mm:ss.SSSSS', maxLength='0'
    dirname='C:/Users/jroque/Downloads/ActiveTestManager/ConnectIoT/Logs/Instances/ControllerTest_FleetManager_CleanRoom/Server', filename='DriverOpcUA_%DATE%.log', options='mode=511, flags='a''
    datePattern='YYYY-MM-DD', maxSize='10m', maxFiles='30d', zippedArchive='false'
    applications='[AutomationController, Driver*]'
Driver(s):
  retryAttempts='30', sleepBetweenAttempts='1000' ms
  processCommunication: useSsl='false'
  monitorProcessCommunication: useSsl='false'
14:21:20.06306 debug: This application is not allowed to use this transport (ManagerAndMonitor)
14:21:20.06306 debug: This application is not allowed to use this transport (Jobs)
14:21:20.06306 warn: *** Minimum log entries set to 'debug' ***
14:21:20.06306 info: Starting OPC UA driver with pid "21624".
14:21:20.06406 debug: Command line: _=[], id='AutomationDriverInstance/2310311623180000032', managerId='ActiveTestManager', manager-id='ActiveTestManager', componentId='Server', component-id='Server', serverHost='localhost', sh='localhost', server-host='localhost', monitorPort=49371, mp=49371, monitor-port=49371, monitorHost='localhost', mh='localhost', monitor-host='localhost', monitorToken='monitorSecurityToken', mt='monitorSecurityToken', monitor-token='monitorSecurityToken', monitorSsl='false', config='C:\Users\jroque\Downloads\ActiveTestManager\config.json', dev='false', entityName='ControllerTest_FleetManager_CleanRoom', entity-name='ControllerTest_FleetManager_CleanRoom', inspect-brk=true, inspectBrk=true, serverPort=0, sp=0, server-port=0, $0='index.js'
14:21:20.06406 info:   ConfigurationFile='C:\Users\jroque\Downloads\ActiveTestManager\config.json'
14:21:20.06807 info: Driver "AutomationDriverInstance/2310311623180000032" is booting...
14:21:20.06907 info: Setting up Monitor connection
14:21:20.06907 debug: Establishing connection with Monitor, reconnectInterval=1000, host='localhost', port=49371, securityToken='monitorSecurityToken',
14:21:20.09009 info: Connection with monitor established
14:21:20.09109 info: Connection with Monitor successfully established!
14:21:20.10410 debug: Trying to read control persistency data from "C:\Users\jroque\Downloads\ActiveTestManager\ConnectIoT\Persistency\OPCUA_Server\AutomationDriverInstance_2310311623180000032\connectIoT_ControlFile.json"
14:21:20.10410 debug: No store was found
14:21:20.10410 info: Using port "49380" to communicate with controller
14:21:20.10410 info: Notifying communication details to allow connection with controller
14:21:20.10511 debug:   ConnectionInformation: type='websocket', host='localhost', port=49380, useSsl=false, securityToken='d5f6cdd28ecd535a221788046e56cd3989c3ea5db5a6399fbf098d871adc970369552b9d00cf16504461b868686cf921092ad59ffee3ccb3a56eb94911890d2a131ec25b0955db512f4cb741c62d209fa06a660ffa9698b8500d0fea46e33b03a9a9ced788eb34fee914fa923c8c2c66410c6a2d2f37f47efd056db945c24b76205fe1b125f2c976d1ade54bb4dbea016cb80bf83ee2f6f199274a0a8aad236a99574360154028d583e94c1e9a95c61b054cd268fbd7cadc58204a24ac52cfe96428f90a9f8e5b3e2c4e6e224f29bda81530a0b4ada244c1bf7cd9e078cb418604ce63f045cf31bfce4f8e2eb12434f069a1756bd7050465eb3cb74bb74c8cb6'
14:21:20.11311 info: Received entity information assigned to this process. Name='ControllerTest_FleetManager_CleanRoom', Type='Cmf.Navigo.BusinessObjects.Resource, Cmf.Navigo.BusinessObjects'
14:21:20.11411 info: Communication details notification was accepted
14:21:20.11411 info: OPC UA Driver process started with success
14:21:25.70070 debug: Client Connected! from 49967. Have now 1 clients.
14:21:25.71471 info: Configuring device based on configuration received
14:21:25.71872 debug: Managing certificates
14:21:25.71972 debug: Certificates folder: C:\Users\jroque\AppData\Local\Temp\ConnectIoT\OPC-UA\Certificates\AutomationDriverInstance_2310311623180000032
14:21:25.71972 debug: No certificate/public key define, using default generated.
14:21:25.94194 debug: Added custom data types
14:21:26.18819 info: Configuring device based on configuration received
14:21:26.18919 debug: Managing certificates
14:21:26.19019 debug: Certificates folder: C:\Users\jroque\AppData\Local\Temp\ConnectIoT\OPC-UA\Certificates\AutomationDriverInstance_2310311623180000032
14:21:26.19019 debug: No certificate/public key define, using default generated.
14:21:26.19319 info: Connecting device
14:21:26.19419 info: Creating OPCUA server
14:21:26.19620 info: Communication state changed Unknown -> Connecting
14:21:26.19720 info: Initializing OPCUA server
14:21:27.14414 info: Starting OPCUA server
14:21:27.14715 info: Server listening on 'opc.tcp://127.0.0.1:6020'
14:21:27.14715 info: Communication state changed Connecting -> Setup
14:21:27.14715 info: Device connected and ready to be setup
14:21:27.17618 info: Reporting Setup Result (success='true')
14:21:27.17618 info: Device successfully setup
14:21:27.17618 info: Communication state changed Setup -> Communicating

Very similar to the controller there is a boot up cycle. Interesting things that we see is that we have a persistency being retrieved for the driver, also that this driver uses certificates and that is listening on opc.tcp://127.0.0.1:6020.

Note: All drivers are different and log different things, in this case it is a Server as such it informs where the server is hosting.

Notice that it connects to the Monitor application and also to the controller. The connection via websocket to the controller is how we will notify it of new information (i.e events).

14:22:05.43544 info: Received a Command Execution for 'JobStatusUpdate' with id:'8ee8ad50-314c-4c31-a0a1-cc9437a14ea1' with command parameters '{"TransportStatus":"Completed","SLCID":"test","AMRID":"asdas"}'
14:22:05.43544 debug: Notify connect.iot.driver.opcua.OnCommand.JobStatusUpdate for 'JobStatusUpdate' with command params '{"TransportStatus":"Completed","SLCID":"test","AMRID":"asdas"}'
14:22:18.31932 debug: Heartbeat received from Monitor
14:22:27.19720 info: Getting values for properties: JobStatusUpdate-HServiceType
14:22:28.11500 info: Replying to command with id '8ee8ad50-314c-4c31-a0a1-cc9437a14ea1' with outputs: [{"dataType":5,"value":0,"arrayType":0},{"dataType":12,"value":"No Error","arrayType":0}]

In this log we see that there was a command execution in the OPC-UA server. The command was the JobStatusUpdate and it was sent with the parameters {"TransportStatus":"Completed","SLCID":"test","AMRID":"asdas"}. Then we see this request being forwarded to the controller. We get a request and we answer with a reply for the same id 8ee8ad50-314c-4c31-a0a1-cc9437a14ea1.

Pretty simple, right. Let’s look at the workflow to see if it matches.

All in all, we just receive a request and reply with a constant set of values. But you already new this, as we already took that trip through the logs!!!

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. Writing a Post
3. Deployment
4. Future

Thank you

If you are reading this, you are thinking of contributing, so you are already half way there 😉. A blog only survives by the quality and volume of content, so scalability and diversity of content are key.

“But why will I contribute, tomorrow we move to another platform and everything is lost” - Well, this is a legitimate concern, in the past we had this happen to us, when we relied on proprietary systems to store our information, only to later realize exporting information was a major hassle. But fret not, this time we use markdown for all this content, so if tomorrow we want to move to an even better platform due to having so much content and contributors, we can just tweak a bit and use as is.

Overview

The DevBlog is built in hugo, using a template heksagon. The source code resides in DevDocs under the folder DevBlogPortal.

In order to start locally the blog, install hugo in a powershell run choco install hugo-extended, then simply clone the repository, open a command shell in the folder DevBlogPortal and run hugo serve. This is very helpful to validate how your post looks before commiting the changes, you can also perform edits while the server runs and it will refresh automatically.

The important folders for contributors are the DevBlogPortal/content/blog here is where you will create new contributions are write the markdown for your blog post and DevBlogPortal/static/blogPosts here is where all static content will reside (i.e images).

Note: Please use videos/gifs with moderation as they cumulatively consume a lot of space.

Writing a Post

To write the markdowns there are a lot of available markdown editors, what was used to write this post as VSCode, VSCode has a preview mode for markdown.

In order to write a new post go to DevBlogPortal/content/blog and create a new markdown file with the name <yearmonthday>_<areaofinterest>_<yourpost> (i.e 20230413_iot_mes_as_configuration_owner). It’s very important your markdown has the following header:

---
author: <authorname>
title: <title>
date: <yearmonthday>
description: <description
tags: [<tag1>,<tag2>]
categories: [<category1><category2>]
ShowToc: true
TocOpen: true
---

It’s very important you add relevant tags and categories as the template we are using, offers a very powerful search based on tags and categories, and as the number of post grows it is a very easy way to sort and search.

We also strongly incentivize for you to add a footer

<br></br>

---

<br></br>
## Author
###  <Small text saluting the reader and explaining who you are and how they can reach you>

Skills: <particular focus or areas of interest you have>
<image of yourself (use your cmf photo if possible) this photos should be added under `DevBlogPortal/static/blogPosts/contributors`>

Things to look out for

The least intuitive thing, is that for images, the path for deployment and for preview here in your markdown editor are different. For example, it would be intuitive to say that an image is under DevBlogPortal/static/blogPosts/contributors/johnDoe.jpg, but when hugo deploy this path will not have meaning, you should use /blogPosts/contributors/johnDoe.jpg. Nevertheless, you can use hugo serve in order to see how the site will render.

When you’ve finished writing your post and are happy with the result, create a Pull Request against the branch DevDocs. Someone from the DevDocs team will revise your post and perform the deployment.

Future

The CMF CLI already has a cmf doc-gen, so why not add a doc-gen for the blog, that creates this structure! As more contributors join, we get their feedback and understand how needed is this type of tooling.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. Use Case IoT
3. Use Case MES (extra)

Overview

Connect IoT is not just customizable, but it is also extendable. This means that you have entry points that are already given out of the box, for a more advanced implementation that can leverage the system. For example, in the automation controller the code task, provides a point of extensibility where you can create a task with your own set of code. This is a rather straightforward point of extensibility, but also the driver is extensible and the use of both together is what is going to be shown today.

The driver is traditionally described in the Automation Driver Definition, where you will complete all the required communication interface with a particular equipment in a given protocol. Nevertheless, it may be the case where you may not want to have such a static definition, and may want to resolve what to subscribe or get by logic outside of the normal driver definition.

This is an advanced scenario as you are skipping the normal system behavior and are replacing this mechanism with your own.

The use case is on Track In, depending on the attached resource instrument, we want to dynamically resolve which properties to retrieve and post to a Data Collection. So, in this case we don’t know before hand what are the properties we want to retrieve. The business logic is the owner of what the system must retrieve and is even orchestrated outside the Automation Controller.

The implementation will be done using OPC-UA as protocol, but should be more or less generic to every protocol.

Use Case IoT

We will try to focus first on the use case from a Connect IoT perspective and will then explain a bit more what was the full feature, using the MES. From an IoT perspective the use case will be that we require on Track In to dynamically Get Properties and post to a DataCollectionInstance opened for a Material.

Create a DEE in Track In

We will have a DEE that will publish to IoT on the topic Cmf.Trackin.LineClearance with an object like:

{
    "materialName": "materialExample",
    "parameterTag":
        { "Temperature": {
            "tag": "ns=3;s=AirConditioner_1.Temperature",
            "deviceType": "Long",
            "dataType": "Double",
            },
          "Pressure": {
            "tag": "ns=3;s=AirConditioner_1.Pressure",
            "deviceType": "Long",
            "dataType": "Double",
            },
        }
}

Then the DEE must be something like this (here we show the snippet pertinent to IoT), it will collect the material name and then for each parameter collect all the necessary information to be able to get the property and post to a datacollection.

(...)
var parameters = dc.DataCollectionParameters.Select(dcp => dcp.TargetEntity.Name);

// This is the object that will hold the parameter and as value, everything required to perform a get
JObject parameterTag = new JObject();
foreach (DataRow row in ds.Tables[0].Rows)
{
    var rowParam = row["Parameter"].ToString();
    if (parameters.Contains(rowParam))
    {
        JObject tagInfo = new JObject();
        tagInfo["tag"] = row["DeviceID"].ToString();
        tagInfo["deviceType"] = row["DeviceType"].ToString();
        tagInfo["dataType"] = row["DataType"].ToString();

        parameterTag[row["Parameter"].ToString()] = tagInfo;
    }
}

// This code implies there is an Instance associated between the Resource, the Controller and Driver
var instance = resource.GetAutomationControllerInstance();
if (instance == null)
    throw new Exception("Resource not connected to any IoT instance");
else
{
    JObject data = new JObject();
    data["materialName"] = material.Name;
    data["parameterTag"] = parameterTag;

    string topic = "Cmf.Trackin.LineClearance";

    // This Code will publish a message bus broadcast for the specified topic, with the specified data
    instance.Publish(topic, data);
}

Note: The payloads being sent in the MessageBus should be the least required for the implementation. Beware of sending big payloads, creating traffic, without a very strong reason, if you fall in this case, you are probably doing something wrong in terms of component coupling architecture.

Looking at the Automation Controller

The implementation in Connect IoT, will now be split in to four parts: Receiving Messages, Transform Data, Dynamically Get Properties, Post to Data Collection.

Receiving Messages

Now we have a DEE that publishes a message with topic Cmf.Trackin.LineClearance on TrackIn with the information regarding the material tracked in and what parameters to collect and post. Now we need to add a subscriber for that topic by using a On System Event Documentation, this task will be notified whenever a message is published in the MessageBus for this topic.

We have created a link between the information in the DEE and our workflow.

Transform Data

We will use two Code tasks to transform the data from the parameterTags provided by the DEE into the input content the Send Notification To Driver is expecting. Then we will transform that information into what is needed to successfully post the data.

Create the content object to send to the driver

In this case we are not worried about the extended data, so we will just use the default.

import { Framework } from 'framework';

export default class {

    (/** Allows accessing external functions */
    private framework: Framework;

    constructor(framework: Framework) {
        this.framework = framework;
    }

    /*
     * Entry point of the class (IMPORTANT: don't change the signature of this method)
     * Should return an object containing the values for each output to emit
     * If necessary, use the parameter "outputs" to emit data while running the code.
     */)
    public async main(inputs: any, outputs: any): Promise<any> {
        const values = new Map<string, Tag>(Object.entries(inputs.parameterTag));

        let content = [];
        for (const [key, value] of values.entries()) {
            const prop = {
                name: value.tag,
                deviceId: value.tag,
                deviceType: value.deviceType,
                dataType: value.dataType,
                extendedData: value?.extendedData ?? JSON.parse("{\"$id\":\"1\",\"customDataType\":\"\",\"arrayType\":\"Scalar\",\"dimensions\":\"\",\"path\":\"\",\"isPersisted\":\"false\",\"defaultValue\":\"\",\"minimumSamplingInterval\":\"1000\"}")
            };
            content.push(prop);
        }

        outputs.content.emit(content);
        outputs.parameterTag.emit(inputs.parameterTag);
    }
}

export interface Tag {
    tag: string;
    deviceType: string;
    dataType: string;
    extendedData: any;
}

Create the rawData object to send to the DataCollection Task

The translation from what we receive from the Send Notification To Driver into rawData of the task Data Collection Documentation.

import { Framework } from 'framework';

export default class {

    /** Allows accessing external functions */
    private framework: Framework;

    constructor(framework: Framework) {
        this.framework = framework;
    }

    /*
     * Entry point of the class (IMPORTANT: don't change the signature of this method)
     * Should return an object containing the values for each output to emit
     * If necessary, use the parameter "outputs" to emit data while running the code.
     */
    public async main(inputs: any, outputs: any): Promise<any> {
        const values = new Map<string, Tag>(Object.entries(inputs.parameterTag));
        const reply = Array.from(inputs.reply) as Array<TagReply>;

        let dcp = new Map<string, DataCollectionPoint>();
        for (const [key, value] of values.entries()) {

            for (const replyValue of reply) {
                if (value.tag === replyValue.propertyName) { 
                    const point = {
                        value: replyValue.value,
                        sampleId: "Sample 1",
                    };
                    dcp.set(key, point);
                    break;
                }
            }
        }
        outputs.dataCollectionPoints.emit(dcp);
    }

}
export interface Tag {
    tag: string;
    deviceType: string;
    dataType: string;
    extendedData: any;
}
export interface TagReply {
    propertyName: string;
    originalValue: any;
    value: string;
}
export interface DataCollectionPoint {
    value: string;
    sampleId: any;
}

Dynamically Get Properties

We will now dynamically invoke the Send Notification to Driver Documentation. We can use the extensibility to use in runtime, without having to pre declare in the driver definition all the information regarding events, commands and properties. In this case, we will dynamically GetProperties.

Extensibility

If we check the documentation for OPC-UA Documentation, in the section under methods, it describes how we can use the extensibility feature. In this case we want to get properties, to collect OPC-UA tags. In order to do that, we need to send a Request for the driver with the topic connect.iot.driver.opcua.getPropertiesValues and as the documentation explains, it requires a name, deviceId, dataType, deviceType.

In order to use the extension entry point we could invoke directly from the Code Task, in this case we want to separate the business logic, so we will use the default task for this Send Notification To Driver. We will connect the data from the On System Event to the Code Task retrieving the object parameterTag. Notice these are the keys of the object created in the DEE.

Using Only the Code Task

Post to Data Collection

Now we have everything we need to perform a post to the material DataCollectionInstance, we use a Synchronize task, to make sure we have everything ready when we post. We use the converter Get Object Property path: materialName, type string, to retrieve the material name from the data output and then the converter Create System Entity Instance, with entityType Material and identifier Name to create a shell object of the Material. The Data Collection task will receive a material and the rawData corresponding the to dataCollectionPoints emitted by the code task.

This was to show how you can dynamically get properties, similar variations exist for all actions, set properties, execute commands and also notify and register events.

Use Case MES

We will now take some time to explain the full use case.

In this example we will create a generic table that relates Resource <-> Parameter <-> Device Id <-> DataType <-> DeviceType. In the TrackIn, we will retrieve the resource instruments attached to the resource and resolve the table, in order to retrieve the Device Ids to collect from the OPC-UA server. Then invoke Connect IoT to collect and post to the DataCollectionInstance that was opened in the TrackIn for the material.

Create a Generic Table

We created a Generic Table with the Resource (Name) and Parameter (Name) as keys, the DeviceId and DeviceType as text fields and the DataType as the Enum Cmf.Foundation.BusinessObjects.AutomationDataType. The DeviceType is part of the extension data of the Protocol, so we have no way of providing that information for the table, without being protocol specific.

We can add some data to the table matching resource instruments, parameters and OPC-UA tags.

Create a DEE in Track In

We will now create a DEE that will, resolve the instruments attached to the resource, and correlate the CurrentDataCollectionInstance of the Material with the parameters defined in the table for the attached instruments.

Here we are using some specific project utilities, but bellow the pertinent code is shown.

UseReference("System.Data.dll", "System.Data");
UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects");
UseReference("Cmf.Foundation.BusinessOrchestration.dll", "");
UseReference("", "Cmf.Foundation.Common.Exceptions");
UseReference("", "Cmf.Foundation.Common");
UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects");
UseReference("Cmf.Navigo.BusinessOrchestration.dll", "");
UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.MaterialManagement.InputObjects");
UseReference("Newtonsoft.Json.dll", "Newtonsoft.Json.Linq");
UseReference("%MicrosoftNetPath%Microsoft.CSharp.dll", "");
UseReference("Cmf.Custom.Project.Common.dll", "Cmf.Custom.Project.Common");
UseReference("Cmf.Common.CustomActionUtilities.dll", "Cmf.Common.CustomActionUtilities");
UseReference("Cmf.Custom.Project.Common.dll", "Cmf.Custom.Project.Common");

IMaterial material = DeeContextHelper.GetContextParameter("material") as IMaterial;
IResource resource = DeeContextHelper.GetContextParameter("resource") as IResource;

material.Load();
resource.Load();

// Retrieve Resource Instruments
if(resource.RelationCollection == null || !resource.RelationCollection.ContainsKey("ResourceInstrument")) {
    resource.LoadRelations("ResourceInstrument");
}

var filters = new List<IFilter>();
if(resource.RelationCollection != null && resource.RelationCollection.ContainsKey("ResourceInstrument")) {
    foreach(ResourceInstrument item in resource.RelationCollection["ResourceInstrument"]) {
        filters.Add(new Foundation.BusinessObjects.QueryObject.Filter()
        {
            Name = "Resource",
            Operator = Cmf.Foundation.Common.FieldOperator.IsEqualTo,
            Value = item.TargetEntity.Name,
            LogicalOperator = Cmf.Foundation.Common.LogicalOperator.OR
        });
    }
}

DataSet ds = Cmf.Custom.Project.Common.ProjectUtilities.GetGenericTableData("CustomIoTDataCollectionContext", filters.ToArray());

if (ds.HasData())
{
    IDataCollectionInstance dataCollectionInstance = material.CurrentDataCollectionInstance;
    dataCollectionInstance.Load();
    
    IDataCollection dc = dataCollectionInstance.DataCollection as IDataCollection;
    dc.LoadRelations("DataCollectionParameter");

    if(!dc.DataCollectionParameters.Any()){
        throw new Exception("No Parameters found");
    }
    var parameters = dc.DataCollectionParameters.Select(dcp => dcp.TargetEntity.Name);

    // This is the object that will hold the parameter and as value, everything required to perform a get
    JObject parameterTag = new JObject();
    foreach (DataRow row in ds.Tables[0].Rows)
    {
        var rowParam = row["Parameter"].ToString();
        if (parameters.Contains(rowParam))
        {
            JObject tagInfo = new JObject();
            tagInfo["tag"] = row["DeviceID"].ToString();
            tagInfo["deviceType"] = row["DeviceType"].ToString();
            tagInfo["dataType"] = row["DataType"].ToString();

            parameterTag[row["Parameter"].ToString()] = tagInfo;
        }
    }

    // This code implies there is an Instance associated between the Resource, the Controller and Driver
    var instance = resource.GetAutomationControllerInstance();
    if (instance == null)
        throw new Exception("Resource not connected to any IoT instance");
    else
    {
        JObject data = new JObject();
        data["materialName"] = material.Name;
        data["parameterTag"] = parameterTag;

        string topic = "Cmf.Trackin.LineClearance";

        // This Code will publish a message bus broadcast for the specified topic, with the specified data
        instance.Publish(topic, data);
    }
}

This is the code pertaining to Cmf.Custom.Project.Common.ProjectUtilities.GetGenericTableData, invoked in the DEE.

/// <summary>
/// Returns a dataset with the filtered data from a GenericTable
/// </summary>
/// <param name="table">The Generic Table to filter</param>
/// <param name="filters">The filters to apply</param>
/// <returns>A DataSet with the filtered content, null otherwise</returns>
public static DataSet GetGenericTableData(IGenericTable table, params IFilter[] filters)
{
    DataSet output = null;
    if (table != null && table.Id > 0)
    {
        IFilterCollection tableFilters = new FilterCollection();
        if (filters != null)
        {
            tableFilters.AddRange(filters);
            table.LoadData(tableFilters);
        }
        else
        {
            table.LoadData();
        }
        if (table.Data != null)
        {
            output = NgpDataSet.ToDataSet(table.Data);
        }
    }
    return output;
}

/// <summary>
/// Returns a dataset with the filtered data from a GenericTable
/// </summary>
/// <param name="tableName">The name of the Generic Table</param>
/// <param name="filters">The filters to apply</param>
/// <returns></returns>
public static DataSet GetGenericTableData(string tableName, params IFilter[] filters)
{

    IGenericTable table = new GenericTable();
    table.Load(tableName);

    if (table != null)
    {
        return GetGenericTableData(table, filters);
    }
    return null;
}

After the message is published IoT will take care of the Data Collection Post.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. How To

Overview

With the introduction of the code task, the need of customization has decreased substantially. Some of the complexity of customization has shifted to the code task. In software development, visibility and transparency on what is happening is essential, to better understand if the work fits the requirements and does not have undesirable side effects.

How To

Using the visual studio code as administrator we will debug the code task. In the code task add a hook debugger; this hook will provide a breakpoint for the appended execution of the visual studio code debugger to stop.

Start the Automation Manager and open visual studio code and select F1. After selecting F1 a dropdown will appear, in this case we want to append to a node process, so select Debug: Attach to Node Process. It will show several node processes, the automation manager spawns at least 3 processes, one for the monitor, other for the driver and another for the controller. The code task is executed by the controller process, so that is the one we want to attach.

Now, whenever the code task is invoked it will stop in VS Code.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. System Requirements
3. Using the MES GUI
4. Using the Setup
5. Running as a Console
6. Productive Deployment
   1. Running as a Windows Service
   2. Using the DevOps Center

Overview

Connect IoT supports multiple architectures to best adapt to the integration scenario. The architecture of the application is centered on having a main or parent process, the Automation Manager and that process is responsible for spawning all the subprocesses of ConnectIoT.

Each process that constitutes Connect IoT is a nodejs process. The smallest amount of nodejs processes that Connect IoT requires to be fully functional is four. It requires the parent process Automation Manager, the Automation Monitor, then an Automation Controller and an Automation Driver.

Connect IoT also, requires a registry with the packages required to run. These packages may be in a directory or in an NPM registry, also they can be in a directory that the MES has access and then the Automation Manager, may request through REST requests the packages from the MES.

The Automation Manager also generates logging and is able to write to files process data, through a mechanism called persistency.

It is important to have a global idea of the architecture, so as to understand what may required in terms of deployment.

System Requirements

The system requirements vary according to what drivers your implementation uses. Some of them have particularities that are described in the help Connect IoT Requirements.

Regarding hardware it is somewhat hard to know before hand, as it depends on what you are using the manager for and the level of traffic and processing you are expecting. The hardware needs also vary by driver, some are more resource intensive than others. It is accepted that typically the bottleneck is RAM allocation Hardware Requirements.

Using the MES GUI

The first interaction most Connect IoT users has with the application is through the Automation Manager GUI, in that GUI it’s possible to download a zip file with all that is needed to run the Automation Manager. The download will also generate a new authentication token for the user selected as integration user for the download.

It is important to note that the GUI allows you to customize for each particular manager the configuration that you want that manager to export when downloading. Also that configuration can be changed for all Automation Managers in the config /Cmf/System/Configuration/ConnectIoT/ConfigurationTemplate/, or in the specific Automation Manager. This may be very useful, to have pre configured all the configurations for what is relevant in your case.

The zip file that is generated contains the following:

• scripts (folder)
• service (folder)
• src (folder)
• config.json
• npm-shrinkwrap.json
• package.json
• README.md

Some of these are more obvious than others.

Looking at the files:

Regarding versioning and metadata, that is assured by the npm-shrinkwrap.json and package.json 📄. The README.md 📄 contains the documentation on the automation manager entity The config.json 📄, controls all the key configuration for the Automation Manager and is a key part of the deployment process. All the fields of the config.json are described in the documentation under Help IoT Runtime Components Configuration

Looking at the folders:

The src 📁 folder is where the everything needed for the Automation Manager to run sits. In essence it is the source code of the application. The scripts and service folder are utilities that are included in the zip in order to be of easy use. The service 📁 has everything needed for the Automation Manager be installed as a Windows Service. The scripts 📁 has StartConsole.bat which enables to run the Automation Manager as a console application, it also has the InstallService.ps1 this script allows you to install as a windows service.

Using the Setup

The Critical Manufacturing Setup allows the deployment of an Automation Manager or several through a GUI. This method is useful if you want to deploy several managers, making the manual download through the GUI impractical and if you prefer to have a simple GUI abstraction.

In the Help Connect IoT Installation all the setups and configurations are explained. It works by selecting to deploy the Cmf.ConnectIoT.Packages and then it allows for filling in the GUI all relevant fields for the config.json of the manager. It also supports the possibility of installing as a windows service.

Running as a Console

This method is very useful, when developing or when troubleshooting an installation. It consists on running the process as a console, by using the StartConsole.bat script. When you run as a console, all the logs are aggregated as console output, so it may be easier to spot errors that are occurring.

Productive Deployment

There are several different ways to have a productive deployment of the Automation Manager, but before starting with the differences, let’s start with what is the same.

In a ❗High Availability❗ scenario, if a manager goes down, another must start to take it’s place. It’s important that whenever a new process starts, it doesn’t start from scratch and that it inherits the context that the previous manager was working with. This is very impactful when thinking about implementations that are very reliant on a persistent layer for tracking or data collection. If manager 1 is tracking a material A and for some reason manager 1 goes down, it’s important that the new manager 2 has a notion of material A. That is why typically, we suggest that the Storage/Persistency layer be stored in a shared folder accessible to all manager nodes, of course this depends on if the implementation uses the persistency layer for any logic.

It is also important to configure the logging appropriately, this is critical for very verbose drivers and implementations. You can configure retention times, rollovers, verbosity, etc Help Logging Configuration. If it’s important to keep the logs for a long time and a relevant amount, consider to also offload the logs to a shared folder, for risk mitigation.

Certificates: It is required that a certificate is defined for the Automation Manager. The certificate must be placed (or path to the certificate) in an environment variable NODE_EXTRA_CA_CERTS Certificate Troubleshooting. For development purposes the flag NODE_TLS_REJECT_UNAUTHORIZED can be set to 0 to use the certificate exactly as received. For more information and as a curiosity Extra CA Certificates Nodejs & Nodejs original issue

Running as a Windows Service

This was traditionally the main approach of a production environment and is still very much a possibility today. Some drivers, like OIB and OPC-DA are windows based, one because it is built on NetFramework and the other because it uses the DCOM, which are windows exclusive. As explained before, both the download action in the MES GUI and the Setup provide easy ways to install the Automation Manager as a Windows Service Library used for Creating nodejs windows services. It is important to highlight that the user used to run the windows service is also very important, as the permissions, or lack thereof may cause side effects. Keep in mind also, that if the password has expiration this will impact the automation manager, that is running that user.

High Availability Running as a Windows Service

Achieving High Availability using windows services is possible using a microsoft technology the Windows Server Failover Cluster Help Windows Server Failover Clustering. This allows for creating a cluster of different machines, where each machine will has installed the windows service we want to monitor. The Failover Cluster guarantees from the nodes, one and only one is always running. If one of the nodes goes down, meaning a windows service is shutdown, it will start in the other node a new windows service. The key and important part here is that node 1 and node 2 and node N all share the same accesses, configurations and so on, this enables it to be as smooth as possible when the failover occurs.

Using the DevOps Center

Critical Manufacturing following the industry trend and simplifying the deployment process, created a portal to be able to remotely perform the deployments, this is the DevOps Center Help DevOps Center. It also supports different orchestrators for deployment namely docker swarm and kubernetes and also different deployment targets Help Deployment Targets.

Using the Agent

Assuming you have an agent running in your machine, that agent will have the possibility to deploy automation managers. Doing that is a matter of, in your Infrastructure, deploying new Environments, MES Example, for each automation manager.

In order to deploy an Automation Manager select the Deployment Package, Automation Manager 10.1.0, or whichever version you need. Then select the target for the deployment. In the General Data, specify the Manager Id that you are deploying and the manager configuration related to system Help System Configuration. Regarding service resources, the automation manager will just run one instance of itself so just one replica. Volumes, will be where you specify where you will place the storage/persistency and the logs, also where the Connect IoT packages are accessible.

For certain drivers it is also important to create volumes, this is the case for example for all drivers that require file monitoring and processing. The DevOps Center allows you to set those mounts through the GUI as well. Currently, the only way to set open particular ports or ranges is by editing configuration yaml in the manager deployment.

Using the Agent, the agent will automatically deploy the stack.

Standalone Installation

The Standalone Installation works in the same way as using the agent, but it will generate a zip file that can be manually executed. In other words it is the previous method but with a manual deployment.

The zip file that is generated depends on the Target selected, but will consist on scripts to deploy and remove the stack and then the needed configuration yaml files.

Enabling Inbound Traffic: When using a driver that requires an external system to connect to it, instead of it connecting to an external system, a port will need to be opened for this access. Currently, the DevOps Center does not support setting it through the GUI. In order to do this, it will require a manual change in the configuration yaml, or in the system that is managing your stack, i.e portainer to open the port.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview
2. Use Case Overview
3. Use Case Implementation
4. Bonus

CAUTION: Just because you can, doesn’t mean you should. Using the Message Bus as broadcaster between controllers can be very useful and powerful, but in a sense it’s an anti-pattern. It bleeds over functionality between Controllers and bypasses the MES. Also, if not correctly setup it may be broadcasting payloads to a large number of active subscribers. The bigger the payload and the bigger the number of subscribers, the bigger the impact on the traffic in the network. You should always, only send exactly what you need and be as Peer to Peer as possible. Also, this is not transaction, this is an asynchronous method of communication. Even though we can send messages and wait for a reply.

Overview

Connect IoT supports multiple architectures to best adapt to the integration scenario. When a machine has multiple interfaces, typically we can have one controller that orchestrates multiple drivers. The MES typically holds and persists the key process information and orchestrates and enriches the Connect IoT as needed, this way being able to save history, provide analytics and being more transparent to the end user.

Nevertheless, there are scenarios, where communication between controllers is needed, in a close to synchronous fashion. Where the push of information for a particular controller is not the machine nor the MES but the other controller. Critical Manufacturing provides a tool for just this type of broadcasts, the Message Bus. This tool enables us to subscribe to topics and to broadcast messages. This scenario is also very helpful for interactions between the GUI or in pipelining DEEs for the Connect IoT.

This tutorial will focus only in a very simple example in MQTT. Other scenarios may be addressed in future posts.

Use Case Overview

The use case for demonstrating this type of interaction, will be two lines. Where one controller is focusing on controlling one particular line. The Line 1 Controller will receive a PartsFinished and will then forward that information to the Line 2 Controller, which will iterate and set the value for the property partsToBeProduced. It will also perform a Line Clear that will clear the counter on Line 2.

This is a very simple interaction, but you can imagine the case, where you want segregation between two lines, and which are functionally different, but according to some logic you want the following line to be informed immediately on what hast been produced and is now to to be produced on the next line. Some parts of this, of course may make sense to represent in the MES and to not only broadcast a message but also, to persist in the MES this information.

Use Case Implementation

For this implementation we will use an MQTT broker Mosquitto.

First, let’s create the Automation Protocol, just keep the default values:

Now we need to create two different Automation Driver Definitions. For Line 1 we will have two events, one for the PartFinished and in Line 2 we will set a property value.

For Line 2, let’s define the property that we want to set:

Line 1 Configuration

Now we need to create two Automation Controllers, starting on the Line 1. Leave the Setup as default and create a new page Part Finished. For Part Finished we will receive an Event and broadcast a message to our new topic Cmf.Line.PartsFinished.

To try and make our solution a bit more generic, lets add context to our event and instead of just being a passthrough, lets also add the line as a context. This is done in the task Expression Evaluator.

Checking Message Bus Message through the GUI

In the Administration/MessageBus section, we can generate and subscribe to Message Bus messages. If you want to subscribe all messages, use the special character >. For our use case we can see the message being broadcast by adding a subscription to Cmf.Line.PartsFinished.

Line 2 Configuration

Leave the Setup as default and create a new page Receive PartsFinished. In this page we will subscribe to the message bus topic Cmf.Line.PartsFinished and we will retrieve the last value sent and set the property with the last value.

IMPORTANT: In this case we are using the context to filter messages that are from line 1 with a Switch, this allows to be more dynamic, on what messages to consume or not, this can also be done by generating unique topics that only the two peers know, allowing for a peer to peer communication. It’s very important to try and minimize traffic in the network, so you want to as most as possible to only broadcast the message to who you want as a consumer.

Getting it to Work

Create an Automation Manager and Connect the Automation Controllers. In my case, I have a manager called the MQTT_Manager.

First, let’s start the broker with mosquitto.

.\mosquitto.exe -v

We will need a publisher to broadcast the message of the parts finished and a subscriber to receive the parts to be produced.

.\mosquitto_sub.exe -h localhost -t "Line.2.PartsToBeProduced"

Start the Automation Manager, this will start both Controllers and Drivers.

Let’s start sending messages and see how the system behaves. For now, we will send two messages with 1.

.\mosquitto_pub.exe -h localhost -t "Line.1.PartFinished" -m "1"

You can also see the value changing on the subscriber.

Right, now we already have a communication between both controllers through the Message Bus.

Bonus

Let’s implement a Reset, so the use case would be to have a line clearance, where you want to cascade reset the counters.

In the driver definition create a new property for the topic Line.1.ClearLine and create a new event OnLineClear. For the Line 1 Controller create a new page Line Clear. In this page we will receive an event and activate a Message Bus Message for the topic Cmf.Line.Clear, in this case we will not add context.

In Line 2 we will now have a new page Reset Counters where we persist the value of the counter as 0 and set the property as 0.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Overview and Motivation
2. Dependencies & Installing
3. Generating Documentation
4. Creating your Templates

Overview and Motivation

One of the more difficult tasks a software developer has is producing documentation. Documentation is a very hard process, because it requires someone that understands profoundly the topics he is writing about, but at the same time is able to write in a way that is free from the assumptions and biases he has. Templating is good way to guide the writer into have a standardized approach to writing documentation and helping him with a set structure of topics he has to gather and fill.

The goal of this tool is to make it easier to write documentation and standardize the documentation through different use cases and in different teams. Also, the goal is to have it in such a way, where even people that are not developers are able to use and contribute.

Dependencies & Installing

This tool requires the pre installation of:

• Visual Studio Code
• CMF CLI

The tool is based on a visual studio code extension called VSCode Folder Templates, also for convenience it installs an extension for mermaid VSCode Markdown Mermaid. Mermaid is a tool to generate diagrams and more information on how to generate diagrams can be found here Mermaid and they also provide a very helpful Mermaid Live Editor.

To install the tool:

npm install @criticalmanufacturing/doc-gen --registry=https://dev.criticalmanufacturing.io/repository/npm-public --global

The tool provides a set of predefined templates that you can import in your vscode, these templates can also receive inputs.

Generating Documentation

Right now, the tool only has templates for IoT documentation. So the command available is to install IoT templates.

In order to install the IoT templates, you can run:

cmf doc-gen install iot

The command supports generating to a particular folder using pathForTemplates. The command also allows to add an exception in your .gitignore file, to not commit your templates with --persistInProject, if false it will add to the exclusions. It also has another option which is --localInstall, if this is passed as false it will install globally and be available every time you open vscode.

By default it adds the exclusion in your .gitignore file if it exists and creates locally.

The templates will be generated in a folder called .fttemplates:

In order to use them, just right-click and select Create New Templated Folder, select the template and fill the inputs:

Ok, we have created our template and now we can edit it!!!

Creating your Templates

If you want to add your own templates, you can edit what is under .fttemplates, and edit existing or add new one’s. If, for example in your project you want to add templates, make sure to commit the folder .fttemplates, so you don’t lose them.

All the documentation on how to create templates is in VSCode Folder Templates and also, you can check all our examples. Feel free to suggest new templates and to contribute. The source code for this plugin is currently under Devops doc-gen

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Notification
2. Setting up the MES
3. Adding to our Workflow
4. Enriching our Notification

Notification

A Notification is an asynchronous way of broadcasting a message to all or some MES users. It allows to generate an alarm within the MES and without namely through sending emails. This is very useful to call the users to act upon a particular event. In this use case we will setup for a case, where we want to be notified whenever the system is disconnected.

Info: Notifications are part of Alarm Management, which is a Critical Manufacturing optional module.

Info: The user must have an Employee object associated in order to be able to use Notifications.

Setting up the MES

In order to create a notification we must create a Rule that will invoke a DEE that will create a notification. This DEE may have inputs that will feed information to our Notification.

Creating a DEE

In this use case we will want to send a notification on disconnect. I will create a DEE called CustomSendNotificationOnDisconnected with classification ConnectIoT. I want my DEE to be flexible in creating a notification, this can be achieved with a table or in our case with configuration.

The first thing in our DEE is importing Foundation:

UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects");

var serviceProvider = (IServiceProvider)Input["ServiceProvider"];

Now we can retrieve all our configs, in my case they live under /Notification/:

Config.TryGetConfig("/Notification/Name", out var notificationName);
Config.TryGetConfig("/Notification/Description", out var notificationDescription);
Config.TryGetConfig("/Notification/Details", out var notificationDetails);
Config.TryGetConfig("/Notification/EmailDistributionList", out var notificationEmailDistributionList);
Config.TryGetConfig("/Notification/SendEmailToAssignedParty", out var notificationSendEmailToAssignedParty);
Config.TryGetConfig("/Notification/Severity", out var notificationSeverity);
Config.TryGetConfig("/Notification/Title", out var notificationTitle);
Config.TryGetConfig("/Notification/Type", out var notificationType);
Config.TryGetConfig("/Notification/NotificationAssignedToRole", out var notificationAssignedToRole);

Feel free to create the configurations and populate with default values.

Now we can create our notification:

Cmf.Foundation.BusinessObjects.Abstractions.INotification notification = serviceProvider.GetService<Cmf.Foundation.BusinessObjects.Abstractions.INotification>();

notification.Name = notificationName.Value + Guid.NewGuid().ToString();
notification.Description = notificationDescription.Value as string;
notification.Details = notificationDetails.Value as string;
notification.EmailDistributionList = notificationEmailDistributionList.Value as string;
notification.SendEmailToAssignedParty = ((bool?) notificationSendEmailToAssignedParty.Value) ?? false;
notification.Severity = notificationSeverity.Value as string;
notification.Title = notificationTitle.Value as string;
notification.Type = notificationType.Value as string;

if (!string.IsNullOrEmpty(notificationAssignedToRole.Value as string))
{
	IRole role = serviceProvider.GetService<IRole>();
	role.Load(notificationAssignedToRole.Value as string);
	notification.AssignedToRole = role;
}

notification.Create();

For now we are just using the configurations to provide information to our notification.

Now we can execute our DEE (select Execute button) and see the notification being created.

Creating a Rule

The DEE is working, we need the action that will wrap it and be used to trigger it. The action must be of Scope ConnectIoT and of course it needs to select the DEE that was created.

Adding to our Workflow

We have a Rule and this rule can be invoked in the Connect IoT Workflow. In our example, we want to create a notification if we disconnect. The communication state is given by the task On Equipment Setup output OnCommunicationChange. This output will always emit the new communication state. In order to address, this we will create a switch for the state machine of the communication state.

After having our switch we can use the Disconnected output to feed our Rule. In order to invoke rules in Connect IoT we use the task Execute Action. In this task, select the Rule that we’ve created.

Info: Only Rules created under the scope ConnectIoT will show in the task to be selected.

Connecting everything:

Running the Automation Manager

In order to test this implementation, simply start the Automation Manager and then Stop it. This will trigger a Disconnected communication state that will trigger the notification.

You can see the notification is called and it will show as the one generated when we executed the DEE.

Enriching our Notification

In our previous use case, even with configurations, we had a very static definition of our notification. Let’s then save an error message in the Notification details. This could come from the Error output of a task, but in this case let’s just create a constant message saying Controller Disconnected.

In the Execute Action task create an Input ErrorMessage and as default value Controller Disconnected. In the DEE we will need to change it to receive an external input called ErrorMessage and change the details of the notification to log our error message.

var errorMessage = Input["ErrorMessage"] as string;

notification.Details = errorMessage;

Save the DEE as a new effective version.

Now, repeating the process of starting and stopping the automation a new notification is create and has as details our expected message.

This finishes our post, as you can see the mechanisms of communication between the Connect IoT and the MES are very powerful and can provide a lot of synergies.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. What is a Unit Test?
2. What do we use Unit Testing for?
3. Creating a Unit Test
4. Creating a Unit Test for Tasks
5. Bonus Content - Coverage

What is a Unit Test?

Unit testing is a type of testing which we try to isolate the components or object of testing. In this type of test we are testing, not the interaction with multiple components, but the most atomic set of functional code. For example, testing a workflow execution and seeing if a request for MES is at least an interaction between several tasks (which are blocks of code components) and an external system, like an MES or even a persistence layer.

In this post I want to talk about testing an IoT Task. An IoT Task is something that is mostly contained, it may or may not have settings, inputs and outputs and it may have colateral effects. Typically, a task behaves mostly as a function with inputs and outputs and some configurations, so it’s a prime target for Unit Testing.

The main advantage of unit testing are the consistency checks, speed of execution and “noise reduction”. A unit test will maintain that give conditions A a task will always give output B and will not be burden by externalities, if for some reason a change is introduced in the task and now A is C, the consistency is no longer and the test will fail. They are very fast as they are only code executions or function executions, so its small scope and focus ensure atomic executions.

As mentioned previously this allows you to concentrate in testing the component without having to worry about interactions with other systems or components, of course they don’t replace End to End tests or integration tests as those interactions are outside the scope of this type of test.

Another advantage is that they can be run at commit time and not at build time. Being very fast, they can be run using git hooking tools like husky. This can be a big advantage, because if the tests are going to fail we want them to fail as soon as possible and not wait for a cumbersome build to fail. So we can save time and resources.

I hope it is clear what are the advantages and disadvantages. For more information about testing I would check GeeksForGeeks or Atlassian.

What do we use Unit Testing for?

The IoT Tasks scaffolding already gives everything you need to start testing. It will generate a /test/ folder in your taskPackage, also in also creates the package.json with all the devDependencies you need to run the tests.

It will mainly use two tools mocha and chai.

Mocha is a test framework for javascript tests. The tests will have a test suite under describe, they can have some steps before executing in a beforeEach, before testing is where the containers for the normal components should be overriden by the test components, more on this further ahead. A test is defined by it followed by what it should do. For example, it('should return a string with Hello World', ... ).

The addition of chai is to further strenghten our assertion to be as word like as possible. For example, using chai we can write our test in way that is much clearer. We are not asserting that result result === "Hello World", we are expecting result to be equal to Hello World expect(result).to.equal('Hello World'); which is much more transparent.

it('should return a string', async () => {
        
        (...)

        expect(result).to.equal('Hello World');
        expect(result).to.be.a('string');
    });

There is more variation, I suggest reading the documentation on both these tools to unlock their full potential.

Creating a Unit Test

Let’s start with the simplest and most atomic business logic unit of IoT, the converter. I am using an example of a converter that was generated but no tests have been created yet, so it’s a good opportunity to use a real use-case. The template has generated something like this:

import "reflect-metadata";
import { Task, System, TYPES, DI, Converter } from "@criticalmanufacturing/connect-iot-controller-engine";
import EngineTestSuite from "@criticalmanufacturing/connect-iot-controller-engine/test";
import * as chai from "chai";
import { CustomArrayDeltaCalculationConverter } from "../../../../src/converters/customArrayDeltaCalculation/customArrayDeltaCalculation.converter";

describe("Array Delta Calculation converter", () => {

    let converter: Converter.ConverterContainer;

    beforeEach(async () => {
        converter = await EngineTestSuite.createConverter({
            class: CustomArrayDeltaCalculationConverter
        });
    });

    it("should convert", async (done) => {
        /* Example int to string*/
        let result: string = await converter.execute(123, {
            parameter: "something"
        });

        chai.expect(result).to.equal("123");
        
        done();
    });

});

Taking a glance at this we can see somethings that are familiar to us. We have the import of chai, so we know we can use our assertion methods, also we are importing the controller engine and its test package, finally we have our own custom converter. In the body we can see our describe from mocha, so this is where our test suite begins, we see it’s already creating our converter container, and we already have a dummy test showing how we can invoke our converter.

In order to start building tests let’s see what our converter does. We can see in the comment it receives an array, a type and extracts the value from that position and typecasts it. This is common, converter tasks shouldn’t be bloated in terms of functionality, they should be straightforward.

/**
 * Extracts a value from a position of an array and type casts it
 * @param position position that we want to extract from array
 * @param type type of the value to be extracted from the position
 */
    transform(value: object, parameters: { [key: string]: any; }): any {

    if (!(value instanceof Array)) {
        throw new Error("Given input value is not an instance of an Array");
    }  else {
        const result = Utilities.convertValueToType(value[parameters["position"]], parameters["type"], undefined);
        return result;
    }
}

Before starting to code, let’s add the vscode launch setting in order to be able to debug our tests. So in our case we have a custom task package, generically called utilities, we are saying that we want to run from the package, so our workspaceRoot is where our console will launch and we want to run mocha and our tests are under test and end in .test.js.

{
   "name": "Run tests on controller-engine-custom-utilities-tasks",
   "type": "node",
   "request": "launch",
   "program": "${workspaceRoot}/node_modules/mocha/bin/_mocha",
   "stopOnEntry": false,
   "args": ["test/**/*.test.js", "test/*.test.js", "--no-timeouts"],
   "cwd": "${workspaceRoot}",
   "runtimeExecutable": null,
   "sourceMaps": true,
   "outputCapture": "std"
}

In VSCode we can now Run and Debug and select our test and add breakpoints and debug our test. A useful command that comes out of the box from the scaffolding is:

npm run watchTest

This command will always compile the tests when a change is detected. There is a watchPackage for the src and it behaves the same and also a watch which makes sure to compile both are always keep running the unit tests.

Our first test will be to make sure that if we receive an invalid input, in other words not an array, we want to receive an error. We send an integer value and catch the exception and compare the message to the expected value. This way we cover that our converter will only work when expected.

it("should fail if value is not an array", async () => {
    chai.expect(
        await converter.execute(123, {
            type: Task.TaskValueType.Integer,
            position: 0
        }).catch(error => error.message)
    ).to.equal("Given input value is not an instance of an Array");
});

Awesome, very simple, now let’s do a happy test scenario, we will give it an array with two positions, select the second and cast it to number.

NOTE: Sometimes it’s useful to add it.only, to our test, this will make it so it only runs that particular test.

it("should succeed in retrieving second array position and converting it to number", async () => {

    const result = await converter.execute(["test", 1], {
                        type: Task.TaskValueType.Integer,
                        position: 1
                    });

    chai.expect(result).to.be.a('number');
    chai.expect(result).to.equal(1);
});

In this test, I splitted the execution to a property to be more legible. The test validates the output is a number and its value is 1. Pretty simple, right! 👏

Creating a Unit Test for Tasks

As you can imagine, tasks have more variability, but the good news is that the jist is more or less the same. So let´s give it a try. Again, there’s a lot that already comes in the scaffolding, so let’s take a look at what comes in the scaffolding, in this case this was generated for a task called CustomResolveMetadataFeature.

import "reflect-metadata";
import { Task, System, TYPES, DI } from "@criticalmanufacturing/connect-iot-controller-engine";
import EngineTestSuite from "@criticalmanufacturing/connect-iot-controller-engine/test";

// import { DataStoreMock } from "@criticalmanufacturing/connect-iot-controller-engine/test/mocks/dataStore.mock";
import * as chai from "chai";

import {
    CustomResolveMetadataFeatureTask,
    CustomResolveMetadataFeatureSettings
} from "../../../../src/tasks/customResolveMetadataFeature/customResolveMetadataFeature.task";
import CustomResolveMetadataFeatureTaskModule from "../../../../src/tasks/customResolveMetadataFeature/index";

describe("CustomResolveMetadataFeature Task tests", () => {


    // Optional: See container handling under customResolveMetadataFeatureTestFactory
    // let dataStoreMock: DataStoreMock;
    beforeEach(() => {
        // dataStoreMock = new DataStoreMock();
    });

    const customResolveMetadataFeatureTestFactory = (  settings: CustomResolveMetadataFeatureSettings | undefined,
                                            trigger: Function,
                                            validate: Function): void => {

        const taskDefinition = {
            class: CustomResolveMetadataFeatureTaskModule,
            id: "0",
            settings: settings || <CustomResolveMetadataFeatureSettings>{
                message: ""
            }
        };

        EngineTestSuite.createTasks([
            taskDefinition,
            {
                id: "1",
                class: Task.Task({
                    name: "mockTask"
                })(class MockTask implements Task.TaskInstance {
                    [key: string]: any;
                    _outputs: Map<string, Task.Output<any>> = new Map<string, Task.Output<any>>();

                    async onBeforeInit(): Promise<void> {
                        this["activate"] = new Task.Output<any>();
                        this._outputs.set("activate", this["activate"]);
                        // Create other custom outputs (for the Mock task) here
                    }

                    // Trigger the test
                    async onInit(): Promise<void> {
                        trigger(this._outputs);
                    }

                    // Validate the results
                    async onChanges(changes: Task.Changes): Promise<void> {
                        validate(changes);
                    }
                })
            }
        ], [
            { sourceId: "1", outputName: `activate`, targetId: "0", inputName: "activate", },
            { sourceId: "0", outputName: `success`, targetId: "1", inputName: "success", },
            { sourceId: "0", outputName: `error`, targetId: "1", inputName: "error", },
            // Add more links needed here...
        ],
        undefined,
        (containerId) => {
            // Change what you need in the container
            // Example:
            // containerId.unbind(TYPES.System.PersistedDataStore);
            // containerId.bind(TYPES.System.PersistedDataStore).toConstantValue(dataStoreMock);
        });
    };

    /**
     * Instructions about the tests
     * It is assumed that there are two tasks:
     *    0 - CustomResolveMetadataFeature Task
     *    1 - Mockup task
     *
     * All Outputs of Mock task are connected to the inputs of the CustomResolveMetadataFeature task
     * All Outputs of CustomResolveMetadataFeature Task are connected to the Mock task inputs
     *
     * You, as the tester developer, will trigger the outputs necessary for the CustomResolveMetadataFeature to be activated
     * and check the changes to see if the CustomResolveMetadataFeature task sent you the correct values
     *
     * Note: This is just an example about how to unit test the task. Not mandatory to use this method!
     */

    it("should get success when activated", (done) => {
        customResolveMetadataFeatureTestFactory(undefined,
            (outputs: Map<string, Task.Output<any>>) => {
                // Trigger an output
                outputs.get("activate").emit(true);
            }, (changes: Task.Changes) => {
                // Validate the input
                chai.expect(changes["success"].currentValue).to.equal(true);
                // Report the test as a success
                done();
            });
    });
});

Pretty intimidating, right? But already we see things that are familiar to us. We see a describe for our test suite, we see a beforeEach and an it for our test run and also an EngineTestSuite.createTasks, similar to the EngineTestSuite.createConverter we saw and used above. There is a lot of information that is comented and that can throw you in a loop, but let’s try and see if we can untangle it.

Philosophy: The philosophy behind this, is we kind of want to have our own workflow, but just for this task. We are able to inject it with inputs and then receive as inputs the output of our task.There’s a common practice to build a diagram to try and explain how the test will work:

    /*
     *               +----------------------------------------+       +-----------------------------------------------------------+
     *               |            ==mockTask (0)==            |       |         ==CustomResolveMetadataFeature (1)==              |
     *               |                                        |       |                                                           |
     *    (*1) ----> | () barcodeOut             barcodeIn () | ----> | () barcodeIn                            barcodeOut     () | ----> (*1)
     *    (*2) ----> | () entityOut               entityIn () | ----> | () entityIn                             entityOut      () | ----> (*2)
     *    (*3) ----> | () activateOut            eventType () | ----> | () eventType                            activateOut    () | ----> (*3)
     *    (*4) ----> | () notActivateOut            inputs () | ----> |                                         notActivateOut () | ----> (*4)
     *               |----------------------------------------|       |-----------------------------------------------------------|
     *    (*5) ----> | () success                 activate () | ----> | () activate                                    success () | ----> (*5)
     *    (*6) ----> | () error                               |       |                                                  error () | ----> (*6)
     *               +----------------------------------------+       +-----------------------------------------------------------+
     */

The diagram makes clear what are the inputs and outputs of our task. I chose a somewhat complex task on purpose, so we could see some more challenges.

Right off the bat we see that our focus, before anything else should be on the createTasks, and trying to setup what will be our task execution Mock.

Ok, let’s fill in our settings with the settings of the task and some dummy values:

const taskDefinition = {
    class: CustomResolveMetadataFeatureTaskModule,
    id: "1",
    settings: settings || <CustomResolveMetadataFeatureSettings>{
        functionalityName: "functionalityName",
        enableFunctionalityName: "IsEnabledFunctionalityName",
        configurationTable: "TestSmartTable"
    }
};  

Next off, we are injecting the outputs of our mock task to our actual task, we need to add those to the EngineTestSuite.createTasks in the onBeforeInit to make sure they are instantianted. We can leave the onInit and onChanges as is.

async onBeforeInit(): Promise<void> {
    // barcodeIn
    this['barcodeIn'] = new Task.Output<string>();
    this.outputs.set('barcodeIn', this['barcodeIn']);

    // entityIn
    this['entityIn'] = new Task.Output<EntityType>();
    this.outputs.set('entityIn', this['entityIn']);

    // eventType
    this['eventType'] = new Task.Output<string>();
    this.outputs.set('eventType', this['eventType']);

    // activate
    this['activate'] = new Task.Output<boolean>();
    this.outputs.set('activate', this['activate']);
}

Adding links, the diagram is really helpful to understand this. We want to connect the mock task (0) to our task (1).

NOTE: Outputs and inputs have the same names, just for convenience but you can give it any name you want.

[
    // Inputs of task ->
    { sourceId: '0', outputName: `barcodeIn`, targetId: '1', inputName: 'barcodeIn' },
    { sourceId: '0', outputName: `entityIn`, targetId: '1', inputName: 'entityIn' },
    { sourceId: '0', outputName: `eventType`, targetId: '1', inputName: 'eventType' },
    { sourceId: '0', outputName: `activate`, targetId: '1', inputName: 'activate' },
    // Outputs of task ->
    { sourceId: '1', outputName: `entityOut`, targetId: '0', inputName: 'entityOut' },
    { sourceId: '1', outputName: `barcodeOut`, targetId: '0', inputName: 'barcodeOut' },
    { sourceId: '1', outputName: `notActivateOut`, targetId: '0', inputName: 'notActivateOut' },
    { sourceId: '1', outputName: `activateOut`, targetId: '0', inputName: 'activateOut' },
    { sourceId: '1', outputName: `success`, targetId: '0', inputName: 'success' },
    { sourceId: '1', outputName: `error`, targetId: '0', inputName: 'error' }
]

We are not overriding the driver proxy, so you can leave it undefined. Now the interesting bit, in this task it’s going to resolve a smart table and also persit data in the Connect IoT Persistency. So we need to inject our mock container for Datastore and for System.Proxy.

For Datastore, because this is very common the template helps you along. In the beforeEach uncomment the commented code, also do the same in the import:

import { DataStoreMock } from "@criticalmanufacturing/connect-iot-controller-engine/test/mocks/dataStore.mock";

...

let dataStoreMock: DataStoreMock;
beforeEach(() => {
    dataStoreMock = new DataStoreMock();
});

In our EngineTestSuite.createTasks in the (containerId) => { section we can bind our mock containers. For our Datastore we just unbind and bind ours, declared above, for the System.Proxy we need to declare what we are going to reply when we get the resolve table smart table request.

Binding the Datastore (I changed containerId to container):

(container) => {
    container.unbind(TYPES.System.PersistedDataStore);
    container.bind(TYPES.System.PersistedDataStore).toConstantValue(dataStoreMock);
}

In creating our new container to intercept the request, we need to specify it implementsSystem.Proxy, and it will require you to implement a set of functions. We will implemnt them and say if they are called throw an exception.

(container) => {
    (...)
    class MockSystemAPI implements System.SystemProxy {
        getMetadata(): Promise<Foundation.BusinessObjects.AutomationController> {
            throw new Error('Unexpected call');
        }
        subscribeActionGroup(actionGroup: Foundation.Common.DynamicExecutionEngine.ActionGroup | string,
            callback: System.SystemEventCallback): void {
            throw new Error('Unexpected call');
        }
        unsubscribeActionGroup(
            actionGroup: Foundation.Common.DynamicExecutionEngine.ActionGroup | string,
            callback: System.SystemEventCallback): void {
            throw new Error('Unexpected call');
        }
        executeQuery(queryObject: System.QueryObject.QueryObject, parameterCollection?: System.QueryObject.QueryParameterCollection,
            settings?: Utilities.SystemApiUtilsSettings): Promise<System.LBOS.System.Data.DataSet> {
            throw new Error('Unexpected call');
        }
        getObjectById(id: string, type: string, levelsToLoad?: number, typeIsTypeId?: boolean,
            settings?: Utilities.SystemApiUtilsSettings): Promise<any> {
            throw new Error('Unexpected call');
        }
        getObjectByName(name: string, type: string, levelsToLoad?: number, typeIsTypeId?: boolean,
            settings?: Utilities.SystemApiUtilsSettings): Promise<any> {
            throw new Error('Unexpected call');
        }
        loadAttributes(entity: any, specificAttributes?: string[], settings?: Utilities.SystemApiUtilsSettings): Promise<any> {
            throw new Error('Unexpected call');
        }
        adjustState(entity: any, newState: string, settings?: Utilities.SystemApiUtilsSettings): Promise<any> {
            throw new Error('Unexpected call');
        }
}

Now we implement our call:

NOTE: There may be imports you need to add (i.e import Foundation = System.LBOS.Cmf.Foundation;)

(container) => {
    (...)
    class MockSystemAPI implements System.SystemProxy {
    (...)
    async call(input: System.LBOS.Cmf.Foundation.BusinessOrchestration.BaseInput): Promise<any> {

        // Mock the execution of the LBO call
        if (input instanceof System.LBOS.Cmf.Foundation.BusinessOrchestration.TableManagement.InputObjects.ResolveSmartTableInput) {
            const output = new System.LBOS.Cmf.Foundation.BusinessOrchestration.TableManagement.OutputObjects.ResolveSmartTableOutput();

            // Validate Service Requests
            chai.expect(input.SmartTable.Name).to.be.equal("TestSmartTable");
            chai.expect(input["Values"].get("Resource")).to.be.equal("Test");
            chai.expect(input["Values"].get("Name")).to.be.equal("IsEnabledFunctionalityName");

            // Inject reply
            output.Result = {
                    [`T_ST_${input.SmartTable.Name}`]: [
                        {
                            "Resource": "Test",
                            "ResourceType": "",
                            "Name": "IsEnabledFunctionalityName",
                            "Value": "functionalityName",
                            "Area": "TestArea"
                        }
                    ]
                };

            return Promise.resolve(output);
        } else {
            return Promise.reject('Unexpected input type.');
        }
    }
}

If the request is what we want to test, we will check it’s inputs and inject our mock output.

Ok, enough setting up, let’s build a test. I like to start with fail scenarios, in this case if the entityIn input is not an entity the task will send an error saying “Please provide a valid entity”. Pretty obvious, right? Let’s do it!

    it("should get fail due to incorrect entity input", () => {
        customResolveMetadataFeatureTestFactory(undefined,
            (outputs: Map<string, Task.Output<any>>) => {

                // Trigger an entityIn with an invalid entity
                outputs.get('entityIn').emit({});
                outputs.get('activate').emit(true);

            }, (changes: Task.Changes) => {

                // Validate the input
                chai.expect(changes["error"].currentValue.message).to.equal("Please provide a valid entity");
            });
    });

We have two phases, the trigger phase and validate phase. In the trigger we inject an activate true and we give an invalid value for the entityIn. In the validate phase we use chai to make an assertion saying that we are expecting a specific error message.

Now let’s move on to a happy test scenario.

    it("should get success when activated", (done) => {
        const entity = new System.LBOS.Cmf.Navigo.BusinessObjects.Resource();
        entity.Name = "Test";
        const barcode = "testBarcode";

        customResolveMetadataFeatureTestFactory(undefined,
            (outputs: Map<string, Task.Output<any>>) => {
                // Trigger an output

                outputs.get('barcodeIn').emit(barcode);
                outputs.get('entityIn').emit(entity);
                outputs.get('eventType').emit("testEvent");
                outputs.get('activate').emit(true);

            }, (changes: Task.Changes) => {

                // Validate the input
                chai.expect(changes["entityOut"].currentValue).to.equal(entity);
                chai.expect(changes["barcodeOut"].currentValue).to.equal(barcode);
                chai.expect(changes["activateOut"].currentValue).to.equal(true);
                chai.expect(changes["notActivateOut"]).to.equal(undefined);
                chai.expect(changes["success"].currentValue).to.equal(true);

                // Validate the persistency
                const persistedValue = Object.values(dataStoreMock["_persistedDataStore"].get('TestSmartTablePersisted'))[0][0]["result"];

                chai.expect(persistedValue["Area"]).to.equal("TestArea");
                chai.expect(persistedValue["Name"]).to.equal("IsEnabledFunctionalityName");
                chai.expect(persistedValue["Resource"]).to.equal("Test");
                chai.expect(persistedValue["Value"]).to.equal("functionalityName");

                // Report the test as a success
                done();
            });
    });

Wow, that’s a lot more code, but if you take a closer look, it’s all just assertions. Same as before, we have two phases, in the trigger phase we will send some values. In the validate phase we have two different validations. We validate the outputs of the task and also the persistency layer. Remember that this task, not only emits outputs, but also persists information. Also, now we create a know entity called Resource to surpass the guard we saw in the previous test.

Bonus Content - Coverage

A very interesting advantage of these kinds of tests is that it’s very easy to extract metrics. The scaffolding already has in the package.json the command vs:test defined. In order to extract coverage you just need to run npm run vs:test.

Statements   : 67.89% ( 203/299 )
Branches     : 50.21% ( 117/233 )
Functions    : 61.82% ( 34/55 )
Lines        : 73.73% ( 188/255 )

I saw an interesting post talking about what this means feel free to take a look here softwaretestinghelp. In order to generate this report we are using cobertura and mocha-junit-reporter.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Should you use it?
2. Pre-requisites and setting up an environment
3. Connecting to the Database
4. Executing Queries
5. Database Polling

Should you use it?

No… ok blog post ended…. .

I am hearing you now in my mind, “but it’s really the only way”, “we really have to use it”. Ok ok, settle down. I will try to make a case not to use this approach.

1. Separation of Concerns - Databases are focused in persisting information in a structured way (for relational database). They are not concerned with guard clauses or with interfaces.

2. Change - Databases change over time, they can have schema changes, and because you don’t have an interface that is contracting between you and the database, things may change that impact you, without you knowing. Also, partition changes, fail-overs, load balancing, etc are a host of concerns that you are forced to have when interacting with a database, so there are a lot of points of failure.

3. Catastrophic Failure - A database may not be the brain of a system, but it is it’s memory. If you are directly interacting the database the margin for catastrophic failure is much higher. When you have an API in front of the database, the api can guard against bad requests and by being a controlled point of entry to the database can optimize for caching and for indexing, so when you bypass all that, a bad write may crash or throttle the database.

4. Security - A database should be a highly secure and the most protected component of your application. It may have sensitive data and is the greatest point of failure of an application. Having a third party application as Connect IoT doing requests is an easy security exploit. Imagine the following scenario: Upon configuring Connect IoT, by mistake the user given has admin privileges in the database, so someone decides to execute a query for droping tables, you can see how that can be an issue. Ok, this is a pretty dramatic example, but also bear in mind that the database was not made to enforce great role based levels of security, so it’s common to end up with much more permissions than you need, and if you have permission you gain responsibility for any future problem down the line.

So the database driver is one of those Uncle Ben’s kind of things: “With great power comes great responsibility”. Disclaimer aside, let’s get rolling.

Pre-requisites and setting up an environment

Currently, Connect IoT only supports MSSQL databases. The way it’s implemented it’s not hard to add engines for other types like oracle, but we haven’t had a use case for that yet and for the user that would be abstracted, so what’s valid for MSSQL is valid for everything else.

For this example, I am going to use a windows pc, with wsl and I am going to use a local mssql database as a container. Microsoft has made a great tutorial on this and that is what I used MSSQL Tutorial. I will assume that you have a CM MES system available to use.

Connecting to the Database

So, first things first, lets create a database. I created a TestDB by running:

CREATE DATABASE TestDB;

Now in the Microsoft SQL Server Management Studio we will have our database:

Now, we have a sql database up and running. 👏 👏

1. Create - Automation Protocol for the database package. Feel free to already:

   1. Change server to the correct IP
   2. Change password to the correct User (i.e sa)
   3. Change password to the correct Password (i.e 123456789)
   4. Change database name to the generated database (in this case TestDB)

   

   

2. Create - Automation Driver Definition, for now, just leave everything empty (use the previous created protocol), we just want to test the connection.

3. Create - Automation Controller (use the automation driver definition created) and also leave as default

4. Create - Automation Manager and Connect to a Resource

5. Download and Run - Automation Manager

Executing Queries

We now have a running Connect IoT Manager and it connects to the database, hurray 👏 👏

Now for the fun stuff, let’s start of by doing a Select Query. But before that run, this will create a table and insert some dummy data:

USE [TestDB];
CREATE TABLE [dbo].TestTable(
	[A] [int] NULL,
	[name] [nvarchar](50) NULL,
	[quantity] [int] NULL
) ON [PRIMARY];

INSERT INTO TestTable VALUES (1, 'banana', 150); 
INSERT INTO TestTable VALUES (2, 'orange', 154);
GO

Explaining a bit how this works. A query is a command and a command has parameters. So for example if I have Select A as Magic from TestTable where A = 1. The Connect IoT database driver interprets this as a Command of type Select and argument TestTable. Then it has parameters, in the parameters is where things get spicy.

We have two different concepts Parameter Value and Parameter Query, this matches the format of the query, the query has two parts one for defining what you want to Select the Parameter Value and what are the conditions for that selection Parameter Query.

In this example, we would add two parameters, one for the selection and another for the condition. So we can add one, for example Magic of type String, Column Name A and Protocol Data Type INT and is a ParameterValue. Then we need the condition, can be of Name MagicCondition and type String, Protocol Data Type INT, Column Name A and type ParameterQuery. In the ParameterQuery we can say that it has default value 1 and Operation is an Equal.

Key Concept: In other words the decomposition of Select A as Magic from TestTable where A = 1 would be -> [Command Type] [ParameterValue1, ..., ParameterValueN] [Command Argument] [ParameterQuery1, ..., ParameterQueryN].

For defining the output, the database driver gives you 3 modes, one if you have multiple results, so if your query will return multiple rows Array of Results, other if you are expecting multiple outputs of just one row Single Result, which will output a key-value pair of column name and value and finally one if you just want one value of a column Single Value.

Recap

We want a Command Select:

And our two Command Parameters:

Now let’s go back to the Automation Controller view and edit our workflow. Now we have a new command, so drag and drop the Execute Equipment Command task and we can select our command.

We are just testing it out, so I am going to also drag a Timer task and I’m going to connect my timestamp to Activate the Execute Equipment Command and also drag a Log Message task to log the reply. Now we can start the automation manager and take a look at the console.

Taking a look at the console we can see an array of key-value pairs with column name and the value.

Ok, so we got the basics rolling. There are other commands particularly the Raw command if you want to do very advanced queries without having to declare everything, it’s not advisable, because you lose transparency. Nevertheless, these concepts of ParameterValue and ParameterQuery hold throughout.

Database Polling

I would say 9 out 10 times, you need the database driver to do some kind of polling. Basically, you want to be notified when something changes, either a row changes or a value on a row or a row is added. You could do that already with timers and a clever use of the command tasks, but the driver already implements a configurable way to do that. These will be what the driver considers as events.

It has two modes, FireAndForget and ContinueOnProcessingConfirmation, as the name entails, the fire and forget emits whenever a change is detected, the continue on processing confirmation will only continue looking for changes when the first change is acknowledge or timed out.

For this example, we will focus on the ContinueOnProcessingConfirmation. The use case will be very simple, we want to log whenever a new row is inserted in the table and wait for 1s, before proceeding.

1. Create an Event - Go back to the driver definition and now create an event. The event will be looking at our table TestTable, we need to give it a column for it to use as reference (using * will not work) in our case I used A and we also have to give a column for the order by column I also gave it A. We are checking for new rows, that is why we selected OnNewRows. We also wan to to wait for confirmation, so we left of ContinueOnProcessingConfirmation.

2. Create a ConfirmDataProcessing Command - Remember that the event will be produced and will await for confirmation of being processed, in order to do that we will create a Command of type ConfirmDataProcessing. The Command also requires a Command Parameter of type EventReply which will specify to what event you are confirming.

3. Edit Workflow - In the Automation Controller worflow add an On Equipment Event, Timer, Execute Equipment Event and Log Message task. On the equipment event task and equipment command should the respective events and commands. Use the event task to log and to activate the timer. The Success of the timer will activate the confirmation command. Change the timer to not auto activate and to be of type sleep, and decrease the timeout.

NOTE: Timers are only activated by a boolean input (1, 0, true, false), so you will need to add a converter to constant true for $A. Timer timeout must be lesser than the timeout of the confirmation timeout, as the driver will discard value if it has finished the defined amount of retries (in our case 3).

4. Run Automation Manager - If you already have data in the table you can either go to the driver definition and check the Ignore Initial Values. I deleted all rows that I had from the previous example and will now insert some rows.

INSERT INTO TestTable VALUES (1, 'banana', 150); 
INSERT INTO TestTable VALUES (2, 'orange', 151);
INSERT INTO TestTable VALUES (3, 'grape', 152);
INSERT INTO TestTable VALUES (4, 'apple', 153);
INSERT INTO TestTable VALUES (5, 'tangerine', 154);
INSERT INTO TestTable VALUES (6, 'mango', 155);
INSERT INTO TestTable VALUES (7, 'blueberry', 156);

What is happening is the Automation Manager starts and performs the trigger query and sees no values. After that we executed the inserts, so now the database has 7 columns. The driver detects a change and is now going to process one by one and wait for confirmation. We could have it to process all the rows after a change by selecting the result data format AllNew, but normally for confirmation scenarios, you want to process one by one, to assure processing.

This finishes our post, as you can see it is a bit complex when you first look at it, but after understanding the driver syntax it becomes quite powerful.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps

Topics

1. Configuration
2. Anti-Pattern
3. Pattern

Configuration

The Automation Manager on boot, already retrieves a lot of context information, such as what drivers to start and what is the logic in the Automation Controllers that it will need to execute. Nevertheless, when implementing an integration scenario sometimes it’s very helpful to try and maintain our controller more abstract and have a dynamic resolution for some configuration such as ports, ips, what interfaces to enable, etc.

If we for example, want to use an Automation Manager to control several Resources, but want to reuse the same Automation Driver and Automation Controller we want to be dynamic enough in our Automation Controller to support this scenario.

Anti-Pattern

The common anti-patterns are two different approaches that seem ok, but don’t scale or obfuscate too much.

Workflow - Static

The first one is a no-brainer, just leave it static, leave it in the OnEquipmentSetup or in the driver definition. This works well for immutable configurations, things that won’t change, when reusing the controller. This obviously is not very dynamic, because if Resource A and Resource B need different ports or ip, or any other configuration we will need to create a new Automation Controller.

As we can see in this example, we have the ResourceA as part of file Path, if we want to reuse this implementation for ResourceB it won’t work.

Workflow - Tokens

One of the clever solutions we can use is tokenize, so instead of explicitly stating ResourceA we can write ${entityName} and Connect IoT will replace it in runtime with the correct value for the entity that is appended to the execution.

NOTE: In my opinion, I don’t like tokenization, because it results in obfuscation of the code and that is why for me it’s an anti pattern. It adds cognitive load to whomever is reading the workflow to know that tokens exist and to what they are replaced with. So I would advise caution when using tokens, but it has helped solve problems in the past, so it’s important to mention it.

Pattern

The common patterns described here, may not be the only way, but are beaten paths that have worked well in the past.

NOTE: One consideration that always has to be made is the level of dynamism in correlation to the addition of complexity in you workflow.

Leveraging the MES Entity

If the Connect IoT Controller is running embedded with the context of a MES Entity, it seems obvious to add the needed configuration to that context. This works well when we require few configurations and we can solve our issue just by adding attributes to our entity.

So this seems like it solves all problems right? Well take a look at this Resource:

We can see the list grows fast and bloats the definition of the resource. It also forces us to create some kind of query if we want to see all the configurations for Connect IoT. So this can be a too decentralized solution. It also does not allow for more dynamic resolution than just the entity, like a particular vendor, type, etc. Also, the type of user with access to the Resource Entity, may not be the user you want to have access to these types of configurations.

Using Configs

An alternative, is the use of an MES Configuration. We can create a tree of configurations and then retrieve them in the workflow before starting. We can define the config tree appropriate to our needs and then use an Execute Service task to retrieve our configs. We can get the configurations with the service GetConfigByPath.

This will return a JSON object, which we will have to convert and extract the value.

This approach, enables us to group the information in a Configuration tree and behind the Administration tab, giving a more secure and aggregated location to our implementation configurations.

Smart Table

The last pattern we will discuss is the Smart Table pattern. This enables dynamic resolution of configurations based on several different columns. Let’s try and see an example of this.

Imagine that depending on a context, for example, Resource, Resource Type or Area, we want to collect a particular subset of configurations. The configurations will have a name and value. We can create a Smart Table, like this:

This enables us to have a priority of resolution, history of changes and also we can have DEE Rules triggered on changes.

In the workflow we can then use a Resolve Table task, that will automatically be populated with the inputs and outputs and will resolve the Smart Table according to what are the inputs given.

In this case we had Name and Value, to simplify you can have only value and then have the object to manipulate there. We can also access either the rows output which will have all the columns. This use case requires a bit of logic downstream to treat the value and use them.

An example of the rows output, would be something like this:

[
  {
    "ROWID": "1",
    "Resource": "1805111613350000001",
    "Name": "Path",
    "Value": "C:\\",
    "CreatedBy": "System",
    "CreatedOn": "2018-05-11T15:13:42.030Z",
    "ModifiedBy": "System",
    "ModifiedOn": "2023-04-12T05:54:16.563Z",
    "LastServiceHistoryId": "2304121347210000011",
    "LastOperationHistorySeq": "638168756565633333"
  },
  {
    "ROWID": "2",
    "Resource": "1805111613350000001",
    "Name": "Type",
    "Value": "Test",
    "CreatedBy": "System",
    "CreatedOn": "2020-10-01T20:31:46.333Z",
    "ModifiedBy": "System",
    "ModifiedOn": "2023-04-12T05:54:16.660Z",
    "LastServiceHistoryId": "2304121347210000011",
    "LastOperationHistorySeq": "638168756566600000"
  }
]

The goal would then be to extract the values and according to name create our configuration.

This is probably the most powerful and dynamic method of configuration, but as we saw, it requires some more configuration and know-how. So use it with balance and consideration, in order to not introduce too much complexity to our solution.

Summary

There are multiple approaches to tackling the same issue. We should always strive for balance a solution that is simple, but retains an adequate level of dynamism. Unfortunately, this is not a one size fits all post, but I hope it helps illustrate the thought process that you must have when thinking about these types of issues.

Author

Hello 👏 , my name is João Roque ✌️

I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn

Skills: Connect IoT / DevOps