This post is a part of the Learning Lab initiative - [Productivity Month] Boost your Productivity: Unlock your Potential and is a result of the talk [Learning Lab] Task Automation.
Topics
- Overview and Motivation
- Problem Statement
- Using what is Near
- Making it a Part of the Whole
Overview and Motivation
“Sometimes you just need to stop and think.”
There are several reasons to automate processes, I’ll give you mine: “This is a waste of time (aka boring)”, “hum, what did I have to run to do this (*checks digital notes, *then analogue notes, *then under the table)” and “Ok… param 1 run ERROR:, ok now again param1 param2 ERROR: ah yeah right param2 param1 run SUCCESS: You just wasted 30mins on this.
The first part of automating processes is understanding what are your processes and workflows. As an engineer you want to keep at the creative and engaging tasks and try to offload what is deterministic and repetitive to a known procedure. This procedure may not even be possible to fully automate, but the more you document and catalog the known path to achieve success, the more you can tweak and improve it and of course automate it.
Scripts are a very dangerous tool, a script when wrongly used is a great way to obfuscate bad practices. A script is not a spell, it shouldn’t be a set of wires glued together in order to transform the frog into a princess. It’s important to understand the process and the logical steps that you are automating, and sometimes this exercise is great to reduce the noise that exists in our work process. We will show also how to mitigate the second problematic part of script, which I call the pandemic script. This is a script that was done by someone, sometime, somewhere and has been shared a million times and each time it has mutated. Sometimes it mutates into features, sometimes it just mutates into a mutant.
This drive to improve our work and be proactive is a great impulse and has bred a whole tribe in CM “Platform Engineering”. One of their jobs is to create and help us create amazing tooling that makes our life easier and allows us to focus on the creative work. This also ensures that a script is created in a moment in time to fix one issue, but has a development lifecycle where it is maintained and improved and becomes a proper work tool.
Problem Statement
The goal for this talk was not to be an abstract presentation on automation, but to show actual helpful examples of things we found that would be or were helpful to automate.
I focused on a use-case that is near to me and that matches three out of three of my criteria to automate. The problem is that in order to be able to launch separately and debug a Connect IoT component (monitor, controller or driver), I am first, required to manually boot each one.
If for example I have a manager with a controller and a driver file-raw:
- Go to the cache of the automation monitor and for each package that I wanted to reroute to my code source, create a symbolic link with the same name of the package the manager expected. Then I would have to start the components.
Start the Automation Monitor
node ./monitor.js --dev --config='../config.downloaded.json' --mp=88
Start the Automation Controller
- In order to start the controllers, I need to see the monitor (or go to the MES) to see what are the Ids the Automation Monitor is expecting and what is the controller package it is expecting.
node index.js --id=AutomationControllerInstance/2403060210100000002 --monitorPort=88 --config=C:/Users/<user>/Downloads/<manager>/config.downloaded.json --dev=true
Start the Automation Driver for File-Raw
- In order to start the controllers, I need to see the monitor (or go to the MES) to see what are the Ids the Automation Monitor is expecting and what is the controller package it is expecting.
node index.js --id=AutomationDriverInstance/2403060210100000004 --monitorPort=88 --config=C:/Users/<user>/Downloads/<manager>/config.downloaded.json --dev=true
NOTE: If I had more drivers or more controllers, I would have to start them as well. Of course the one I wanted to test, I would have to launch it from the solution I wanted to debug.
As you can imagine this is a very cumbersome process.
Using what is Near
The easiest way to create scripts that easy to use in Windows is to use the scripting language for Windows Powershell. Powershell has the concept of profile, where you can declare a set of functionality that is loaded whenever you open a shell. For me this is a good starting point as it allows me to have very fast and lean iterations and it’s instantly available in the shell without having to install or use third party components.
My first try at it was kind of bad, but that is fine and is part of a proof of concept approach. This is key when trying to automate processes, sometimes it’s not clear if automating a process will be easy, and if actually the process that you think is repetitive is actually repetitive and deterministic. So having a first approach to prove that the script works and is usable is helpful for me.
So I added the following script to my profile:
Function StartAutomationManagerComponents {
param(
[Parameter(Mandatory = $true)][string] $ManagerLocation,
[Parameter(Mandatory = $true)][hashtable] $Controller,
[Parameter(Mandatory = $true)][hashtable] $Driver,
[Parameter(Mandatory = $false)][string] $LinkDir
)
# Check if I am in a Manager Directory
$path = (Get-Childitem -Path $ManagerLocation -Include package.json -Recurse | Where-Object { $_.FullName -notlike "*\node_modules\*" } | Select-Object -First 1).fullName
$name = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "name"
if ($name -ne "@criticalmanufacturing/connect-iot-manager") {
throw "You are not in a Directory with an Automation Manager"
}
# Check what is my config
# Order is config.downloaded.json -> config.full.json -> config.json
$config = Get-Childitem -Path $ManagerLocation -Include config.downloaded.json -Recurse
if (!$config) { Get-Childitem -Path $ManagerLocation -Include config.full.json -Recurse }
if (!$config) { Get-Childitem -Path $ManagerLocation -Include config.json -Recurse }
if (!$config) { throw "Couldn't find a valid config file" }
# Now we will check for the components in the cache
$cache = (Get-Content $config.FullName) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "cache"
# Validate for Temp
if ($cache.Contains('${temp}')) {
$cache = $cache.Replace('${temp}', $env:TEMP)
}
# Make Links
if ($LinkDir) {
$folders = Get-ChildItem -Path $LinkDir
}
$Monitor = "$($ManagerLocation)/src/"
$execMonitor = "node monitor.js --dev --config=$config --mp=88"
$exec = ""
$execControllers = ""
$execDrivers = ""
$exec = 'wt --maximized --title "Monitor" -d $Monitor pwsh -c $execMonitor'
$Controller.keys | ForEach-Object {
$action = "split-pane"
if (!$execControllers) {
$action = 'new-tab --title "Automation Controllers"'
}
if (!$Controller[$_]) {
if (!$managerVersion) {
$managerVersion = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "version"
}
$Controller[$_] = "connect-iot-controller@$managerVersion"
}
elseif (!$Controller[$_].Contains("connect-iot-controller")) {
$Controller[$_] = "connect-iot-controller@$_"
}
$value = $Controller[$_]
# Write-Host "Using Controller: $value for instance $_"
$packageName = $Controller[$_].Split("@")[0]
$controllerPath = Get-ChildItem -Path $cache -Include $value -Recurse
if (-not $controllerPath) { throw -join ("Expected ", $value, " in ", $cache, " does not exist") }
if ($folders) {
Foreach ($folder in $folders) {
if ($packageName.Replace("connect-iot-", "") -eq $folder.Name) {
if (Test-Path -Path $controllerPath) {
Remove-Item $controllerPath -Force -Recurse
}
new-item -itemtype symboliclink -path $cache -name $Controller[$_] -value $folder.FullName
break;
}
}
}
$execControllers += " ``; $action -d $controllerPath pwsh -c node src/index.js --id=AutomationControllerInstance/$_ --monitorPort=88 --config=$config --dev=true"
}
$Driver.keys | ForEach-Object {
$action = "split-pane"
if (!$execDrivers) {
$action = 'new-tab --title "Automation Drivers"'
}
if (!$Driver[$_].Contains("connect-iot-")) {
$Driver[$_] = -join ("connect-iot-", $Driver[$_])
}
if (!$Driver[$_].Contains("@")) {
if (!$managerVersion) {
$managerVersion = (Get-Content $path) -join "`n" | ConvertFrom-Json | Select-Object -ExpandProperty "version"
}
$Driver[$_] = -join ($Driver[$_], "@", $managerVersion)
}
$value = $Driver[$_]
# Write-Host "Using Driver: $value for instance $_"
$packageName = $value.Split("@")[0]
$driverPath = Get-Childitem -Path $cache -Include $value -Recurse
if (-not $driverPath) { throw -join ("Expected ", $value, " in ", $cache, " does not exist") }
if ($folders) {
Foreach ($folder in $folders) {
if ($packageName.Replace("connect-iot-", "") -eq $folder.Name) {
if (Test-Path -Path $driverPath) {
Remove-Item $driverPath -Force -Recurse
}
new-item -itemtype symboliclink -path $cache -name $value -value $folder.FullName
break;
}
}
}
$execDrivers += " ``; $action -d $driverPath pwsh -c node src/index.js --id=AutomationDriverInstance/$_ --monitorPort=88 --config=$config --dev=true `;"
}
$exec = $exec + $execControllers + $execDrivers
Invoke-Expression $exec
}
The code itself it’s not important. The important part is that right away we have a lot of repeated code blocks, we have a load of inputs and honestly this could would be very hard to maintain without knowing the use-case very well. But even if all that is true, this script is fast to write and proved that scripting this task was possible and helpful and, importantly, sparked ideas on how it could be changed and remade.
Notice that we start the automation manager, we download the default packages. We then kill the automation manager and execute our script. The script will receive the ids of the components and the directory to link the packages. Then it will create an automation workspace.
Making it a Part of the Whole
Critical Manufacturing already has a structure to incorporate tooling in a way that is simple to use, that is maintainable and easy to share with control throughout the company. This is using the CM CLI, the CLI is something that everyone knows how to use and that we can add our own plugins to. This command is linked to how the development process should be used, so instead of creating a plugin from scratch, which I could have done, I used the existing cmf dev plugin.
The source code for the plugins resides at Dev Ops Plugins.
Looking at the source code it’s easy to organize and orientate our script in this tool set mindset. I want to be able to just run cmf dev iot startComponents.
In order to do this in the dev project we will create a folder to hold our iot related dev commands and two cs files. Our parent command IoTCommand and our command StartComponentsCommand;
The IoTCommand is a parent command so it’s very simple, it serves just to aggregate commands. Notice the attributes in the method header, those will be important to map to the command console execution.
[CmfCommand("iot", Id = "dev_iot")]
public class IoTCommand : BaseCommand
{
public IoTCommand() : this(new FileSystem())
{
}
public IoTCommand(IFileSystem fileSystem) : base(fileSystem)
{
PluginExecutionContext.Initialize(this.fileSystem);
}
(...)
Now, our actual command. A command is divided into two parts: the Configure, where we treat and prepare all the arguments that will be used by the command and the Execute.
[CmfCommand("startComponents", Id = "iot_startComponents", ParentId = "dev_iot")]
public class StartComponentsCommand : IoTCommand
{
/// <summary>
/// Configure command
/// </summary>
/// <param name="cmd"></param>
public override void Configure(Command cmd)
{
cmd.AddArgument(new Argument<IDirectoryInfo>(
name: "managerLocation",
parse: argResult => Parse<IDirectoryInfo>(argResult)
)
{
Description = "Location of your downloaded manager"
});
cmd.AddOption(new Option<IDirectoryInfo>(
aliases: new string[] { "--linkDir", "-ld" },
parseArgument: argResult => Parse<IDirectoryInfo>(argResult),
description: $"This specifies the directory you wish to use to create links in order to debug or to use your code."));
cmd.AddOption(new Option<string>(
aliases: new string[] { "--linkVersion", "-lv" },
description: $"Specify version that will be used for the link packages, to allow using a linked package with a different version that the one defined for the runtime (note: will be used if there are no packages in the cache)"));
// Add the handler
cmd.Handler = CommandHandler.Create<IDirectoryInfo, IDirectoryInfo, string>(Execute);
}
(...)
Notice how the tooling already does most of the heavy-lifting, it already allows to differentiate between optional and mandatory arguments, allows to add helpful descriptions for the user and the use of aliases. We have one mandatory arguments that is the manager location and two optional arguments, the directory where we want to link our cache and in the case of an empty cache the link version we want to give to the symbolic links.
Let’s take a look at the execution phase. The first phase is validation, we will validate the correct operating system, then we validate that the manager location is actually a manager location and we will replace a very common token in the config, by a location the filesystem understands. Also, we will interpret the configs with the correct order of precedence.
(...)
/// <summary>
/// Executes the specified target
/// </summary>
/// <exception cref="PluginException"></exception>
public void Execute(IDirectoryInfo managerLocation, IDirectoryInfo linkDir, string linkVersion)
{
using var activity = ExecutionContext.ServiceProvider?.GetService<ITelemetryService>()?.StartExtendedActivity(this.GetType().Name);
if (!OperatingSystem.IsWindows())
{
throw new PluginException(Cmf.CLI.Plugin.Common.Messages.OSNotSupported);
}
IFileInfo configJson;
try
{
// Check if directory is a Manager Directory
dynamic packageJson = managerLocation.GetFile("package.json");
if (packageJson == null || packageJson.name != "@criticalmanufacturing/connect-iot-manager")
{
throw new PluginException($"Manager Location either does not contain a package.json or the packagen.json does not match the expected package @criticalmanufacturing/connect-iot-manager");
}
// Check config to use:
// Start with downloaded -> full -> config.json
configJson = managerLocation.GetFiles("config.downloaded.json").FirstOrDefault();
configJson ??= managerLocation.GetFiles("config.full.json").FirstOrDefault();
configJson ??= managerLocation.GetFiles("config.json").FirstOrDefault();
}
catch
{
throw new PluginException($"This Directory {managerLocation} does not contain an Automation Manager");
}
// Read the config.json to see where the package dependencies are
dynamic configJsonContent = JsonConvert.DeserializeObject(configJson.ReadToString());
string cacheLocation = configJsonContent.cache.ToString();
// Replace temp token with temp location
if (cacheLocation.Contains("${temp}"))
{
cacheLocation = cacheLocation.Replace("${temp}", Environment.GetEnvironmentVariable("TEMP"));
}
(...)
The next phase is preparing the links. If a link directory is passed, we must change the cached package names and create symbolic links to the packages in the directory link. If there are packages in the cache we will match those with the ones in the directory link and perform just the link for those. If the cache is empty, we will link all the packages, here we can either link them with their own version or use the linked version argument version.
(...)
#region Prepare Links
// Check if I have to do links
if (linkDir != null)
{
IDirectoryInfo cache = this.fileSystem.DirectoryInfo.New(cacheLocation);
var cachedPackages = cache.GetDirectories();
// If there are packages we will link the ones that match, else we will link all
Dictionary<string, string> packagesToLink = new();
if (cachedPackages != null && cachedPackages.Any())
{
foreach (var cachedPackage in cachedPackages)
{
// if the package as already processed or if it us a link we will skip it
if (cachedPackage.LinkTarget != null || cachedPackage.Name.Contains("_CACHE_ORIGINAL"))
{
Log.Debug($"This packaged is already a link, so it will not be relinked {cachedPackage.Name}");
continue;
}
// We will go to each cached package and see the name and the version of the package
IFileInfo cachedPackageJson = cachedPackage.GetFiles("package.json").FirstOrDefault();
dynamic content = JsonConvert.DeserializeObject(cachedPackageJson.ReadToString());
packagesToLink.Add(content.name.ToString(), content.version.ToString());
}
}
// Check matches between cache and folders in the directory
// Rename existing folders to _CACHE_ORIGINAL_
// Make Links
var linkedDirectories = linkDir.GetDirectories();
List<IDirectoryInfo> createdLinkedDirectories = new();
foreach (var linkedDirectory in linkedDirectories)
{
IFileInfo linkedPackageJson = linkedDirectory.GetFiles("package.json").FirstOrDefault();
dynamic content = JsonConvert.DeserializeObject(linkedPackageJson.ReadToString());
// We will check if the package name exists in the list of cached packages
// Or if there are no cached packages in that case we link all
if (!packagesToLink.Any() || packagesToLink.ContainsKey(content.name.ToString()))
{
// Remove scope
var packageVersion = packagesToLink.Any() ? packagesToLink[content.name.ToString()] : linkVersion ?? content.version.ToString();
var packageName = $"{content.name.ToString().Split("/")[1]}@{packageVersion}";
var dirName = Path.Join(cache.FullName, packageName);
// If it's a downloaded package we will rename it
// Else we will just delete it
if (packagesToLink.Any()) { this.fileSystem.Directory.Move(dirName, dirName + "_CACHE_ORIGINAL"); }
var directory = this.fileSystem.DirectoryInfo.New(dirName);
if (directory.Exists) { directory.Delete(); }
// Create a Link between the cache and our package with the correct name <packageName>@<cachedPackageVersion>
this.fileSystem.Directory.CreateSymbolicLink(dirName, linkedDirectory.FullName);
createdLinkedDirectories.Add(this.fileSystem.DirectoryInfo.New(dirName));
}
}
}
#endregion
(...)
I really wanted to avoid the need for me to know the ids of the components and the arguments to start them. So I decided a different approach, I start the automation monitor for a dummy execution and sniff the output of the console for the parts where the monitor tells me what I need to start. Then I kill the monitor and perform a clean start. If you are curious for the messages being intercepted, try to start only the monitor and see what it logs, and you will see the messages we are intercepting. We wait a set amount of time for the monitor to finish and then kill the process.
(...)
#region Dummy Execution Phase
// Start Monitor
// Intercept Messages regarding Drivers and Controllers
// We will sniff the console output of the monitor and see what are the components that must be started and their Ids and packages
// Then the monitor we will kill the monitor and start everything
CommandLineRunner runMonitor = new(fileSystem.DirectoryInfo.New(Path.Join(managerLocation.FullName, "src")));
List<ComponentExecution> componentExecutions = new List<ComponentExecution>();
string idInForkPattern = "(?<=id=')[^']+";
string pathInForkPattern = "(?<=path=')[^']+";
string idInArgumentsPattern = "(?<=--id=)(.*?)(?= --managerId)";
string argumentsInArgumentsPattern = "--id=(.*)";
bool killedMonitor = false;
Action<string> monitorConsoleHandler = output =>
{
if (output != null)
{
// In this Message we will know the Id of the component and the package it needs to boot
if (output.Contains("Fork requested *NOT* executed"))
{
var id = Regex.Match(output, idInForkPattern).Value;
var path = Regex.Match(output, pathInForkPattern).Value;
componentExecutions.Add(new ComponentExecution(id, path));
}
// In this Message we will know the arguments required to execute the component
else if (output.Contains("Arguments: "))
{
var id = Regex.Match(output, idInArgumentsPattern).Value;
var arguments = Regex.Match(output, idInArgumentsPattern).Value;
var comp = componentExecutions.FirstOrDefault(x => x.Id == id);
comp.Arguments = Regex.Match(output, argumentsInArgumentsPattern).Value;
}
else if (output.Contains("error"))
{
runMonitor.KillProcess();
throw new PluginException($"Something went wrong when booting up the Monitor, please fix it before running the command again - {output}");
}
else if (output.Contains("Processes and packages processed"))
{
// When all packages are processed we will kill the Monitor Process that we started
runMonitor.KillProcess();
killedMonitor = true;
}
}
};
// Grab a random port in the 81-90 range
Random random = new();
int randomPort = random.Next(81, 90);
var monitorExec = $"node {Path.Join(managerLocation.FullName, "src", "monitor.js")} --dev --config={configJson.FullName} --mp={randomPort}";
runMonitor.Run(monitorExec, waitForExit: true, outputHandler: monitorConsoleHandler, logOutputOnError: false, errorCodeToIgnore: -1);
// We will wait a set amount of time for the monitor process to log all components
try
{
WaitFor(60, () =>
{
if (killedMonitor)
{
// Wait for the monitor to finish
return true;
}
return false;
}, "No Fork was Requested");
}
catch (Exception)
{
runMonitor.KillProcess();
throw new PluginException("No Fork was Requested from monitor in the allotted 60s period");
}
Log.Debug($"Number of Components found - {componentExecutions.Count}");
#endregion
(...)
In the last phase we construct the windows terminal workspace, similar to what we were already doing in the script.
(...)
#region Run All Components
CommandLineRunner runAll = new(fileSystem.DirectoryInfo.New(Path.Join(managerLocation.FullName, "src")));
var exec = $"wt --maximized --title \"Monitor\" pwsh -c {monitorExec}";
// For each component type we will open a new tab and for each component of the same type we will spit the tab
ComponentType? previousType = null;
foreach (var component in componentExecutions.OrderBy(x => x.Type))
{
var action = "split-pane";
if (previousType != component.Type)
{
action = $"new-tab --title \"Automation {component.Type}\"";
}
else
{
}
exec += $" ``; {action} pwsh -c node {component.Path} {component.Arguments}";
previousType = component.Type;
};
// Controller - Collect Ids and Package versions
// Driver - Collect Ids, Package Names and Versions
// Start Controllers and Drivers
Log.Debug($"Executing Command - {exec}");
runAll.Run(exec, waitForExit: false);
#endregion
(...)
In order to execute our command
To just start the components:
cmf dev iot startComponents <managerlocation>
To start the components and make links, with an existing cache:
cmf dev iot startComponents <managerlocation> -ld <linkDirectory>
To start the components and make links, without an existing cache, but with a specific version:
cmf dev iot startComponents <managerlocation> -ld <linkDirectory> -lv <linkVersion>
With this we finish the loop, we started with an idea on how to improve a part of our workflow that was very cumbersome and prone to error and now we have a code block with head, torso and members, this way closing the anatomy of a script.
Author
Hello 👏 , my name is João Roque ✌️
I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn
Skills: Connect IoT / DevOps