I want to take a look at a problem regarding node-gyp and Python3.1.2 and use that opportunity to talk a bit about both.
Topics
- The problem
- What is node-gyp
The Problem
The other day, everything broke down… Depending on external systems makes us gain a lot of features, maturity and experience that was gathered by third parties, but you abdicate control. It’s not uncommon, for small updates to generate disruption in our workflow. Being mindful of these trade-offs should be at the forefront of the decision making process of every developer, questions like stability, updates, community and documentation are key factors when deciding what dependencies to chose and build on.
Knowing how to read is also an important feature of a software engineer, mainly reading logs. We wasted some minutes not doing that and jumping to conclusions, but when we stopped to read the stack trace, we saw the following message:
This was a strange issue, as it was working on our local machines. So we already knew that the problem was either in the pipeline or in the dependencies installed in the system that ran the pipeline.
Looking carefully, we can see that the problem was on a dependency called node-gyp. The dependency finds the vs tools to build C++, but it subsequently fails when it reaches the Python312 execution.
A quick search on the internet gave us some clues distutils is removed in Python 3.12 and in the official documentation. Looking further [DOCS] The latest version of python (3.12) doesn't support node-gyp, so we know that the node-gyp latest version supports Python312. Stupid question, what is the latest version and are we on it?
In the node-gyp github releases we know the latest version is 10.0.1. Let’s see what is ours by running in our package npm list node-gyp, we could also see globally by running npm list node-gyp -g
We see that we have version 9.4.1 being used in the angular/[email protected]. In the github history we can see that the fix was applied to gyp-next and this was released in gyp-next releases v0.16.0, looking back at node-gyp we can see that the bump to gyp-next v0.16.0 only happens for node-gyp v10.x.
In order to fix this, we could update node-gyp but node-gyp is a sub-dependency of a product dependency, which may be impacted by a version change. The best approach is to follow the python official documentation and install the setup tools.
Run:
python3 -m pip install setuptools
In order to check your python version:
python3 --version
If your version is above 3.1.12 you will be impacted.
What is node-gyp?
Why does node which is a javascript engine, need python. Actually, the dependency that we’ve been talking about node-gyp is a cross-platform command-line tool for compiling native addon modules for Node.js. This allows developers to write code in C, C++, or Rust and integrate them with JavaScript running in Node.js. node-gyp is a wrapper around GYP (Generate Your Projects), a meta-build system that generates project files for use with various IDEs and build systems, such as Visual Studio, Xcode, and make 12.
Notice, how this matches what we saw with our original problem, where the fix was not only in node-gyp but was in gyp. So actually, it’s not node-gyp that needs python, but it’s the parent dependency gyp that is built on python.
Author
Hello 👏 , my name is João Roque ✌️
I’ve been working for some years at Critical Manufacturing. I split my time working in the IoT Team and working with the project’s teams. You can visit me at https://j-roque.com/ or check me at LinkedIn
Skills: Connect IoT / DevOps