NEMS Linux 1.6 Changelogs (2023-2024)

The NEMS 1.6 release cycle will focus on hardware porting and peripherals.

Release Dates:

• NEMS Linux 1.6 Alpha was released to the beta team December 2020.

• NEMS Linux 1.6 Beta and NEMSeOS 0.1 Alpha Build 2 were released to Patrons May 21, 2021.

• NEMS Linux 1.6 for Raspberry Pi released to Patrons January 8, 2023.

• NEMS Linux 1.6 Virtual Appliance (OVA/VHD/QCOW2) released to Patrons January 21, 2023.

• NEMS Linux 1.6 for Raspberry Pi and Cloud Instance released to public January 28, 2023.

• NEMS Linux 1.6 for ODROID XU3/XU4/HC1/HC2/MC1 released to public February 5, 2023.

• NEMS Linux 1.6 for Indiedroid Nova released to public May 11, 2023.

• NEMS Linux 1.6 for the ASUS Tinker Board released to public August 12, 2023.

• NEMS Linux 1.6 for ODROID N2/N2+ released to public August 16, 2023.

• NEMS Linux 1.6 for Amazon Web Services released to public October 23, 2023.

• NEMS Linux 1.6 Build 2 for Raspberry Pi released to public October 31, 2023.

Key Development Features and Goals:

• Public release of NEMS Warning Light.

• Public release of NEMS Extender OS.

• Updated packages.

• Supported single board computer list to be reduced to focus on most used, with option for users to request ports if they do not exist.

• Port NEMS Linux to Docker platform. Development funding provided by Patrons (THANK YOU).

• Consistent networking configuration across platform (for setting static IPs, for example).

Development Changelogs Since 1.5:

December 2019

• Merged PR 4, adding escalations for hosts / services / advanced services and fixing various bugs.

• Upgrade NEMS NConf jquery code to 3.3.1.

• Tap NEMS NConf into user’s wallpaper settings.

• Output the final line from syntax check to screen when deploying config in NEMS NConf.

• Changed “Generate Nagios config” to clearer “Deploy Config to NEMS” in NEMS NConf.

• Moved all the legal stuff off the landing page of NEMS NConf and moved to new “Legal Disclaimer” section.

• Prepare Migrator for 1.6 and integrate new NConf configs.

• Upgrade Nagios Core to 4.4.5.

• Integrated NagiosTV as per BastyJuice.

January-February 2020

• Upgrade Monit to 5.26.0.

• Move build tmpdir out of /tmp/ and into /usr/src/

• NEMS NConf 1.6 tooltip font color fixed.

• NEMS NConf 1.6 removal of JQuery Accordion from Generate page, replaced with terminal output.

• New Deploy button for NEMS NConf Generate Config page.

• Change notify-service-by-telegram to point to the modern script, rather than the old path (which is a symlink these days). mydogboris noted that when using the symlink, his Telegram notifications were wonky.

• Added the NEMS MOTD to Docker.

• General filesystem cleanup: remove some old cruft.

• NagiosTV theme customized to match user theme settings (such as background image).

• Added humidity sensor to check_temper.

• NRPE upgraded to 4.0.0.

• Fixed NRPE installer script if run on non-NEMS system, as per Issue 2 by Xelo.

• Revert NRPE to 3.2.1 temporarily, until I can figure out how to make it talk to older installs (upgrading to 4.0.0 breaks all existing 3.2.1 installs).

• check_speedtest_cli.sh now records a log file in /var/log/nems/speedtest.log. This log is now parsed by NEMS TV Dashboard and displayed at all times. Further improvements will be added before public release. As requested by ITManLT.

• Replace words “Down” and “Up” in NEMS TV Dashboard speedtest stats with font-awesome icons.

March-April 2020

• Move 1.6 base to Debian 11 (bullseye).

• Upgrade Monitorix to 3.12.0.

• Added mod_wsgi to 1.6, which is no longer auto-installed with Apache2 in bullseye.

• Initial user files now placed correctly in /etc/skel for user creation.

• Change default body background color of NEMS NConf to dark grey (almost black) to prevent the appearance of the screen flashing white during page transition.

• Moved old python-pip, python-django and python-paramiko to modern python3-pip, python3-django and python3-paramiko packages respectively.

• Upgrade Adagios to 1.6.6.

• Migrate to several experimental branches of Adagios components to begin testing support under Python 3.8. Current stable only supports Python 2.7 which is EOL this year.

May-June 2020

• NEMS Linux 1.6 development merged (some would say “downgraded”) with NEMS Linux 1.5.2 as I wait for Adagios support for Python 3.7+. NEMS Linux 1.6 will be buster-based and use Python 2.7.

• Change ownership of /var/log/nems to nagios user and group to allow check commands to write to logs. This enables the speedtest log, which in turn shows speedtest data on NEMS TV Dashboard.

• TEMPer calibration added to NEMS SST and integrated into all TEMPer output. Now, for the sake of accuracy, users may adjust the thermal and humidity sensor offset by simply dragging a slider.

• Hide TEMPer calibration option if TEMPer device is not present.

• Add Argon ONE scripts for Raspberry Pi 4.

• Argon ONE support complete.

• Added repos feature to nems-info.

• Added new check_mrtgtraf_nems check command in preparation for a very sleek MRTG integration.

• Set Argon ONE to log to /var/log/nems/argonone.log if system is rebooted or shutdown safely using the power button (with timestamp).

• Move rpimonitor off the repository and instead manually install. The source repository (http://giteduberger.fr/rpimonitor/) became unstable during the 1.5 release cycle (up and down) and since rpimonitor appears to be a dead project (hasn’t been updated in > 4 years) I’ve instead removed the repository requirement and installed it manually. Also, fixed the distribution name on the template.

July-August 2020

• Change logrotate for NEMS logs to rotate logs based on log size, not date. Delayed compression added.

• check_mrtgtraf_nems now rounds the floats of the traffic results to two decimal places.

• Added check_apc to check the state of an APC UPS. As requested by Neptunum.

• Python version default will now be hard set at 995 during build process. This is in preparation for python3 down the road (pending Adagios compatibility).

• I noticed logrotate rotating some NEMS logs that really shouldn’t be rotated, such as cloudauth.log. Modified logrotate.d conf to omit logs that should not be rotated.

• Change permissions of /var/log/nagios/nagios.log to allow group write. Required to remove root user requirement on nems-mailtest, which will be needed to integrate it into NEMS Dashboard.

• nems-mailtest is now integrated into NEMS SST. See Video

• nems-mailtest in NEMS SST now runs nems-mailtest using the information entered in the form, not saved to the config. This way, a user may test their SMTP config before saving the changes.

• Reverted MySQL database to previous push of 1.6 branch as I had accidentally pushed the 1.5 database to the 1.6 branch when I added check_apc. I’ve corrected it, and re-added check_apc. No users impacted by this since 1.6 hasn’t yet been released.

• Added nems-info [dht11|dht22] which outputs json response from either the DHT11 or DHT22 sensors.

• Add user to gpio group during initialization. This will allow access to the GPIO pins without root access.

September-October 2020

• Minor improvements to output of DHTxx check command for Humidity values.

• Add nemsadmin user to gpio group, just to ensure the demo check commands work out of the box, before initialization takes place.

• Upgrade check_mk Livestatus to 1.6.0p17. This gets the sock working on the NEMS Linux 1.6 AWS development system, which is required for Adagios and NEMS TV Dashboard.

• Added phpmyadmin, disabled by default and interface access protected behind NEMS user login/password when enabled.

• Change check_local_disk to support unit selection (KB, MB, GB, TB) and set default for NEMS Local disk check sample to GB.

• New Feature: NEMS PHP Server Agent can now be configured and downloaded in NEMS System Settings Tool.

• New documentation launched, with the help of Bill Marshall plus submissions from Don Jenkins.

November-December 2020

• NEMS Tools now automatically detects the IP address of the running NEMS Server on the same subnet and creates its configuration file.

• NEMS Tools GPIO Extender client now uses the config file to determine NEMS Server IP address.

• Omzlo Warning Light pHAT now transmits and receives via NEMS GPIO Extender, allowing you to plug the pHAT into your NEMS Tools GPIO Extender Receiver.

• Fixed a typo in Warning Light that incorrectly determined all platforms to be a Raspberry Pi.

• Update nems-tools GPIO Extender to detect NEMS Servers via gpioe-server on port 9595 rather than looking for a host reply from nems-api.

• Moved nems-tools.conf to /boot, making it extremely easy to modify the conf on nems-tools Extender OS.

• NEMS SST will no longer warn of requirement to add a password for the PHP Agent if a password already exists in the config.

• NEMS PHP Agent 1.1 released. Now the keyphrase is encrypted (rather than base_64 encoded). Also added “.” current folder disk space check and fixed several bugs with disk and var checks.

• Fixed the directions on NEMS SST which directed a user to add their encypription/decryption passphrase to the “General” tab, since that feature was moved to the NEMS Migrator tab long ago. Pointed out by UltimateBugHunter-NitPicker.

• NEMS Extender OS will now illuminate all lights if the NEMS Server goes offline or cannot be contacted.

• Improve output in nems-init if user tries to initialize with same name as already running user.

• Added qemu-guest-agent to improve integration with Proxmox VE as per UltimateBugHunter-NitPicker.

• Delay for a random amount of time (up to 2 hours) when running offsite backup tasks to prevent 1,000+ NEMS Servers clobbering the API all in the same moment. Issue pointed out by UltimateBugHunter-NitPicker who was seeing errors with his daily backup.

• NEMS Migrator Off Site Backup will now log if the Internet was down when the backup ran. Also added a ‘now’ cli option that will force it to run without delaying.

• Changed Migrator patches failsafe to determine if Quickfix / fixes was running for more than 120 minutes (previous setting was 90) before killing.

• Adjusted NEMS Off Site Backup to trigger at midnight but delay for a random amount of time up to 4 hours before running the backup. This will further reduce strain on the NEMS Cloud Services servers as NEMS’ userbase continues to grow.

• Added missing Set::IntSpan package, required by check_mikrotik_switch. Modify the check script to provide better (cleaner) output if CLI type not provided.

• All check_mikrotik_switch check commands renamed from the old check_mt_ and now are check_mikrotik_. Also fixed argument count and improved descriptions in NEMS NConf for each of the MikroTik checks.

• Removed check_minecraft. It hasn’t been kept up to date by its developer, and unless there is a demand for it, I don’t want to have to take over maintenance on such a niche plugin.

• Added check_ncpa along with two sample checks: check_ncpa_processes and check_ncpa_mem.

• Fix bug in PiVoyager daemon that falsely detected PiVoyager hardware on some setups where it didn’t actually exist.

• Several small web interface fixes in nems-www.

• Fix running user detection in nems-info to prevent www-data from attempting to use user cache.

• Removed NEMS host from HTTP Advanced Service. It was included as a sample, but since the interface can run quite slow on some low-powered SBC’s, it causes timeout notifications which tend to confuse users into thinking there’s actually a problem.

• Added check_synology as per AlbertPauw. Added several Advanced Services samples to Synology host group.

• Improved Warning Light’s detection of Omzlo Warning Light pHAT to prevent log bloat.

• Improve NEMS Extender OS’s browser-based output to include an iteration, which will help in event of a hung service: If the iteration (counter) no longer increases, it becomes more apparent that something has gone awry.

• Disable the TEMPer thermal and humidity checks by default to prevent new users seeing a warning that they are missing the sensor. Leave the demo checks in NEMS NConf for easy re-activation.

• Fixed permission issue on NIC cache if root is automatically detecting NIC while user is simultaneously logging in as non-root user. This bug was seen in Novaspirit Tech’s video introduction to NEMS Linux 1.5.2 when signing in via SSH. The MOTD did not display, and in pausing the video I see it is in fact the nemsadmin user momentarily not having access to the NIC cache. So I fixed it.

• Rearranged NEMS Warning Light daemon to ensure the Omzlo Warning Light pHAT over a NEMS GPIO Extender is synchronized to the GPIO pin output (I.E., don’t delay for an iteration: instantly change states).

• Improved the output of all disk checks in NEMS PHP Agent. Now the mountpoint will be listed in brackets after the percentage, making it easier to see the actual state.

• NEMS Migrator Off-Site Backup schedule maintenance automatically keeps backups tidy now. Current schedule is that you have access to every daily backup for the past month, and a weekly backup for the past year.

• Added new check command check_nems_osb which will notify if a NEMS Migrator Off-Site Backup fails.

• NEMS PHP Agent 1.2 released with the following improvements: Network usage now uses ifstat and generates a more accurate average usage number based on all network interfaces on the server with a 5 second average. New agent will only run the equations and tests for the requested check. For example, don’t run a 5 second network bandwidth test when the requested check is for the load average. Fixed bug where nettx was in fact reporting netrx.

• Updates to NConf to improve output. Add AJAX spinner during generation, remove horizontal scroller, etc.

• Block error output when detecting NIC to prevent MOTD being broken during first login.

• Determine the fk_id of the NEMS Host and adapt nems-restore to use this (NEMS 1.6 branch only). Keep 1.5 branch separate and improve compatibility with 1.6.

• When a Patron opens the NEMS Dashboard (I.E., they have a valid NEMS Cloud Services account) a link is now available to visit the latest Patron-exclusive posts.

• NEMS Hardware ID is now blurred by default and hidden from display in NEMS Server Overview. I saw a YouTuber who opened NEMS Server Overview willy-nilly on his video and did not blur this information in post production, so I’ve enforced it by default. The HWID can be revealed by double-clicking the blurred area.

January-February 2021

• Migrator tab now has more intuitive output when a new NEMS Cloud Services account is activated (I.E., Notice re. waiting 24 hours for first OSB).

• NEMS Tactical Overview (NagiosTV) now running 0.6.5 and now uses Chris Carey’s implementation of the Livestatus connector rather than the previous Nagios CGI method. In lay speak, this means when you open NEMS Tactical Overview, you’ll no longer have to supply a password within five seconds to login to Nagios Core. Huge thanks to Chris for making this change for us!

• Pause general development to work through major issue with WMIC following changes to Microsoft Windows starting with Windows 10 Version 2004.

March-April 2021

• The WMIC issue has been fixed. Final stages of development for NEMS Linux 1.6 can resume.

• Many updates to NagiosTV bringing it up to 0.7.3, which improves error handling on connection loss and much provides overall performance improvements. These updates are with thanks to Chris Carey, who we are pleased to have welcomed to our Beta Team earlier this year.

• Remove the usage of Ookla’s speedtest service and replace it seamlessly with Netflix’s fast.com. This is due to changes in Ookla’s licensing agreement, but also means the check commands have been rewritten with better error handling. I forked the fast-cli project so I could make necessary changes to port this to arm64. fast-cli does not work on arm processors since it uses puppeteer which depends on the x86 Chromium headless browser. My fork instead calls for the version of Chromium built specifically for each system, which means it will work on all platforms running NEMS Linux.

• Set service timeout to 120 seconds rather than the previous 60 seconds. The short timeout was causing timeouts with speedtest since that takes extra time to execute on some connections.

• Created a new --reset option for nems-quickfix which allows resetting a previously-applied patch. This can be useful should the patch fail for whatever reason.

• Made the new speedtest script load a cache file if it is already running. In an event where the user had initiated multiple simultaneous speedtests, many processes could spawn resulting in crippling bandwidth usage. Noted by UltimateBugHunter-NitPicker.

• Backported the speedtest update to NEMS Linux 1.5.

• nems-quickfix now resets the log but appends both runs to it each time it runs. This will give me the ability to investigate QuickFix issues more easily.

• nems-quickfix now ensures all previous package installations are complete before running patches. Was an issue if user had previously rebooted their NEMS Server during an update leaving some packages broken.

• Several compatibility fixes added to the nems-speedtest patch (000015) to ensure backward compatibility with as many NEMS Servers as possible.

• Continued work on WMIC, which has posed challenges in porting across platform.

• Launched new NEMS Linux repository for 1.6. By doing so, I plan to compile WMIC on each architecture and then be able to install via apt, rather than going through so much trouble compiling from scratch on every single board. Inevitably this will also mean moving all NEMS packages off github (as far as how NEMS Servers obtain the code) and instead maintain a single apt repository. This could mean much easier cross-platform support from my perspective, and faster adoption of updates across all boards.

• Modified the speedtest script to wait 100 seconds for the task to finish and then forcibly kill it. If this happens, NEMS will report “0 Error” instead of a speed result. By doing this, I prevent slower connections from having the speedtest check killed by Nagios (service timeout) leaving a compounding number of Chromium tasks running, eventually leading to high load and slow operation.

• Upgrade CheckMK to 1.6.0p23 and move source to Github (rather than a zip from their site, which became deadlink after a recent redesign).

May-August 2021

• Clean up a lot of the old build scripts and nems-upgrade packages to accommodate the changes in NEMS Linux 1.6.

• Deprecation of Samba wmic complete: NEMS Linux 1.6 now entirely moved to new custom Python replacement.

• Upgrade Nagios to 4.4.6.

• Move Nagios configuration base to 1.6.

• Remove deprecated Speedtest Server output from NEMS Server Overview.

• Use the new hw-detect (32-bit or 64-bit) system rather than hw_model (32-bit only) in preparation for future 64-bit release (1.7).

• Added check_by_ssh check command to sample database as requested by AlphaPiAlpha.

• Small fix to prevent changelog duplication in the nemsadmin home folder.

• Checkboxes have been missing from NEMS NConf for quite some time, since upgrading JQuery to a more current version. This has been fixed by adapting the code to changes in the JQuery-UI widget callbacks.

• Moved 9590, hw-detect, nems-migrator, nems-scripts, nems-www and wmic to dpkg repository.

• Adapted nems-update to upgrade all possible apps via apt-get rather than git.

• Stripped out a lot of legacy code from NEMS Migrator and NEMS Scripts.

• Reworked NEMS Migrator’s MySQL base settings. Now take significantly less space.

• Moved NEMS Merch store to https://shop.nemslinux.com/

• Upgraded NRPE to 4.0.3.

• The file check_rpi_temperature has been renamed to match its check command, check_sbc_temperature since it was upgraded to support more than just RPi a while ago.

• Include Fahrenheit instead of just Celsius in check_sbc_temperature / NRPE CPU temperature check as per tripled

• Moved all included Nagios check command plugins to nems-plugins package on the DPKG Repository to ease the update process should fixes or new features be implemented down the road.

• Upgrade check_ncpa.py to 1.2.4.

• Add NCPA sample check_commands to NEMS NConf as per joeluzzi: check_ncpa_mem (Memory Usage) and check_ncpa_processes (Running Processes).

• Update boot for USB boot on Raspberry Pi as per Kevin Shumaker.

• Add 1-Wire Temperature Check as per jtoland.

• Add SONOFF / Tasmota IoT device monitoring as per AndroBuilder.

• All speedtest commands have now been re-written (again). The fast.com fix from the end of 1.5.x to replace Ookla was heavy (required a headless Chromium task), sometimes unreliable and didn’t work on all platforms. So I’ve re-written the system entirely, now using Cloudflare’s Speedtest to test the Internet connection speed. The results are cached for 2 minutes, so if a user accidentally runs multiple instances, it will not bottleneck nor give incorrect results due to high bandwidth usage.

• Nagios i fully integrated, along with check commands for IBM i platform as requested by chris_hird.

• IBM i integration may now be enabled in NEMS SST under optional services. It is disabled by default since it requires an additional daemon be running.

• Add option to NEMS SST for “disabled by default” optional services that must be enabled manually.

• Adapt NRPE configuration to support newly named check_sbc_temperature script.

• Direct NEMS Linux to pull the NRPE config file from NEMS Migrator package rather than downloading from Github (as would be the case when installing on a non-NEMS system).

• NEMS Linux 1.6 Beta Build 1 released to Patrons.

• NEMSeOS 0.1 Alpha Build 2 released to Patrons.

• Update fk_id for NEMS Migrator restore, and temporarily disable nems-restore feature until more testing can be done.

• Remove state text (UNKNOWN, WARN, CRITICAL, OK) from check_nems_osb - let the output be based on the exit codes. Also change date output to show full day, not 3-character short-form.

• Add nems-plugins to nems-update procedure.

• Remove NEMS Hardware ID from RPi-Monitor. Thanks to UltimateBugHunter-NitPicker for pointing this out. NEMS Server Overview is the correct place to get this now.

• Add NEMS Server’s platform name to RPi-Monitor.

• Temporarily reverted NRPE host installer to 1.5.x compatibility since 1.6 uses a newer version of NRPE. Thanks to smccloud for pointing this out.

• Added libnumber-format-perl as a dependency for wmic package.

• Improved the error output of check_mrtgtraf_nems after PixelSlayer Bob reported messy error output.

September-December 2021

• Upgraded NagVis to 1.9.27.

• Create new check_truepool check command.

• Added pool share percentage to output of check_truepool and formatted output.

• Further improved output of check_truepool and added a cache and some error handling to ensure accuracy.

• check_speedtest_cli updated to 2.1. NEMS Linux requirement removed so it can be used on non-NEMS systems. Set paths dynamically and prepare for backport to NEMS Linux 1.5.x.

• If speedtest is missing any of the required components, they will now be installed rather than just providing an UNKNOWN state.

• Pipe speedtest logs differently to avoid errors if the check is run by a user other than nagios.

• Fixed array associations with speedtest output.

• Made it so if someone scheduled/ran the speedtest task more than once per 2 minutes, it will pull the response from cache rather than running multiple speedtests simultaneously (which would skew the results).

• Speedtest updates backported to NEMS Linux 1.5.x.

January-July 2022

• Add TEMPer sensor support for TEMPerGold_V3.3

• Update speedtest to automatically kill the process if it has been running > 10 minutes.

• Fix speedtest to load the log data first, ensuring correct output even if the task is already running.

• Set livestatus version to 1.5.0p13.

• Update moment.js to 2.29.3 to patch against directory traversal bug and backport this patch to 1.5.x.

• Update WSGI to Python 3 version.

• Add libraspberrypi-bin (particularly for vcgencmd so I can add a check for under-voltage).

• Added detection of other Pi models with Model IDs between 150 and 199 (the new Raspberry Pi 4 Rev. 1.5, for example) so these will be recognized by NEMS Linux as Raspberry Pi. Previously the detection looked for Model IDs between 0 and 9.

• Add undervoltage check to nems-info to detect power state of Raspberry Pi.

• Upgraded NagiosTV to 0.8.5.

• Adagios moved to 2.0.1, deprecating its dependency on Python 2.7 and moving to Python 3.9 in a venv.

• Add some extra fonts that are required by Monitorix to fix broken fonts.

• pnp4nagios has been missing for a long time since development stopped and it was pulled from repositories. Write a new compiler to re-add pnp4nagios to NEMS Linux.

• Several packages whose names were changed have been replaced in NEMS Linux with their modern counterparts. NetworkManager, PyWBEM and ModSecurity to name a few examples.

• Ensure all Adagios code is compiled within a Python Virtual Environment.

• Allow authorized users (members of the nagios group) to execute nems-info commands that require root access without needing a password.

• Add int option to nems-info undervoltage to send an integer response rather than plain text.

• Remove extraneous output from nems-info undervoltage - we only want to know if it’s under voltage, or not.

• Adapt wmic to Python 3 code rather than the old Python 2 version.

• nems-tv moved to Debian repository.

• nems_sendmail_host and nems_sendmail_service now save state to /tmp/email_failure.tmp if the email failed to send, and upon a successful mail send will follow up by sending that log. This means if Internet goes down, notifications that occurred during that time will later be sent (for example). Previously, if the notification couldn’t send, it would just not go out.

• nems-plugins - Added check_snmp_apc_env for APC NetBotz environmental sensors.

• checkmk livestatus upgraded from version 1.5.0p13 to 2.1.0p9.

August-December 2022

• Upgraded NEMS PHP Agent to 1.5. Now will report CRITICAL state if a disk mountpoint is specified but not mounted. This will ensure an alert is received if a mounted drive fails or otherwise disconnects.

• Upgrade Nagios to 4.4.7.

• Disabled check_for_updates in Nagios config per https://github.com/NagiosEnterprises/nagioscore/issues/861 due to Nagios instability when enabled.

• Created NEMS Hero, which allows NEMS Linux tech support to access a NEMS Server in event of a support request. This access is subject to firewall rules and only allowed if the connection is made within 15 minutes of a reboot. This is in preparation for upcoming NEMS SAAS, as well as a means of restoring a user in event of a lost / forgotten password.

January 8, 2023 - NEMS Linux 1.6 RC for Raspberry Pi released.

February 22, 2023 - Fixed issue with new NEMS Servers not being imported to system, resulting in failed CheckIn account setup.

Visit the official NEMS Linux Git Changelog for more.