NEMS Linux 1.6 Changelogs

The NEMS 1.6 release cycle will focus on hardware porting and peripherals.

Key Development Features and Goals:

  • Python moved to v3 stack due to 2.7 EOL.
  • Public release of NEMS Warning Light.
  • check_mk livestatus upgraded to 1.5 (requires C++17).
  • Supported single board computer list to be reduced to focus on most used, with option for users to request ports if they do not exist.
  • Port NEMS Linux to Docker platform. Development funding provided by Patrons (THANK YOU).
  • Firewall - NEMS Linux to include ConfigServer Security & Firewall (csf) with Login Failure Daemon (lfd). Internal LAN IP addresses are whitelisted by default, but if a user tries to access your NEMS server from outside the LAN (ie., if you have your NEMS server accessible on the Internet) they will get locked out if they try an incorrect password too many times.
  • Add IPMI (freeipmi) if not added already.
  • Begin work on smart home assistant integration: NEMS Skill for Alexa and Google Assistant.
  • Consistent networking configuration across platform (for setting static IPs, for example).

Release Dates:

NEMS Linux 1.6 Alpha was released December 2020. Beta release is planned for January 2021 with the public release to follow in Q1 2021.

Development Changelogs Since 1.5:

December 2019

  • Merged PR 4, adding escalations for hosts / services / advanced services and fixing various bugs.
  • Upgrade NEMS NConf jquery code to 3.3.1.
  • Tap NEMS NConf into user’s wallpaper settings.
  • Output the final line from syntax check to screen when deploying config in NEMS NConf.
  • Changed “Generate Nagios config” to clearer “Deploy Config to NEMS” in NEMS NConf.
  • Moved all the legal stuff off the landing page of NEMS NConf and moved to new “Legal Disclaimer” section.
  • Prepare Migrator for 1.6 and integrate new NConf configs.
  • Upgrade Nagios Core to 4.4.5.
  • Integrated NagiosTV as per BastyJuice.

January-February 2020

  • Upgrade Monit to 5.26.0.
  • Move build tmpdir out of /tmp/ and into /usr/src/
  • NEMS NConf 1.6 tooltip font color fixed.
  • NEMS NConf 1.6 removal of JQuery Accordion from Generate page, replaced with terminal output.
  • New Deploy button for NEMS NConf Generate Config page.
  • Change notify-service-by-telegram to point to the modern script, rather than the old path (which is a symlink these days). mydogboris noted that when using the symlink, his Telegram notifications were wonky.
  • Added the NEMS MOTD to Docker.
  • General filesystem cleanup: remove some old cruft.
  • NagiosTV theme customized to match user theme settings (such as background image).
  • Added humidity sensor to check_temper.
  • NRPE upgraded to 4.0.0.
  • Fixed NRPE installer script if run on non-NEMS system, as per Issue 2 by Xelo.
  • Revert NRPE to 3.2.1 temporarily, until I can figure out how to make it talk to older installs (upgrading to 4.0.0 breaks all existing 3.2.1 installs).
  • check_speedtest_cli.sh now records a log file in /var/log/nems/speedtest.log. This log is now parsed by NEMS TV Dashboard and displayed at all times. Further improvements will be added before public release. As requested by ITManLT.
  • Replace words “Down” and “Up” in NEMS TV Dashboard speedtest stats with font-awesome icons.

March-April 2020

  • Move 1.6 base to Debian 11 (bullseye).
  • Upgrade Monitorix to 3.12.0.
  • Added mod_wsgi to 1.6, which is no longer auto-installed with Apache2 in bullseye.
  • Initial user files now placed correctly in /etc/skel for user creation.
  • Change default body background color of NEMS NConf to dark grey (almost black) to prevent the appearance of the screen flashing white during page transition.
  • Moved old python-pip, python-django and python-paramiko to modern python3-pip, python3-django and python3-paramiko packages respectively.
  • Upgrade Adagios to 1.6.6.
  • Migrate to several experimental branches of Adagios components to begin testing support under Python 3.8. Current stable only supports Python 2.7 which is EOL this year.

May-June 2020

  • NEMS Linux 1.6 development merged (some would say “downgraded”) with NEMS Linux 1.5.2 as I wait for Adagios support for Python 3.7+. NEMS Linux 1.6 will be buster-based and use Python 2.7.
  • Change ownership of /var/log/nems to nagios user and group to allow check commands to write to logs. This enables the speedtest log, which in turn shows speedtest data on NEMS TV Dashboard.
  • TEMPer calibration added to NEMS SST and integrated into all TEMPer output. Now, for the sake of accuracy, users may adjust the thermal and humidity sensor offset by simply dragging a slider.
  • Hide TEMPer calibration option if TEMPer device is not present.
  • Add Argon ONE scripts for Raspberry Pi 4.
  • Argon ONE support complete.
  • Added repos feature to nems-info.
  • Added new check_mrtgtraf_nems check command in preparation for a very sleek MRTG integration.
  • Set Argon ONE to log to /var/log/nems/argonone.log if system is rebooted or shutdown safely using the power button (with timestamp).
  • Move rpimonitor off the repository and instead manually install. The source repository (http://giteduberger.fr/rpimonitor/) became unstable during the 1.5 release cycle (up and down) and since rpimonitor appears to be a dead project (hasn’t been updated in > 4 years) I’ve instead removed the repository requirement and installed it manually. Also, fixed the distribution name on the template.

July-August 2020

  • Change logrotate for NEMS logs to rotate logs based on log size, not date. Delayed compression added.
  • check_mrtgtraf_nems now rounds the floats of the traffic results to two decimal places.
  • Added check_apc to check the state of an APC UPS. As requested by Neptunum.
  • Python version default will now be hard set at 995 during build process. This is in preparation for python3 down the road (pending Adagios compatibility).
  • I noticed logrotate rotating some NEMS logs that really shouldn’t be rotated, such as cloudauth.log. Modified logrotate.d conf to omit logs that should not be rotated.
  • Change permissions of /var/log/nagios/nagios.log to allow group write. Required to remove root user requirement on nems-mailtest, which will be needed to integrate it into NEMS Dashboard.
  • nems-mailtest is now integrated into NEMS SST. See Video
  • nems-mailtest in NEMS SST now runs nems-mailtest using the information entered in the form, not saved to the config. This way, a user may test their SMTP config before saving the changes.
  • Reverted MySQL database to previous push of 1.6 branch as I had accidentally pushed the 1.5 database to the 1.6 branch when I added check_apc. I’ve corrected it, and re-added check_apc. No users impacted by this since 1.6 hasn’t yet been released.
  • Added nems-info [dht11|dht22] which outputs json response from either the DHT11 or DHT22 sensors.
  • Add user to gpio group during initialization. This will allow access to the GPIO pins without root access.

September-October 2020

  • Minor improvements to output of DHTxx check command for Humidity values.
  • Add nemsadmin user to gpio group, just to ensure the demo check commands work out of the box, before initialization takes place.
  • Upgrade check_mk Livestatus to 1.6.0p17. This gets the sock working on the NEMS Linux 1.6 AWS development system, which is required for Adagios and NEMS TV Dashboard.
  • Added phpmyadmin, disabled by default and interface access protected behind NEMS user login/password when enabled.
  • Change check_local_disk to support unit selection (KB, MB, GB, TB) and set default for NEMS Local disk check sample to GB.
  • New Feature: NEMS PHP Server Agent can now be configured and downloaded in NEMS System Settings Tool.
  • New documentation launched, with the help of Bill Marshall plus submissions from Don Jenkins.

November-December 2020

  • NEMS Tools now automatically detects the IP address of the running NEMS Server on the same subnet and creates its configuration file.
  • NEMS Tools GPIO Extender client now uses the config file to determine NEMS Server IP address.
  • Omzlo Warning Light pHAT now transmits and receives via NEMS GPIO Extender, allowing you to plug the pHAT into your NEMS Tools GPIO Extender Receiver.
  • Fixed a typo in Warning Light that incorrectly determined all platforms to be a Raspberry Pi.
  • Update nems-tools GPIO Extender to detect NEMS Servers via gpioe-server on port 9595 rather than looking for a host reply from nems-api.
  • Moved nems-tools.conf to /boot, making it extremely easy to modify the conf on nems-tools Extender OS.
  • NEMS SST will no longer warn of requirement to add a password for the PHP Agent if a password already exists in the config.
  • NEMS PHP Agent 1.1 released. Now the keyphrase is encrypted (rather than base_64 encoded). Also added “.” current folder disk space check and fixed several bugs with disk and var checks.
  • Fixed the directions on NEMS SST which directed a user to add their encypription/decryption passphrase to the “General” tab, since that feature was moved to the NEMS Migrator tab long ago. Pointed out by UltimateBugHunter-NitPicker.
  • NEMS Extender OS will now illuminate all lights if the NEMS Server goes offline or cannot be contacted.
  • Improve output in nems-init if user tries to initialize with same name as already running user.
  • Added qemu-guest-agent to improve integration with Proxmox VE as per UltimateBugHunter-NitPicker.
  • Delay for a random amount of time (up to 2 hours) when running offsite backup tasks to prevent 1,000+ NEMS Servers clobbering the API all in the same moment. Issue pointed out by UltimateBugHunter-NitPicker who was seeing errors with his daily backup.
  • NEMS Migrator Off Site Backup will now log if the Internet was down when the backup ran. Also added a ‘now’ cli option that will force it to run without delaying.
  • Changed Migrator patches failsafe to determine if Quickfix / fixes was running for more than 120 minutes (previous setting was 90) before killing.
  • Adjusted NEMS Off Site Backup to trigger at midnight but delay for a random amount of time up to 4 hours before running the backup. This will further reduce strain on the NEMS Cloud Services servers as NEMS’ userbase continues to grow.
  • Added missing Set::IntSpan package, required by check_mikrotik_switch. Modify the check script to provide better (cleaner) output if CLI type not provided.
  • All check_mikrotik_switch check commands renamed from the old check_mt_ and now are check_mikrotik_. Also fixed argument count and improved descriptions in NEMS NConf for each of the MikroTik checks.
  • Removed check_minecraft. It hasn’t been kept up to date by its developer, and unless there is a demand for it, I don’t want to have to take over maintenance on such a niche plugin.
  • Added check_ncpa along with two sample checks: check_ncpa_processes and check_ncpa_mem.
  • Fix bug in PiVoyager daemon that falsely detected PiVoyager hardware on some setups where it didn’t actually exist.
  • Several small web interface fixes in nems-www.
  • Fix running user detection in nems-info to prevent www-data from attempting to use user cache.
  • Removed NEMS host from HTTP Advanced Service. It was included as a sample, but since the interface can run quite slow on some low-powered SBC’s, it causes timeout notifications which tend to confuse users into thinking there’s actually a problem.
  • Added check_synology as per AlbertPauw. Added several Advanced Services samples to Synology host group.
  • Improved Warning Light’s detection of Omzlo Warning Light pHAT to prevent log bloat.
  • Improve NEMS Extender OS’s browser-based output to include an iteration, which will help in event of a hung service: If the iteration (counter) no longer increases, it becomes more apparent that something has gone awry.
  • Disable the TEMPer thermal and humidity checks by default to prevent new users seeing a warning that they are missing the sensor. Leave the demo checks in NEMS NConf for easy re-activation.
  • Fixed permission issue on NIC cache if root is automatically detecting NIC while user is simultaneously logging in as non-root user. This bug was seen in Novaspirit Tech’s video introduction to NEMS Linux 1.5.2 when signing in via SSH. The MOTD did not display, and in pausing the video I see it is in fact the nemsadmin user momentarily not having access to the NIC cache. So I fixed it.
  • Rearranged NEMS Warning Light daemon to ensure the Omzlo Warning Light pHAT over a NEMS GPIO Extender is synchronized to the GPIO pin output (I.E., don’t delay for an iteration: instantly change states).
  • Improved the output of all disk checks in NEMS PHP Agent. Now the mountpoint will be listed in brackets after the percentage, making it easier to see the actual state.
  • NEMS Migrator Off-Site Backup schedule maintenance automatically keeps backups tidy now. Current schedule is that you have access to every daily backup for the past month, and a weekly backup for the past year.
  • Added new check command check_nems_osb which will notify if a NEMS Migrator Off-Site Backup fails.
  • NEMS PHP Agent 1.2 released with the following improvements: Network usage now uses ifstat and generates a more accurate average usage number based on all network interfaces on the server with a 5 second average. New agent will only run the equations and tests for the requested check. For example, don’t run a 5 second network bandwidth test when the requested check is for the load average. Fixed bug where nettx was in fact reporting netrx.
  • Updates to NConf to improve output. Add AJAX spinner during generation, remove horizontal scroller, etc.
  • Block error output when detecting NIC to prevent MOTD being broken during first login.
  • Determine the fk_id of the NEMS Host and adapt nems-restore to use this (NEMS 1.6 branch only). Keep 1.5 branch separate and improve compatibility with 1.6.
  • When a Patron opens the NEMS Dashboard (I.E., they have a valid NEMS Cloud Services account) a link is now available to visit the latest Patron-exclusive posts.
  • NEMS Hardware ID is now blurred by default and hidden from display in NEMS Server Overview. I saw a YouTuber who opened NEMS Server Overview willy-nilly on his video and did not blur this information in post production, so I’ve enforced it by default. The HWID can be revealed by double-clicking the blurred area.

January-February 2021

  • Migrator tab now has more intuitive output when a new NEMS Cloud Services account is activated (I.E., Notice re. waiting 24 hours for first OSB).
  • NEMS Tactical Overview (NagiosTV) now running 0.6.5 and now uses Chris Carey’s implementation of the Livestatus connector rather than the previous Nagios CGI method. In lay speak, this means when you open NEMS Tactical Overview, you’ll no longer have to supply a password within five seconds to login to Nagios Core. Huge thanks to Chris for making this change for us!
  • Pause general development to work through major issue with WMIC following changes to Microsoft Windows starting with Windows 10 Version 2004.

March-April 2021

  • The WMIC issue has been fixed. Final stages of development for NEMS Linux 1.6 can resume.
  • Many updates to NagiosTV bringing it up to 0.7.3, which improves error handling on connection loss and much provides overall performance improvements. These updates are with thanks to Chris Carey, who we are pleased to have welcomed to our Beta Team earlier this year.
  • Remove the usage of Ookla’s speedtest service and replace it seamlessly with Netflix’s fast.com. This is due to changes in Ookla’s licensing agreement, but also means the check commands have been rewritten with better error handling. I forked the fast-cli project so I could make necessary changes to port this to arm64. fast-cli does not work on arm processors since it uses puppeteer which depends on the x86 Chromium headless browser. My fork instead calls for the version of Chromium built specifically for each system, which means it will work on all platforms running NEMS Linux.
  • Set service timeout to 120 seconds rather than the previous 60 seconds. The short timeout was causing timeouts with speedtest since that takes extra time to execute on some connections.
  • Created a new --reset option for nems-quickfix which allows resetting a previously-applied patch. This can be useful should the patch fail for whatever reason.
  • Made the new speedtest script load a cache file if it is already running. In an event where the user had initiated multiple simultaneous speedtests, many processes could spawn resulting in crippling bandwidth usage. Noted by UltimateBugHunter-NitPicker.
  • Backported the speedtest update to NEMS Linux 1.5.
  • nems-quickfix now resets the log but appends both runs to it each time it runs. This will give me the ability to investigate QuickFix issues more easily.
  • nems-quickfix now ensures all previous package installations are complete before running patches. Was an issue if user had previously rebooted their NEMS Server during an update leaving some packages broken.
  • Several compatibility fixes added to the nems-speedtest patch (000015) to ensure backward compatibility with as many NEMS Servers as possible.
  • Continued work on WMIC, which has posed challenges in porting across platform.
  • Launched new NEMS Linux repository for 1.6. By doing so, I plan to compile WMIC on each architecture and then be able to install via apt, rather than going through so much trouble compiling from scratch on every single board. Inevitably this will also mean moving all NEMS packages off github (as far as how NEMS Servers obtain the code) and instead maintain a single apt repository. This could mean much easier cross-platform support from my perspective, and faster adoption of updates across all boards.
  • Modified the speedtest script to wait 100 seconds for the task to finish and then forcibly kill it. If this happens, NEMS will report “0 Error” instead of a speed result. By doing this, I prevent slower connections from having the speedtest check killed by Nagios (service timeout) leaving a compounding number of Chromium tasks running, eventually leading to high load and slow operation.
  • Upgrade CheckMK to 1.6.0p23 and move source to Github (rather than a zip from their site, which became deadlink after a recent redesign).

May-July 2021

  • Clean up a lot of the old build scripts and nems-upgrade packages to accommodate the changes in NEMS Linux 1.6.
  • Deprecation of Samba wmic complete: NEMS Linux 1.6 now entirely moved to new custom Python replacement.