Nms current design

Design Information

All systems are deliberately being designed on UNDER POWERED machines.

• Frontend + Backend + Daemons running on a 2 core, 2 Gig RAM, 100GB Drive KVM Guest
• as of 07-15-23, ~80 hosts, and several hundred checks end up with 300MB RAM use with little to no IO wait
• 4.5% userland and 3.5% system in use
• Currently using Apache for frontend and backend on different ports. SSL enabled on API but not frontend

Backend

The backend is PHP 8.X using the Slim4 framework. Composer is used for additional functionality, however most of the installed packages are not actually being used. These are going to have to be removed and get down to the core packages needed before posting the code publicly.

• Some additional custom classes were written to simplify some tasks that were more of a pain to deal with using default classes than I liked.
• locations: app/ templates/ src/Infrastructure/Shared/
• Several previously serial tasks can be changed to include functions so they can be recycled elsewhere, but that is future.
• apiKeys are working on the backend
• initial testing for events require apiKey to be set as X-Api-Key in the headers for curl
• JWT works
• User access levels work
• Still some minor warnings to clean up due to PHP version change
  • As I have gotten better with PHP looking at older APIs makes me think some of them should be re-implemented
  • Focus more on CRUD for every API, and make sure information leakage is always minimal
  • Keep putting notes in your PHP! Even snarky crap can be useful

Frontend

Frontend is Bootstrap5 and PHP8.x

1. No database connections will EVER be allowed from the frontend. Anything to do with the DB must use an API.
2. Attempting to keep complex PHP out of the frontend. Mostly specific to rendering and curl calls
3. RRD graphs are rendered on the API server, not on the frontend. Do not pass unnecessary things to the frontend for security
4. AFAIK all icons and images are copyright free. If any are found not to be pull ASAP or create your own
5. No passwords stored directly on frontend
   1. single exception is a failsafe that will alarm anytime it is used
   2. This will be hard-coded so it cannot be silenced as it is last resort only
   3. API keys will be leveraged beyond user auth. Only "blessed" UI's will be able to talk to the APIs for sensitive data
      1. Making the assumption that there could be a kiosk UI that is different than an Ops UI outside of User access levels
      2. investigate support for a kiosk mode as a general user with minimal access and a long JWT timer?
      3. This is still at the spitball stage, and may never get coded out
6. User auth is still at the implementation level. It does work, but has not been wired in
   1. JWT implemented and works, along with userAccess levels, but need to tie into frontend
   2. Initial design will likely save JWT as a cookie to keep it simple initially
   3. JWT lifetime currently set at 1 hour, when live likely default to 12 hours
7. Kidnap or blackmail a real UX dev to get this beyond kindergarten level of UX

Database

• Currently using MySQL 5.7.
  • This is a deliberate choice follow a KISS principle where there are not new features in on version making it mandatory to upgrade to version X every time there is an upgrade. It also makes it easier to transfer between MySQL, MariaDB, and Postgres, or really ANY database that follows a SQL format.
• Old Ubuntu 12.10 host 4core, 1GB RAM, 40GB Drive, KVM Guest (many, many! multiple other services running on this as well)

1. Next plan is MariaDB latest stable
2. can almost guarantee a simple upgrade due to the way the database was designed
3. investigate a versioning path for upgrades on the software.
   1. basically apply patches that reflect if they were applied or not.
   2. think newznab and how they update the database on patch and version changes
   3. consider possibility of user bypass for patches as well. There may be reasons for a user to say no, not going to happen.
4. begin investigation of PostgreSQL support.
   1. May be less painful than anticipated due to classes.
   2. Begin building a default class to deal with Postgres, as what I use now is MySQL specific

Metrics

1. RRD stored locally on the API server. API will generate and display the images for the frontend to consume. Raw data is NEVER sent to the frontend for rendering unless there is no other option. JavaScript will be the death of me. I suck at it so badly, and really it seems to make the site SO MUCH SLOWER to have client side work done for complex things.
2. Graphite is running on 4core, 4GB RAM, 100GB drive KVM Guest with other apps running on it. Initially my thought was to use Graphite as the main graph engine, but thinking about small business, as well as being able to do future predictions I felt RRD was the best way to start this, and have Graphite supported but not the default graph engine.
   1. The graphite support I have in place currently is going to have to be redone in the "template"style so that adhoc stuff can be more easily added in the future. right now it is a series of regexes and database hacks that I am not happy with.
3. Beginning storage of application metric data as well to get a sense of devices >> service check load and latency for scaling
4. Reporting of application metrics will have to be baked in as a default report.

Templates

These all live on the API host within the templates directory. These are going to have to be expanded much further. Additionally the "templates" table is going to have to be leveraged more going forward. Currently templates exist to push metric data to rrd, graphite, and the database. Additional templates are in place for rendering RRD files. I prefer this route so that we can change templates easier going forward.

Daemons

Daemon controls have all been written in PHP. This likely is not the best language to write a daemon in, but I wanted consistency with what the rest of the application is written in. If I ever get additional support, I will likely have this revamped to Python.

SNMPTRAP

The snmptrapd system forwards all SNMP traps to a PHP script which parses and forwards events into the system after being processed.

NRPE

Leveraging the Nagios Plugin system for adhoc or external checks to add additional functionality. Currently supports a basic metric ingestion into RRD, but does not have a good rendering tie-in yet.

SNMP

• Currently using the net-snmp package via shell exec.
• I wanted more control over how SNMP responded than I could get with SNMP Classes or composer installed support Classes.
  • Some of them made things pretty but dropped legit info all over
  • Some composer packages altered the data enough that they choked on unexpected but valid data
• Also I FIRMLY believe if a machine is doing something a human MUST be able to duplicate the behavior. Using shell commands makes this easier to work with.
  • If a human cannot duplicate an event or failure, then you are guessing that it is legit. Always have a way to confirm "truth" whenever possible.

Installer

1. Currently nothing has been designed as a proper "installer".
2. Still considering if flat pack or bin installer is a valid solution. (unlikely, but would be kinda nice)
   1. Likely going with 3 tarball explodes. Frontend, Backend, Daemons.
   2. Daemons will be a hybrid of the backend with API's removed
3. Daemon controls written for init.d and systemd. Initial will likely be systemd only.
4. Investigate a docker solution for VERY SMALL installations (say 50 hosts or less, with a reasonable number of checks)

Recommendations to reproduce a minimal install

• Designed on Debaian system
• net-snmp
• snmptrapd
• nagios-plugins
• nrpe-server
• MySQL (version really irrelevant) >= 5.7
• Apache, or Nginx.
  • Debugging API can also be done via

     nohup php -S 0.0.0.0:8002 -t public 2>&1 >> ./logs/someLogSomewhere.log & 

• PHP composer (only used on backend currently)
• PHP 8.1 or greater. Testing 8.3 soonish
• 100 GB HDD.
• 2 Cores
• 2 GB RAM
• Write notes on oddball changes to php.ini, and php packages installed via apt.
  • libapache2-mod-php7.4 php7.4 php7.4-cli php7.4-common php7.4-curl php7.4-fpm php7.4-gd php7.4-gmp php7.4-intl php7.4-json php7.4-mbstring php7.4-mysql php7.4-opcache php7.4-readline php7.4-rrd php7.4-snmp php7.4-xml php7.4-zip
  • some packages not used currently like SNMP and XML, but may be used in the future.
• no Pecl used