10 ★ 8 ↺
napierge boosted

[?]o gyptazy »

Hey community! I would like to hear your thoughts on how you usually update your Proxmox nodes and clusters. How do you handle minor Proxmox and package upgrades with ?

What would you think about a new API endpoint that lets you run unattended upgrades with a simple call like:

/nodes/{node_name}/apt/upgrade
At the moment you need to use the node’s HTML5 console to perform upgrades. Other methods exist such as running unattended Debian upgrade scripts, using patch management tools like or , or automating the process with over SSH. My idea is to have an API based solution that relies on Proxmox authentication and authorization. This would also allow third party tools such as to provide automated patch management and even handle guest rebalancing in a way that is similar to DRS without requiring direct SSH access.
I have already been running this approach on several internal clusters since the release of PVE 8 without issues. Now I am interested to hear if you would use unattended upgrades in general or if you are already running them today.


A Proxmox Node running apt-get -y dist-upgrade command via API in a non interactive and unattended way by a patched and extended API from gyptazy

Alt...A Proxmox Node running apt-get -y dist-upgrade command via API in a non interactive and unattended way by a patched and extended API from gyptazy

    ...
    Older...

    [?]o JimmyChezPants 🇨🇦 »

    @gyptazy I'm a bare Linux guy, but I used ProxMox for a while when I was learning, not here to tell you not to use it jftr

    But I am going to encourage you, if the number of machines you manage is less than twenty, to always pull up a terminal and run your updates by hand. Most of the time it's just gonna be you staring at a text scroll, but now and then you'll catch an easy problem that would likely stymie an unattended update, and you might even get the occasional warning that you're about to break something.

    Once you get into large numbers of machines (I once ran a 500-node render farm for a couple years) that calculus changes, but it becomes relaxing even. :>

      ...
      2 ★ 0 ↺

      [?]o gyptazy »

      Personally, I'm running clusters all over the world (for my @BoxyBSD@bsd.cafe) project - Germany, Netherlands, Italy, France, USA, Canada, Singapore, Japan etc. - so there's already a higher node count and this made me create . Since PVE8, I always run unattended upgrade and never had any issues:

      * Node Validation
      * Patch node
      * Validate if patches req. reboot
      If yes:
      * Move all guests to another node by evaluating free resources (DRS alike style)
      * Reboot
      * Rebalance cluster again
      * goto next node ;)

      Even if there would be any issue - what is my benefit by seeing this on the console rather than being notified by the monitoring afterwards and just looking at the logs of the specific node. I'm fully with you for major upgrades, but for minors I never encountered any issues - but that's exactly why I ask - my references may not fit for all setups ;) So, thanks a lot for your feedback - appreciate it!

        [?]o Cinux »

        @gyptazy

        Privat habe ich nur ein node. Da ist nur fire and forget 🤣 via cli.

        Nur weil es eine API gibt muss man die ja nicht nutzen wenn man es bereits anders automatisiert hat. Für ProxLB scheint das sinnvoll zu sein.

        Wegen einem Node nutze ich das Privat nicht. Aber kann es mir vorstellen später im unternehmen bei uns einzusetzen. Aber da sind wir noch komplett im aufbauen.

          ...
          0 ★ 0 ↺

          [?]o gyptazy »

          Danke für Dein Feedback! Fire & Forget scheint bei Dir soweit ja auch ebenfalls recht problemfrei zu laufen :)

            [?]o Andreas Scherbaum »

            @gyptazy Only have one node (still trying to figure out something smaller as backup) for my home network. It has backups and all, but simply using the web interface is fine for me so far.

            I can see that if you want to automate the upgrade, the API also sends status updates. Would that be possible? Otherwise automation might just run into a dead endpoint while the system is upgrading.

              ...
              0 ★ 0 ↺

              [?]o gyptazy »

              @ascherbaum@mastodon.social hey Andres, thanks for your feedback! You can query the status by a different api path, indeed. Currently, I'm trying to find out if people would be open in general to use an unattended upgrade way (if existing) - or already go this way by different toolings. For it's interesting to know for #ProxLB, because I have multiple approaches to solve this issue for automated node patching but would prefer an API driven way.

                ...

                [?]o Andreas Scherbaum »

                @gyptazy Yes, I get that point. That's why I was thinking how I would use that if I ever automate this.

                  ...
                  1 ★ 0 ↺

                  [?]o gyptazy »

                  Three ways:

                  * Validating the upgradeable packages (more or less, it could still run into a race condition)
                  * A get instead post on the API path
                  * Polling the state of the returned job_id

                    [?]o stdevel »

                    @gyptazy That’s a good idea. Currently I’m using either tools like and to mirror the packages or just trigger updates using SSH. Having a dedicated API call would make it more streamlined.

                      ...
                      1 ★ 0 ↺

                      [?]o gyptazy »

                      Nice way, pretty similar to some other setups here :)

                        History

                        about this site - powered by snac/2.82-dev