Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/neondatabase/autoscaling. Pull mirroring updated .
  1. Feb 16, 2024
  2. Feb 15, 2024
  3. Feb 14, 2024
  4. Feb 13, 2024
    • Heikki Linnakangas's avatar
      plugin: Fix k8s status message if pod scheduling fails (#804) · fda961a8
      Heikki Linnakangas authored
      Saw this when I did "kubectl describe pod compute-...", on a VM pod:
      
          Events:
            Type     Reason            Age    From                 Message
            ----     ------            ----   ----                 -------
      Warning FailedScheduling 4m53s autoscale-scheduler 0/1 nodes are
      available: 1 Not enough resources for pod.
      Warning FailedScheduling 4m51s autoscale-scheduler running Reserve
      plugin "AutoscaleEnforcer": Not enough resources to reserve non-VM pod
      
      That message is wrong, because this was a VM. Fix the message to not
      specify whether it's a VM or non-VM pod.
      fda961a8
  5. Feb 12, 2024
    • Em Sharnoff's avatar
      plugin: Handle panics from agent requests (#785) · d46b3089
      Em Sharnoff authored
      Closes #760.
      
      AFAIK this hasn't been an issue in the past, but as we're trying to
      improve reliability, it's good to get this out of the way before it
      becomes an issue.
      
      Note that this PR is quite minimal - expanding the existing tech debt we
      have around how the scheduler plugin handles HTTP requests.
      It's probably ok *enough* for now. I don't expect we'll be making too
      many changes to it in the near future. See also: #13.
      
      Tested locally by forcing it to panic on every request:
      
      diff --git a/pkg/plugin/run.go b/pkg/plugin/run.go
      index 007554a..6da7728 100644
      --- a/pkg/plugin/run.go
      +++ b/pkg/plugin/run.go
      @@ -262,8 +262,10 @@ func (e *AutoscaleEnforcer) handleAgentRequest(
       		}
       	}
      
      -	pod.vm.mostRecentComputeUnit = &e.state.conf.ComputeUnit
      -	return &resp, 200, nil
      +	panic(errors.New("test panic!"))
      +
      +	// pod.vm.mostRecentComputeUnit = &e.state.conf.ComputeUnit
      +	// return &resp, 200, nil
       }
      
       // getComputeUnitForResponse tries to return compute unit that the agent supports
      
      The change appears to work as intended.
      d46b3089
  6. Feb 09, 2024
  7. Feb 07, 2024
  8. Feb 06, 2024
  9. Feb 02, 2024
    • Em Sharnoff's avatar
      neonvm: Use crictl to change container CPU, ditch cgroup (#738) · d30687bb
      Em Sharnoff authored
      NB: This PR is conditionally enabled via the --enable-container-mgr flag
      on neonvm-controller. There are no effects without that.
      
      ---
      
      We recently realized[^1] that under cgroups v2, kubernetes uses cgroup
      namespaces which has a few effects:
      
      1. The output of /proc/self/cgroup shows as if the container were at the
         root of the hierarchy
      2. It's very difficult for us to determine the actual cgroup that the
         container corresponds to on the host
      3. We still can't directly create a cgroup in the container's namespace
         because /sys/fs/cgroup is mounted read-only
      
      So, neonvm-runner currently *does not* work as expected with cgroups v2;
      it creates a new cgroup for the VM, at the top of the hierarchy, and
      doesn't clean it up on exit.
      
      How do we fix this? The aim of this PR is to remove the special cgroup
      handling entirely, and "just" go through the Container Runtime Interface
      (CRI) exposed by containerd to modify the existing container we're
      running in.
      
      This requires access to /run/containerd/containerd.sock, which a
      malicious user could use to perform priviledged operations on the host
      (or in any other container on the host).
      Obviously we'd like to prevent that as much as possible, so the CPU
      handling is now runs alongside neonvm-runner as a separate container.
      neonvm-runner does not have access to the containerd socket.
      
      On the upside, one key benefit we get from this is being able to set cpu
      shares, the abstraction underlying container resources.requests.
      The other options weren't looking so great[^2], so if this works, this
      would be a nice compromise.
      
      [^1]: https://neondb.slack.com/archives/C03TN5G758R/p1705092611188719
      [^2]: https://github.com/neondatabase/autoscaling/issues/591
      d30687bb
    • Em Sharnoff's avatar
    • Em Sharnoff's avatar
      neonvm: Add .status.restartCount, check it in e2e (#754) · eff3888c
      Em Sharnoff authored
      Summary of changes:
      
      - Add `.status.restartCount`, type *int32
      - restartCount is non-nil when .status.phase != "", incremented every
        subseqent time the VM enters the "Pending" phase
      - `(*VirtualMachine).Cleanup()` no longer modifies `.status.phase`
      - VirtualMachine restart handling sets .status.phase to "Pending" on
        restart, not ""
      - in e2e tests, add `restartCount: 0` to all VM object assertions
      - add `restart-counted` e2e test
      
      This is a pre-req for backwards-compatibility testing (#580), both so
      that we can ensure the VM doesn't slip to a newer neonvm-runner version
      by restarting, and so that we don't end up with newer versions causing
      triggering restarting.
      eff3888c
    • Em Sharnoff's avatar
      neonvm-controller: Use --concurrency-limit=128 (#783) · d6e95c8d
      Em Sharnoff authored
      We noticed issues at at 8, then 16, then 32, and even 64 for some large
      regions.
      
      128 appears to be stable in prod (even though it's overcommitting
      controller CPU limits 32:1).
      
      ref https://neondb.slack.com/archives/C03TN5G758R/p1706725735037289?thread_ts=1706160071.213319
      d6e95c8d
  10. Feb 01, 2024
    • Em Sharnoff's avatar
      neonvm: Use container statuses, not pod phase, to trigger restart (#749) · 7f17032a
      Em Sharnoff authored
      Extracted from #738, which adds a second container to the runner pods.
      Because of that second container, if only one container exits, the pod
      will still have `.status.phase = Running`, so we need to proactively
      notice that one of the containers has stopped and propagate that status
      to the VM itself.
      
      This also introduces some funky logic around how we handle restarts:
      Because the `Succeeded` and `Failed` phases no longer imply that QMEU
      itself has stopped, we need to explicitly wait until either the pod is
      gone or the neonvm-runner container has stopped; otherwise we could end
      up with >1 instance of the VM running at a time.
      7f17032a
  11. Jan 31, 2024
    • Em Sharnoff's avatar
      43a9b322
    • Em Sharnoff's avatar
      plugin: Unify reserve and unreserve logic (#666) · 3fa3fa35
      Em Sharnoff authored
      We had the same thing implemented in a few places, with some TODOs to
      unify them eventually - so, here it is.
      
      Summary of changes:
      
      - Merge all handling of pod/VM "start" / "reserve" logic to go through
        (*AutoscaleEnforcer).reserveResources()
        - Combined handlePodStarted/handleVMStarted -> handleStarted
      - Merge all handling of pod/VM "deletion" / "unreserve" logic to go
        through (*AutoscaleEnforcer).unreserveResources()
      - Pass around *corev1.Pod objects 
      
      This also fixes the issue behind #435 ("Handle bypassed Reserve ...").
      3fa3fa35
  12. Jan 30, 2024
  13. Jan 26, 2024
    • Em Sharnoff's avatar
      CI: Fix e2e-tests failure from empty $SHA on main (#767) · a7943db2
      Em Sharnoff authored
      ref https://github.com/neondatabase/autoscaling/actions/runs/7673566092
      
      ${{ github.event.pull_request }} doesn't exist for 'push' events
      :face_palm:
      a7943db2
    • Em Sharnoff's avatar
      CI: Use separate workflows for image builds (#748) · ca9e9447
      Em Sharnoff authored
      Brief summary of changes:
      
      1. Add new workflow `build-images.yaml` in that builds images and pushes
         them to dockerhub (e.g. neonvm-controller, autoscaler-agent, etc.)
      2. Add new workflow `build-test-vm.yaml` that builds vm-builder and
         makes the postgres:15-bullseye VM image.
          - Also uploads vm-builder as an artifact if requested
      3. In `e2e-test.yaml`, use images from (1) and (2)
          - Also uploads the rendered manifests as an artifact if requested
      4. In `release.yaml`, use images from (1) and (2), run tests with (3),
         and use vm-builder and manifests from (1) and (3).
      5. Adds `make load-example-vms` and equivalents, which load images
         without building
      
      Refer to the PR description for more info.
      ca9e9447
  14. Jan 25, 2024
  15. Jan 24, 2024
    • Shayan Hosseini's avatar
      neonvm: Custom reconciler metrics (#757) · 09eec06c
      Shayan Hosseini authored
      Providing custom metrics for reconciler objects.
      
      - `reconcile_failing_objects` represents the number of objects that are
      failing to reconcile for each specific controller.
      
      Fixes #247 (along with #739).
      09eec06c
Loading