KubeCon Europe 2025: days 2-3 (at last)

I’ve (finally) finished writing my my notes from the remainder of KubeCon; see my previous post for some background and initial thoughts. This is a lot later than I’d like, I was pretty tired after each day in the end and then life and other work has taken over a little bit since then. Hopefully it’s not all too murky to make sense of my notes. At least by now, the videos are up, so I can jog my memory as I go. I’ll add links to the recording of each talk, and also go back and do the same for my previous post.

This’ll be a pretty long one, as a paragraph or two for all the sessions I attended adds up fast. My general summary is sadly a bit removed from my feelings at the time, but I’ll do my best to remember my feelings at the time.

Day 2

Keynotes

This was the first day we made it to the keynotes. Unfortunately, there were a few issues with the clicker and screens for the speakers today. Today’s keynotes were the “end-user showcase”, where “end-user” in CNCF jargon means “company which has deployed Kubernetes” (not “someone who visits a website served by Kubernetes”). As a result, I didn’t get as much from these keynotes as I did from the talks in general or the next day’s, especially as almost all of the end-users were operating on a vastly larger scale than we do.

My notes from the flurry of end-users on stage are pretty brief, as each one only had a few minutes, and I had to take their (obviously sugar-coated) takes on Kubernetes tech at face value without being able to get into much detail.

HSBC

My only notes here are “stunning shirt” and “way bigger than anything we do”.

Peptone

As they did for the whole conference, my ears metaphorically pricked up when I saw a research-y organisation appear on stage, in this case a biotech company modelling disordered proteins or something (biology was never my strongest science). My main takeaway from this one was that they found it easy to migrate from dev machines with local GPUs to using Nvidia cards in the DGX cloud.

Spotify

This talk was focused on their Backstage “Internal Developer Platform”, which is a UI to let developers (and presumably SREs, platform engineers, devops engineers and whatever the trendy job title is these days) see what is going on across Kubernetes. It was interesting to hear how they had to balance company-internal needs and those coming from the open-source community as they built improvements.

Apple

The first of many times that I saw Katie Gamanji on stage at this conference, today with her Apple hat on (rather than as part of the Technical Oversight Committee as she appeared later on). Her topic was about how Apple used gRPC to connect the Apple Intelligence system (running locally on iPhones/iPads/Macs) to private clusters. Surprisingly enough for an Apple project it was written in Swift, and as a result it was too hard to resist mentioning that it was Memory Safe (as opposed to all the other CNCF projects written in Go).

End User Awards

This was mostly a promotion for various engagement activities run by the CNCF for end users. It was the first time I’d heard “observability” being reduced to the numeronym “o11y”, which for some terrible reason the speaker pronounced as “olly”. Numeronyms in general have a slightly bad reputation, which might not be entirely deserved, but I don’t think there’s ever a reason not to pronounce them as the thing they abbreviate.

Some other initiatives that seemed interesting were the Academic Accreditation Program to certify academic courses at universities and other educational institutes, and GitJobs.dev, a jobs board promoting careers opportunities with time allocated to working on open source and upstream projects.

CERN/Linux Foundation/OpenInfra

I thought the HSBC talk earlier was a big estate, but CERN’s system puts almost everything else to shame. The top level numbers for (I think) just the ATLAS project come to over 10,000 servers with several petabytes of RAM between them. They were dealing with petabytes of data in the early 2000s, and are now looking at exabytes.

Michelin

This was another end-user example running on a scale much larger than our dozen-node, several-hundred-pod deployment. Still, the problem they were talking about, vendor migrations, is something that applies at all scales, and building a solution entirely on open-source technologies is a great way to remain vendor-agnostic.

Red Hat

It had been a while since anyone talked about AI, so it was about time for a run of talks about running AI models on Kubernetes. The trend is for pressures on cloud infrastructure and technology to push towards solutions based on agents and orchestration models working with smaller models, which in theory is a good fit for Kubernetes, but the problem is how to handle state, at a much larger and more dynamic way than traditional workloads have demanded.

Panel discussion

The panel featured Joseph Sandoval from Adobe, Liz Rice from Cisco, and the return of Katie Gamanji of Apple to talk about the history of the CNCF around its tenth anniversary. The CNCF took credit for the convergence of the industry on Kubernetes as the standard orchestration engine, as a vendor-neutral foundation with ownership of the project. Focuses for the future include multi-cluster observability across providers; cost management, sustainability and hardware management; and secret management (which is definitely something I’d be interested to do better!).

Solo.io

This was pretty much just an advert for Solo.io’s sidecarless service mesh. I still don’t know what a service mesh is or why I’d use one. It also includes some Kgateway thing that lets you connect to LLM providers, because it had been too long since anyone had mentioned AI.

Mirantis

This was more or less another advert, one much more focused on AI, showing how their new control plane is great for firing up LLM workloads. It included the amazing line “Kubernetes has the opportunity to win open source”. I wasn’t previously aware that open source was a competition you could win, but apparently Kubernetes can do it.

OTel sucks!

I’ve subtitled this one “and so does the conference wifi”, since no amount of technical expertise on the event side will compete with 13,000 geeks trying to use several devices each at the same time. I was hoping this would be something of an introduction to OpenTelemetry, which seemed to be everywhere. Monitoring and alerting is something I’ve wanted to improve at work and I was hopeful that OTel was something we could put to use on that front. The format used in this “X Sucks!” talk is to have a bunch of community members give (edited) complaints about X, then re-play the full video in which it is revealed that the complaints are actually strong points. My notes contain a lot more downsides than advantages, which is perhaps indicative of how this format can end up a bit back-slappy.

The quote I’ve jotted down perhaps sums up OTel: “It is a lot”. The speaker counted up 13 distinct APIs and a huge number of SDKs for different languages. Autoinstrumentation can be a bit overzealous in collecting unnecessary data and overload the collection process. The one pro I’ve got is that the “stability of the semantic convention is taken seriously” which seems a little opaque. The semantic convention came up almost every time OTel was mentioned, and it’s clear that it’s a very important idea.

Strengthening Auth in Kubernetes

I’m not really sure why I decided to go to this session: perhaps I was hoping to understand better how to handle multiple users in one Flux cluster. In any case, I did pick up one or two interesting and useful things from this talk. I’ve split my notes into APIs which have “graduated” and those which are “upcoming”.

In the graduated section, I have noted down a new way to pass CA roots into Pods, which is not something I see myself using any time soon (although we have occasionally run into issues before we migrated to LetsEncrypt with the certificate on our GitLab container registry expiring). There was also something about new authorisation1 for the Kubelet API where it’s now possible to restrict things to the /configz, /healthz and/or /pods endpoints.

The “upcoming” part had some things which were a bit more relevant. It turns out that if an image which requires authentication had already been downloaded, the Pod’s credentials aren’t checked. I was pretty surprised that hadn’t already been the case, but soon that little hole will be closed. I’ve also written down “Service Account pull credentials rather than separate secret options”. I’m not sure what this means, and I can’t find the point in the talk recording which prompted me to make this note. Hopefully it means something to someone.

KubeCon Family Fortune

Another “fun” presentation, taking the popular game show format (also known as “Family Feud”) to Kubernetes. I appreciated them framing it as “Tabs v Spaces”, although a serious opportunity was missed by putting Tabitha on the “Spaces” team, or at the very least not by pointing out this contradiction.

GPU Sharing at CERN: Cutting the Cake Without Losing a Slice!

A talk by CERN promised to be interesting, if again beyond our scale. This had a bit of a classic computer science bin-packing flavour to it, about optimising GPU usage between teams and workloads. Traditionally, a GPU would be assigned to one team, but would often be unused between training runs or similar. This would suggest a move towards a more centralised setup to reduce idle time. At that point you run the risk of demand outstripping supply, so GPU sharing is vital to effectively use your resources.

There are a few different technical approaches to sharing a GPU across workloads. The simplest is time-slicing, a strategy which dates all the way back to the mainframes of old. It’s well-understood and straightforward, but has issues when it comes to memory-sharing and context switching. One alternative is known as Multi Process Service (MPS) where a control daemon process manages cooperative multitasking. This means that there is no context switching overhead, and also it’s impossible for one process to starve others.

Another approach is Multi Instance GPU (MIG), which requires compatible GPUs which can be sliced into independent partitions. This is a nice idea in the abstract, but it’s a bit inflexible as you can’t readily re-partition the GPUs as workloads change and the configuration is per-node. This can be solved by the upcoming Dynamic Resource Allocation tooling (which was mentioned in a lot of talks).

The speaker finished with some benchmarks, showing that the different types of workload can behave very differently on MPS vs timeslicing, so it’s important for a platform team to educate the users on which works best for their compute tasks.

Image Snapshotters for Efficient Container Execution in Particle Physics

Another talk by CERN was my last one of day 2. They started with the stat that 85% of the bytes downloaded are not used by a container. Physicists’ containers often end up gigabytes in size in such a way that they can’t be slimmed down. This means that images have to wait a long time before the whole container is downloaded before they can start. One way to reduce this startup latency is lazy pulling, where the storage layer intelligently fetches only the bytes that the container actually needs, at the cost of some runtime performance.

Lazy pulling requires some metadata indices to the container so that the runtime knows what bytes are needed for each request by the container. The main solution presented here was CUMFS, a FUSE-based filesystem developed at CERN. It’s a fully global lazy filesystem, which is generally mounted at /cumfs by nodes at CERN. It serves something like 4 billion files running to multiple petabytes (they said it had been proven up to 100 PB). They showed some every impressive looking graphs, but warned about using lots of small files, and the fly crawling across the lens of the projector was a little distracting.

Day 3

Keynotes

Google/Bytedance

This was basically an announcement for their new Inference Gateway for self-hosting LLMs (the first hint at something of a theme for the keynotes this morning). One issue is how LLM requests are quite variable and unpredictable in terms of input and output size. There are also issues where, for supply chain and hardware availability reasons, clusters might run a range of different GPUs. Overall, traditional microservice requests (small, predictable, cheap) are very different to LLM requests (large, variable, expensive), so the Inference Gateway provides more intelligent load balancing.

Oracle/Red Bull Racing

This was a pretty content-free talk really, just Oracle making the most of sponsoring the F1 team. They spoke a little about how they were using AI for real time insights for race strategy, and also getting input on FIA regulatory desicions. They also managed to make it seem like they were rather ignorant of recent politics by saying that this was all happening “right here in the EU” on a stage in post-Brexit London.

Panel: cloud native in telecoms

I’m not sure how informative this panel discussion was really. The main takeaways I had were that telcos tend to be reasonably conservative, with physical boxes wired together virtually. They spoke about an ambition to upgrade to 6G by some sort of rolling upgrade or operator subscription, which sounds pretty ambitious.

Cutting Through the Fog: Clarifying CRA Compliance in Cloud Native

This was the first detail I’d heard about the Cyber Resiliance Act, upcoming EU legislation for “products with digital elements”. It’s unclear to me what a scientific institute working (generally) outside the EU, albeit often partnering with EU-based universities and research centres, has to worry about the CRA. Still, it is a codification of mostly good practices, but how that shakes out for open source projects rather than companies remains to be seen. One interesting aspect of this is the notion of “stewards” as an intermediate between maintainers and manufacturers, which is likely the role that the CNCF and Linux Foundation would assume.

Lessons Learned in LLM Prompt Security

It had been a whole two talks since we last talked about LLMs, so it was time to bring them back up again. Despite being a sponsored keynote, I did find this one a little bit interesting, although more in the way of morbid fascination. If you are running an LLM to which users can submit queries, how can you make sure their prompts aren’t dangerous (whatever that means in your usage context)? The obvious way, if you are that deep into the hype cycle, is of course to run another LLM to classify prompts as “safe” or “not safe”2. However, as we saw in the earlier talk, LLM queries tend to be slow and expensive, which is not what you want when you are building a filter into your load balancer. More “advanced” techniques, such as text filtering(!), are required.

221B Cloud Native Street

It’s Katie Gamanji again! This time, it was a presentation from the Technical Oversight Committee (TOC), giving their update on what they’d been up to. They explained what the CNCF project maturity levels meant and let us know where the Technical Advisory Group (TAG) reform was going. They also gave an update on the End User Technical Advisory Board, which is currently working on “Feedback Loops”, “Reference Architectures” and “Gaps”.

Science at Light Speed

Another talk focused on science, but unfortunately again at a scale way beyond the few gigabytes a year we’ll be collecting. The Square Kilometre Array (SKA) is a radio telescope currently under construction in Australia and South Africa. It will produce something like 600 PB of data each year, and this presentation went into how you can possibly build infrastructure to handle that. The starting point will be local processing centres in the same country as the telescope and then distributed resources around the world.

The project has 14 SKA Regional Centres (SRCs), which independently use whatever infrastructure they choose to provide a unified service layer, as much as possible with off-the-shelf tools. The speaker was from the Swiss SRC and explained how they used worker nodes from Swiss supercompute facilities. There was also a quick demo of using the network to access some data, and, as seems to be the case for every single data science demo, the example used was to cookie-cut a subset of the data out.

Ensuring Quality in Kubernetes: The Graduation Process From Alpha To GA

This talk pairs with the other one I attended about the lifecycle of deprecations in Kubernetes, and describes how APIs go from alpha to beta to general availability, and a bit of describing what each step along that process means.

Alpha APIs should work and not be buggy, although the interface is not necessarily stable yet. They emphasised the point that API stability is not just about syntax, but also semantics and behaviour.

Alpha and beta APIs are disabled by default, and part of the consideration as they progress is the scalability and performance requirements. The testing requirements also increase as the feature moves towards GA. The talk finished with a bit of a discussion about the project’s strong policy about avoiding flaky tests, and how much work had been done to reach that standard.

The State of Prometheus and OpenTelemetry Interoperability

I got most of the sort of introductory overview of Prometheus and OTel I was hoping for out of this talk, even if it wasn’t intended as such. There was a lot of describing the philosophical differences between the two systems.

The first difference is how Prometheus pulls data while OTel pushes it. Pulling data means that the instrumented service has to keep its metrics in memory for when it is queried, and doesn’t let you only submit data when it changes, which are two points in favour of push-based architecture. A key downside of pushing data is that your instrumentation finds it difficult to tell the difference between your app not existing and your app being down. I’ve scribbled down the following table to compare the use cases for the two:

Prometheus OpenTelemetry
Time series database Instrumentation framework
Monitoring and alerting Creation, collection, processing and export of metrics

Another issue of interoperability is metric names: OTel supports the whole of Unicode, whereas (until version 3.0) Prometheus only supports alphanumerics, _ and -.

A goal of the Prometheus project is to be the best backend for OpenTelemetry. One way to bring the two together is to use versioned reads in Prometheus to access metrics which were renamed from the Prometheus naming style to the OTel one. OTel deltas can be converted to cumulative metrics, but resource attributes (like information about OS, architecture, process IDs or Docker) have taken a few iterations to translate into Prometheus:

  • turn them all into labels, but this led to cardinality issues

  • create a target_info metric which holds all the resource attributes and use PromQL JOINs to access them, but PromQL JOINs are very hard to use and form a barrier for many users

  • let the user configure which resource attributes to turn into labels, but this is then an admin nightmare

User research for the best solution is still ongoing.

Finally, since we are using Parquet for storing time series in the current project at work, my ears pricked up when they mentioned a Parquet storage working group, even if it is still early days.

And that’s it! After that last Prometheus/OTel talk we had to leave the conference to catch the train back to Lancaster. It had been an exhausting few days but I think I learned a lot about Kubernetes. It was also quite eye-opening to see a conference of that scale, drink all the coffee, eat all the cakes and meet lots of people. Maybe I’ll go again next time, and not take 3 months to write up all of my notes.

  1. It always feels weird spelling “authorisation” in the British English way when it’s so frequently abbreviated to “authz”.↩︎

  2. What could possibly go wrong‽‽‽‽↩︎