Youtube video

hello everyone i’m so happy to be here um let’s start with some presentations so my name is martin fuentes i am senior product manager at instagram for kubernetes observability and i am here today with my colleagues eddie

hey i’m cedric i’m a product manager for our distributed tracing and open telemetry in particular nice to meet you all okay so today and we have a bunch of slides to share with you we are going to talk about kubernetes resource management uh what are the different metrics that kubernetes takes into account for a scheduling pods and containers and what are the different scaling approaches and then i will hand it over to my colleague cedric who is going to talk about how you can actually observe kubernetes workloads uh giving an introduction to open telemetry the different ways you can instrument applications without contributing with some examples and then also some like the demo and with this i will just start let’s so as i mentioned let’s go ahead and talk about kubernetes resource management so when we talk about a resource at kubernetes we mainly talk about cpu and memory and those are the most important resources that kubernetes is managed and the ones that are taking into account when a current is scheduled pots or containers in the cluster um so cpu is one of them as i mentioned one cpu unit is equivalent to one physical or virtual cpu that’s the way that a component measure it you can actually allocate a fractional parts of a cpu to a workload and the minimum that you can actually assign a request for a for a container is one mili cpu which is of course the 1000 part of a cpu then for memory memory is measured in bytes and bytes are just bytes it supports a two different suffixes you can use quantity suffixes like beta tera giga mega kilo and so on and as well it also supports power of two equivalents like pb bytes the tb bytes gigabytes and and so on so there’s one other resource that it’s probably less taking into account unless use which is the local ephemeral stretch but for some workers it might be important that you make sure that your container is going to run in a node that actually has this kind of storage and it’s a also a and

enough space also available um so it’s also measured in bytes it’s also supported two different quantity suffixes and and power of two and a one thing that it’s important to remark if you’re actually taking this a kind of resource uh into account that it’s not guaranteed that it will be long-term availability so that’s something important and it’s only about the the ephemeral storage that lives inside the boat

so for every container in your cluster you can actually set up what it’s called request and limits um so the request is the minimal amount of resources either cpu or memory that your a workload will need to run and you can specify that so kubernetes will know uh in which node of the cluster schedule your containers or pods the limits are more or less similar but it says to kubernetes which is the threshold that this workload shouldn’t be exiting so it shouldn’t be consuming more than x amount of cpus or x amount of bytes of memory the cube scheduler which is one of the one part of the kubernetes components will decide in which node the port will run depending on the availability of resources in that node and the requests that were configured for that workload and for that container then the cubelet will reserve at least that amount of resources that were requested in the node to make sure that it’s available for the container to run and also the kubernetes will be the one enforcing that the limits are respected so we can prevent that other containers that are running there will have less resources than they requested or they need to actually run

now this is how a request and limits are actually configured for a container so this is a very simple conflict map with two containers running in a single pod for each of the containers there is a configuration for a memory and cpu requested and at the same time limits for those same applications as you can see here it’s very straightforward and very simple as i mentioned here is um requesting a 64 mb bytes of memory and 250 millicourse to run this a specific container

now what happens with request and limits how do they actually impact your workloads and their containers running there so if a container hits the cpu limit what kubernetes will do is to throttle the container meaning that the application running there will be probably less performance than it was before but it won’t be terminated or depicted so you will have less performance but your application will still run uh at the other hand if a container hits or exists the memory limit it will be actually terminated by kubernetes the pot will die and depending if the pot was managed by an application controller like a deployment stateful set or demonstrate um kubernetes will make so actually the controller will make sure to spin up a replacement for that pot that just died because of memory consumption so the decided state is always respected it’s important to take into account that this process could actually happen like in a loop so you if you have a memory leak in your application it might happen that your application will be dying very often because kubernetes will always make sure that the the memory limit is not surpassed

and uh it’s important also to try to take a look at how your applications are consuming these resources against the request and limits um i try to bring this example to show that uh it’s not only important to allocate as the minimum that your application needs but also trying to not request more than it actually needs because that that resources will be reserved for the application while it will not will be actually using it so here in this chart for example you see for memory consumption the actual usage of the application is below the request that means that there is some space here of a memory that is not really used by the application but it’s still reserved for it because it was requested so it’s really important to make sure that you have also like a visual way to to to see how the the configuration of requests and limits is doing um together with the consumption of those resources by the application

moving forward i was also i wanted to talk about the scaling so there are two different types of scaling at kubernetes you have horizontal scaling that also have two different meanings depending if it’s for the node or for a pod at the node level it actually means adding more nodes to a given cluster so you will have more servers in that cluster to allocate workloads while at the pot level it means that you are adding more running replicas uh to an application for example you have um an application that is taking requests from end users and at certain point you see that the application performance is degrading so you will actually spin up more applica more replicas of that applications to actually allocate more end user requests and in that case it will be horizontally scaling your pot in the case of vertical scaling at the node level it actually means to modify the attribute resources of each node so if you have for example a virtual machine running on a a physical server you can mod if you modify what are the resources available in that virtual machine you’re actually vertically scaling that node and and for a pod it means a playing a bit with request and limit so when you as i mentioned before it’s important to not request more than what your application needs when you are doing that this fine tune of request and limits taking into account the actual consumption of your application you are actually vertically scaling your containers or pots

i would uh also like to do a um have one more slide about uh the horizontal and vertical part of the scaling so focusing on pots here uh because at the end that’s why that that’s what is going to be running in your cluster and the configuration that you have um the possibility to impact there is a a way to automatically scale your containers in kubernetes and it’s called hpa and it stands for a horizontal put out scaling and in that case you can set up a query that will tell to kubernetes starting on which threshold it will start spinning up or down more replicas of that container that you have deployed and this is not possible for any for all the application controllers or workloads but you can do it for deployment stateful sets or set for example in the case of a vertically vertical pod auto scaling uh so there is a mechanism that will allow a kubernetes to dynamically adjust the cpu and memory attributes of your pods so you will actually be um as i mentioned before modifying the requests for that part but it can happen also in an automated way by kubernetes

so now um i also wanted to bring you a summary of how you would look at this information or data in an observability tool uh and and it’s important to mention that uh you need to have like visibility for all the different components of the cluster it’s important that the observability tool that you’re using allows to see also not only the request and limits for specific containers or pots but also a kind of a summary or roll up of this request limits and resource usage at different levels of the cluster you can look at that at the node level by namespace or even at the the whole cluster

with this and probably before i hand it over to my colleague cedric i will just take a look if there are questions

okay so i don’t really see any question from our audience i encourage you to send us any question or doubt that you have we’re here to answer them but looks that they’re not so i’m handing the presentation now to my colleague cedric to take it over

oh i can see so wait just one second said it because um question just popped out um so i have a question from deepankar who asked do we vertically so do we vertical scale using golang so in this case it’s actually um not depending on the technology or the language that you’re using as long as you can you know what are the different resources that your application will use you have a way to actually tell kubernetes which is the query that it has to run and the different thresholds uh to scale up or down vertically your bots uh but you uh it’s not depending on on the uh the language that your application is writing i hope to answer your question with this thank you deepankar

thanks questions are always great let’s talk to the actual observing part or is there more i see one more question do you want to uh answer that

so uh asks during vertical scaling is the container recreated or you know if it if the scaling was done on the fly um so if it’s an auto scaling so if you’re actually setting up kubernetes to scale the cluster in that case it will be actually really great

oh sorry note i am talking about sorry stories you’re talking about vertical out scaling it’s done dynamically so it’s not going to be recreated it just does it on the fly it dynamically modifies the request for you for your workload sorry

all right now it looks to be thanks i think we can catch up with everything let’s uh then go to the uh observing workloads part and uh i think um this part deserves a bit more uh preparation right we are here for observability and um i think we should talk about what what we actually mean when we want to you know establish or facilitate observability and um it’s the same um whether we talk about you know kubernetes whether we talk about a more traditional scale uh scheduler like nomad or um just your plain whole host-based application that you deploy via floppy disks or whatever um so observability is actually inferring the state of a workload by looking at its inputs and outputs right so we want to consume signals from a from a given service and when we talk signals we we usually mean traces metrics and logs and then we want to infer the state of the application we want to go analyze the logs see if there are any errors in them we want to take a look at the metrics see if the numbers are healthy for example you know the processing rate latency is a very good measure and for tracing actually um it gets a bit more interested it’s interesting right um with tracing we usually mean distributed tracing so it’s not only one service involved in a transaction but it’s rather a distributed transactions or you have your bank application on your mobile phone and when you tap you know you you wire some money to a friend um the request will actually go from your mobile it will go into you know some edge service and from the edge service will probably distribute it into some kind of bank management system maybe it even goes to a to a mainframe in the seller um and um yeah these kinds of transactions get increasingly complex and that’s why the distributed part here is very important we want to collect these signals from services and infrastructure that you are running either inside traditional data centers or inside kubernetes um then once you have the signals um and you know your services are emitting these things you want to enable connection on them right so there needs to be some kind of tool that would catch all the traces catch all the metrics catch all the logs it would maybe allow you some post processing to remove personal information from from the data and then you want to store the signals if you wanted to take a deeper look you know five minutes later or the other day you need to have a storage that is capable of storing these um these signals for you um which directly caters into the next part um at some point in time you will want to analyze all the signals with an analytical engine and um all in all the you know the traditionally all that has been very very complex and it was a space that was dominated by some vendors and they did the data collection for you and they would provide the analytics engine and that has shifted a bit and let’s take a look at the next slide

what is already in the in the title of this talk is open telemetry and um so open telemetry is really an open standard that cares about um observability as a whole they care about all the different steps in the process they have a component for collecting the data which is called the collector which receives processes and exports all the signals at your at your disposal basically they provide some instrumentation libraries and these are the things that are in your processes and they you know analyze the better flow in your in your process probably do some instrumentation on the code level so they would be intercepting for example in java via bytecode or whatever and um infer traces logs and metrics for you and what’s interesting is um the project even provides auto instrumentation for some runtimes such as java node.js python.net which basically means removing a lot of manual work of the data collection and just including the um you know these open telemetry in process components into your application including the java agent into your java applications the java 80 will automatically instrument your your workload and you have minimum time to that even with this um this open source option um then the op inflammatory project cares as well about some deployment helpers so yeah you need to deploy all of these components you need to instrument your workloads you need to deploy a collector you need to make sure that the networking and your kubernetes cluster is set up all of that and the community is taking care of that for example there is a helm chart that will deploy the collector for you there is no there is a kubernetes operator that will auto instrument your workload so they will do some transparent modification of your kubernetes workload definitions and automatically inject instrumentation for some runtimes which is a really cool feature and yeah probably one of the best things that this project has established is a shared protocol so without the telemetry when i think about open telemetry um there is this open telemetry line protocol which is available it covers you know the data transmission between the collection components and a collector or a vendor and this is a standard standard thing so if you are not satisfied with your current observability solution you know you can you can you can pack your things and just direct your workloads to report to another vendor without having to reinstrument everything which is cool the project is governed by the cncf um it’s an open process everyone can participate and it’s entirely open on github so if you’re interested um head towards github look for telemetry um tons and tons of great people and discussions there next slide please

so um let’s take a look at the instrumentation what does it actually look like so if you are not in a position where you can or want your workloads to be automatically instrumented you are probably familiar with the case that you need to pull in a vendor library um you need to you know add some code builds um in and around your um your business logic to facilitate um collecting traces for example right we would need to encapsulate all of your business logic into some mechanism that would define hey this is a transaction um take care of it and export it to a observability tool with open temperature and especially with java this is super easy so what i what i brought today is um the snippet of a or a snippet from a dockerfire so you can see it’s basically just putting the plane open jdk 17 and then it is pulling from github the open telemetry java agent um which is hr file and then in the cmd line we are basically incorporating that java file we are basically using it as a java agent we are attaching it to our jvm and what that will do it will automatically automatically modify your code on known code paths such as popular web frameworks um and it will wrap around it do that wrapping for you basically and automatically collect the um the trace signals in that case and you know that’s it there is no further for the work to be done you can enhance or augment your experience here with an sdk but it’s entirely optional right you just put it in um have it work and it will even automatically find its way to the open telemetry collector if the collector is available at the standard port um the connection is made automatically

cool so it’s for so much for java i brought another example this is not jazz it’s a bit more complex so you see that with open telemetry the landscape is not homogeneous it’s more it’s a very diverse community and all the different projects that are at various stages of maturity but um automatic instrumentation is um possible with node.js as well so one option would be to use their operator in kubernetes to automatically do what’s on this slide but since we’re here to learn something um i thought it would make sense we take a look at the snippets so at the left hand side there is the docker file once again we pull the plain note 17 image um we copy some stuff and what you will notice is that in line 13 we are actually requiring a file that is basically prepended to everything that the the application does so we are requiring that tracer.js upfront before starting the application and the contents is on the right hand side we are configuring all the telemetry here we are configuring basically the or specifically the node sdk portion of open telemetry and we wire the otlp trace and metric exporters here you can see in line 10 line 14 they can they can consume an a canonical environment variable which you can set on your workload so that they find their correct door and then all you have to do is you know start the sdk line 24 and you’re done and by means of the instrumentations config in line 21 and the automatic resource detection on 920 you are covered so what this will do is it will automatically start reporting it will automatically recognize its environment it will detect um whether you are running on gcp whether you’re running on aws an lambda function in a fargate container in an azure function or in a google cloudtran container that’s facilitated by means of resource detectors and every signal will be annotated with this information so that you can easily consume it later down the road and the auto instrumentation registration in 921 just make sure that you know standard libraries in the node runtime and even some community libraries are automatically instrumented so when you have an express app and you um you invoke a controller or a route that you have um it will automatically collect spam data for you so collect trades later

cool that’s it and and that’s pretty it instrumentation is done next slide please

so if you want to learn more about instrumentation we have a demo application up on github it’s a github.com standard hotel shop we have some examples for java python node.js even golang a instrumented nginx instrumented page 2 web server and you can check the project out i think and the examples are fairly straightforward forward and by that let’s uh check out if there are any questions what about go laying is ultra instrumentation available great question i think why is the question great go with a compiled language right and it’s it’s inherently hard to um auto instrument these while there are some proprietary options available for auto instrumentation the open telemetry project is not currently at the point where they are investing and to also instrumenting and go applications but that could very well be an enhancement driven by the community thank you thanks for the question

next slide

so remember my first slide that now we need to collect all the data that our workloads are emitting um and as an open telemetry collector we have chosen uh we’ve chosen a specific artifact so instead of using the open telemetry connector which is a project and it’s a specification at the same time um let’s deploy the instantan agent as an open telemetry collector that is our host based agent basically that you would roll out onto your production systems and it would observe anything out of the box and it can ingest all the telemetry for for easy augmentation of our already available automatic instrumentation so um this specific example um takes care of creating a namespace in your kubernetes cluster if it doesn’t exist it’s not an agent and it will then you know deploy that helm chart that is specified it will set some default configuration and then in in the second to last line we said open telemetry.enabled equips true and that’s all you need that facilitates populating our agent and your cluster as an ingress point for telemetry data it’s it’s addressable via a dns name in your cluster and it listens on the standard part so it’s a really transparent process and now combine that with um configuring your workloads we will take a look at that but basically data collection is done you deploy the agent or any other open telemetry collector and collection is taken care of

next slide

yes configuration of kubernetes workloads so as i said um we would be really looking forward towards an easy configuration and in reality all you will probably need is the old underscore exporter underscore otp underscore endpoint environment variable on the process you modify your deployment or pod specification and you just inject this environment variable um make it uh point at the internal dns name for your open telemetry collector which in this case refers to our agent um it could really be any other connector and in addition to that you would set the hotel service name environment variable which is the only required thing in open telemetry everything needs to be recognizable right so it requires us to set the service name here and it’s easiest to be done from the outside with environment configuration and by that our workload configuration is done as well easy next slide now we are getting to the beefy part right so we have taken care of of a lot of things but now we want to analyze on our telemetry data so you would now select the vendor of choice for your observability and needs and you would make sure that your collection mechanism so your open telemetry collector for example would shuffle the data from your premises to the vendor windows premises and um in this example we already have chosen uh we have taken the care of that decision right so we want we we are in standard we want to use our product in that case because we know it well and how to use it and and what value it brings so we thought we um we would take a few minutes on a on a very short demo but basically by deploying the installer agent um we already um we already uh yeah have an analytics backend and we already configured it i see there is a rather large question and i would like to address that in the q a part after after the short demo thanks okay hi uh marcin can you uh let me shut the screen

thank you

so this is enzana you can all see it now this is a dashboard for for a kubernetes cluster you are recognizing it because martin already showed it to you right you see um that we denote all the different object types and kubernetes clusters here and we neatly group them under specific clusters this is our demo environment which we use for customer presentations and webinars so it has a bunch of demo data in here and one particular thing i wanted to show you is the hotel shop in action basically i promised you that there would be a demo project and i think this is a very good um very good example so since in stana is all about giving you full context um throughout analyzing your your observability signals um you can always go from infrastructural elements like this kubernetes cluster to for example trace data or logging data connected and let’s take a look at our analytics section which is here and we say hey dear unbounded analytics feature please give me all the spam data that we that you have for this specific kubernetes cluster and what it will do it will um it will do it right so what we what we have here is all the trace data produced by that specific cluster grouped by a namespace and um you can see that we recognize objects like services so for example the different hotel shop components and we can analyze trace data for this to get a better picture what the hotel shop really is let’s look at our applications perspective section application perspectives is a concept that we wrap around the uh individual services it’s basically a way for you as a customer or a user of instant to kind of segregate your services um into more cohesive units so in this case let’s take a look at the uh hotel trump which is one of the application perspectives that i defined taking a look at configuration first and it’s basically denoted here by means of the physical hosts zone so a ready-made dashboard for you to go analyze on the health of your application but what is my application made of i see something here it says auto okay open timer tree shop yeah okay probably an online shop but what does it consist of um we have a tool that we call the dependency map over here and this is really a fl a way to analyze your application data flow so uh don’t mind that uh okay shop web server is over here it’s just sitting around probably doing nothing um the apache 2 instrumentation is not very mature yet so i assume an issue with the actual instrumentation but take a look at the hotel shop engine x front service by by looking at the name this is the your front proxy it receives calls from a load generator and we can see all the services that it will talk to right um so a user might check out the shipping options for objects in your store right um they might want to change their passwords through the hotel shop user service they want to rate products via the rating service and then the writing service for example it calls into something else and you can see that the tiny pop-up here

this is the writings database and the writing sector base just receives calls from the rating service as it should it’s all observed all inferred out of the box but what we have over here hotel shop card so the card service is talking to the shipping service which makes sense because you want to know shipping options analyzed for the things that you have in your cart and we see that this is a http service and what we can do for every individual service we go through the dashboard and this is once again an opinionated dashboard um and we can go see all the calls that were created by the open telemetry auto instrumentation in that case the shipping service is a is a java application and so all of these out of the transactions that you can see here are automatically created there is no additional polishing from our site at least not that i would remember um so let’s take a look at this one it’s an http post that was created by the auto instrumentation and we natively blend open telemetry data alongside our auto instrumentation that we have on our more or less proprietary side of things so you can mix and match open telemetry and jump and it’s done out and auto instrumentation whenever you like but this set of services is really all about open telemetry so here’s the call graph and now it gets interesting right one of the um one of the measures of application healthy for healthiness for example is um call latency so you want to know how long is my call taking but you don’t only want to look at that very specific call you also want to take a look at all the other other calls that were made to that endpoint so we give you a way to um you know very transparently look at the numbers and see um all the points in your distributed transactions so there’s a this call it enters through the front proxy goes inside the shipping service there is an http call that’s outgoing from the nginx front to the hotel shipping service and then there is some internal stuff happening here there is a controller involved and then there is a card helper involved yeah so i can do a lot of with this but imagine you are a developer or a devops persona and you want to track down a production issue you want to see an issue with calls that happen to a specific service and you can do that by means of checking a box it’s not visible right now let me let me clear out some filters here

so we see that we have erroneous coins in our systems and we would not like to investigate those either because we are on call and we received an issue that is being tracked or you know a customer is asking why the transaction has failed and we see that oh my gosh there are a lot of fair transactions it’s red red means contains errors take a look at that

once again the distributed transaction and we immediately see that the payment service actually failed and we can now infer where the where the distributed transaction actually went wrong so we know that we have a status quo error and then open telemetry there is a specific label that can be applied so there was a status code 500 so something in my applications logic is apparently wrong so i can go now fix it um and i think it’s very straightforward to to do the analysis um over here so that was a tiny glimpse into using open telemetry tracing with minstana we have a lot more in store ranging from user and user monitoring to our own tracing libraries with some more expert knowledge but i would really recommend that you check out our demo project take a look at the instrumentation and enjoy the power of telemetry um and you know make it your own we’ll stop stop sharing the screen and take it care about some questions maybe

so first of all thank you since my team put up the slide thing thank you thanks a lot right there is once again the link to the demo application on github and if you want to play with instagram we have a demo environment accessible to um to you you can access that um from your browser i think you don’t have to really register you you need to leave your email address and you can um you know then play with asana in the play with environment looking at questions consider the small micro services segment below

there is a question about an slo bridge as at ingress node a and at the same time we want to find anomaly alerts in containers b5 and b8

is there a way we can quickly plot a flowchart using the traces or any mechanism to find the issues quickly so we can find the root cause the goal is to to reduce the mttr minimum time to respond and maintain the healthy slow civil service level objective and use the error budget effectively whoa dude you’re so i i suppose you are advanced in your in your devops journey so you are playing around with um slows and slis which is great by the way we can support you with that um the question is can you plot a flow chart using traces to find the issues um yes so it depends on your analytics engine in theory your distributed tracing provider or the telemetry and the instrumentation does it as well they will you know supply you with a graph of a call they will supply you with all of the history um all of the services that a call traveled through and you have it at your disposal right you have a trace id you have a span id for the individual um individual sections and sub-components and you can really use that too to analyze more slos is a bit more in the hard analytics topics where you need an analytics engine that takes care of analyzing data over time keeping state it’s not a trivial thing to do um so our product has capabilities to do so but since we are talking about open telemetry which is not yet at the analytic stage i would say yes you can do that you just need a vendor that supports it

and then the follow-up question from uh we dip can we calculate slots with open temperature yeah for sure so it collects tracing metrics data log data and whatever your objectives are made of um you can of course use open telemetry for that right so you would really be dependent on your vendor or your data analytics engine if you want to host it yourself for example with a prompt stack or um you know some some grafana product for example um but very popular choices um there are mechanisms in these analytics engines um but it’s really up to to you on your tool to settle for a best practices approaching and there’s nothing that open telemetry will provide as a best practice it’s up to you to define your goals can the tool monitor the flow between the clusters interesting question can you kill me on my screen

because the answer answer is yes i will show you a very cool feature of winston that i that i failed to highlight and this is really a pity it’s my bed it’s my bed so um the first things first open telemetry does not care if your distribut if your transactions just happen on a local host but they can be distributed by nature they can go beyond a single cluster they can go to your payment provider they can go through your cloud edge provider for example if they support tracing um for example you know cloudflare has an option to do that and it’s really about the data collection if you can get the data sure you have this distributed transaction and it really doesn’t care where it’s running is it run starting in a kubernetes cluster ending in a lambda function traveling through a cloud run container and then um maybe at some point also going through a mainframe that’s perfectly possible but one thing what tool does is we do infrastructure correlation right i i set it briefly but for every span so for every sub interaction that we denote here we can pinpoint to the specific process with with its physical context here’s a kubernetes a linux machine it’s hosting a kubernetes cluster um there is a docker container involved kubernetes port and i can directly jump to that specific python process and see its metrics i can directly judge the health and if you are talking about like really really taking the street the distributed aspect beyond what i just said if you want to make infrastructure correlation you will need some additional help but it’s perfectly possible it’s actually supported it’s it’s the reason why distributed tracing exists

okay yeah that’s all for questions

any any other questions

i just see that we have five minutes left

no further questions great that does either either mean it was too much information or you’re all busy googling of telemetry right now which you should

our positioning against dynatrain okay now we are getting to the uh to the hard questions so since we are not focusing on competitive positioning here i would rather like to check the discussion offline so my contact infos are available if you want to have that discussion hit me up

all right well thank you so much to martine and cedric for their time today and thank you to all the participants who joined us as a reminder this recording will be on the linux foundation youtube page later today and we hope you’re able to join us for future webinars have a wonderful day thank you everyone have a wonderful day thanks for joining