tstorage (Time series embedded db)

Mon, 07 Oct 2024 00:00:00 +0000

Project Info

Embedded time series database for usage in a Go application.

Example

Example on how to use the tstorage project can be found here

Internal Document

Google doc diagram and explanation can be found inside here.

The document contains the following information:

Structure of tstorage.memoryPartition which is used as container to collect metric information in memory

Breakdown of data stored inside tstorage.memoryPartition in terms of metrics.

Diagram outlining the structure of tstorage.storage showing the linked list information of all parititions (memory and disk).

Steps breakdown of when a new partition is created.

Splitting of reading partition file from disk into memory mapped and meta.json into meta structure.

Internal representation of the Gorilla compression

Go Concurrency Notes

Fri, 19 Apr 2024 00:00:00 +0000

Restricting Writers

Scenario

Building library that perform expensive I/O operation (eg: writing to a file). The library uses a writer code to write the data and it limits the number of writers to the number of CPU cores. This will safeguard the library write operation to work correctly and consistently in the event if the caller is calling the writer in an uncontrollable manner. Imagine an app that calls the writer in 100 goroutines in the hope it speeds up the write operation, which in reality does not happen as I/O operation are bound to the speed of the I/O peripherals.

Solution

The caller calls the writer function in a normal way. This will works normal in a sequential manner, the number of channels does not make any difference. All 10 calling will be processed.

package mainimport ("log""time")type storage struct {workersLimitCh chan struct{}}func (s *storage) InsertRows(i int) error {insert := func() error {j := idefer func() { <-s.workersLimitCh }()log.Println("Sleeping...", j)time.Sleep(2 * time.Second)return nil}log.Println("Selecting...")select {case s.workersLimitCh <- struct{}{}:log.Println("insert()....")return insert()default:log.Println("defaulting...")}return nil}func main() {s := storage{}s.workersLimitCh = make(chan struct{}, 10)for i := 0; i <= 10; i++ {s.InsertRows(i)}for {}}

The caller spins up goroutine to call the writer function. Even when the client spins up 10 goroutine to call the function, the function will limit it to only the number of channels available (which is 4). The rest of the 6 goroutine will be ignored.

package mainimport ("log""time")type storage struct {workersLimitCh chan struct{}}func (s *storage) InsertRows(i int) error {insert := func() error {j := idefer func() { <-s.workersLimitCh }()log.Println("Sleeping...", j)time.Sleep(2 * time.Second)log.Println("Done Sleeping...", j)return nil}log.Println("Selecting...")select {case s.workersLimitCh <- struct{}{}:log.Println("insert()....")return insert()default:log.Println("defaulting...")}return nil}func main() {s := storage{}s.workersLimitCh = make(chan struct{}, 4)for i := 0; i <= 10; i++ {go func(i int) {s.InsertRows(i)}(i)}for {}}

In-Depth with Ollama

Fri, 19 Apr 2024 00:00:00 +0000

Ollama

The Ollama project is a Go project that has gained a lot of traction with 52,000 stars and forked more than 3600 times. The project can be used as a standalone application to interact with different large language models.. Not only that but the project makes it easy to interface for using the LLM functionality (eg: chat) by exposing REST API.

Architecture

The diagram shows the high level architecture between the application (ollama and llama) with the large language model. The ollama project works like a thin wrapper to the llama project which is responsible for interacting with the model.

We will make reference in article to both llama and ollama as they both serve different purposes even working together as a single unit.

llama.cpp or llama

In Figure-1 we can see that the ollama (Go) is run as a normal process. The llama.cpp will be run as a child process of ollama when a request comes in to load the model. The interaction with llama.cpp (https://github.com/ggerganov/llama.cpp) project is via http. The llama.cpp project is an inference project for LLAMA (and other models).

An inference project is simply a project that involves working with the Large Language Model (LLM). Each model in the LLM has its own way of interacting with systems. So, in an inference project, we use the LLM to do different tasks, like understanding text or generating new content. It’s like using a tool for specific jobs, but each tool works a bit differently.

The interaction that is happening internally inside llama.cpp for example, is not by calling functions to a large language model, but rather through a series of calculations that are fed into memory to be calculated. These calculations can nest deep in several layers of calculation.

To get a better visualization of how models works here is a very good interactive website that walked through the input/output parameters performed internally by llm https://bbycroft.net/llm

ollama or llama.go

The llama.go project simplifies working with Large Language Models (LLMs) by offering the following features:

Local Model Execution: Easily download and run LLM models on your computer’s CPU or GPU. The project supports a variety of models tailored to different tasks.
REST API Access: Access the models through a user-friendly REST API, making it simple to integrate them into your applications. This allows for features like chat functionality and natural language processing.
Compatibility Across Systems: llama.go works seamlessly across different operating systems and hardware architectures, ensuring compatibility regardless of your setup.
Mac Applications: Specific applications are available for macOS users, providing a local interface to interact with LLM models. These applications offer similar features to ChatGPT but for local use on Mac devices.

With llama.go, users can effortlessly harness the capabilities of LLM models for various tasks, with flexibility, ease of use, and compatibility in mind.

Complete list of the models can be found in this link https://github.com/ollama/ollama?tab=readme-ov-file#model-library

Source Layout

The following main branch shows the directories of the ollama source code

We will take a look at some of the directories to explain what kind of source code it host:

Directory	Description
app	Build app for Windows platform
auth	Supporting package for authorization key
cmd	CLI handlers for the different options available by ollama
convert	Conversion function to create model based on Gemma and Mistral
gpu	GPU interfaces for different graphic cards and operating system
integration	Integration tests
llm	Native integration layer between `llama.cpp` and `ollama`. This directory contains the crux of the logic that llama provides by interacting with the `llama.cpp` project. The `llama.cpp` source code lives inside this directory as can be seen in the screenshot below
macapp	Electron based app for Mac platform
openai	Provides middleware for partial compatibility with the OpenAI REST API
parser	HTTP handler and functions for creating model
scripts	Different build scripts for operating systems
server	Contains server related function relevant to models

Building Source

To build the source code we will need to use the latest Go compiler, in this article we are using 1.22.1. The build and packaging process is done by executing the following command in the root directory of the project:

go generate ./...

The generate command perform the following steps:

Execute operating system specific script. This article uses Linux OS so it will use the gen_linux.sh script which is specified inside generate_linux.go source:
```
package generate//go:generate bash ./gen_linux.sh
```
The bash script will checkout llama.cpp submodule from https://github.com/ggerganov/llama.cpp.git into llm/llama.cpp directory as shown in the screenshot
Once successfully checked out the next step will be to perform file patching using the patch file inside ollama/llm/patches directory.

Following are the log snippets from the output of running go generate command:

+ set -o pipefail+ echo 'Starting linux generate script'Starting linux generate script...-- Build files have been written to: /home/nanik/Downloads/GoProjects/ollama/llm/build/linux/x86_64_static...-- Build files have been written to: /home/nanik/Downloads/GoProjects/ollama/llm/build/linux/x86_64/cpu...[100%] Built target ollama_llama_server...+ gzip -n --best -f ../build/linux/x86_64/cpu/bin/ollama_llama_server...+ gzip -n --best -f ../build/linux/x86_64/cpu_avx/bin/ollama_llama_server...+ gzip -n --best -f ../build/linux/x86_64/cpu_avx2/bin/ollama_llama_server...+ cd ../llama.cpp/+ git checkout CMakeLists.txtUpdated 1 path from the index++ cd ../build/linux/x86_64/cpu_avx2/..++ echo cpu cpu_avx cpu_avx2+ echo 'go generate completed.  LLM runners: cpu cpu_avx cpu_avx2'go generate completed.  LLM runners: cpu cpu_avx cpu_avx2

On completion the build directory will look like the following on a Linux machine

The next command to run is now to build the Go binary using the following command from root directory:

go build .

The build command will produce an executable file called ollama that also contains the .gz files produced earlier by the go generate command which contains the llama.cpp project.

Running Ollama

The ollama project runs as an http server that accepts requests for a number of things related to the models. To get started open a terminal window and run it using the following command:

ollama serve

The application will start up printing the following log in the console:

time=2024-04-08T23:51:13.450+10:00 level=INFO source=images.go:793 msg="total blobs: 6"time=2024-04-08T23:51:13.451+10:00 level=INFO source=images.go:800 msg="total unused blobs removed: 0"[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env:export GIN_MODE=release - using code:gin.SetMode(gin.ReleaseMode)[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.PullModelHandler (5 handlers)…[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)…time=2024-04-08T23:51:13.480+10:00 level=WARN source=amd_linux.go:367 msg="amdgpu detected, but no compatible rocm library found.  Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install"time=2024-04-08T23:51:13.480+10:00 level=WARN source=amd_linux.go:99 msg="unable to verify rocm library, will use cpu: no suitable rocm found, falling back to CPU"time=2024-04-08T23:51:13.480+10:00 level=INFO source=routes.go:1144 msg="no GPU detected"

The server is now ready to accept requests. The same ollama application is also used to communicate with the server. Open another terminal window and run the following command:

ollama run llama2

The command will communicate with the server and download the llama2 model to be run locally, when the application successfully download and initialize the model it will give you a prompt that you can type to asked any question as shown below:

The ollama application provide several commands that you can use shown below:

Commands	Description
serve	Start ollama
create	Create a model from a Modelfile
show	Show information for a model
run	Run a model
pull	Pull a model from a registry
push	Push a model to a registry
list	List models
cp	Copy a model
rm	Remove a model

Packaging

As mentioned previously the ollama binary file is packaged with the llama.cpp binary. The build process embeds these binary files into the final binary ollama file. The following screenshot shows the different generated files for an AMD processor with AVX support.

| | | ||———-|———-| – || | | || | | |

The above screenshot shows 3 different .gz files generated for normal CPU, AVX and AVX2, and all of them are named ollama_llama_server.gz. These files are packaged together into the final ollama binary file. This is done by using the embed standard library as shown in the code snippet below.

package llmimport "embed"//go:embed build/linux/*/*/bin/*var libEmbed embed.FS

The go:embed specifies the files that will be packaged which in this case is under build/ directory.

Ollama Internals

As explained previously the ollama application is run as an http server that accepts request, the following tables shows the different endpoints that are available:

HTTP Method	Url
POST	/api/pull
POST	/api/generate
POST	/api/chat
POST	/api/embeddings
POST	/api/create
POST	/api/push
POST	/api/copy
DELETE	/api/delete
POST	/api/show
POST	/api/blobs/:digest
HEAD	/api/blobs/:digest
POST	/v1/chat/completions
GET	/
GET	/api/tags
GET	/api/version
HEAD	/
HEAD	/api/tags
HEAD	/api/version

The commands that are available when running ollama use the above url endpoints, for example: running ollama run llama2 will call the the /api/pull endpoint to download the model and then it uses the /api/chat to accept chat requests and respond to it.

With the availability of the different endpoints, ollama gives the flexibility to develop tools of our own using simple tools like cURL or any programming language.

Debugging

Ollama provides a debugging flag that can be turned on when running it. These flags provide verbose log information on certain internal parts of the application. The following table outlined some of the available flags:

Flag	Value	Description
OLLAMA_DEBUG	true	Provide debug log information for ollama application
LLAMA_TRACE	1	Provide debug log information for llama
GGML_DEBUG	1

For example if we want to debug some issues in ollama we can turn on the debug flag by using the following command:

OLLAMA_DEBUG=true ollama serve

You will get more debug log information as shown below:

...time=2024-04-09T00:06:25.637+10:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gztime=2024-04-09T00:06:25.637+10:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gztime=2024-04-09T00:06:25.637+10:00 level=DEBUG source=payload.go:160 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gztime=2024-04-09T00:06:25.658+10:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3078763984/runners/cputime=2024-04-09T00:06:25.658+10:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3078763984/runners/cpu_avxtime=2024-04-09T00:06:25.658+10:00 level=DEBUG source=payload.go:68 msg="availableServers : found" file=/tmp/ollama3078763984/runners/cpu_avx2time=2024-04-09T00:06:25.658+10:00 level=INFO source=payload.go:41 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2]"time=2024-04-09T00:06:25.658+10:00 level=DEBUG source=payload.go:42 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"time=2024-04-09T00:06:25.658+10:00 level=INFO source=gpu.go:121 msg="Detecting GPU type"time=2024-04-09T00:06:25.658+10:00 level=INFO source=gpu.go:268 msg="Searching for GPU management library libcudart.so*"time=2024-04-09T00:06:25.658+10:00 level=DEBUG source=gpu.go:286 msg="gpu management search paths: [/tmp/ollama3078763984/runners/cuda*/libcudart.so* … /home/nanik/Downloads/GoProjects/ollama/libcudart.so**]"time=2024-04-09T00:06:25.667+10:00 level=INFO source=gpu.go:314 msg="Discovered GPU libraries: []"...

To enable more verbose logging from the llama project we need to pass in the following flag before running the go generate command to build the native application:

export CGO_CFLAGS="-g"

Must be wondering how does the log looks like when we use the -g above, below can see the log snippets when the llama application is processing a chat request:

time=2024-04-09T08:34:19.787+10:00 level=DEBUG source=server.go:432 msg="llama runner started in 24.340054 seconds"time=2024-04-09T08:34:19.789+10:00 level=DEBUG source=prompt.go:172 msg="prompt now fits in context window" required=1 window=2048[GIN] 2024/04/09 - 08:34:19 | 200 | 24.578298317s |       127.0.0.1 | POST     "/api/chat"{"function":"get_new_id","level":"VERB","line":254,"msg":"new task id","new_id":7,"tid":"133124421846592","timestamp":1712615664}{"function":"add_waiting_task_id","level":"VERB","line":397,"msg":"waiting for task id","task_id":7,"tid":"133124421846592","timestamp":1712615664}{"function":"start_loop","level":"VERB","line":302,"msg":"new task may arrive","tid":"133129545471872","timestamp":1712615664}{"function":"start_loop","level":"VERB","line":314,"msg":"callback_new_task","task_id":7,"tid":"133129545471872","timestamp":1712615664}{"function":"process_single_task","level":"INFO","line":1502,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"task_id":7,"tid":"133129545471872","timestamp":1712615664}{"function":"process_single_task","level":"VERB","line":1507,"msg":"slot data","n_idle_slots":1,"n_processing_slots":0,"slots":[{"dynatemp_exponent":1.0,"dynatemp_range":0.0,"frequency_penalty":0.0,"grammar":"","id":0,"ignore_eos":false,"logit_bias":[],"min_keep":0,"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"/home/nanik/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246","n_ctx":2048,"n_keep":0,"n_predict":-1,"n_probs":0,"next_token":{"has_next_token":true,"n_remain":-1,"num_tokens_predicted":0,"stopped_eos":false,"stopped_limit":false,"stopped_word":false,"stopping_word":""},"penalize_nl":false,"penalty_prompt_tokens":[],"presence_penalty":0.0,"prompt":null,"repeat_last_n":64,"repeat_penalty":1.0,"samplers":["top_k","tfs_z","typical_p","top_p","min_p","temperature"],"seed":4294967295,"state":0,"stop":[],"stream":true,"task_id":-1,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false}],"task_id":7,"tid":"133129545471872","timestamp":1712615664}...{"function":"log_server_request","level":"INFO","line":2730,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":37158,"status":200,"tid":"133124421846592","timestamp":1712615664}{"function":"log_server_request","level":"VERB","line":2739,"msg":"request","request":"","response":"{\"slots_idle\":1,\"slots_processing\":0,\"status\":\"ok\"}","tid":"133124421846592","timestamp":1712615664}{"function":"get_new_id","level":"VERB","line":254,"msg":"new task id","new_id":8,"tid":"133124421846592","timestamp":1712615664}{"function":"add_waiting_task_id","level":"VERB","line":397,"msg":"waiting for task id","task_id":8,"tid":"133124421846592","timestamp":1712615664}...{"function":"start_loop","level":"VERB","line":317,"msg":"update_multitasks","tid":"133129545471872","timestamp":1712615664}{"function":"start_loop","level":"VERB","line":336,"msg":"callback_run_slots","tid":"133129545471872","timestamp":1712615664}{"function":"start_loop","level":"VERB","line":339,"msg":"wait for new task","tid":"133129545471872","timestamp":1712615664}{"function":"log_server_request","level":"INFO","line":2730,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":37158,"status":200,"tid":"133124421846592","timestamp":1712615664}...

The log looks like gibberish as there are a lot of messages, but these messages are very useful when troubleshooting or learning how the interaction with the models works.

Ollama to Llama

So far we have explored the ollama source code and looked at compiling and running it locally. In this section we will go deeper on how it works internally.

We know now that the way the application is architected is to communicate via http to the llama project that is run as a child process as shown in the diagram below

So what does the actual process look like ?, first let’s use the command ps aux | grep -i ollama to get the PID of the ollama running in the local machine which is shown in the screenshot below. The PID shown is 3180960

Using the command pstree -a 3180960 we will be able to see the tree structure of the process as shown below

The diagram shows the ollama spawning a child process executing the llama project which is packaged with the file name ollama_llama_server. The child process is executed by passing several command line parameters used to initialize the LLM models, logging, etc.

Fascinating to observe how the Llama application spawns multiple threads upon receiving a chat request. The threads are used for managing the processing of data to and from the model, as shown in the screenshot provided.

How do we know which port llama is running on ?, this is shown in the log when you access the model by running _ollama run _.

Below is a sample log on how it looks like on my local machine where it shows that the llama server is running on port 4767.

...{"function":"initialize","level":"INFO","line":448,"msg":"initializing slots","n_slots":1,"tid":"137254119753600","timestamp":1713529242}{"function":"initialize","level":"INFO","line":457,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"137254119753600","timestamp":1713529242}{"function":"main","level":"INFO","line":3064,"msg":"model loaded","tid":"137254119753600","timestamp":1713529242}{"function":"main","hostname":"127.0.0.1","level":"INFO","line":3267,"msg":"HTTP server listening","n_threads_http":"15","port":"4767","tid":"137254119753600","timestamp":1713529242}...

Llama Endpoints

When the llama application is launched, it operates as a server capable of handling standard HTTP requests. In this scenario, it awaits connections from ollama. Following are the endpoints that are exposed by llama:

HTTP Method	Url	Description
GET	/health	Check health status of `llama`
GET	/slots	Obtain information about the slots initialized internally
GET	/metrics	Used to get metric information about processed tokens
GET	/completion	Used for chat functionality
GET	/tokenize	Used to tokenize the information input by the user
GET	/detokenize
GET	/embdedding	Used for RAG based applications

Slots

Internally llama processes token by the number of allocated slots available, for eg - if there is 1 slot allocated then llama will process 1 token at a time.

The diagram below shows the high level flow when the /completions endpoint is called, this endpoint is used for chat. The slots are initialized when the endpoint is called, and once it is initialized it will go through the Decoding Process.

The Decoding Process is the brain of the operation performed inside the llama process. Basically what it does is it prepares the model which is now in memory to perform calculation in different layers until it gets an output.

Once output is obtained it will send the output to the caller, in this case the caller of the /completions endpoint.The Decoding Process will keep on going until there are no more output returned from the model or it is said that the process has stopped processing token, which means it has reached the end.

Accessing the llama endpoint using the following cURL command:

curl http://localhost:<llama_port>/slots

Will show you information about the slot information:

[  {    "dynatemp_exponent": 1,    "dynatemp_range": 0,    "frequency_penalty": 0,    "grammar": "",    "id": 0,    "ignore_eos": false,    "logit_bias": [],    "min_keep": 0,    "min_p": 0.05000000074505806,    "mirostat": 0,    "mirostat_eta": 0.10000000149011612,    "mirostat_tau": 5,    "model": "/home/nanik/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246",    "n_ctx": 2048,    "n_keep": 0,    "n_predict": -1,    "n_probs": 0,    "next_token": {      "has_next_token": true,      "n_remain": -1,      "num_tokens_predicted": 0,      "stopped_eos": false,      "stopped_limit": false,      "stopped_word": false,      "stopping_word": ""    },    "penalize_nl": false,    "penalty_prompt_tokens": [],    "presence_penalty": 0,    "prompt": null,    "repeat_last_n": 64,    "repeat_penalty": 1,    "samplers": [      "top_k",      "tfs_z",      "typical_p",      "top_p",      "min_p",      "temperature"    ],    "seed": 4294967295,    "state": 0,    "stop": [],    "stream": true,    "task_id": -1,    "temperature": 0.800000011920929,    "tfs_z": 1,    "top_k": 40,    "top_p": 0.949999988079071,    "typical_p": 1,    "use_penalty_prompt_tokens": false  }]

Model

The diagram shows how the model looks like internally as it contains different metadata information describing the model such as license, weight, tensors, etc.

In simple terms Tensors are mathematical formulas that are used to process input to derive the required output. Every model have layers of this tensors built-in internally that are used to perform complex calculation. Data that need to be processed are converted into tokens which will be feed into the model to extract the relevant value which are again re-converted to text.

In the example of the model used by ollama the llama2 model have in total 291 tensors. The following ollama example log shows the tensor information read from llama2 model

...{"function":"load_model","level":"INFO","line":403,"msg":"llama_init_from_gpt_params","tid":"136660906297216","timestamp":1713528385}llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /home/nanik/.ollama/models/blobs/sha256-8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))llama_model_loader: - tensor    0, split  0:                token_embd.weight q4_0     [  4096, 32000,     1,     1 ]llama_model_loader: - tensor    1, split  0:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor    2, split  0:            blk.0.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]...llama_model_loader: - tensor  164, split  0:            blk.4.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  165, split  0:            blk.4.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  166, split  0:              blk.4.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  167, split  0:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  168, split  0:              blk.4.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  169, split  0:         blk.4.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  170, split  0:              blk.4.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  171, split  0:              blk.4.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  172, split  0:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  173, split  0:            blk.5.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  174, split  0:            blk.5.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  175, split  0:              blk.5.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  176, split  0:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  177, split  0:              blk.5.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  178, split  0:         blk.5.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  179, split  0:              blk.5.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  180, split  0:              blk.5.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  181, split  0:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  182, split  0:            blk.6.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  183, split  0:            blk.6.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  184, split  0:              blk.6.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  185, split  0:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  186, split  0:              blk.6.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  187, split  0:         blk.6.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  188, split  0:              blk.6.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  189, split  0:              blk.6.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  190, split  0:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  191, split  0:            blk.7.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  192, split  0:            blk.7.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  193, split  0:              blk.7.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  194, split  0:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  195, split  0:              blk.7.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  196, split  0:         blk.7.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  197, split  0:              blk.7.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  198, split  0:              blk.7.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  199, split  0:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  200, split  0:            blk.8.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  201, split  0:            blk.8.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  202, split  0:              blk.8.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  203, split  0:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  204, split  0:              blk.8.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  205, split  0:         blk.8.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  206, split  0:              blk.8.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  207, split  0:              blk.8.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  208, split  0:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  209, split  0:            blk.9.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  210, split  0:            blk.9.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  211, split  0:              blk.9.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  212, split  0:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  213, split  0:              blk.9.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  214, split  0:         blk.9.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  215, split  0:              blk.9.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  216, split  0:              blk.9.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  217, split  0:                    output.weight q6_K     [  4096, 32000,     1,     1 ]llama_model_loader: - tensor  218, split  0:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  219, split  0:           blk.24.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  220, split  0:           blk.24.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  221, split  0:             blk.24.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  222, split  0:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  223, split  0:             blk.24.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  224, split  0:        blk.24.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  225, split  0:             blk.24.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  226, split  0:             blk.24.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  227, split  0:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  228, split  0:           blk.25.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  229, split  0:           blk.25.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  230, split  0:             blk.25.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  231, split  0:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  232, split  0:             blk.25.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  233, split  0:        blk.25.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  234, split  0:             blk.25.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  235, split  0:             blk.25.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  236, split  0:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  237, split  0:           blk.26.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  238, split  0:           blk.26.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  239, split  0:             blk.26.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  240, split  0:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  241, split  0:             blk.26.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  242, split  0:        blk.26.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  243, split  0:             blk.26.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  244, split  0:             blk.26.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  245, split  0:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  246, split  0:           blk.27.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]...llama_model_loader: - tensor  269, split  0:        blk.29.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  270, split  0:             blk.29.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  271, split  0:             blk.29.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  272, split  0:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  273, split  0:           blk.30.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  274, split  0:           blk.30.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  275, split  0:             blk.30.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  276, split  0:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  277, split  0:             blk.30.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  278, split  0:        blk.30.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  279, split  0:             blk.30.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  280, split  0:             blk.30.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  281, split  0:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  282, split  0:           blk.31.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]llama_model_loader: - tensor  283, split  0:           blk.31.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  284, split  0:             blk.31.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]llama_model_loader: - tensor  285, split  0:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: - tensor  286, split  0:             blk.31.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  287, split  0:        blk.31.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  288, split  0:             blk.31.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  289, split  0:             blk.31.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]llama_model_loader: - tensor  290, split  0:               output_norm.weight f32      [  4096,     1,     1,     1 ]llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output....

Models that are compatible with llama.cpp must be in GGUF format and can be searched using the following link https://huggingface.co/models?sort=trending&search=gguf

GGUF

We know now that model that can be used by llama have to be in GGUF format, so what is it ?. According to the author https://github.com/ggerganov/ggml/blob/master/docs/gguf.md

GGUF is a file format for storing models for inference with GGML and executors based on GGML. GGUF is a binary format that is designed for fast loading and saving of models, and for ease of reading. Models are traditionally developed using PyTorch or another framework, and then converted to GGUF for use in GGML.`

The following website provide an in-depth explanation about the format and how the model works internally https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/

Concurrent map in Go

Thu, 11 Mar 2021 00:00:00 +0000

From golang faq

Why are map operations not defined to be atomic?

After long discussion it was decided that the typical use of maps did not require safe access from multiple goroutines, and in those cases where it did, the map was probably part of some larger data structure or computation that was already synchronized. Therefore requiring that all map operations grab a mutex would slow down most programs and add safety to few. This was not an easy decision, however, since it means uncontrolled map access can crash the program.

The language does not preclude atomic map updates. When required, such as when hosting an untrusted program, the implementation could interlock map access.

Map access is unsafe only when updates are occurring. As long as all goroutines are only reading—looking up elements in the map, including iterating through it using a for range loop—and not changing the map by assigning to elements or doing deletions, it is safe for them to access the map concurrently without synchronization.

As an aid to correct map use, some implementations of the language contain a special check that automatically reports at run time when a map is modified unsafely by concurrent execution.

Other languages such as Java provides map implementation that supports concurrent read/write.

Time will come when we will need to design apps that requires map to be read in different part of the app requiring concurrent read and write. Conceptually to implement a map that supports concurrent read/write is done by introducing some sort of synchronisation. This article goes through few ways on what to use if an app require to have concurrent read/write map functionality.

concurrent-map

concurrent-map is quite an old project last updated 2months ago. In terms of implementation it still works.

Internally the framework is using struct called ConcurrentMapShared

type ConcurrentMapShared struct {items        map[string]interface{}sync.RWMutex // Read Write mutex, guards access to internal map.}

Internally the library creates 32 different buckets of the ConcurrentMapShared type and use sharding when it does reading/writing values.

Using the library is very simple in app as shown below.

func main() {m := New()elephant := Animal{"elephant"}monkey := Animal{"monkey"}m.Set("elephant", elephant)m.Set("monkey", monkey)}

ristretto

Another option is to use cache library. Most cache libraries stores data in a map as follows

map[string]interface{}

For this article we going to take a look at Ristretto project. According to the project

Ristretto is a fast, concurrent cache library built with a focus on performance and correctness.

The library is not built specifically to address map based concurrency issue. However, if we look at the implementation it can pretty much do what a concurrent map can.

The function calls are not exactly the same like a map operation, nevertheless it is quite close.

package mainimport ("fmt""github.com/dgraph-io/ristretto""os""time")func main() {c, err := ristretto.NewCache(&ristretto.Config{NumCounters:        10,MaxCost:            1000,BufferItems:        64,IgnoreInternalCost: true,})if err != nil {os.Exit(1)}valid := c.Set("test", "value", 1000)if valid {fmt.Println("Stored successfully")}time.Sleep(100 * time.Millisecond)s, found := c.Get("test")if !found {fmt.Println("Get not successful")} else {str := fmt.Sprintf("%v", s)fmt.Println(str)}}

Cache library are designed to be asynchronous in nature as there is no hurry for the data to be made available directly. Unlike map when you store something you will get it back directly. Hence, the time.sleep(..) in the code. If the time.sleep code is removed the app would never find the key.

This kind of cache library are very focused on performance and are memory efficient which helped if the app is going to store big sets of data. As a bonus concurrency is built into the library which is something the app does not have to worry about.

Good reading material about the project including performance comparison with other open source cache libraries.

go-workers

Tue, 09 Mar 2021 00:00:00 +0000

go-workers is an open source project that helped developers in making it easier to build concurrency into their app. The project is suitable for small to medium project. The project removes developers from the headache of worrying about channels, etc.

Developers just have to define the workers that it need to do work asynchronously and different workers can work together passing data in and out between them. We defined our worker as follows:

type Worker struct {}

and we need to have the implementation of Work like follows:

func (wo *Worker) Work(w *worker.Worker, in interface{}) error {....}

Once we have implementation ready we can execute it using the library as follows:

workerOne := worker.NewWorker(ctx,  &SingleWorker{}, 1).Work()

The worker object now ready to receive messages by using the following command:

workerOne.send(<can_send_anything>)

The following example shows how 2 workers processing the same data independently.

package mainimport ("context""fmt"worker "github.com/catmullet/go-workers""os""path/filepath""time")func main() {ctx := context.Background()t := time.Now()f, err := os.Create(filepath.Join(os.TempDir(), "output.txt"))if err != nil {os.Exit(1)}workerOne := worker.NewWorker(ctx, NewWorker(), 1).SetWriterOut(f).Work()workerTwo := worker.NewWorker(ctx, NewTestWorkerObject(), 10)workerTwo.InFrom(workerOne).Work()workerOne.SetWriterOut(f).Work()for i := 0; i <= 2000; i++ {workerOne.Send(i)}defer f.Close()workerOne.Close()totalTime := time.Since(t).Milliseconds()fmt.Printf("total time %dms\n", totalTime)}type Worker struct {}type NewTestWorker struct {}func NewWorker() *Worker {return &Worker{}}func NewTestWorkerObject() *NewTestWorker {return &NewTestWorker{}}func (wo *NewTestWorker) Work(w *worker.Worker, in interface{}) error {i := in.(int)w.Out(i)return nil}func (wo *Worker) Work(w *worker.Worker, in interface{}) error {i := in.(int)w.Println(i)return nil}

The following is an example on using the library to GET data from cat-fact every 5 seconds.

package mainimport ("context"worker "github.com/catmullet/go-workers""io/ioutil""log""net/http""os""os/signal""syscall""time")func main() {ctx := context.Background()workerOne := worker.NewWorker(ctx, NewSingleWorker(), 1).SetWriterOut(os.Stdout).Work()c := make(chan os.Signal)signal.Notify(c, os.Interrupt, syscall.SIGTERM)go func() {<-cworkerOne.Close()os.Exit(0)}()for {workerOne.Send(nil)time.Sleep(5 * time.Second)}}type SingleWorker struct {}func NewSingleWorker() *SingleWorker {return &SingleWorker{}}func (wo *SingleWorker) Work(w *worker.Worker, in interface{}) error {download()return nil}func download() {resp, err := http.Get("https://cat-fact.herokuapp.com/facts")if err != nil {log.Fatalln(err)}//We Read the response body on the line below.body, err := ioutil.ReadAll(resp.Body)if err != nil {log.Fatalln(err)}//Convert the body to type stringsb := string(body)log.Printf(sb)}

Dissecting minict

Mon, 04 Jan 2021 00:00:00 +0000

minict is a small interesting project that shows how containers works, here is the link to the github.

You can think of the project as Docker v0.0.1 that provide very simple functionality to play around with containers.

Here are some of the functionality it provides out-of-the box

COMMANDS:   pull             Pull an image from Dockerhub or a different container registry.   run              Run a new container.   start            Start an existing container that was exited.   rm               Remove an existing container.   list-containers  List all current containers.   list-images      List all images.   help, h          Shows a list of commands or help for one command

This article will tear down the project to look at how it works internally and to see how it is able to provide the container functionality using other open source projects.

pull

This command is to pull container image from docker.io registry and store it locally. This command should be executed first before any other command.

minict pull --image ubuntu:20.04

Following is the log output (to enable more verbose logging look under the Hacking section)

DEBU[0000] reference rewritten from 'docker.io/library/ubuntu:20.04' to 'docker.io/library/ubuntu:20.04' DEBU[0000] Trying to pull "docker.io/library/ubuntu:20.04" DEBU[0000] Credentials not found                        DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000]  No signature storage configuration found for docker.io/library/ubuntu:20.04 DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io DEBU[0000] GET https://registry-1.docker.io/v2/         DEBU[0001] Ping https://registry-1.docker.io/v2/ status 401 DEBU[0001] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fubuntu%3Apull&service=registry.docker.io DEBU[0004] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/20.04 DEBU[0005] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb DEBU[0005] Source is a manifest list; copying (only) instance sha256:4e4bc990609ed865e07afc8427c30ffdddca5153fd4e82c20d8f0783a291e241 DEBU[0005] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:4e4bc990609ed865e07afc8427c30ffdddca5153fd4e82c20d8f0783a291e241 DEBU[0006] IsRunningImageAllowed for image docker:docker.io/library/ubuntu:20.04 DEBU[0006]  Using default policy section                DEBU[0006]  Requirement 0: allowed                      DEBU[0006] Overall: allowed                             Getting image source signaturesDEBU[0006] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json] DEBU[0006] Downloading /v2/library/ubuntu/blobs/sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8 DEBU[0006] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8 DEBU[0009] Detected compression format gzip             DEBU[0009] Using original blob without modification     Copying blob da7391352a9b [======================================] 27.2MiB / 27.2MiBDEBU[0043] Downloading /v2/library/ubuntu/blobs/sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d Copying blob da7391352a9b doneDEBU[0044] Detected compression format gzip             DEBU[0044] Using original blob without modification     DEBU[0044] Downloading /v2/library/ubuntu/blobs/sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601 Copying blob da7391352a9b doneCopying blob 14428a6d4bcd doneCopying blob da7391352a9b doneCopying blob 14428a6d4bcd doneCopying blob 2c2d948710f2 doneDEBU[0046] Downloading /v2/library/ubuntu/blobs/sha256:f643c72bc25212974c16f3348b3a898b1ec1eb13ec1539e10a103e6e217eb2f1 DEBU[0046] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:f643c72bc25212974c16f3348b3a898b1ec1eb13ec1539e10a103e6e217eb2f1 DEBU[0047] No compression detected                      DEBU[0047] Using original blob without modification     Copying config aa23411143 doneWriting manifest to image destinationStoring signatures2021/01/05 20:05:27  info Image pulled successfully.Process finished with exit code 0

The image is downloaded inside /var/lib/minict/images and it looks like the following

/var/lib/minict/├── containers└── images    └── ubuntu        ├── blobs        │   └── sha256        │       ├── 14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d        │       ├── 2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601        │       ├── 5d52e1388dedc5da07eebd41b0b1c189183c537be51b9a9d6d5e82385373b2f6        │       ├── aa23411143b1e053a8458b3ea4252fb14570a0621ba0e3d26f6143616a874db1        │       └── da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8        ├── index.json        └── oci-layout

Following outlined the different files downloaded from the pull process.

index.json

This file contains information about the image (eg: platform)

    schemaVersion: 2    manifests: [      {        "mediaType": "application/vnd.oci.image.manifest.v1+json",        "digest": "sha256:5d52e1388dedc5da07eebd41b0b1c189183c537be51b9a9d6d5e82385373b2f6",        "size": 658,        "annotations": {          "org.opencontainers.image.ref.name": "20.04"        },        "platform": {          "architecture": "amd64",          "os": "linux"        }      }    ]

oci-layout

This file contains information about the oci-layout information used to store the image

    imageLayoutVersion: "1.0.0"

blobs/sha256

5d52e1388dedc5da07eebd41b0b1c189183c537be51b9a9d6d5e82385373b2f6

This file contains information about the files that has been downloaded including size, type and shasum

    schemaVersion: 2    config: {      "mediaType": "application/vnd.oci.image.config.v1+json",      "digest": "sha256:aa23411143b1e053a8458b3ea4252fb14570a0621ba0e3d26f6143616a874db1",      "size": 2427    }    layers: [      {        "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",        "digest": "sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f8",        "size": 28563271      },      {        "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",        "digest": "sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d",        "size": 847      },      {        "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",        "digest": "sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f2601",        "size": 162      }    ]

aa23411143b1e053a8458b3ea4252fb14570a0621ba0e3d26f6143616a874db1

This file contains information about the container, for example: it contains information about the command that will be executed when the container startsup, in our example it will be /bin/bash

    created: "2020-11-25T22:25:29.546718343Z"    architecture: "amd64"    os: "linux"    config: {      "Env": [        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"      ],      "Cmd": [        "/bin/bash"      ]    }    rootfs: {      "type": "layers",      "diff_ids": [        "sha256:bacd3af13903e13a43fe87b6944acd1ff21024132aad6e74b4452d984fb1a99a",        "sha256:9069f84dbbe96d4c50a656a05bbe6b6892722b0d1116a8f7fd9d274f4e991bf6",        "sha256:f6253634dc78da2f2e3bee9c8063593f880dc35d701307f30f65553e0f50c18c"      ]    }    history: [      {        "created": "2020-11-25T22:25:26.245907708Z",        "created_by": "/bin/sh -c #(nop) ADD file:4f15c4475fbafb3fe335e415e3ea1ac416c34af911fcdfe273c5759438aa8eb4 in / "      },      {        "created": "2020-11-25T22:25:27.346756278Z",        "created_by": "/bin/sh -c set -xe \t\t&& echo '#!/bin/sh' > /usr/sbin/policy-rc.d \t&& echo 'exit 101' >> /usr/sbin/policy-rc.d \t&& chmod +x /usr/sbin/policy-rc.d \t\t&& dpkg-divert --local --rename --add /sbin/initctl \t&& cp -a /usr/sbin/policy-rc.d /sbin/initctl \t&& sed -i 's/^exit.*/exit 0/' /sbin/initctl \t\t&& echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup \t\t&& echo 'DPkg::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' > /etc/apt/apt.conf.d/docker-clean \t&& echo 'APT::Update::Post-Invoke { \"rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true\"; };' >> /etc/apt/apt.conf.d/docker-clean \t&& echo 'Dir::Cache::pkgcache \"\"; Dir::Cache::srcpkgcache \"\";' >> /etc/apt/apt.conf.d/docker-clean \t\t&& echo 'Acquire::Languages \"none\";' > /etc/apt/apt.conf.d/docker-no-languages \t\t&& echo 'Acquire::GzipIndexes \"true\"; Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/docker-gzip-indexes \t\t&& echo 'Apt::AutoRemove::SuggestsImportant \"false\";' > /etc/apt/apt.conf.d/docker-autoremove-suggests"      },      {        "created": "2020-11-25T22:25:28.342445422Z",        "created_by": "/bin/sh -c [ -z \"$(apt-get indextargets)\" ]",        "empty_layer": true      },      {        "created": "2020-11-25T22:25:29.343142847Z",        "created_by": "/bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container"      },      {        "created": "2020-11-25T22:25:29.546718343Z",        "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",        "empty_layer": true      }    ]

The code uses library from github.com/containers/image to download and store the image from a remote registry. This library is quite extensive and provide a lot of functionality to work with OCI based images, in order it’s functionality there is another tool that uses this library extensively called skopeo

run

minict run --image ubuntu:20.04 --name ubuntu-nanik

This command is to run the downloaded images as container. In our example we are running the ubuntu:20.04 image that we have previously downloaded, the container will be given label ubuntu-nanik.

Following is the output showing minict is reading the image files and trying to start it up.

2021/01/05 20:00:38  info Container process exited. It can be started again using the 'start' option.2021/01/05 20:00:38  info unpacking bundle ...     2021/01/05 20:00:38  info unpack rootfs: /var/lib/minict/containers/ubuntu-nanik/rootfs2021/01/05 20:00:38  info unpack layer: sha256:da7391352a9bb76b292a568c066aa4c3cbae8d494e6a3c68e3c596d34f7c75f82021/01/05 20:00:39  info unpack layer: sha256:14428a6d4bcdba49a64127900a0691fb00a3f329aced25eb77e3b65646638f8d2021/01/05 20:00:39  info unpack layer: sha256:2c2d948710f21ad82dce71743b1654b45acb5c059cf5c19da491582cef6f26012021/01/05 20:00:39  info ... done                 2021/01/05 20:00:39  info computing filesystem manifest ...2021/01/05 20:00:40  info ... done                 2021/01/05 20:00:40  info unpacked image bundle: /var/lib/minict/containers/ubuntu-nanik2021/01/05 20:00:40 Failed to mount tmpfs to /dev due to invalid argument2021/01/05 20:00:40 Failed to mount devpts to /dev/pts due to no such file or directory2021/01/05 20:00:40 Failed to mount shm to /dev/shm due to no such file or directory2021/01/05 20:00:40 Failed to mount mqueue to /dev/mqueue due to no such file or directory

Once the container is up and running you will be given bash command to work with.

The run command is using the umoci project to work with containers. The way the umoci library works to start an image as container is as follows

Perform validation for:
- check to make sure the the oci-layout file is valid
- the blobs directory exist
- index.json file exist

Do unpacking when using the Unpack(..) function as follows:

read the index.json
convert the layer information (from 5d52e1388dedc5da07eebd41b0b1c189183c537be51b9a9d6d5e82385373b2f6 file) to internal struct
create all the necessary directories inside /var/lib/minict/containers/ubuntu-nanik to host the filesystem (including rootfs)
unpack filesystem from all the image files into /rootfs

create config.json containing the following information

{"ociVersion": "1.0.0","process": {        "terminal": true,        "user": {                "uid": 0,                "gid": 0        },        "args": [                "/bin/bash"        ],        "env": [                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",                "TERM=xterm",                "HOME=/root"        ],        "cwd": "/",        "capabilities": {                "bounding": [                        "CAP_AUDIT_WRITE",                        "CAP_KILL",                        "CAP_NET_BIND_SERVICE"                ],                "effective": [                        "CAP_AUDIT_WRITE",                        "CAP_KILL",                        "CAP_NET_BIND_SERVICE"                ],                "inheritable": [                        "CAP_AUDIT_WRITE",                        "CAP_KILL",                        "CAP_NET_BIND_SERVICE"                ],                "permitted": [                        "CAP_AUDIT_WRITE",                        "CAP_KILL",                        "CAP_NET_BIND_SERVICE"                ],                "ambient": [                        "CAP_AUDIT_WRITE",                        "CAP_KILL",                        "CAP_NET_BIND_SERVICE"                ]        },        "rlimits": [                {                        "type": "RLIMIT_NOFILE",                        "hard": 1024,                        "soft": 1024                }        ],        "noNewPrivileges": true},"root": {        "path": "rootfs"},"hostname": "umoci-default","mounts": [        {                "destination": "/proc",                "type": "proc",                "source": "proc"        },        {                "destination": "/dev",                "type": "tmpfs",                "source": "tmpfs",                "options": [                        "nosuid",                        "strictatime",                        "mode=755",                        "size=65536k"                ]        },        {                "destination": "/dev/pts",                "type": "devpts",                "source": "devpts",                "options": [                        "nosuid",                        "noexec",                        "newinstance",                        "ptmxmode=0666",                        "mode=0620"                ]        },        {                "destination": "/dev/shm",                "type": "tmpfs",                "source": "shm",                "options": [                        "nosuid",                        "noexec",                        "nodev",                        "mode=1777",                        "size=65536k"                ]        },        {                "destination": "/dev/mqueue",                "type": "mqueue",                "source": "mqueue",                "options": [                        "nosuid",                        "noexec",                        "nodev"                ]        },        {                "destination": "/sys",                "type": "bind",                "source": "/sys",                "options": [                        "rbind",                        "nosuid",                        "noexec",                        "nodev",                        "ro"                ]        },        {                "destination": "/etc/resolv.conf",                "type": "bind",                "source": "/etc/resolv.conf",                "options": [                        "noexec",                        "nosuid",                        "rbind",                        "ro"                ]        }],"annotations": {        "org.opencontainers.image.architecture": "amd64",        "org.opencontainers.image.author": "",        "org.opencontainers.image.created": "2020-11-25T22:25:29.546718343Z",        "org.opencontainers.image.exposedPorts": "",        "org.opencontainers.image.os": "linux",        "org.opencontainers.image.stopSignal": ""},"linux": {        "namespaces": [                {                        "type": "pid"                },                {                        "type": "ipc"                },                {                        "type": "uts"                },                {                        "type": "mount"                },                {                        "type": "user"                }        ],        "maskedPaths": [                "/proc/kcore",                "/proc/latency_stats",                "/proc/timer_list",                "/proc/timer_stats",                "/proc/sched_debug",                "/sys/firmware",                "/proc/scsi"        ],        "readonlyPaths": [                "/proc/asound",                "/proc/bus",                "/proc/fs",                "/proc/irq",                "/proc/sys",                "/proc/sysrq-trigger"        ]}}

read config.json to construct internal config object
construct the command that will be executed to run the container and apply namespaces

Hacking

Add the following code inside oci/pull.go under function func PullImage(..) to enable the download reporting
```
return copy.Image(ctx, policyContext, destRef, srcRef, &copy.Options{ ReportWriter: os.Stdout})
```
Add the following inside main.go to increase the logging level
```
logrus.SetLevel(8)
```

gvisor

Fri, 02 Oct 2020 00:00:00 +0000

This article will explain about the container runtime called gvisor. The article focus more on how Sentry provides extra security layer to gvisor and how it intercepts system calls and interact with the application running inside container.

Building gvisor

Make sure your docker is working properly as the build system required docker container to build the binaries.
Build gvisor issuing the command
```
make runsc
```
the binary is located inside ./bazel-gvisor/bazel-out/k8-opt/bin/runsc/linux_amd64_pure_stripped copy the runsc binary to
```
/usr/local/bin/
```

Copy the following to a file called daemon.json inside /etc/docker

{"default-runtime": "runsc","runtimes": {"runc-local": {"path": "runc"},"runsc": {"path": "/usr/local/bin/runsc","runtimeArgs": ["--debug-log=/tmp/runsc/","--debug","--strace"]}}}

Run the following docker command to test runsc

docker run --runtime=runsc -it ubuntu  dmesg

The result will be as follows

[    0.000000] Starting gVisor...[    0.380255] Letting the watchdogs out...[    0.671842] Waiting for children...[    1.111248] Moving files to filing cabinet...[    1.537250] Checking naughty and nice process list...[    1.632947] Creating process schedule...[    2.025866] Mounting deweydecimalfs...[    2.054713] Reading process obituaries...[    2.239299] Daemonizing children...[    2.485680] Reticulating splines...[    2.558058] Consulting tar man page...[    2.741916] Ready!

Run the following to test docker using the normal runc

docker run --privileged  --runtime=runc-local -it ubuntu  dmesg

The result is as follows

[    0.000000] microcode: microcode updated early to revision 0x2f, date = 2019-11-12[    0.000000] Linux version 5.4.0-7642-generic (buildd@lcy01-amd64-007) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #46~1598628707~20.04~040157c-Ubuntu SMP Fri Aug 28 18:02:16 UTC  (Ubuntu 5.4.0-7642.46~1598628707~20.04~040157c-generic 5.4.44)[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.4.0-7642-generic root=/dev/mapper/data-root ro quiet splash vt.handoff=7[    0.000000] KERNEL supported cpus:[    0.000000]   Intel GenuineIntel[    0.000000]   AMD AuthenticAMD[    0.000000]   Hygon HygonGenuine[    0.000000]   Centaur CentaurHauls............

Sentry the ‘Secret Sauce’

Noticed the difference in output that you get running runsc compared to runc ?. The dmesg output from runsc shows that the kernel that it is running is NOT the host kernel, unlike what we see in the output when using runc. This is the ‘special sauce’ about gvisor.

gvisor provides a ‘usermode’ Linux Kernel or known as UML. Inside runsc there is a sub-module called Sentry which act as thin layer of kernel, this does not mean that it’s job is to replace Kernel. Sentry provide an extra layer of ‘safety’ that intercepts system calls from application that is running inside a container. This prevents the application from having any direct interaction with the host through syscalls.

gVisor supports two methods of redirecting syscalls * ptrace-mode - uses ptrace in the Linux kernel to forward syscalls to the sentry and * KVM-mode - uses KVM to trap syscalls before they hit the Linux kernel so they can be forwarded to the sentry.

Read the gvisor docs to get more intro about what is it all about and what kind of security it provides

Internals

We will discuss more in details at how gvisor really works internally. This is based on my personal experiment and research, so there could be some information that is not correct.

Following are some of the topics and concepts that you need to understand to really understand how Sentry works and the security functionality it provides for gvisor.

To get a better understand how the whole container stack works (docker, containerd, etc) refer to my other posting

Execution

When container is launched using runsc it does what is shown in the diagram below

What the diagram shows there are few things that is happening behind the scenes before your container is executed. The first command that is triggered by docker to run runsc is as follows

/usr/local/bin/runsc --debug-log=/tmp/runsc/ --debug --strace --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af --pid-file /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/init.pid --console-socket /tmp/pty993587804/pty.sock e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af

Following on executing the above command runsc will executed itself few times as follows sequence:

(1) execute gofer for setting up Capabilities for directories

/proc/self/exe --root=/var/run/docker/runtime-runc/moby --debug=true --log=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format=json --debug-log=/tmp/runsc/ --strace=true --log-fd=3 --debug-log-fd=4 gofer --bundle /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af --spec-fd=5 --mounts-fd=6 --io-fds=7 --io-fds=8 --io-fds=9 --io-fds=10

(2) execute gofer to run as normal

/proc/self/exe --root=/var/run/docker/runtime-runc/moby --debug=true --log=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format=json --debug-log=/tmp/runsc/ --strace=true --log-fd=3 --debug-log-fd=4 gofer --bundle /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af --spec-fd=5 --mounts-fd=6 --io-fds=7 --io-fds=8 --io-fds=9 --io-fds=10 --apply-caps=false --setup-root=false

(3) execute sandbox to setup root

/proc/self/exe --root=/var/run/docker/runtime-runc/moby --debug=true --log=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format=json --debug-log=/tmp/runsc/ --strace=true --log-fd=3 --debug-log-fd=4 boot --bundle=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af --controller-fd=5 --mounts-fd=6 --spec-fd=7 --start-sync-fd=8 --io-fds=9 --io-fds=10 --io-fds=11 --io-fds=12 --stdio-fds=13 --stdio-fds=14 --stdio-fds=15 --setup-root --cpu-num 4 e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af]

(4) execute sandbox to setup everything else (Sentry, ptrace, etc)

/proc/self/exe --root=/var/run/docker/runtime-runc/moby --debug=true --log=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format=json --debug-log=/tmp/runsc/ --strace=true --log-fd=3 --debug-log-fd=4 boot --bundle=/run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af --controller-fd=5 --mounts-fd=6 --spec-fd=7 --start-sync-fd=8 --io-fds=9 --io-fds=10 --io-fds=11 --io-fds=12 --stdio-fds=13 --stdio-fds=14 --stdio-fds=15 --cpu-num 4 e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af

(5) start container to run the sandbox along with the container and it’s application

/usr/local/bin/runsc --debug-log=/tmp/runsc/ --debug --strace --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af/log.json --log-format json start e488ee2bef4cfa13e82dc38c7e7db759ad3a8f89e02218a0339adb79a47193af

The following diagram shows the different process that was created during the lifecycle of runsc. The number in the diagram inside bracket is the PID of the runsc process when it was running.

vDSO

vdso stands for virtual ELF dynamic shared object. It is a small shared library that the kernel automatically maps into the address space of ALL user-space applications. The reason why we have vdso is to make syscall faster, but due to it’s small space only 4 different functions are provided

clock_gettimegetcpugettimeofdaytime

Sentry implement it’s own VDSO which is located inside vdso directory in the source code.

When application is loaded into memory to be executed the ELF loader populate the vdso address using the auxiliary vectors. These vectors are the mechanism to transfer some OS specific information to the program interpreter (e.g. ld) and the process. This task is normally done by the elf loader in Linux but in gvisor it is done by sentry/loader package. These vectors are put on the process stack along with other information like argc, argv, envp. After stack initialization, normal stack will look something like this:

position            content                     size (bytes) + comment------------------------------------------------------------------------stack pointer ->  [ argc = number of args ]     4            [ argv[0] (pointer) ]         4   (program name)            [ argv[1] (pointer) ]         4            [ argv[..] (pointer) ]        4 * x            [ argv[n - 1] (pointer) ]     4            [ argv[n] (pointer) ]         4   (= NULL)            [ envp[0] (pointer) ]         4            [ envp[1] (pointer) ]         4            [ envp[..] (pointer) ]        4            [ envp[term] (pointer) ]      4   (= NULL)            [ auxv[0] (Elf32_auxv_t) ]    8            [ auxv[1] (Elf32_auxv_t) ]    8            [ auxv[..] (Elf32_auxv_t) ]   8            [ auxv[term] (Elf32_auxv_t) ] 8   (= AT_NULL vector)            [ padding ]                   0 - 16            [ argument ASCIIZ strings ]   >= 0            [ environment ASCIIZ str. ]   >= 0(0xbffffffc)      [ end marker ]                4   (= NULL)(0xc0000000)      < bottom of stack >           0   (virtual)

In Sentry all the heavy lifting of the vDSO, etc are done inside the loader.go during the time when application is being loaded into memory as shown below.

[pkg/sentry/loader/loader.go]func Load(ctx context.Context, args LoadArgs, extraAuxv []arch.AuxEntry, vdso *VDSO) (abi.OS, arch.Context, string, *syserr.Error) {// Load the executable itself.loaded, ac, file, newArgv, err := loadExecutable(ctx, args)......// Add generic auxv entries.auxv := append(loaded.auxv, arch.Auxv{arch.AuxEntry{linux.AT_UID, usermem.Addr(c.RealKUID.In(c.UserNamespace).OrOverflow())},arch.AuxEntry{linux.AT_EUID, usermem.Addr(c.EffectiveKUID.In(c.UserNamespace).OrOverflow())},arch.AuxEntry{linux.AT_GID, usermem.Addr(c.RealKGID.In(c.UserNamespace).OrOverflow())},arch.AuxEntry{linux.AT_EGID, usermem.Addr(c.EffectiveKGID.In(c.UserNamespace).OrOverflow())},// The conditions that require AT_SECURE = 1 never arise. See// kernel.Task.updateCredsForExecLocked.arch.AuxEntry{linux.AT_SECURE, 0},arch.AuxEntry{linux.AT_CLKTCK, linux.CLOCKS_PER_SEC},arch.AuxEntry{linux.AT_EXECFN, execfn},arch.AuxEntry{linux.AT_RANDOM, random},arch.AuxEntry{linux.AT_PAGESZ, usermem.PageSize},arch.AuxEntry{linux.AT_SYSINFO_EHDR, vdsoAddr},}...)auxv = append(auxv, extraAuxv...)sl, err := stack.Load(newArgv, args.Envv, auxv)......

specifically this particular line of code which sets the vDSO address

arch.AuxEntry{linux.AT_SYSINFO_EHDR, vdsoAddr},

Seccomp

seccomp (short for secure computing mode) is a computer security facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will terminate the process with SIGKILL or SIGSYS.[1][2] In this sense, it does not virtualize the system's resources but isolates the process from them entirely.

gvisor is known as ‘seccomp on steroids’ because it uses seccomp to setup security filters. Which means that any application running inside a container is not calling the host kernel, but instead it will be calling gvisor kernel.

Following is an example of the seccomp filters that are installed by gvisor (obtained from the log)

I0929 04:00:54.926295       1 seccomp.go:61] Installing seccomp filters for 58 syscalls (action=kill process)D0929 04:00:54.926349       1 seccomp.go:171] syscall filter read: [] => 0x616c6c6f77D0929 04:00:54.926371       1 seccomp.go:171] syscall filter write: [] => 0x616c6c6f77D0929 04:00:54.926388       1 seccomp.go:171] syscall filter close: [] => 0x616c6c6f77D0929 04:00:54.926398       1 seccomp.go:171] syscall filter fstat: [] => 0x616c6c6f77D0929 04:00:54.926408       1 seccomp.go:171] syscall filter lseek: [] => 0x616c6c6f77D0929 04:00:54.926582       1 seccomp.go:171] syscall filter mmap: [( * * * == 0x1 ) ( * * * == 0x22 ) ( * * * == 0x32 )] => 0x616c6c6f77D0929 04:00:54.926612       1 seccomp.go:171] syscall filter mprotect: [] => 0x616c6c6f77D0929 04:00:54.926619       1 seccomp.go:171] syscall filter munmap: [] => 0x616c6c6f77D0929 04:00:54.926624       1 seccomp.go:171] syscall filter rt_sigprocmask: [] => 0x616c6c6f77D0929 04:00:54.926630       1 seccomp.go:171] syscall filter rt_sigreturn: [] => 0x616c6c6f77D0929 04:00:54.926636       1 seccomp.go:171] syscall filter pread64: [] => 0x616c6c6f77D0929 04:00:54.926641       1 seccomp.go:171] syscall filter pwrite64: [] => 0x616c6c6f77D0929 04:00:54.926647       1 seccomp.go:171] syscall filter sched_yield: [] => 0x616c6c6f77D0929 04:00:54.926652       1 seccomp.go:171] syscall filter madvise: [] => 0x616c6c6f77D0929 04:00:54.926658       1 seccomp.go:171] syscall filter dup: [] => 0x616c6c6f77D0929 04:00:54.926666       1 seccomp.go:171] syscall filter nanosleep: [] => 0x616c6c6f77D0929 04:00:54.926675       1 seccomp.go:171] syscall filter getpid: [] => 0x616c6c6f77D0929 04:00:54.926681       1 seccomp.go:171] syscall filter accept: [] => 0x616c6c6f77D0929 04:00:54.926687       1 seccomp.go:171] syscall filter sendmsg: [( * * == 0x0 ) ( * * == 0x4040 )] => 0x616c6c6f77D0929 04:00:54.926697       1 seccomp.go:171] syscall filter recvmsg: [( * * == 0x60 ) ( * * == 0x62 )] => 0x616c6c6f77D0929 04:00:54.926710       1 seccomp.go:171] syscall filter shutdown: [( * == 0x2 )] => 0x616c6c6f77D0929 04:00:54.926717       1 seccomp.go:171] syscall filter socketpair: [( == 0x1 == 0x80005 == 0x0 )] => 0x616c6c6f77D0929 04:00:54.926725       1 seccomp.go:171] syscall filter clone: [( == 0xd0f00 * == 0x0 == 0x0 * ) ( == 0x50f00 * == 0x0 == 0x0 * )] => 0x616c6c6f77D0929 04:00:54.926741       1 seccomp.go:171] syscall filter exit: [] => 0x616c6c6f77D0929 04:00:54.926754       1 seccomp.go:171] syscall filter fcntl: [( * == 0x3 ) ( * == 0x4 ) ( * == 0x1 ) ( * == 0x409 )] => 0x616c6c6f77D0929 04:00:54.926768       1 seccomp.go:171] syscall filter fsync: [] => 0x616c6c6f77D0929 04:00:54.926773       1 seccomp.go:171] syscall filter ftruncate: [] => 0x616c6c6f77D0929 04:00:54.926779       1 seccomp.go:171] syscall filter fchmod: [] => 0x616c6c6f77D0929 04:00:54.926785       1 seccomp.go:171] syscall filter gettimeofday: [] => 0x616c6c6f77D0929 04:00:54.926790       1 seccomp.go:171] syscall filter sigaltstack: [] => 0x616c6c6f77D0929 04:00:54.926801       1 seccomp.go:171] syscall filter fstatfs: [] => 0x616c6c6f77D0929 04:00:54.926808       1 seccomp.go:171] syscall filter mlock: [( * == 0x1000 )] => 0x616c6c6f77D0929 04:00:54.926815       1 seccomp.go:171] syscall filter arch_prctl: [( == 0x1002 )] => 0x616c6c6f77D0929 04:00:54.926828       1 seccomp.go:171] syscall filter gettid: [] => 0x616c6c6f77D0929 04:00:54.926842       1 seccomp.go:171] syscall filter futex: [( * == 0x80 * * == 0x0 ) ( * == 0x81 * * == 0x0 ) ( * == 0x0 * * ) ( * == 0x1 * * )] => 0x616c6c6f77D0929 04:00:54.926859       1 seccomp.go:171] syscall filter getdents64: [] => 0x616c6c6f77D0929 04:00:54.926866       1 seccomp.go:171] syscall filter restart_syscall: [] => 0x616c6c6f77D0929 04:00:54.926872       1 seccomp.go:171] syscall filter clock_gettime: [] => 0x616c6c6f77D0929 04:00:54.926877       1 seccomp.go:171] syscall filter exit_group: [] => 0x616c6c6f77D0929 04:00:54.926886       1 seccomp.go:171] syscall filter epoll_ctl: [] => 0x616c6c6f77D0929 04:00:54.926892       1 seccomp.go:171] syscall filter tgkill: [( == 0x1 )] => 0x616c6c6f77D0929 04:00:54.926904       1 seccomp.go:171] syscall filter openat: [] => 0x616c6c6f77D0929 04:00:54.926910       1 seccomp.go:171] syscall filter mkdirat: [] => 0x616c6c6f77D0929 04:00:54.926916       1 seccomp.go:171] syscall filter mknodat: [] => 0x616c6c6f77D0929 04:00:54.926921       1 seccomp.go:171] syscall filter fchownat: [] => 0x616c6c6f77D0929 04:00:54.926927       1 seccomp.go:171] syscall filter newfstatat: [] => 0x616c6c6f77D0929 04:00:54.926932       1 seccomp.go:171] syscall filter unlinkat: [] => 0x616c6c6f77D0929 04:00:54.926938       1 seccomp.go:171] syscall filter renameat: [] => 0x616c6c6f77D0929 04:00:54.926943       1 seccomp.go:171] syscall filter linkat: [] => 0x616c6c6f77D0929 04:00:54.926949       1 seccomp.go:171] syscall filter symlinkat: [] => 0x616c6c6f77D0929 04:00:54.926954       1 seccomp.go:171] syscall filter readlinkat: [] => 0x616c6c6f77D0929 04:00:54.926960       1 seccomp.go:171] syscall filter ppoll: [] => 0x616c6c6f77D0929 04:00:54.926965       1 seccomp.go:171] syscall filter utimensat: [] => 0x616c6c6f77D0929 04:00:54.926973       1 seccomp.go:171] syscall filter epoll_pwait: [( * * * * == 0x0 )] => 0x616c6c6f77D0929 04:00:54.926987       1 seccomp.go:171] syscall filter fallocate: [( * == 0x0 )] => 0x616c6c6f77D0929 04:00:54.926995       1 seccomp.go:171] syscall filter eventfd2: [( == 0x0 == 0x0 )] => 0x616c6c6f77D0929 04:00:54.927008       1 seccomp.go:171] syscall filter getrandom: [] => 0x616c6c6f77D0929 04:00:54.927019       1 seccomp.go:171] syscall filter memfd_create: [] => 0x616c6c6f77

Syscalls

As can be seen from the above diagram when the container app is doing syscall it will be using the syscalls that is inside Sentry and not the host kernel.

When running a container using runsc run

cat /proc/self/maps

We will see the following memory maps

562059612000-562059614000 r--p 00000000 00:0e 19                         /usr/bin/cat562059614000-562059619000 r-xp 00002000 00:0e 19                         /usr/bin/cat562059619000-56205961c000 r--p 00007000 00:0e 19                         /usr/bin/cat56205961c000-56205961d000 r--p 00009000 00:0e 19                         /usr/bin/cat56205961d000-56205961e000 rw-p 0000a000 00:0e 19                         /usr/bin/cat56205961e000-56205963f000 rw-p 00000000 00:00 0                          [heap]7f74f24eb000-7f74f2ceb000 rw-p 00000000 00:00 0                          [stack]7f986152b000-7f986154d000 rw-p 00000000 00:00 0 7f986154d000-7f9861572000 r--p 00000000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f9861572000-7f98616ea000 r-xp 00025000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f98616ea000-7f9861734000 r--p 0019d000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f9861734000-7f9861735000 ---p 001e7000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f9861735000-7f9861738000 r--p 001e7000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f9861738000-7f986173b000 rw-p 001ea000 00:0e 30                         /usr/lib/x86_64-linux-gnu/libc-2.31.so7f986173b000-7f9861741000 rw-p 00000000 00:00 0 7f9861743000-7f9861744000 r--p 00000000 00:00 0                          [vvar]7f9861744000-7f9861746000 r-xp 00000000 00:00 0 7f9861746000-7f9861747000 r--p 00000000 00:0e 27                         /usr/lib/x86_64-linux-gnu/ld-2.31.so7f9861747000-7f986176a000 r-xp 00001000 00:0e 27                         /usr/lib/x86_64-linux-gnu/ld-2.31.so7f986176a000-7f9861772000 r--p 00024000 00:0e 27                         /usr/lib/x86_64-linux-gnu/ld-2.31.so7f9861773000-7f9861774000 r--p 0002c000 00:0e 27                         /usr/lib/x86_64-linux-gnu/ld-2.31.so7f9861774000-7f9861775000 rw-p 0002d000 00:0e 27                         /usr/lib/x86_64-linux-gnu/ld-2.31.so7f9861775000-7f9861776000 rw-p 00000000 00:00 0 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

in comparison following is from local Linux

55b8227e4000-55b8227e6000 r--p 00000000 fd:01 11272329                   /usr/bin/cat55b8227e6000-55b8227eb000 r-xp 00002000 fd:01 11272329                   /usr/bin/cat55b8227eb000-55b8227ee000 r--p 00007000 fd:01 11272329                   /usr/bin/cat55b8227ee000-55b8227ef000 r--p 00009000 fd:01 11272329                   /usr/bin/cat55b8227ef000-55b8227f0000 rw-p 0000a000 fd:01 11272329                   /usr/bin/cat55b823606000-55b823627000 rw-p 00000000 00:00 0                          [heap]7f2e3747a000-7f2e3749c000 rw-p 00000000 00:00 0 7f2e3749c000-7f2e38396000 r--p 00000000 fd:01 11409222                   /usr/lib/locale/locale-archive7f2e38396000-7f2e383bb000 r--p 00000000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e383bb000-7f2e38533000 r-xp 00025000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e38533000-7f2e3857d000 r--p 0019d000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e3857d000-7f2e3857e000 ---p 001e7000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e3857e000-7f2e38581000 r--p 001e7000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e38581000-7f2e38584000 rw-p 001ea000 fd:01 11808327                   /usr/lib/x86_64-linux-gnu/libc-2.31.so7f2e38584000-7f2e3858a000 rw-p 00000000 00:00 0 7f2e385a0000-7f2e385a1000 r--p 00000000 fd:01 11800376                   /usr/lib/x86_64-linux-gnu/ld-2.31.so7f2e385a1000-7f2e385c4000 r-xp 00001000 fd:01 11800376                   /usr/lib/x86_64-linux-gnu/ld-2.31.so7f2e385c4000-7f2e385cc000 r--p 00024000 fd:01 11800376                   /usr/lib/x86_64-linux-gnu/ld-2.31.so7f2e385cd000-7f2e385ce000 r--p 0002c000 fd:01 11800376                   /usr/lib/x86_64-linux-gnu/ld-2.31.so7f2e385ce000-7f2e385cf000 rw-p 0002d000 fd:01 11800376                   /usr/lib/x86_64-linux-gnu/ld-2.31.so7f2e385cf000-7f2e385d0000 rw-p 00000000 00:00 0 7fffbe390000-7fffbe3b1000 rw-p 00000000 00:00 0                          [stack]7fffbe3bd000-7fffbe3c0000 r--p 00000000 00:00 0                          [vvar]7fffbe3c0000-7fffbe3c1000 r-xp 00000000 00:00 0                          [vdso]ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Loading, Executing and Switching app/kernel

Since Sentry works like a Linux kernel it is responsible in reading, loading and executing the executable that the container is going to run. This means it provides functionality like how an ELF loader does in Linux. The kernel code can be found inside sentry/kernel folder

As shown in the diagram below when the application is doing a system call (eg: writing to a file, console, etc) it will raise a signal that will be intercepted by Sentry. Once the system call is done byt Sentry it goes back to the application, and the cycle continue.

This is possible with the help of ptrace.

ptrace

When gvisor starts up it create what is called ‘master’ thread and it creates different thread by cloning this ‘master’ thread and it enable ptracing on the thread to enable it to capture system calls when the application is run.

Basically the application is ‘wrapped’ inside a goroutine that is part of gvisor and this way gvisor (including Sentry) has capability to control the application.

Log

Following is log that shows the main functionality of running the app and it’s interaction with Sentry. The log message is not the full logs generated by gvisor, only the one that is interesting (which has my name attached to it) is shown here.

I1002 20:55:52.477178   57407 container.go:328] Nanik - container.NewI1002 20:55:52.477202   57407 container.go:868] Nanik - createGoferProcessI1002 10:55:52.517996   57434 loader.go:491] Nanik - createPlatformI1002 10:55:52.518030   57434 ptrace.go:207] Nanik - ptrace.NewI1002 10:55:52.518359   57434 subprocess_linux.go:135] Nanik - beforeForkI1002 10:55:52.518557   57434 subprocess_linux.go:150] Nanik - afterFork pid & tgid 57454I1002 10:55:52.519704   57434 subprocess_amd64.go:238] Nanik - Attempt an emulation.I1002 10:55:52.519733   57434 subprocess_amd64.go:243] Nanik -  t.waitI1002 10:55:52.519747   57434 subprocess_amd64.go:262] Nanik -  NOT syscallEvent | syscall.SIGTRAPI1002 10:55:52.519752   57434 subprocess_amd64.go:238] Nanik - Attempt an emulation.I1002 10:55:52.519759   57434 subprocess_amd64.go:243] Nanik -  t.waitI1002 10:55:52.519769   57434 subprocess_amd64.go:246] Nanik -  syscallEvent | syscall.SIGTRAPI1002 10:55:52.519921   57434 subprocess_linux.go:55] Nanik - createStubI1002 10:55:52.520130   57434 subprocess_linux.go:135] Nanik - beforeForkI1002 10:55:52.520263   57434 subprocess_linux.go:150] Nanik - afterFork pid & tgid 57455I1002 10:55:52.520605   57434 ptrace_unsafe.go:125] Nanik - clone..cloning 57455 -- 57455I1002 10:55:52.520694   57434 subprocess.go:178] Nanik - requests to create threads...cloning the firstThread 57455I1002 10:55:52.752987   57434 loader.go:581] Nanik - run  l.createContainerProcess(true, l.sandboxID, &l.root, ep);I1002 10:55:52.753274   57434 loader.go:734] Nanik - setupContainerFSI1002 10:55:52.784717   57434 path.go:37] Nanik - ResolveExecutablePath args &{Filename: File:<nil> Argv:[/hello] Envv:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=a8e612468f0b] WorkingDirectory:/ Credentials:0xc0002b85a0 FDTable:   fd:0 => name host:[1]I1002 10:55:52.784827   57434 path.go:48] Nanik - ResolveExecutablePath path.IsAbs(name)I1002 10:55:52.785174   57434 loader.go:755] Nanik - createContainerProcess l.k.CreateProcess(info.procArgs) {Filename:/hello File:<nil> Argv:[/hello] Envv:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=a8e612468f0b HOME=/] WorkingDirectory:/ Credentials:0xc0002b85a0 FDTable:       fd:0 => name host:[1]I1002 10:55:52.788418   57434 loader.go:180] Nanik - loadExecutable elfMagicI1002 10:55:52.788437   57434 elf.go:643] Nanik - loadELF for /hello I1002 10:55:52.789010   57434 task_start.go:243] Nanik - assignTIDsLocked 1I1002 10:55:52.789870   57434 kernel.go:1068] Nanik - Kernel Start()I1002 10:55:52.789899   57434 task_start.go:307] Nanik - task_start Start 1I1002 10:55:52.789959   57434 task_run.go:70] Nanik - run task_run.go 1I1002 10:55:52.790006   57434 task_usermem.go:35] Nanik - task_usermem ActivateI1002 10:55:52.790172   57434 ptrace_unsafe.go:125] Nanik - clone..cloning 57455 -- 57455I1002 10:55:52.793677   57434 subprocess.go:178] Nanik - requests to create threads...cloning the firstThread 57455I1002 10:55:52.793942   57434 subprocess_linux.go:254] Nanik - createStub 57515I1002 10:55:52.794118   57434 ptrace_unsafe.go:125] Nanik - clone..cloning 57515 -- 57515I1002 10:55:52.794232   57434 subprocess.go:178] Nanik - requests to create threads...cloning the firstThread 57515I1002 10:55:52.794447   57434 task_run.go:81] Nanik - run task_run.go t.interruptSelf()  1I1002 10:55:52.794482   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.794508   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.794529   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.794540   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.794549   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.794598   57434 ptrace_unsafe.go:125] Nanik - clone..cloning 57515 -- 57515I1002 10:55:52.794691   57434 subprocess.go:178] Nanik - requests to create threads...cloning the firstThread 57515I1002 10:55:52.794789   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.794947   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.794966   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.794980   57434 subprocess.go:589] Nanik - outside Is it a system call?I1002 10:55:52.795038   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795051   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795061   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795075   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.795123   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.795138   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.795152   57434 subprocess.go:589] Nanik - outside Is it a system call?I1002 10:55:52.795194   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795206   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795215   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795232   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.795273   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.795285   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.795298   57434 subprocess.go:589] Nanik - outside Is it a system call?I1002 10:55:52.795338   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795350   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795359   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795372   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.795403   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.795414   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.795427   57434 subprocess.go:589] Nanik - outside Is it a system call?I1002 10:55:52.795466   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795478   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795494   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795508   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.795547   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.795561   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.795575   57434 subprocess.go:579] Nanik - a system callI1002 10:55:52.795592   57434 task_syscall.go:200] Nanik -  doSyscallEnter 158I1002 10:55:52.795685   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795722   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795733   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795748   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.795781   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.795793   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.795807   57434 subprocess.go:579] Nanik - a system callI1002 10:55:52.795820   57434 task_syscall.go:200] Nanik -  doSyscallEnter 218I1002 10:55:52.795863   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.795872   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.795880   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.795894   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518..........I1002 10:55:52.796112   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.796130   57434 ptrace.go:103] Nanik - going to go inside switchToAppI1002 10:55:52.796147   57434 subprocess.go:495] Nanik -switchToAppI1002 10:55:52.796170   57434 subprocess.go:547] Nanik - NOT isSingleStepping 57518I1002 10:55:52.796219   57434 subprocess.go:558] Nanik - after t.waitI1002 10:55:52.796247   57434 subprocess.go:561] Nanik - Refresh all registersI1002 10:55:52.796283   57434 subprocess.go:579] Nanik - a system callI1002 10:55:52.796305   57434 task_syscall.go:200] Nanik -  doSyscallEnter 231I1002 10:55:52.796642   57434 task_run.go:97] Nanik -  executeI1002 10:55:52.796683   57434 task_run.go:97] Nanik -  execute......

The command ran to generate the above log is as follows

docker run --runtime=runsc hello-world

Other Notes

gettimeofday() on Linux is what’s called a vsyscall and/or vdso.

0x00000034f408c2d4 : mov    $0xffffffffff600000,%rax0x00000034f408c2db : callq  *%rax

The address 0xffffffffff600000 is the vsyscall page (on x86_64). The mechanism maps a specific kernel-created code page into user memory, so that a few “syscalls” can be made without the overhead of a user/kernel context switch, but rather as “ordinary” function call.

Syscalls generally create a lot of overhead, and given the abundance of gettimeofday() calls, one would prefer not to use a syscall. To that end, Linux kernel may map one or two special areas into each program, called vdso and vsyscall. Your implementation of gettimeofday() seems to be using vsyscall:

mov    $0xffffffffff600000,%rax

This is the standard address of vsyscall map. Try cat /proc/self/maps to see that mapping. The idea behind vsyscall is that kernel provides fast user-space implementations of some functions, and libc just calls them.

Read this nice article for more details.

Resources

gvisor

https://research.cs.wisc.edu/multifacet/papers/vee20_blending.pdf
https://blog.loof.fr/2018/06/gvisor-in-depth.html
User Mode Linux is a good book and gvisor application kernel works the same like that.

https://wenboshen.org/posts/2018-12-01-sectainer.html

This is straight forward, sentry process acts as tracer while application process is the tracee. Application process system call will stop by PTRACE and wait for sentry to handle. Sentry can emulate the system call, replace the real system with getpid(), and return.

https://archive.fosdem.org/2020/schedule/event/containers_k8s_runtimes/attachments/slides/3751/export/events/attachments/containers_k8s_runtimes/slides/3751/below_kubernetes.pdf
Understanding about Ring and Sentry
Inside gvisor

Seccomp

ptrace

user mode Linux & Linux in general

go-rest-api project

Sun, 15 Mar 2020 00:00:00 +0000

References

https://github.com/golang-standards/project-layout

Janos Web Utility

Sun, 08 Mar 2020 00:00:00 +0000

Reference: https://github.com/janos/web

    ├── client│   ├── api│   └── http├── file-server├── log│   └── access├── maintenance├── minifier├── recovery├── server├── servers│   ├── grpc│   │   └── internal│   │       └── hello│   ├── http│   └── quic└── templates

Framework revolves around Handlers

Samples

file-server
- hasher.go
  - hashing functions
- options.go
  - contains struct defining handler and other http server related options
  - contains global var for handler such as DefaultNotFoundHandler, DefaultForbiddenHandler and DefaultInternalServerErrorHandler
- server.go
  - HashedPath –> create hashed path from a string
recovery
- recovery.go
  - provide handler and other utility for handling ‘panic’ situation in http server environment
server
- internal.go
  - provide lots of internal handlers for troubleshooting - /status, /data, /debug/pprof. The handler are triggered when specify the port in Options.ListenInternal
- handlers.go
servers/http
- http.go - code that takes care of http related functionality - servers, request, handlers, etc

    package mainimport (    "fmt"    "net/http"    "resenje.org/logging"    recovery "resenje.org/recovery"    s "resenje.org/web/server")var (    responseBody = "----- response body -----"    handler      = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {        fmt.Fprint(w, responseBody)    }))func main() {    rs := &recovery.Service{}    logger, _ := logging.GetLogger("default")    logger.SetLevel(8)    o := s.Options{        Name:               "servertest",        Version:            "1.0",        BuildInfo:          "buildinfo",        ListenInternal:     "localhost:9200",        ListenInternalTLS:  "",        InternalTLSCert:    "",        InternalTLSKey:     "",        ACMECertsDir:       "",        ACMECertsEmail:     "",        EmailService:       nil,        RecoveryService:    rs,        MaintenanceService: nil,        Logger:logger,    }    ho := s.HTTPOptions{        Handlers:     nil,        Name:         "",        Listen:       "localhost:9000",        ListenTLS:    "",        TLSCerts:     nil,        IdleTimeout:  0,        ReadTimeout:  0,        WriteTimeout: 0,    }    ho.SetHandler(handler)    ss, _ := s.New(o)    ss.WithHTTP(ho)    ss.Serve()    select {}}

Internals of Container Networking

Wed, 26 Feb 2020 00:00:00 +0000

References:
There are 2 different ways network are setup and used in container world:
- CNM (Container Network Model) or CNI (Container Network Interface)
- CNI implementation github.com/containernetworking/plugins/
CNI plugins available
- IPAM
  - dhcp
  - host-local
  - static
- Main
  - bridge
  - host-device
  - windows
  - ipvlan
  - vlan
  - loopback
  - ptp
  - macvlan
- Meta
  - bandwidth
  - portmap
  - firewall
  - sbr
  - flannel
  - tuning
- CNI network are implemented as plugins github.com/containernetworking/plugins. The plugins are configured by calling the cni executable and passing a configuration file outlining the network setup config like the one below
```
    {    "cniVersion": "0.4.0",    "name": "openfaas-cni-bridge",    "plugins": [      {        "type": "bridge",        "bridge": "openfaas0",        "isGateway": true,        "ipMasq": true,        "ipam": {            "type": "host-local",            "subnet": "10.62.0.0/16",            "routes": [ { "dst": "0.0.0.0/0" } ]               }      },      {        "type": "firewall"      }    ]}
```
reading the config file the ‘bridge’ plugin will be executed (which resides inside /opt/cni/bin directory).
- go client implementation resides in https://github.com/containerd/go-cni the project support
In Linux environment network are configured and managed using netlink
containerd uses CNI
Docker uses CNM
- Docker uses CNM which uses the libnetwork project.
The libnetwork project uses netlink call to create network virtual connection (endpoints, etc). Docker create different namespaces required for isolation as shown in the example below

    docker container listCONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMESbbea5d31d581        ubuntu:latest       "/bin/bash"         13 seconds ago      Up 12 seconds                           peaceful_faraday

    docker inspect -f '{{.State.Pid}}' bbea5d31d58119025

    sudo ls /proc/19025/ns/ -latotal 0dr-x--x--x 2 root root 0 Feb 06 21:08 .dr-xr-xr-x 9 root root 0 Feb 06 21:07 ..lrwxrwxrwx 1 root root 0 Feb 06 21:08 cgroup -> 'cgroup:[4026531835]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 ipc -> 'ipc:[4026533665]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 mnt -> 'mnt:[4026533663]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 net -> 'net:[4026532008]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 pid -> 'pid:[4026533666]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 pid_for_children -> 'pid:[4026533666]'lrwxrwxrwx 1 root root 0 Feb 06 21:08 user -> 'user:[4026531837]'lrwxrwxrwx 1 root root 0 Feb 06 12:08 uts -> 'uts:[4026533664]'

However running the ‘ip’ command does not find the namespace, as Docker does not create the symlink under the /var/run/netns folder. The ‘ip’ command looked under /var/run/netns for inspecting network namespaces. To allow us to use ‘ip’ tool we can symlink the process as follows

    # (as root)pid=$(docker inspect -f '{{.State.Pid}}' ${container_id})mkdir -p /var/run/netns/ln -sfT /proc/$pid/ns/net /var/run/netns/$container_id# e.g. show stats about eth0 inside the container ip netns exec "${container_id}" ip -s link show eth0

Another alternative is to use the ‘nsenter’ tool, this does not requires creating symlink:

sudo nsenter -t <container_pid> -n <command>

Can use ‘ip monitor’ to look at the netlink activity:

sudo ip monitor

The following output shows the activity performed inside netlink when the command ‘docker network rm test’ is executed

Deleted dev br-b076a0ef8a35 lladdr 02:42:fb:1c:c1:e4 PERMANENTDeleted dev br-b076a0ef8a35 lladdr 02:42:fb:1c:c1:e4 PERMANENT9: br-b076a0ef8a35: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default     link/ether 02:42:fb:1c:c1:e4 brd ff:ff:ff:ff:ff:ffDeleted 224.0.0.22 dev br-b076a0ef8a35 lladdr 01:00:5e:00:00:16 NOARPDeleted 224.0.0.251 dev br-b076a0ef8a35 lladdr 01:00:5e:00:00:fb NOARPDeleted 9: br-b076a0ef8a35    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-b076a0ef8a35       valid_lft forever preferred_lft foreverDeleted local 172.18.0.1 dev br-b076a0ef8a35 table local proto kernel scope host src 172.18.0.1 Deleted inet br-b076a0ef8a35 Deleted inet6 br-b076a0ef8a35 Deleted 9: br-b076a0ef8a35: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default     link/ether 02:42:fb:1c:c1:e4 brd ff:ff:ff:ff:ff:ff

Docker takes care of setting up the network namespace for containers by utilizing the project github.com/vishvananda/netlink. The netlink project provides all the necessary API to setup network infrastructure via netlink
containerd uses the same method to create network for containers, the only difference it uses a different API than Docker. The CNI API is used internally by containerd, the shim
Tools
- nsenter
- ip

First look at the internals of containerd and runc

Sun, 05 Jan 2020 00:00:00 +0000

containerd

Following slides outline the role containerd plays including what kind of services it provides.

Understand what is and isn’t provide inside containerd. This document provide the full scope of the project
History background on the reason why networking was left out from containerd
containerd-shim – After runc runs the container, it exits (allowing us to not have any long-running processes responsible for our containers). The shim is the component which sits between containerd and runc to facilitate this. Containers does not died when dockerd orcontainerd died as it is ‘attached’ to the containerd-shim process. The containerd-shim process job is to monitor stdin(out) and report back the error code returned from exiting the container
Some of containerd Makefile task:
- make bin/containerd-shim – building the containerd-shim app
Following are some explanation about containerd source code:
- cmd/containerd – contains the containerd daemon source code
- cmd/containerd-shim – containerd-shim code
- cmd/containerd-shim-runc-v1 – containerd-shim-v1 code
- cmd/containerd-shim-runc-v2 – containerd-shim-v2 code

Examples how to use containerd

Running ubuntu interactively

Make sure image is pulled using ‘ctr image pull’

Run the following commands to run it interactively and kill

sudo ./ctr run -t docker.io/library/ubuntu:latest u2 [ run the image ]

sudo ./ctr container info ubuntulatest

{    "ID": "ubuntulatest",    "Labels": {        "io.containerd.image.config.stop-signal": "SIGTERM"    },    "Image": "docker.io/library/ubuntu:latest",    "Runtime": {        "Name": "io.containerd.runc.v2",        "Options": {            "type_url": "containerd.runc.v1.Options"        }    },    "SnapshotKey": "ubuntulatest",    "Snapshotter": "overlayfs",    "CreatedAt": "2020-01-01T00:24:30.509643667Z",    "UpdatedAt": "2020-01-01T00:24:30.509643667Z",    "Extensions": null,    "Spec": {        "ociVersion": "1.0.1-dev",        "process": {            "user": {                "uid": 0,                "gid": 0            },            "args": [                "/bin/bash"            ],            "env": [                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"            ],            "cwd": "/",            "capabilities": {                "bounding": [                    "CAP_CHOWN",                    "CAP_DAC_OVERRIDE",                    "CAP_FSETID",                    "CAP_FOWNER",                    "CAP_MKNOD",                    "CAP_NET_RAW",                    "CAP_SETGID",                    "CAP_SETUID",                    "CAP_SETFCAP",                    "CAP_SETPCAP",                    "CAP_NET_BIND_SERVICE",                    "CAP_SYS_CHROOT",                    "CAP_KILL",                    "CAP_AUDIT_WRITE"                ],                "effective": [                    "CAP_CHOWN",                    "CAP_DAC_OVERRIDE",                    "CAP_FSETID",                    "CAP_FOWNER",                    "CAP_MKNOD",                    "CAP_NET_RAW",                    "CAP_SETGID",                    "CAP_SETUID",                    "CAP_SETFCAP",                    "CAP_SETPCAP",                    "CAP_NET_BIND_SERVICE",                    "CAP_SYS_CHROOT",                    "CAP_KILL",                    "CAP_AUDIT_WRITE"                ],                "inheritable": [                    "CAP_CHOWN",                    "CAP_DAC_OVERRIDE",                    "CAP_FSETID",                    "CAP_FOWNER",                    "CAP_MKNOD",                    "CAP_NET_RAW",                    "CAP_SETGID",                    "CAP_SETUID",                    "CAP_SETFCAP",                    "CAP_SETPCAP",                    "CAP_NET_BIND_SERVICE",                    "CAP_SYS_CHROOT",                    "CAP_KILL",                    "CAP_AUDIT_WRITE"                ],                "permitted": [                    "CAP_CHOWN",                    "CAP_DAC_OVERRIDE",                    "CAP_FSETID",                    "CAP_FOWNER",                    "CAP_MKNOD",                    "CAP_NET_RAW",                    "CAP_SETGID",                    "CAP_SETUID",                    "CAP_SETFCAP",                    "CAP_SETPCAP",                    "CAP_NET_BIND_SERVICE",                    "CAP_SYS_CHROOT",                    "CAP_KILL",                    "CAP_AUDIT_WRITE"                ]            },            "rlimits": [                {                    "type": "RLIMIT_NOFILE",                    "hard": 1024,                    "soft": 1024                }            ],            "noNewPrivileges": true        },        "root": {            "path": "rootfs"        },        "mounts": [            {                "destination": "/proc",                "type": "proc",                "source": "proc",                "options": [                    "nosuid",                    "noexec",                    "nodev"                ]            },            {                "destination": "/dev",                "type": "tmpfs",                "source": "tmpfs",                "options": [                    "nosuid",                    "strictatime",                    "mode=755",                    "size=65536k"                ]            },            {                "destination": "/dev/pts",                "type": "devpts",                "source": "devpts",                "options": [                    "nosuid",                    "noexec",                    "newinstance",                    "ptmxmode=0666",                    "mode=0620",                    "gid=5"                ]            },            {                "destination": "/dev/shm",                "type": "tmpfs",                "source": "shm",                "options": [                    "nosuid",                    "noexec",                    "nodev",                    "mode=1777",                    "size=65536k"                ]            },            {                "destination": "/dev/mqueue",                "type": "mqueue",                "source": "mqueue",                "options": [                    "nosuid",                    "noexec",                    "nodev"                ]            },            {                "destination": "/sys",                "type": "sysfs",                "source": "sysfs",                "options": [                    "nosuid",                    "noexec",                    "nodev",                    "ro"                ]            },            {                "destination": "/run",                "type": "tmpfs",                "source": "tmpfs",                "options": [                    "nosuid",                    "strictatime",                    "mode=755",                    "size=65536k"                ]            }        ],        "linux": {            "resources": {                "devices": [                    {                        "allow": false,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 1,                        "minor": 3,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 1,                        "minor": 8,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 1,                        "minor": 7,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 5,                        "minor": 0,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 1,                        "minor": 5,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 1,                        "minor": 9,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 5,                        "minor": 1,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 136,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 5,                        "minor": 2,                        "access": "rwm"                    },                    {                        "allow": true,                        "type": "c",                        "major": 10,                        "minor": 200,                        "access": "rwm"                    }                ]            },            "cgroupsPath": "/default/ubuntulatest",            "namespaces": [                {                    "type": "pid"                },                {                    "type": "ipc"                },                {                    "type": "uts"                },                {                    "type": "mount"                },                {                    "type": "network"                }            ],            "maskedPaths": [                "/proc/acpi",                "/proc/asound",                "/proc/kcore",                "/proc/keys",                "/proc/latency_stats",                "/proc/timer_list",                "/proc/timer_stats",                "/proc/sched_debug",                "/sys/firmware",                "/proc/scsi"            ],            "readonlyPaths": [                "/proc/bus",                "/proc/fs",                "/proc/irq",                "/proc/sys",                "/proc/sysrq-trigger"            ]        }    }}

sudo ./ctr t metrics ubuntulatest

ID              TIMESTAMP                                  ubuntulatest    2020-01-01 00:29:16.295322149 +0000 UTC    METRIC                   VALUE                                                                                memory.usage_in_bytes    1433600                                                                              memory.limit_in_bytes    9223372036854771712                                                                  memory.stat.cache        0                                                                                    cpuacct.usage            19273664                                                                             cpuacct.usage_percpu     [205817 2386548 4771841 148926 5529908 227097 2037362 0 2107441 0 1017213 841511]    pids.current             1                                                                                    pids.limit               0

sudo ./ctr shim –id ubuntulatest state

{    "id": "ubuntulatest",    "bundle": "/run/containerd/io.containerd.runtime.v2.task/default/ubuntulatest",    "pid": 16032,    "status": 2,    "stdin": "/run/containerd/fifo/249557939/ubuntulatest-stdin",    "stdout": "/run/containerd/fifo/249557939/ubuntulatest-stdout",    "stderr": "/run/containerd/fifo/249557939/ubuntulatest-stderr",    "exited_at": "0001-01-01T00:00:00Z"}

sudo ./ctr t kill u2 [ kill the container labelled u2 ]

Another example to download hello world

sudo ./ctr image pull docker.io/library/hello-world:latest

docker.io/library/hello-world:latest:                                             resolved       |++++++++++++++++++++++++++++++++++++++| index-sha256:4fe721ccc2e8dc7362278a29dc660d833570ec2682f4e4194f4ee23e415e1064:    done           |++++++++++++++++++++++++++++++++++++++| manifest-sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a: done           |++++++++++++++++++++++++++++++++++++++| layer-sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced:    done           |++++++++++++++++++++++++++++++++++++++| config-sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e:   done           |++++++++++++++++++++++++++++++++++++++| elapsed: 3.7 s                                                                    total:  4.8 Ki (1.3 KiB/s)                                       unpacking linux/amd64 sha256:4fe721ccc2e8dc7362278a29dc660d833570ec2682f4e4194f4ee23e415e1064...

sudo ./ctr container create docker.io/library/hello-world:latest demo
sudo ./ctr task start demo

containerd utilise kernel feature called ‘reaper’ to reparent the container proces to the shimFollowing shows the process structure when running a container using containerd

nanik     7741  3230  0  2019 ?        00:02:54  \_ /usr/libexec/gnome-terminal-servernanik     7750  7741  0  2019 pts/1    00:00:00  |   \_ bash..........nanik    19294  7741  0 13:45 pts/5    00:00:00  |   \_ bashroot      5123 19294  0 17:51 pts/5    00:00:00  |   |   \_ sudo ./ctr run -t docker.io/library/ubuntu:latest u13root      5124  5123  0 17:51 pts/5    00:00:00  |   |       \_ ./ctr run -t docker.io/library/ubuntu:latest u13nanik    18313  7741  0 16:11 pts/11   00:00:00  |   \_ bash.........................root      5884  3230  0 17:52 ?        00:00:00  \_ /usr/bin/containerd-shim-runc-v2 -namespace default -id u13 -address /run/containerd/containerd.sockroot      5906  5884  0 17:52 ?        00:00:00      \_ /bin/bash..........

As can be seen containerd uses shim called ‘containerd-shim-run-v2’. Runc has been terminated after running the container and the shim takes over as the parent of the container. Containerd supports shim v2

The shim is executed out-of-process (executed with exec(..)) and the following are used to execute it:

 0 = {string} "-namespace" 1 = {string} "default" 2 = {string} "-address" 3 = {string} "/run/containerd/containerd.sock" 4 = {string} "-publish-binary" 5 = {string} "/tmp/___containerd" 6 = {string} "-id" 7 = {string} "u8" 8 = {string} "-debug" 9 = {string} "start"

The /tmp/__containerid contains the containerd executable.

Comment by Michael Crosby about shim

The shim allows for daemonless containers.  It basically sits as the parent of the container's process to facilitate a few things.First it allows the runtimes, i.e. runc,to exit after it starts the container.  This way we don't have to have the long running runtime processes for containers.  When you start mysql you should only see the mysql process and the shim.Second it keeps the STDIO and other fds open for the container incase containerd and/or docker both die.  If the shim was not running then the parent side of the pipes or the TTY master would be closed and the container would exit.  Finally it allows the container's exit status to be reported back to a higher level tool like docker without having the be the actual parent of the container's process and do a wait.

containerd uses FIFO for reporting event and exit code and also for stdout and stdin

/run/containerd/fifo/195093460/<something_something>_stdout/run/containerd/fifo/195093460/<something_something>_stdin

runc

A lightweight binary that supports the OCI runtime-spec for running containers. Deals with the low-level interfacing with Linux capabilities like cgroups, namespaces, etc…
runc looked for temp directory using “XDG_RUNTIME_DIR” eg:/run/user/1000/runc/
runc have heavy dependencies on libcontainer.

How ‘runc’ is used/executed inside containerd ?. Following are some explanation:

Containerd runs as a server where it receive GRPC command. The ctr CLI tool is the way to send command to run, stop, etc containers in containerd
When containerd receive command as suchsudo ./ctr run -t docker.io/library/ubuntu:latest u67, it will go through services/tasks/service.go source code to prepare all the necessary data to spin off ‘containerd-shim-runc-v2’ executable.

Following is an example of the executable argument prepared when executing ‘containerd-shim-runc-v2’

 0 = {string} "-namespace" 1 = {string} "default" 2 = {string} "-address" 3 = {string} "/run/containerd/containerd.sock" 4 = {string} "-publish-binary" 5 = {string} "/tmp/___containerd" 6 = {string} "-id" 7 = {string} "u8" 8 = {string} "-debug" 9 = {string} "start"

Full command used – /usr/bin/containerd-shim-runc-v2 -namespace default -id u67 -address /run/containerd/containerd.sock

Following are the log output (debug log were added to trace soure code) when ‘containerd-shim-runc-v2’ is running:

time="2020-01-02T23:38:00.438682328+11:00" level=info msg=setupDumpStacks...time="2020-01-02T23:38:00.438927931+11:00" level=info msg="calling newServer..."time="2020-01-02T23:38:00.439033554+11:00" level=info msg="registering ttrpc server"time="2020-01-02T23:38:00.439097725+11:00" level=info msg="calling serve..."time="2020-01-02T23:38:00.439198636+11:00" level=info msg="calling handleSignals..."time="2020-01-02T23:38:00.439220268+11:00" level=info msg="nanik starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/u67 pid=17162time="2020-01-02T23:38:00.439861913+11:00" level=info msg="Create is called inside  RegisterTaskService"time="2020-01-02T23:38:00.439974576+11:00" level=info msg="container NANIK "time="2020-01-02T23:38:00.440214591+11:00" level=info msg="--- CreateTaskRequest  &CreateTaskRequest{ID:u67,Bundle:/run/containerd/io.containerd.runtime.v2.task/default/u67,Rootfs:[&types.Mount{Type:overlay,Source:overlay,Target:,Options:[workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/133/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/133/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs],XXX_unrecognized:[],}],Terminal:true,Stdin:/run/containerd/fifo/383837931/u67-stdin,Stdout:/run/containerd/fifo/383837931/u67-stdout,Stderr:,Checkpoint:,ParentCheckpoint:,Options:&types1.Any{TypeUrl:containerd.runc.v1.Options,Value:[],XXX_unrecognized:[],},XXX_unrecognized:[],}"time="2020-01-02T23:38:00.440329033+11:00" level=info msg="--- rootfs  /run/containerd/io.containerd.runtime.v2.task/default/u67/rootfs"time="2020-01-02T23:38:00.440350982+11:00" level=info msg="--- opts.BinaryName  "time="2020-01-02T23:38:00.440367236+11:00" level=info msg="--- opts.Bundle  /run/containerd/io.containerd.runtime.v2.task/default/u67"time="2020-01-02T23:38:00.441364775+11:00" level=info msg="--- calling p.create with context.Background.WithValue(type namespaces.namespaceKey, val default).WithValue(type metadata.mdOutgoingKey, val <not Stringer>).WithValue(type ttrpc.metadataKey, val <not Stringer>).WithValue(type shim.OptsKey, val <not Stringer>).WithValue(type log.loggerKey, val <not Stringer>).WithCancel.WithCancel AND ... "time="2020-01-02T23:38:00.441809346+11:00" level=info msg="------- inside init.gocontext.Background.WithValue(type namespaces.namespaceKey, val default).WithValue(type metadata.mdOutgoingKey, val <not Stringer>).WithValue(type ttrpc.metadataKey, val <not Stringer>).WithValue(type shim.OptsKey, val <not Stringer>).WithValue(type log.loggerKey, val <not Stringer>).WithCancel.WithCancel/run/containerd/io.containerd.runtime.v2.task/default/u67&{<nil> /run/containerd/io.containerd.runtime.v2.task/default/u67/init.pid 0xc000154520 false false false []}" runtime=io.containerd.runc.v2time="2020-01-02T23:38:00.503607036+11:00" level=info msg="Start is called inside  RegisterTaskService"time="2020-01-02T23:38:00.503640676+11:00" level=info msg="v2/service Start"ime="2020-01-02T17:33:35.454812846+11:00" level=info msg="v2/service Delete"

The final function that will execute ‘runc’ is inside containerd/go-runc/runc.go

func (r *Runc) Create(context context.Context, id, bundle string, opts *CreateOpts) error {}

Logging code was added inside the Create(..) function and following is the output:

--- args  [create --bundle /run/containerd/io.containerd.runtime.v2.task/default/u67]--- cmd  /home/nanik/AndroidProjects/docker/docker/runc --root /run/containerd/runc/default --log /run/containerd/io.containerd.runtime.v2.task/default/u67/log.json --log-format json create--bundle /run/containerd/io.containerd.runtime.v2.task/default/u67 --pid-file /run/containerd/io.containerd.runtime.v2.task/default/u67/init.pid --console-socket /tmp/pty415594316/pty.sock u67The command used to execute 'runc' is as follows"/home/nanik/AndroidProjects/docker/docker/runc --root /run/containerd/runc/default --log /run/containerd/io.containerd.runtime.v2.task/default/u67/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/default/u67 --pid-file /run/containerd/io.containerd.runtime.v2.task/default/u67/init.pid --console-socket /tmp/pty415594316/pty.sock u67"

To use runc to see docker containers that are running

sudo ./runc --root /run/docker/runtime-runc/moby  list ID                                                                 PID         STATUS      BUNDLE                                                                                                                 CREATED                          OWNERf182f95645673b94af95495ea4c2a7c0f58dcce523f3d4e7174d7e482e136e08   12212       running     /run/containerd/io.containerd.runtime.v1.linux/moby/f182f95645673b94af95495ea4c2a7c0f58dcce523f3d4e7174d7e482e136e08   2020-01-05T21:28:05.371871841Z   root

Good explanation from here

The runtime (runc) uses so-called runtime root directory to store and obtain the information about containers. Under this root directory, runc places sub-directories (one per container), and each of them contains the state.json file, where the container state description resides.The default location for runtime root directory is either /run/runc (for non-rootless containers) or $XDG_RUNTIME_DIR/runc (for rootless containers) - the latter also usually points to somewhere under /run (e.g. /run/user/$UID/runc).When the container engine invokes runc, it may override the default runtime root directory and specify the custom one (--root option of runc). Docker uses this possibility, e.g. on my box, it specifies /run/docker/runtime-runc/moby as the runtime root.That said, to make runc list see your Docker containers, you have to point it to Docker's runtime root directory by specifying --root option. Also, given that Docker containers are not rootless by default, you will need the appropriate privileges to access the runtime root (e.g. with sudo).    So, that's how this should work:    $ docker run -d alpine sleep 1000    4acd4af5ba8da324b7a902618aeb3fd0b8fce39db5285546e1f80169f157fc69    $ sudo runc --root /run/docker/runtime-runc/moby/ list    ID                                                                 PID         STATUS      BUNDLE                                                                                                                               CREATED                          OWNER    4acd4af5ba8da324b7a902618aeb3fd0b8fce39db5285546e1f80169f157fc69   18372       running     /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/4acd4af5ba8da324b7a902618aeb3fd0b8fce39db5285546e1f80169f157fc69   2019-07-12T17:33:23.401746168Z   rootAs to images, you can not make runc see them, as it has no notion of image at all - instead, it operates on bundles. Creating the bundle (e.g. based on image) is responsibility of the caller (in your case - containerd).

libcontainer

Library used inside runc for container operation
Explanation about libcontainer, runc and nsenter. More in-depth explanation

Docker and lower level

Docker CLI (docker) - /usr/bin/docker
Docker is used as a reference to the whole set of docker tools and at the beginning it was a monolith. But now docker-cli is only responsible for user friendly communication with docker.
So the command’s like docker build … docker run … are handled by Docker CLI and result in the invocation of dockerd API.
Dockerd - /usr/bin/dockerd
The Docker daemon - dockerd listens for Docker API requests and manages host’s Container life-cycles by utilizing contanerd
dockerd can listen for Docker Engine API requests via three different types of Socket: unix, tcp, and fd. By default, a unix domain socket is created at /var/run/docker.sock, requiring either root permission, or docker group membership. On Systemd based systems, you can communicate with the daemon via Systemd socket activation, use dockerd -H fd://.
There are many configuration options for the daemon, which are worth to check if you work with docker (dockerd).
My impression is that dockerd is here to serve all the features of Docker (or Docker EE) platform, while actual container life-cycle management is “outsourced” to containerd.Containerd
containerd - /usr/bin/docker-containerd
containerd was introduced in Docker 1.11 and since then took main responsibilty of managing containers life-cycle. containerd is the executor for containers, but has a wider scope than just executing containers. So it also take care of:
```
Image push and pullManaging of storageOf course executing of Containers by calling runc with the right parameters to run containers...Managing of network primitives for interfacesManagement of network namespaces containers to join existing namespaces      
```
containerd fully leverages the OCI runtime specification1, image format specifications and OCI reference implementation (runc). Because of its massive adoption, containerd is the industry standard for implementing OCI. It is currently available for Linux and Windows.
RunC - /usr/bin/docker-runc runc (OCI runtime) can be seen as component of containerd.
runc is a command line client for running applications packaged according tothe OCI format and is a compliant implementation of the OCI spec.
Containers are configured using bundles. A bundle for a container is a directorythat includes a specification file named “config.json” and a root filesystem.The root filesystem contains the contents of the container.
Assuming you have an OCI bundle you can execute the container
containerd-ctr - /usr/bin/docker-containerd-ctr (docker-)containerd-ctr
it’s barebone CLI (ctr) designed specifically for development and debugging purpose for direct communication with containerd. It’s included in the releases of containerd. By that less interesting for docker users.

containerd-shim - /usr/bin/docker-containerd-shim

The shim allows for daemonless containers. According to Michael Crosby it’s basically sits as the parent of the container’s process to facilitate a few things.

First it allows the runtimes, i.e. runc,to exit after it starts the container. This way we don't have to have the long running runtime processes for containers.Second it keeps the STDIO and other fds open for the container in case containerd and/or docker both die. If the shim was not running then the parent side of the pipes or the TTY master would be closed and the container would exit.Finally it allows the container's exit status to be reported back to a higher level tool like docker without having the be the actual parent of the container's process and do a wait.

Complete interaction between docker cli, dockerd, containerd, containerd-shim and runc

dockerd is sent POST Containers Create    ↳ dockerd finds the requested image    ↳ A container object is created and stored for future use    ↳ Directories on the file system are setup for use by the containerdockerd is sent a POST Containers Start    ↳ An OCI spec is created for the container    ↳ containerd is contacted to create the container        ↳ containerd stores the container spec in a database    ↳ containerd is contacted to start the container        ↳  containerd creates a task for the container            ↳  The task uses a shim to call runc create        ↳  containerd starts the task            ↳  The task uses the shim to call runc start        ↳ The shim / containerd continue to monitor the container until completion

Running container with runc

This following is step-by-step example on how to run OCI compliant image using runc. We going to use docker in this example.

Checkout the runc project from https://github.com/opencontainers/runc and build it by running make
Download exportrootfs.sh from https://github.com/estesp/utils/blob/master/exportrootfs.sh and add it to your PATH. Make sure follow the instruction inside the script to compile uidmapshift.c and include that too in the PATH
Make sure you have pull ubuntu:latest image using docker
Run the image using the docker run command:
```
docker run -it ubuntu:latest /bin/bash
```

Get the container id using docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMESebfbbecaf715        ubuntu:latest       "/bin/bash"         38 minutes ago      Up 38 minutes                           zen_kirch

Create another separate directory and cd into that directory. Run the following command
```
sudo env "PATH=$PATH" exportrootfs.sh -u 0 -r 65536 ebf
```
ebf is the container id shown in the output of docker ps

You will have a roootfs directory in your current directory and it will look like the following

rootfs├── bin├── boot├── dev├── etc├── home├── lib├── lib64├── media├── mnt├── opt├── proc├── root├── run├── sbin├── srv├── sys├── tmp├── usr└── var

Create the runtime spec using the command
```
runc spec
```
You will see a new file called config.json
Open config.json and modify the args to the following
```
    "args": [    "/bin/bash"],
```

Execute the image using the following

    sudo env "PATH=$PATH" runc run anycontainername

You will see bash running

    root@runc:/# cat /etc/lsb-release DISTRIB_ID=UbuntuDISTRIB_RELEASE=18.04DISTRIB_CODENAME=bionicDISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"root@runc:/#

As can be seen the runc does not know how to pull, prepare, etc the image. It just knows that there is a root fileysystem with the config.json that it needs to run. The ubuntu container ran by the above example does not have network as this will be taken care by some other project and not by runc.

runc utilize prestart hooks to run some other application required as part of the setup of the containers, as shown in here. The config.json

   .   .   .   "hooks": {        "prestart" : [            {                "path" : "/path/to/netns",                "args" : [                    "",                    "--state-dir", "/path/to/netns/netns-state"                ]            }        ]    },    .    .    .

shows the prestart hook that will be executed to setup the networking state using the netns executable. The netns tool is part of the genuinetools project

References

Concurrency Design Pattern in Go

Mon, 02 Dec 2019 00:00:00 +0000

Design Patterns

Concurrency in Golang is quite simple, and it’s a powerful tool. Understanding concurrency basic is important before jumping any further. Once basic understanding has been established the next step is to go deeper by looking at using different kind of concurrency patterns. Understanding different concurrency pattern is important to make it easier in picking up the right solution to a certain concurrency problem that we are trying to tackle. Also, it will reduce boilerplate code .

Pipeline

The cascading effect of closing channel (from close(done)) will ensure that all the other channels inside generator(..), add(..) and multiply (..) will also closed.

func main(){    generator := func(done <-chan interface{}, integers ...int) <-chan int {        intStream := make(chan int)        go func() {            defer close(intStream)            for _, i := range integers {                select {                case <-done:                    return                case intStream <- i:                }            }        }()        return intStream    }    multiply := func(        done <-chan interface{},        intStream <-chan int,        multiplier int,    ) <-chan int {        multipliedStream := make(chan int)        go func() {            defer close(multipliedStream)            for i := range intStream {                select {                case <-done:                    return                case multipliedStream <- i*multiplier:                }            }        }()        return multipliedStream    }    add := func(        done <-chan interface{},        intStream <-chan int,        additive int,    ) <-chan int {        addedStream := make(chan int)        go func() {            defer close(addedStream)            for i := range intStream {                select {                case <-done:                    return                case addedStream <- i+additive:                }            }        }()        return addedStream    }    done := make(chan interface{})    defer close(done)    intStream := generator(done, 1, 2, 3, 4)    pipeline := multiply(done, add(done, multiply(done, intStream, 2), 1), 2)    for v := range pipeline {        fmt.Println(v)    } }

Resources:

https://blog.golang.org/pipelines

Tee-Channel

Takes a single input channel and an arbitrary number of output channels and duplicates each input into every output. When the input channel is closed, all outputs channels are closed. The eapache channels project provide this kind of pattern. Following code snippet is the implementation.

func tee(input SimpleOutChannel, outputs []SimpleInChannel, closeWhenDone bool) {cases := make([]reflect.SelectCase, len(outputs))for i := range cases {cases[i].Dir = reflect.SelectSend}for elem := range input.Out() {for i := range cases {cases[i].Chan = reflect.ValueOf(outputs[i].In())cases[i].Send = reflect.ValueOf(elem)}for _ = range cases {chosen, _, _ := reflect.Select(cases)cases[chosen].Chan = reflect.ValueOf(nil)}}if closeWhenDone {for i := range outputs {outputs[i].Close()}}}

The library uses Golang’s internal SelectCase package to process multiple channels.

type SelectCase struct {    Dir  SelectDir // direction of case    Chan Value     // channel to use (for send or receive)    Send Value     // value to send (for send)}

The SelectCase structure is like a container of channels depending on the direction specified (Dir).

Fan-in-Fan-out channel

This pattern collate results from goroutines that have been spawned to do background processing.

package mainimport (    "fmt"    "time")func main() {    ch := fanIn(generator("Hello"), generator("Bye"))    for i := 0; i < 10; i++ {        fmt.Println(<- ch)    }}// fanIn is itself a generatorfunc fanIn(ch1, ch2 <-chan string) <-chan string { // receives two read-only channels    new_ch := make(chan string)    go func() { for { new_ch <- <-ch1 } }() // launch two goroutine while loops to continuously pipe to new channel    go func() { for { new_ch <- <-ch2 } }()    return new_ch}func generator(msg string) <-chan string { // returns receive-only channel    ch := make(chan string)    go func() { // anonymous goroutine        for i := 0; ; i++ {            ch <- fmt.Sprintf("%s %d", msg, i)            time.Sleep(time.Second)        }    }()    return ch}

Context

Good for cancelling / terminating goroutine by propagation. In order for this to work effectively the context must be send as part of argument parameters to each of the goroutines. The Done() inside locale(..) method is called as the context.WithTimeout(..) got triggered after 1 second and it propagates all the way to the root context

func main() {    var wg sync.WaitGroup    ctx, cancel := context.WithCancel(context.Background())    defer cancel()    wg.Add(1)    go func() {        defer wg.Done()        if err := printGreeting(ctx); err != nil {            fmt.Printf("cannot print greeting: %v\n", err)            cancel()        }    }()    wg.Add(1)    go func() {        defer wg.Done()        if err := printFarewell(ctx); err != nil {            fmt.Printf("cannot print farewell: %v\n", err)        }    }()    wg.Wait()}func printGreeting(ctx context.Context) error {    greeting, err := genGreeting(ctx)    if err != nil {        return err    }    fmt.Printf("%s world!\n", greeting)    return nil}func printFarewell(ctx context.Context) error {    farewell, err := genFarewell(ctx)    if err != nil {        return err    }    fmt.Printf("%s world!\n", farewell)    return nil}func genGreeting(ctx context.Context) (string, error) {    ctx, cancel := context.WithTimeout(ctx, 1*time.Second)    defer cancel()    switch locale, err := locale(ctx); {    case err != nil:        return "", err    case locale == "EN/US":        return "hello", nil    }    return "", fmt.Errorf("unsupported locale")}func genFarewell(ctx context.Context) (string, error) {    switch locale, err := locale(ctx); {    case err != nil:        return "", err    case locale == "EN/US":        return "goodbye", nil    }    return "", fmt.Errorf("unsupported locale")}func locale(ctx context.Context) (string, error) {    select {    case <-ctx.Done():        return "", ctx.Err()    case <-time.After(1 * time.Minute):        fmt.Println("Expired")    }    return "EN/US", nil}

Bridge Channel

..........................

Error Propagation

..........................

Concurrency Project

This section will discuss in detail about the go-resiliency project. This project is a very useful project to learn more in-depth about concurrency. There are several different implementations and patterns it implemented that are useful to use in an application. The other project that is useful to learn is the channels project, this project is useful to learn different ways using Golang channels.

Blackhole

The function of the class is to receive data and discard it and calculated the total number of data it received. This is useful in situation where we want to check if our app concurrency app is receiving the same amount of data as expected.

package channels// BlackHole implements the InChannel interface and provides an analogue for the "Discard" variable in// the ioutil package - it never blocks, and simply discards every value it reads. The number of items// discarded in this way is counted and returned from Len.type BlackHole struct {input  chan interface{}length chan intcount  int}func NewBlackHole() *BlackHole {ch := &BlackHole{input:  make(chan interface{}),length: make(chan int),}go ch.discard()return ch}func (ch *BlackHole) In() chan<- interface{} {return ch.input}func (ch *BlackHole) Len() int {val, open := <-ch.lengthif open {return val} else {return ch.count}}func (ch *BlackHole) Cap() BufferCap {return Infinity}func (ch *BlackHole) Close() {close(ch.input)}func (ch *BlackHole) discard() {for {select {case _, open := <-ch.input:if !open {close(ch.length)return}ch.count++case ch.length <- ch.count:}}}

Following is a code sample on how to use Blackhole

func TestBlackHole(t *testing.T) {discard := NewBlackHole()for i := 0; i < 1000; i++ {discard.In() <- i}discard.Close()if discard.Len() != 1000 {t.Error("blackhole expected 1000 was", discard.Len())}}

Deadline

There are instances where we want our application to do a background process but there is a time limit imposed on it. Following is a sample snippet on how to use this functionality

func ExampleDeadline() {dl := New(1 * time.Second)err := dl.Run(func(stopper <-chan struct{}) error {// do something possibly slow// check stopper function and give up if timed outreturn nil})switch err {case ErrTimedOut:// execution took too long, oopsdefault:// some other error}}

Circuit Breaker

Implementation is a simple counter measurement of success and failure of a particular task. The only goroutine used is to run the timer to make sure that that there some time limit imposed. The different states - closed, half-open and open

closed – close the circuit breaker as all are good
half-open – transition to this state happens when the timeout timer has expired
open – transition to this state is triggered when error happens, there could be 2 different scenarios - no of errors reached errors threshold or the current state is half-open (timeout occured)

The function processResult(..) is the main logic performing the transition to different states based on the success and errors threshold criteria.

func (b *Breaker) processResult(result error, panicValue interface{}) {b.lock.Lock()defer b.lock.Unlock()if result == nil && panicValue == nil {if b.state == halfOpen {b.successes++if b.successes == b.successThreshold {b.closeBreaker()}}} else {if b.errors > 0 {expiry := b.lastError.Add(b.timeout)if time.Now().After(expiry) {b.errors = 0}}switch b.state {case closed:b.errors++if b.errors == b.errorThreshold {b.openBreaker()} else {b.lastError = time.Now()}case halfOpen:b.openBreaker()}}}

The following diagram explains in detail how the logic works inside processResult. The diagram breaks down the logic into 2 different category - success and errors. Each category have different logic to transition to different states.

Batcher

Below is a high level diagram on how the design pattern interact with the different pieces internally.

This pattern handle situation where we need to do a particular task but we need to wait X amount of time before running it. For example - we are waiting for all parameteres to be collected together in 1 minute before executing a particular function/task for processing the parameter.

Code uses 2 different channels:

future – for communication err value received from the executed task
submit – this channel is used as part of the timeout feature. The channel will be pushed with
- parameter that need to be collected to be sent to the targeted task and
- future channels that will be used to send the error value

The submitWork method is the method that is called when there is a duration specified for the timeout as the task need to be executed asynchronously.

func (b *Batcher) submitWork(w *work) {b.lock.Lock()defer b.lock.Unlock()if b.submit == nil {b.submit = make(chan *work, 4)go b.batch()}b.submit <- w}

The batch() method is the crux of the whole process. The code will keep looping the b.submit channel to extract the params interface{} and future channel.Code will exit the loop when the b.submit channel is closed by the the b.timer() function (as it expired)

Once it’s out of the loop it will execute the task by calling b.doWork(..) and on completion of the task the error obtained (err) will be push to all the future channels.

func (b *Batcher) batch() {var params []interface{}var futures []chan errorinput := b.submitgo b.timer()for work := range input {params = append(params, work.param)   futures = append(futures, work.future)}ret := b.doWork(params)for _, future := range futures {future <- retclose(future)}}

Following diagram outlines the different channels used for this pattern

Deploying k8guard (Work in progress)

Sun, 01 Dec 2019 00:00:00 +0000

Deploying k8guard

Make sure minikube and kubectl is in PATH
Use the following command to start (v1.11.10 is the lowest version possible that will work with minikube master branch)
```
minikube start --memory 4096 --kubernetes-version v1.11.10 
```
```
eval $(minikube docker-env)
```
```
make build-deploy-minikube
```

Deploy using the following commands

kubectl apply -f ./minikube/report/k8guard-report-secrets.yaml.EXAMPLE

kubectl apply -f ./minikube/action/k8guard-action-secrets.yaml.EXAMPLE

k8guard-action

k8guard-action-configmap.yaml         |         |         |    k8guard-action-deployment.yam             |         |         |          k8guard-action-secrets.yaml.EXAMPLE

File k8guard-action-deployment.yaml contains deployment information for the k8guard-action app. It uses docker image called local/k8guard-action which is build doing the above mentioned step.
The env: section declared the env variables that will be populated in the EXPORT section of the bash. The value is obtained from the k8guard-action-configmap.yaml
The k8guard-action-configmap.yaml contains the config map for the different variables

The following environment variables are found inside the k8guard-action container. These values are populated via the .env file

export HOME='/root'export HOSTNAME='k8guard-action-deployment-67c4db498-j4hvd'export K8GUARD_ACTION_CASSANDRA_CAPATH=''export K8GUARD_ACTION_CASSANDRA_HOSTS='k8guard-cassandra-service.default.svc.cluster.local:9042'export K8GUARD_ACTION_CASSANDRA_KEYSPACE='k8guardkeyspace'export K8GUARD_ACTION_CASSANDRA_PASSWORD='cassandra'export K8GUARD_ACTION_CASSANDRA_SSL_HOST_VALIDATION='false'export K8GUARD_ACTION_CASSANDRA_USERNAME='cassandra'export K8GUARD_ACTION_DRY_RUN='false'export K8GUARD_ACTION_DURATION_BETWEEN_CHAT_NOTIFICATIONS='30s'export K8GUARD_ACTION_DURATION_BETWEEN_NOTIFYING_AGAIN='24h'export K8GUARD_ACTION_DURATION_VIOLATION_EXPIRES='120h'export K8GUARD_ACTION_HIPCHAT_BASE_URL=''export K8GUARD_ACTION_HIPCHAT_ROOM_ID=''export K8GUARD_ACTION_HIPCHAT_TAG_NAMESPACE_OWNER='true'export K8GUARD_ACTION_HIPCHAT_TOKEN='REPLACE_ME'export K8GUARD_ACTION_SAFE_MODE='true'export K8GUARD_ACTION_SMTP_FALLBACK_SEND_TO='REPLACE@REPLACE_WITH_DOMAIN.COM'export K8GUARD_ACTION_SMTP_PORT='25'export K8GUARD_ACTION_SMTP_SEND_FROM='DO_NOT_REPLY@REPLACE_WITH_DOMAIN.COM'export K8GUARD_ACTION_SMTP_SEND_TO_NAMESAPCE_OWNER='true'export K8GUARD_ACTION_SMTP_SERVER=''export K8GUARD_ACTION_VIOLATION_EMAIL_FOOTER=''export K8GUARD_ACTION_WARNING_COUNT_BEFORE_ACTION='4'export K8GUARD_CASSANDRA_CREATE_KEYSPACE='true'export K8GUARD_CASSANDRA_CREATE_TABLES='true'export K8GUARD_CASSANDRA_SERVICE_PORT='tcp://10.99.115.177:9042'export K8GUARD_CASSANDRA_SERVICE_PORT_9042_TCP='tcp://10.99.115.177:9042'export K8GUARD_CASSANDRA_SERVICE_PORT_9042_TCP_ADDR='10.99.115.177'export K8GUARD_CASSANDRA_SERVICE_PORT_9042_TCP_PORT='9042'export K8GUARD_CASSANDRA_SERVICE_PORT_9042_TCP_PROTO='tcp'export K8GUARD_CASSANDRA_SERVICE_SERVICE_HOST='10.99.115.177'export K8GUARD_CASSANDRA_SERVICE_SERVICE_PORT='9042'export K8GUARD_CLUSTER_NAME='minikube'export K8GUARD_DISCOVER_SERVICE_PORT='tcp://10.103.122.8:3000'export K8GUARD_DISCOVER_SERVICE_PORT_3000_TCP='tcp://10.103.122.8:3000'export K8GUARD_DISCOVER_SERVICE_PORT_3000_TCP_ADDR='10.103.122.8'export K8GUARD_DISCOVER_SERVICE_PORT_3000_TCP_PORT='3000'export K8GUARD_DISCOVER_SERVICE_PORT_3000_TCP_PROTO='tcp'export K8GUARD_DISCOVER_SERVICE_SERVICE_HOST='10.103.122.8'export K8GUARD_DISCOVER_SERVICE_SERVICE_PORT='3000'export K8GUARD_KAFKA_ACTION_TOPIC='k8guard-to-action-k8s-lab'export K8GUARD_KAFKA_BROKERS='k8guard-kafka-service.default.svc.cluster.local:9092'export K8GUARD_KAFKA_SERVICE_PORT='tcp://10.106.168.209:2181'export K8GUARD_KAFKA_SERVICE_PORT_2181_TCP='tcp://10.106.168.209:2181'export K8GUARD_KAFKA_SERVICE_PORT_2181_TCP_ADDR='10.106.168.209'export K8GUARD_KAFKA_SERVICE_PORT_2181_TCP_PORT='2181'export K8GUARD_KAFKA_SERVICE_PORT_2181_TCP_PROTO='tcp'export K8GUARD_KAFKA_SERVICE_PORT_9092_TCP='tcp://10.106.168.209:9092'export K8GUARD_KAFKA_SERVICE_PORT_9092_TCP_ADDR='10.106.168.209'export K8GUARD_KAFKA_SERVICE_PORT_9092_TCP_PORT='9092'export K8GUARD_KAFKA_SERVICE_PORT_9092_TCP_PROTO='tcp'export K8GUARD_KAFKA_SERVICE_SERVICE_HOST='10.106.168.209'export K8GUARD_KAFKA_SERVICE_SERVICE_PORT='2181'export K8GUARD_KAFKA_SERVICE_SERVICE_PORT_KAFKA='9092'export K8GUARD_KAFKA_SERVICE_SERVICE_PORT_ZK='2181'export K8GUARD_LOG_LEVEL='debug'export K8GUARD_MEMCACHED_SERVICE_PORT='tcp://10.100.96.55:11211'export K8GUARD_MEMCACHED_SERVICE_PORT_11211_TCP='tcp://10.100.96.55:11211'export K8GUARD_MEMCACHED_SERVICE_PORT_11211_TCP_ADDR='10.100.96.55'export K8GUARD_MEMCACHED_SERVICE_PORT_11211_TCP_PORT='11211'export K8GUARD_MEMCACHED_SERVICE_PORT_11211_TCP_PROTO='tcp'export K8GUARD_MEMCACHED_SERVICE_SERVICE_HOST='10.100.96.55'export K8GUARD_MEMCACHED_SERVICE_SERVICE_PORT='11211'export K8GUARD_REPORT_SERVICE_PORT='tcp://10.105.186.144:3001'export K8GUARD_REPORT_SERVICE_PORT_3001_TCP='tcp://10.105.186.144:3001'export K8GUARD_REPORT_SERVICE_PORT_3001_TCP_ADDR='10.105.186.144'export K8GUARD_REPORT_SERVICE_PORT_3001_TCP_PORT='3001'export K8GUARD_REPORT_SERVICE_PORT_3001_TCP_PROTO='tcp'export K8GUARD_REPORT_SERVICE_SERVICE_HOST='10.105.186.144'export K8GUARD_REPORT_SERVICE_SERVICE_PORT='3001'export KUBERNETES_PORT='tcp://10.96.0.1:443'export KUBERNETES_PORT_443_TCP='tcp://10.96.0.1:443'export KUBERNETES_PORT_443_TCP_ADDR='10.96.0.1'export KUBERNETES_PORT_443_TCP_PORT='443'export KUBERNETES_PORT_443_TCP_PROTO='tcp'export KUBERNETES_SERVICE_HOST='10.96.0.1'export KUBERNETES_SERVICE_PORT='443'export KUBERNETES_SERVICE_PORT_HTTPS='443'export OLDPWD='/var/log'export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'export PWD='/var'export SHLVL='1'export TERM='xterm'

Minikube Brain Dump (ALWAYS WIP)

Sun, 01 Dec 2019 00:00:00 +0000

Minikube supports lxc where it has it’s own Go bindings called libvirt-go.
Internally libvirt-go provides Go binding for different containers as virtualbox, kvm, etc
Minikube uses different way to interact with different containers as outlined below:
- virtualbox – Calling executable /usr/bin/VboxManage
- kvm2 – Calling docker-machine-driver-kvm2 which is part of minikube and can be found under cmd/drivers/kvm/main.go. The code is using the libvirt to communicate with kvm
- gvisor – there is some sort of dependencies with containerd and rpc-statd.service as this both service is restarted when gvisor has been download and successfully copy over (https://github.com/kubernetes/minikube/blob/master/deploy/addons/gvisor/README.md)
Without specifying the –vm-driver parameter by default minikube uses the kvm driver.

To specify specific container runtime to use inside the VM use the following command:

start --vm-driver=kvm2 --container-runtime=cri-o --v=8

we can take a look at the kvm running by using virt-manager. Entering into the machine and executing systemctl status will yield the following output

    ● minikube        State: degraded         Jobs: 0 queued       Failed: 1 units        Since: Mon 2019-10-28 21:14:47 UTC; 4min 25s ago       CGroup: /               ├─init.scope               │ └─1 /sbin/init noembed norestore               ├─kubepods               │ ├─besteffort               │ │ ├─pod66fd795f-401c-4181-9cf0-0c2a17663b81               │ │ │ ├─crio-conmon-cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5               │ │ │ │ └─3584 /usr/libexec/crio/conmon --syslog -c cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5 -n k8s_POD_kube-proxy-plfd2_kube-system_66fd795f-401c-4181-9cf0-0c2a17663b81_0 -u cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5/userdata -p /var/run/containers/storage/overlay-containers/cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5/userdata/pidfile -l /var/log/pods/kube-system_kube-proxy-plfd2_66fd795f-401c-4181-9cf0-0c2a17663b81/cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5.log --exit-dir me-arg --root=/run/runc --no-pivotath /var/run/crio --log-level debug --runti--More--                │ │ │ ├─crio-cb4126fb3d539269743b654879485e5fa5f1d863896783a307fb43f488d29ac5               │ │ │ │ └─3598 /pause               │ │ │ ├─crio-conmon-f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2               │ │ │ │ └─3616 /usr/libexec/crio/conmon --syslog -c f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2 -n k8s_kube-proxy_kube-proxy-plfd2_kube-system_66fd795f-401c-4181-9cf0-0c2a17663b81_0 -u f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2/userdata -p /var/run/containers/storage/overlay-containers/f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2/userdata/pidfile -l /var/log/pods/kube-system_kube-proxy-plfd2_66fd795f-401c-4181-9cf0-0c2a17663b81/kube-proxy/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │ │ │ └─crio-f25ed7b9fb31818f0b8ab6d4cb94baead72a124b8915d92145bbb5ce86811ee2               │ │ │   └─3626 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=minikube               │ │ ├─pod89ae5909ffeb743f1dfb204f25a829e6               │ │ │ ├─crio-conmon-fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610               │ │ │ │ └─3079 /usr/libexec/crio/conmon --syslog -c fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610 -n k8s_POD_etcd-minikube_kube-system_89ae5909ffeb743f1dfb204f25a829e6_0 -u fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610/userdata -p /var/run/containers/storage/overlay-containers/fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610/userdata/pidfile -l /var/log/pods/kube-system_etcd-minikube_89ae5909ffeb743f1dfb204f25a829e6/fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │ │ │ ├─crio-conmon-42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9               │ │ │ │ └─3273 /usr/libexec/crio/conmon --syslog -c 42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9 -n k8s_etcd_etcd-minikube_kube-system_89ae5909ffeb743f1dfb204f25a829e6_0 -u 42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9/userdata -p /var/run/containers/storage/overlay-containers/42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9/userdata/pidfile -l /var/log/pods/kube-system_etcd-minikube_89ae5909ffeb743f1dfb204f25a829e6/etcd/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │ │ │ ├─crio-fa20492d16fce683bd7eeeef02b18abdcd46d9418036f70d9cb0764f60178610               │ │ │ │ └─3136 /pause               │ │ │ └─crio-42866e6955f0f3d8bde90c9235c1da99498469db636e6a8404208473c2fca3d9               │ │ │   └─3327 etcd --advertise-client-urls=https://192.168.39.214:2379 --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=https://192.168.39.214:2380 --initial-cluster=minikube=https://192.168.39.214:2380 --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.39.214:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.39.214:2380 --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt               │ │ └─pod9a1966c7-8565-4aed-8609-04abe6bdf785               │ │   ├─crio-21627aa0b547296c821ae7432cb4ee4ed22f2a9707c833fc6e3563d8faaf6dd9               │ │   │ └─3870 /storage-provisioner               │ │   ├─crio-conmon-ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9    76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9 -n k8s_POD_storage-provisioner_kube-system_9a1966c7-8565-4aed-8609-04abe6bdf785_0 -u ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9/userdata -p /var/run/containers/storage/overlay-containers/ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9/userdata/pidfile -l /var/log/pods/kube-system_storage-provisioner_9a1966c7-8565-4aed-8609-04abe6bdf785/ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │ │   ├─crio-ba42f9d76d11545bedcf8b43f19877145bb27efbfee6d11df061b8373bfe61a9               │ │   │ └─3840 /pause               │ │   └─crio-conmon-21627aa0b547296c821ae7432cb4ee4ed22f2a9707c833fc6e3563d8faaf6dd9               │ │     └─3859 /usr/libexec/crio/conmon --syslog -c 21627aa0b547296c821ae7432cb4ee4ed22f2a9707c833fc6e3563d8faaf6dd9 -n k8s_storage-provisioner_storage-provisioner_kube-system_9a1966c7-8565-4aed-8609-04abe6bdf785_0 -u 21627aa0b547296c821ae7432cb4ee4ed22f2a9707c833fc6e3563d8faaf6dd9 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/21627aa0b547296c821ae7432cb4ee4ed22f2a9707c833fc6e3563d8faaf6dd9/userdata -p /var/run/containers/storage/overlay-ed-8609-04abe6bdf785/storage-provisioner/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │ └─burstable               │   ├─pod67888a6f41348f1a41e319a7f77279a2               │   │ ├─crio-90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af               │   │ │ └─3267 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-signing-cert-file=/var/lib/minikube/certs/ca.crt --cluster-signing-key-file=/var/lib/minikube/certs/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --root-ca-file=/var/lib/minikube/certs/ca.crt --service-account-private-key-file=/var/lib/minikube/certs/sa.key --use-service-account-credentials=true               │   │ ├─crio-conmon-027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df               │   │ │ └─3096 /usr/libexec/crio/conmon --syslog -c 027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df -n k8s_POD_kube-controller-manager-minikube_kube-system_67888a6f41348f1a41e319a7f77279a2_0 -u 027e7f3d792498br/run/containers/storage/overlay-containers/027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df/userdata -p /var/run/containers/storage/overlay-containers/027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df/userdata/pidfile -l /var/log/pods/kube-system_kube-controller-manager-minikube_67888a6f41348f1a41e319a7f77279a2/027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-conmon-90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af               │   │ │ └─3249 /usr/libexec/crio/conmon --syslog -c 90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af -n k8s_kube-controller-manager_kube-controller-manager-minikube_kube-system_67888a6f41348f1a41e319a7f77279a2_0 -u 90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af/userdata -p /var/run/containers/storage/overlay-containers/90f7b48799d7f40bf2f81ba9946e74f15afd7348a515ca00b4367b1e208621af/userdata/pidfile -l /var/log/pods/kube-system_kube-controller-manager-minikube_67888a6f41348f1a41e319a7f77279a2/kube-controller-manager/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ └─crio-027e7f3d792498b1e54bfb1b6148c4959d2d366a8af64e3ae5d0ac2aba9679df               │   │   └─3158 /pause               │   ├─pod288cbfbd5565f92cc63b18bcafb084d5               │   │ ├─crio-conmon-8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851               │   │ │ └─3217 /usr/libexec/crio/conmon --syslog -c 8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851 -n k8s_kube-apiserver_kube-apiserver-minikube_kube-system_288cbfbd5565f92cc63b18bcafb084d5_0 -u 8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851/userdata -p /var/run/containers/storage/overlay-containers/8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851/userdata/pidfile -l /var/log/pods/kube-system_kube-apiserver-minikube_288cbfbd5565f92cc63b18bcafb084d5/kube-apiserver/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-8fedc929a6c2f817e0c7855d588f6aa6a0db7e0bd5df4a7e3baa24ea8ccd0851               │   │ │ └─3228 kube-apiserver --advertise-address=192.168.39.214 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt --etcd-cerlib/minikube/certs/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8443 --service-account-key-file=/var/lib/minikube/certs/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/var/lib/minikube/certs/apiserver.crt --tls-private-key-file=/var/lib/minikube/certs/apiserver.key               │   │ ├─crio-conmon-da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d               │   │ │ └─3093 /usr/libexec/crio/conmon --syslog -c da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d -n k8s_POD_kube-apiserver-minikube_kube-system_288cbfbd5565f92cc63b18bcafb084d5_0 -u da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d/userdata -p /var/run/containers/storage/overlay-containers/da45fb084d5/da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ └─crio-da45f694e1afffe3d6cbd945e96f9798b34a728243ecd5232fc329f64446bd2d               │   │   └─3124 /pause               │   ├─pod74dea8da17aa6241e5e4f7b2ba4e1d8e               │   │ ├─crio-993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799               │   │ │ └─3302 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true               │   │ ├─crio-conmon-42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f               │   │ │ └─3110 /usr/libexec/crio/conmon --syslog -c 42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f -n k8s_POD_kube-scheduler-minikube_kube-system_74dea8da17aa6241e5e4f7b2ba4e1d8e_0 -u 42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f/userdata -p /var/run/containers/storage/overlay-containers/429544e1d8e/42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-42954baf61955ccb20af54ca4ceb0d29e04c1661a7252c1555aeaa64b75da42f               │   │ │ └─3147 /pause               │   │ └─crio-conmon-993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799               │   │   └─3278 /usr/libexec/crio/conmon --syslog -c 993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799 -n k8s_kube-scheduler_kube-scheduler-minikube_kube-system_74dea8da17aa6241e5e4f7b2ba4e1d8e_0 -u 993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799/userdata -p /var/run/containers/storage/overlay-containers/993cc803be23925d62509f515bf2b3c82defd9c6aaf02ffdfb66f14764396799/userdata/pidfile -l /var/log/pods/kube-system_kube-scheduler-minikube_74dea8da17aa6241e5e4f7b2ba4e1d8e/kube-scheduler/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   ├─poda551ce90-e9ae-412e-b777-2a622a41b3d1               │   │ ├─crio-42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897               │   │ │ └─3739 /pause    06708faeafa70077d626278-conmon-0c70b067f98ddadd639a90b776e6947a7d083b66a--More--                │   │ │ └─3915 /usr/libexec/crio/conmon --syslog -c 0c70b067f98ddadd639a90b776e6947a7d083b66a06708faeafa70077d626278 -n k8s_coredns_coredns-5644d7b6d9-lgt6b_kube-system_a551ce90-e9ae-412e-b777-2a622a41b3d1_0 -u 0c70b067f98ddadd639a90b776e6947a7d083b66a06708faeafa70077d626278 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/0c70b067f98ddadd639a90b776e6947a7d083b66a06708faeafa70077d626278/userdata -p /var/run/containers/storage/overlay-containers/0c70b067f98ddadd639a90b776e6947a7d083b66a06708faeafa70077d626278/userdata/pidfile -l /var/log/pods/kube-system_coredns-5644d7b6d9-lgt6b_a551ce90-e9ae-412e-b777-2a622a41b3d1/coredns/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-0c70b067f98ddadd639a90b776e6947a7d083b66a06708faeafa70077d626278               │   │ │ └─3926 /coredns -conf /etc/coredns/Corefile               │   │ └─crio-conmon-42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897               │   │   └─3714 /usr/libexec/crio/conmon --syslog -c 42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897 -n k8s_POD_coredns-5644d7b6d9-lgt6b_kube-system_a551ce90-e9ae-412e-b777-2a622a41b3d1_0 -u 42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897/userdata -p /var/run/containers/storage/overlay-containers/ile -l /var/log/pods/kube-system_coredns-5644d7b6d9-lgt6b_a551ce90-e9ae-412e-b777-2a622a41b3d1/42bf005600a5ce7b8b99d371a69fbd64bba8fcd112972ef43a7ee5d71ca18897.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   ├─podc3e29047da86ce6690916750ab69c40b               │   │ ├─crio-conmon-0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50               │   │ │ └─3061 /usr/libexec/crio/conmon --syslog -c 0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50 -n k8s_POD_kube-addon-manager-minikube_kube-system_c3e29047da86ce6690916750ab69c40b_0 -u 0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50/userdata -p /var/run/containers/storage/overlay-containers/0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50/userdata/pidfile -l /var/log/pods/kube-system_kube-addon-manager-minikube_c3e29047da86ce6690916750ab69c40b/0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-conmon-5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2               │   │ │ └─3416 /usr/libexec/crio/conmon --syslog -c 5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2 -n k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_c3e29047da86ce6690916750ab69c40b_0 ---More-u 5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2/userdata -p /var/run/containers/storage/overlay-containers/5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2/userdata/pidfile -l /var/log/pods/kube-system_kube-addon-manager-minikube_c3e29047da86ce6690916750ab69c40b/kube-addon-manager/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │   │ ├─crio-0a85d1a6edd833057f8537f9195fb1aebb0273febb33790a996b58b64cb9bf50               │   │ │ └─3072 /pause               │   │ └─crio-5935820f335661c4ce86e9ef5f97a910b0958f942b2673cf224ee7b80b0c15f2               │   │   ├─3427 bash /opt/kube-addons.sh               │   │   └─5230 sleep 4               │   └─poddd190f17-7822-4345-9141-22a797802e0f               │     ├─crio-conmon-1759f8374de60691c19247b1f93dab3f4c42b25e744c8350001de00524a7e059               │     │ └─3724 /usr/libexec/crio/conmon --syslog -c 1759f8374de60691c19247b1f93dab3f4c42b25e744c8350001de00524a7e059 -n k8s_POD_coredns-5644d7b6d9-26x6p_kube-system_dd190f17-7822-4345-9141-22a797802e0f_0 -u 1759f8374de60691c194c8350001de00524a7e059/userdata -p /var/run/containers/storage/overlay-containers/1759f8374de60691c19247b1f93dab3f4c42b25e744c8350001de00524a7e059/userdata/pidfile -l /var/log/pods/kube-system_coredns-5644d7b6d9-26x6p_dd190f17-7822-4345-9141-22a797802e0f/1759f8374de60691c19247b1f93dab3f4c42b25e744c8350001de00524a7e059.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │     ├─crio-conmon-892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0               │     │ └─3969 /usr/libexec/crio/conmon --syslog -c 892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0 -n k8s_coredns_coredns-5644d7b6d9-26x6p_kube-system_dd190f17-7822-4345-9141-22a797802e0f_0 -u 892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0 -r /usr/bin/runc -b /var/run/containers/storage/overlay-containers/892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0/userdata -p /var/run/containers/storage/overlay-containers/892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0/userdata/pidfile -l /var/log/pods/kube-system_coredns-5644d7b6d9-26x6p_dd190f17-7822-4345-9141-22a797802e0f/coredns/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level debug --runtime-arg --root=/run/runc --no-pivot               │     ├─crio-1759f8374de60691c19247b1f93dab3f4c42b25e744c8350001de00524a7e059               │     │ └─3760 /pause               │     └─crio-892f3d3545ee21ad95a171ae77636fe40c3983f0566d1439939a04430d64adb0               │       └─3980 /coredns -conf /etc/coredns/Corefile               └─system.slice                 ├─nfs-mountd.service                 │ └─1660 /usr/sbin/rpc.mountd                 ├─systemd-timesyncd.service                 │ └─1636 /usr/lib/systemd/systemd-timesyncd                 ├─crio.service                 │ └─2049 /usr/bin/crio --log-level=debug --insecure-registry 10.96.0.0/12                 ├─dbus.service                 │ └─1685 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only                 ├─sshd.service                 │ └─1770 /usr/sbin/sshd -D -e                 ├─kubelet.service                 │ └─3011 /var/lib/minikube/binaries/v1.16.2/kubelet --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --fail-swap-on=false --hostname-override=minikube --image-service-endpoint=/var/run/crio/crio.sock --kubeconfig=/etc/kubernetes/kubelet.conf --network-plugin=cni --node-ip=192.168.39.214 --pod-manifest-path=/etc/kubernetes/manifests --runtime-request-timeout=15m                 ├─system-serial\x2dgetty.slice                 │ └─serial-getty@ttyS0.service                 │   ├─1678 -sh                 │   ├─5263 systemctl status                 │   └─5264 /bin/more                 ├─system-getty.slice                 │ └─getty@tty1.service                 │   └─1679 /sbin/getty -L tty1 115200 vt100                 ├─rpcbind.service                 │ └─1677 /usr/bin/rpcbind                 ├─systemd-logind.service                 │ └─1688 /usr/lib/systemd/systemd-logind                 ├─systemd-resolved.service                 │ └─1656 /usr/lib/systemd/systemd-resolved                 ├─systemd-udevd.service                 │ └─1640 /usr/lib/systemd/systemd-udevd                 ├─systemd-journald.service                 │ └─1093 /usr/lib/systemd/systemd-journald                 └─systemd-networkd.service                   └─1649 /usr/lib/systemd/systemd-networkd

When running cri-o container use the following command to view images

crictl images

IMAGE                                     TAG                 IMAGE ID            SIZEgcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb103       80.8MBk8s.gcr.io/coredns                        1.6.2               bf261d1579144       44.2MBk8s.gcr.io/etcd                           3.3.15-0            b2756210eeabf       248MBk8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.13             6dc8ef8287d38       41.6MBk8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.13             55a3c5209c5ea       51.4MBk8s.gcr.io/k8s-dns-sidecar-amd64          1.14.13             4b2e93f0133d3       43.1MBk8s.gcr.io/kube-addon-manager             v9.0                119701e77cbc4       84.7MBk8s.gcr.io/kube-addon-manager             v9.0.2              bd12a212f9dcb       84.7MBk8s.gcr.io/kube-apiserver                 v1.16.2             c2c9a0406787c       219MBk8s.gcr.io/kube-controller-manager        v1.16.2             6e4bffa46d70b       165MBk8s.gcr.io/kube-proxy                     v1.16.2             8454cbe08dc9f       87.9MBk8s.gcr.io/kube-scheduler                 v1.16.2             ebac1ae204a2c       88.8MBk8s.gcr.io/kubernetes-dashboard-amd64     v1.10.1             f9aed6605b814       122MBk8s.gcr.io/pause                          3.1                 da86e6ba6ca19       747kB

and to view running container use the following command:

crictl  ps

CONTAINER           IMAGE                                                                                                   CREATED             STATE               NAME                      ATTEMPT             POD ID892f3d3545ee2       bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b                                        7 minutes ago       Running             coredns                   0                   1759f8374de600c70b067f98dd       bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b                                        7 minutes ago       Running             coredns                   0                   42bf005600a5c21627aa0b5472       4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c                                        7 minutes ago       Running             storage-provisioner       0                   ba42f9d76d115f25ed7b9fb318       8454cbe08dc9ff820283939d1e509af7746256a3fc0511999c7e29ed8f29f852                                        7 minutes ago       Running             kube-proxy                0                   cb4126fb3d5395935820f33566       k8s.gcr.io/kube-addon-manager@sha256:3e315022a842d782a28e729720f21091dde21f1efea28868d65ec595ad871616   7 minutes ago       Running             kube-addon-manager        0                   0a85d1a6edd8342866e6955f0f       b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed                                        8 minutes ago       Running             etcd                      0                   fa20492d16fce993cc803be239       ebac1ae204a2c8d36411792d8bfea34bc82bd733b499ff81ca10e0f6c082d44c                                        8 minutes ago       Running             kube-scheduler            0                   42954baf6195590f7b48799d7f       6e4bffa46d70bb06f05f9e32d9f73511fe59c0dcd85aa25a56e651c1250b4777                                        8 minutes ago       Running             kube-controller-manager   0                   027e7f3d792498fedc929a6c2f       c2c9a0406787cbb6b206d082f1085215e9b0c98483b6e3dba3d0df6d4653e7b0                                        8 minutes ago       Running             kube-apiserver            0                   da45f694e1aff

Few examples available from client-go project that helped to play around with minikube client-go/examples/
JSON structure for Kubernetes can be found inside k8s.io/kubernetes/staging/src/k8s.io/api/core/v1/types.go
Testing
- Test cases uses a lot of ‘fake’ classes that are made available inside k8s.io/client-go/kubernetes/typed/core/v1/fake folder

For example to access a pod’s log use the following command

curl -k --cert ~/.minikube/apiserver.crt --key ~/.minikube/apiserver.key  -v -XGET https://192.168.99.223:8443/api/v1/namespaces/default/pods/demo-788cf8d6f5-6d6qz/log

pkg/minikube/service/service.go – contains code to connect to kubernetes

type K8sClient interface {    GetCoreClient() (typed_core.CoreV1Interface, error)    GetClientset(timeout time.Duration) (*kubernetes.Clientset, error)}

To communicate with etcd that is running inside minikube we need to ssh to minikube

minikube ssh

Inside minikube we use the following command

/hosthome/nanik/Downloads/temp/packages/src/go.etcd.io/etcd/etcdctl/etcdctl --cacert=/var/lib/minikube/certs/etcd/ca.crt --key=/var/lib/minikube/certs/etcd/ca.key --cert=/var/lib/minikube/certs/etcd/ca.crt get  --prefix=true ""

the ca.crt and ca.key resides inside minikube VM

    ETCDCTL_API=3  /hosthome/nanik/Downloads/temp/packages/src/go.etcd.io/etcd/etcdctl/etcdctl --cacert=/var/lib/minikube/certs/etcd/ca.crt --key=/var/lib/minikube/certs/etcd/ca.key --cert=/var/lib/minikube/certs/etcd/ca.crt get  --from-key '' --keys-only

will get all the keys stored inside

/hostname – is the default mount volume created by minikube to access host directory.

>/registry/apiregistration.k8s.io/apiservices/v1>/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io>/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io>/apiregistration.k8s.io/apiservices/v1.apps>/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io

Minikube Integration Testing (Work in progress)

Sun, 01 Dec 2019 00:00:00 +0000

This article will outline few things that are helpful when running or modifying minikube testing code.

Make sure kubectl is in the PATH
Following command is to disable parallel testing and enable all log output
```
-parallel 1  -test.v
```
the test.v parameter instruct the testing framework to output non-failure messages. The output will not be realtime as it will only be outputted when the test complete.

To run the test use the following command

PATH=/home/nanik/Golang/go/bin:/home/nanik/Downloads:/home/nanik/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/var/lib/snapd/snap/bin:/home/nanik/Downloads/ go test   -test.v -tags integration

The /home/nanik/Downloads is added to the PATH because that’s the location for kubectl and the test cases uses kubectl.

DO NOT USE -parrallel 1 when running ALL the test cases as for some reason it throws an error.

Running make integration command will yield output similar to the following

GOOS="linux" GOARCH="amd64" go build -tags "container_image_ostree_stub containers_image_openpgp go_getter_nos3 go_getter_nogcs" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.5.2 -X k8s.io/minikube/pkg/version.isoVersion=v1.5.1 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="3322c50ceb4abf795ebb8505f99bbb269a144e21-dirty"" -a -o out/minikube-linux-amd64 k8s.io/minikube/cmd/minikubecp out/minikube-linux-amd64 out/minikubego test -v -test.timeout=60m ./test/integration --tags="integration container_image_ostree_stub containers_image_openpgp go_getter_nos3 go_getter_nogcs" === RUN   TestDownloadOnly=== RUN   TestDownloadOnly/group=== RUN   TestDownloadOnly/group/v1.11.10=== RUN   TestDownloadOnly/group/v1.16.2=== RUN   TestDownloadOnly/group/v1.16.2#01=== RUN   TestDownloadOnly/group/ExpectedDefaultDriver=== RUN   TestDownloadOnly/group/DeleteAll=== RUN   TestDownloadOnly/group/DeleteAlwaysSucceeds--- PASS: TestDownloadOnly (33.63s)    --- PASS: TestDownloadOnly/group (33.57s)        --- PASS: TestDownloadOnly/group/v1.11.10 (19.74s)            a_serial_test.go:61: (dbg) Run:  out/minikube start --download-only -p download-20191104T082052.238721463-18873 --force --alsologtostderr --kubernetes-version=v1.11.10            a_serial_test.go:61: (dbg) Done: out/minikube start --download-only -p download-20191104T082052.238721463-18873 --force --alsologtostderr --kubernetes-version=v1.11.10: (19.741382231s)        --- PASS: TestDownloadOnly/group/v1.16.2 (13.46s)            a_serial_test.go:63: (dbg) Run:  out/minikube start --download-only -p download-20191104T082052.238721463-18873 --force --alsologtostderr --kubernetes-version=v1.16.2            a_serial_test.go:63: (dbg) Done: out/minikube start --download-only -p download-20191104T082052.238721463-18873 --force --alsologtostderr --kubernetes-version=v1.16.2: (13.460844164s)        --- PASS: TestDownloadOnly/group/v1.16.2#01 (0.27s)            a_serial_test.go:63: (dbg) Run:  out/minikube start --download-only -p download-20191104T082052.238721463-18873 --force --alsologtostderr --kubernetes-version=v1.16.2        --- SKIP: TestDownloadOnly/group/ExpectedDefaultDriver (0.00s)            a_serial_test.go:94: --expected-default-driver is unset, skipping test        --- PASS: TestDownloadOnly/group/DeleteAll (0.06s)            a_serial_test.go:124: (dbg) Run:  out/minikube delete --all        --- PASS: TestDownloadOnly/group/DeleteAlwaysSucceeds (0.03s)            a_serial_test.go:134: (dbg) Run:  out/minikube delete -p download-20191104T082052.238721463-18873    helpers.go:167: (dbg) Run:  out/minikube delete -p download-20191104T082052.238721463-18873=== RUN   TestAddons=== PAUSE TestAddons=== RUN   TestDockerFlags=== PAUSE TestDockerFlags=== RUN   TestKVMDriverInstallOrUpdate=== PAUSE TestKVMDriverInstallOrUpdate=== RUN   TestHyperKitDriverInstallOrUpdate--- SKIP: TestHyperKitDriverInstallOrUpdate (0.00s)    driver_install_or_update_test.go:101: Skip if not darwin.=== RUN   TestFunctional=== RUN   TestFunctional/serial=== RUN   TestFunctional/serial/StartWithProxy=== RUN   TestFunctional/serial/KubeContext=== RUN   TestFunctional/serial/KubectlGetPods=== RUN   TestFunctional/serial/CacheCmd=== PAUSE TestFunctional=== RUN   TestGvisorAddon--- SKIP: TestGvisorAddon (0.00s)    gvisor_addon_test.go:34: skipping test because --gvisor=false=== RUN   TestChangeNoneUser--- SKIP: TestChangeNoneUser (0.00s)    none_test.go:39: Only test none driver.=== RUN   TestStartStop=== PAUSE TestStartStop=== RUN   TestVersionUpgrade=== PAUSE TestVersionUpgrade=== CONT  TestAddons=== CONT  TestStartStop=== CONT  TestFunctional=== CONT  TestVersionUpgrade=== CONT  TestDockerFlags=== RUN   TestFunctional/parallel=== CONT  TestKVMDriverInstallOrUpdate=== RUN   TestFunctional/parallel/AddonManager=== RUN   TestStartStop/group=== PAUSE TestFunctional/parallel/AddonManager=== RUN   TestFunctional/parallel/ComponentHealth=== PAUSE TestFunctional/parallel/ComponentHealth=== RUN   TestStartStop/group/docker=== RUN   TestFunctional/parallel/ConfigCmd=== PAUSE TestFunctional/parallel/ConfigCmd=== RUN   TestFunctional/parallel/DashboardCmd=== PAUSE TestStartStop/group/docker=== RUN   TestStartStop/group/cni=== PAUSE TestFunctional/parallel/DashboardCmd=== RUN   TestFunctional/parallel/DNS=== PAUSE TestStartStop/group/cni=== PAUSE TestFunctional/parallel/DNS=== RUN   TestStartStop/group/containerd=== RUN   TestFunctional/parallel/StatusCmd=== PAUSE TestFunctional/parallel/StatusCmd=== PAUSE TestStartStop/group/containerd=== RUN   TestFunctional/parallel/LogsCmd=== RUN   TestStartStop/group/crio=== PAUSE TestFunctional/parallel/LogsCmd=== RUN   TestFunctional/parallel/MountCmd=== PAUSE TestStartStop/group/crio=== CONT  TestStartStop/group/docker=== PAUSE TestFunctional/parallel/MountCmd=== CONT  TestStartStop/group/containerd=== RUN   TestFunctional/parallel/ProfileCmd=== CONT  TestStartStop/group/crio=== CONT  TestStartStop/group/cni=== PAUSE TestFunctional/parallel/ProfileCmd=== RUN   TestFunctional/parallel/ServiceCmd=== PAUSE TestFunctional/parallel/ServiceCmd=== RUN   TestFunctional/parallel/AddonsCmd=== PAUSE TestFunctional/parallel/AddonsCmd=== RUN   TestFunctional/parallel/PersistentVolumeClaim=== PAUSE TestFunctional/parallel/PersistentVolumeClaim=== RUN   TestFunctional/parallel/TunnelCmd=== PAUSE TestFunctional/parallel/TunnelCmd=== RUN   TestFunctional/parallel/SSHCmd=== PAUSE TestFunctional/parallel/SSHCmd=== RUN   TestFunctional/parallel/MySQL=== PAUSE TestFunctional/parallel/MySQL=== CONT  TestFunctional/parallel/AddonManager=== CONT  TestFunctional/parallel/MySQL=== CONT  TestFunctional/parallel/MountCmd=== CONT  TestFunctional/parallel/DNS    > docker-machine-driver-kvm2.sha256: 65 B / 65 B [-------] 100.00% ? p/s 0s=== CONT  TestFunctional/parallel/ConfigCmd=== CONT  TestFunctional/parallel/DashboardCmd=== CONT  TestFunctional/parallel/LogsCmd=== CONT  TestFunctional/parallel/ComponentHealth=== CONT  TestFunctional/parallel/StatusCmd=== CONT  TestFunctional/parallel/PersistentVolumeClaim    > docker-machine-driver-kvm2: 0 B / 48.57 MiB [_____________] 0.00% ? p/s ?    > docker-machine-driver-kvm2: 33.55 KiB / 48.57 MiB [>______] 0.07% ? p/s ?    > docker-machine-driver-kvm2: 84.55 KiB / 48.57 MiB [>______] 0.17% ? p/s ?    > docker-machine-driver-kvm2: 169.55 KiB / 48.57 MiB  0.34% 282.61 KiB p/s     > docker-machine-driver-kvm2: 169.55 KiB / 48.57 MiB  0.34% 282.61 KiB p/s     > docker-machine-driver-kvm2: 338.52 KiB / 48.57 MiB  0.68% 282.61 KiB p/s     > docker-machine-driver-kvm2: 662.55 KiB / 48.57 MiB  1.33% 317.38 KiB p/s === CONT  TestFunctional/parallel/SSHCmd    > docker-machine-driver-kvm2: 662.55 KiB / 48.57 MiB  1.33% 317.38 KiB p/s === CONT  TestFunctional/parallel/TunnelCmd=== CONT  TestFunctional/parallel/ServiceCmd    > docker-machine-driver-kvm2: 1.31 MiB / 48.57 MiB  2.70% 317.38 KiB p/s ET    > docker-machine-driver-kvm2: 2.13 MiB / 48.57 MiB  4.38% 459.99 KiB p/s ET    > docker-machine-driver-kvm2: 2.30 MiB / 48.57 MiB  4.74% 459.99 KiB p/s ET    > docker-machine-driver-kvm2: 3.35 MiB / 48.57 MiB  6.89% 459.99 KiB p/s ET    > docker-machine-driver-kvm2: 4.44 MiB / 48.57 MiB  9.14% 684.97 KiB p/s ET    >      > .....    > .....=== CONT  TestFunctional/parallel/AddonsCmd=== CONT  TestFunctional/parallel/ProfileCmd    > docker-machine-driver-kvm2: 21.21 MiB / 48.57 MiB  43.66% 1.86 MiB p/s ET    > docker-machine-driver-kvm2: 21.21 MiB / 48.57 MiB  43.66% 2.08 MiB p/s ET    > docker-machine-driver-kvm2: 22.78 MiB / 48.57 MiB  46.91% 2.08 MiB p/s ET    > docker-machine-driver-kvm2: 23.35 MiB / 48.57 MiB  48.06% 2.08 MiB p/s ET    > docker-machine-driver-kvm2: 24.91 MiB / 48.57 MiB  51.28% 2.34 MiB p/s ET        > .....     > .....     > .....    2019/11/04 08:25:30 [DEBUG] GET http://192.168.39.24:318492019/11/04 08:25:38 [DEBUG] GET http://127.0.0.1:38193/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/--- PASS: TestFunctional (254.26s)    --- PASS: TestFunctional/serial (206.43s)        --- PASS: TestFunctional/serial/StartWithProxy (177.80s)            functional_test.go:120: (dbg) Run:  out/minikube start -p functional-20191104T082125.872658733-18873 --wait=false             functional_test.go:120: (dbg) Done: out/minikube start -p functional-20191104T082125.872658733-18873 --wait=false : (2m57.799561087s)        --- PASS: TestFunctional/serial/KubeContext (0.05s)            functional_test.go:138: (dbg) Run:  kubectl config current-context        --- PASS: TestFunctional/serial/KubectlGetPods (0.44s)            functional_test.go:149: (dbg) Run:  kubectl get pod -A        --- PASS: TestFunctional/serial/CacheCmd (28.14s)            functional_test.go:302: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 cache add busybox            functional_test.go:302: (dbg) Done: out/minikube -p functional-20191104T082125.872658733-18873 cache add busybox: (5.119077944s)            functional_test.go:302: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 cache add busybox:1.28.4-glibc            functional_test.go:302: (dbg) Done: out/minikube -p functional-20191104T082125.872658733-18873 cache add busybox:1.28.4-glibc: (5.002939096s)            functional_test.go:302: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 cache add mysql:5.6            functional_test.go:302: (dbg) Done: out/minikube -p functional-20191104T082125.872658733-18873 cache add mysql:5.6: (18.018114782s)    --- PASS: TestFunctional/parallel (0.00s)        --- PASS: TestFunctional/parallel/MountCmd (4.77s)            fn_mount_cmd.go:62: (dbg) daemon: [../../out/minikube mount -p functional-20191104T082125.872658733-18873 /tmp/mounttest209263981:/mount-9p --alsologtostderr -v=1]            fn_mount_cmd.go:96: wrote "test-1572816292301508417" to /tmp/mounttest209263981/created-by-test            fn_mount_cmd.go:96: wrote "test-1572816292301508417" to /tmp/mounttest209263981/created-by-test-removed-by-pod            fn_mount_cmd.go:96: wrote "test-1572816292301508417" to /tmp/mounttest209263981/test-1572816292301508417            fn_mount_cmd.go:104: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh "findmnt -T /mount-9p | grep 9p"            fn_mount_cmd.go:104: (dbg) Non-zero exit: out/minikube -p functional-20191104T082125.872658733-18873 ssh "findmnt -T /mount-9p | grep 9p": exit status 1 (202.701275ms)                                ** stderr **                 ssh: Process exited with status 1                                ** /stderr **            fn_mount_cmd.go:104: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh "findmnt -T /mount-9p | grep 9p"            fn_mount_cmd.go:118: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh -- ls -la /mount-9p            fn_mount_cmd.go:122: guest mount directory contents                total 2                -rw-r--r-- 1 docker docker 24 Nov  3 21:24 created-by-test                -rw-r--r-- 1 docker docker 24 Nov  3 21:24 created-by-test-removed-by-pod                -rw-r--r-- 1 docker docker 24 Nov  3 21:24 test-1572816292301508417            fn_mount_cmd.go:126: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh cat /mount-9p/test-1572816292301508417            fn_mount_cmd.go:137: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 replace --force -f testdata/busybox-mount-test.yaml            fn_mount_cmd.go:142: (dbg) waiting for pods with labels "integration-test=busybox-mount" in namespace "default" ...            helpers.go:247: "busybox-mount" [cad31f9d-5fe3-423d-b72c-76365e32eaac] Pending            helpers.go:247: "busybox-mount" [cad31f9d-5fe3-423d-b72c-76365e32eaac] Pending / Ready:ContainersNotReady (containers with unready status: [mount-munger]) / ContainersReady:ContainersNotReady (containers with unready status: [mount-munger])            helpers.go:247: "busybox-mount" [cad31f9d-5fe3-423d-b72c-76365e32eaac] Succeeded: Initialized:PodCompleted / Ready:PodCompleted / ContainersReady:PodCompleted            fn_mount_cmd.go:142: (dbg) pods integration-test=busybox-mount up and healthy within 2.006393812s            fn_mount_cmd.go:158: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 logs busybox-mount            fn_mount_cmd.go:170: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh stat /mount-9p/created-by-test            fn_mount_cmd.go:170: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh stat /mount-9p/created-by-pod            fn_mount_cmd.go:79: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh "sudo umount -f /mount-9p"            fn_mount_cmd.go:83: (dbg) stopping [../../out/minikube mount -p functional-20191104T082125.872658733-18873 /tmp/mounttest209263981:/mount-9p --alsologtostderr -v=1] ...        --- PASS: TestFunctional/parallel/ConfigCmd (0.15s)            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config unset cpus            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config get cpus            functional_test.go:326: (dbg) Non-zero exit: out/minikube -p functional-20191104T082125.872658733-18873 config get cpus: exit status 64 (21.49557ms)                                ** stderr **                 Error: specified key could not be found in config                                ** /stderr **            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config set cpus 2            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config get cpus            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config unset cpus            functional_test.go:326: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 config get cpus            functional_test.go:326: (dbg) Non-zero exit: out/minikube -p functional-20191104T082125.872658733-18873 config get cpus: exit status 64 (19.061597ms)                                ** stderr **                 Error: specified key could not be found in config                                ** /stderr **        --- PASS: TestFunctional/parallel/AddonManager (5.06s)            functional_test.go:162: (dbg) waiting for pods with labels "component=kube-addon-manager" in namespace "kube-system" ...            helpers.go:247: "kube-addon-manager-minikube" [713e5c52-a374-40ed-9f57-f14c7d0de26a] Running            functional_test.go:162: (dbg) pods component=kube-addon-manager up and healthy within 5.026986587s        --- PASS: TestFunctional/parallel/LogsCmd (0.59s)            functional_test.go:344: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 logs        --- PASS: TestFunctional/parallel/ComponentHealth (0.07s)            functional_test.go:169: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 get cs -o=json        --- PASS: TestFunctional/parallel/StatusCmd (0.57s)            functional_test.go:194: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 status            functional_test.go:200: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 status -f host:{{.Host}},kublet:{{.Kubelet}},apiserver:{{.APIServer}},kubeconfig:{{.Kubeconfig}}            functional_test.go:210: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 status -o json        --- PASS: TestFunctional/parallel/DNS (8.60s)            functional_test.go:270: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 replace --force -f testdata/busybox.yaml            functional_test.go:275: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [24600016-7278-4daa-8a5d-b54f465d0f19] Pending            helpers.go:247: "busybox" [24600016-7278-4daa-8a5d-b54f465d0f19] Pending / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])            helpers.go:247: "busybox" [24600016-7278-4daa-8a5d-b54f465d0f19] Running            functional_test.go:275: (dbg) pods integration-test=busybox up and healthy within 8.011521017s            functional_test.go:281: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 exec busybox nslookup kubernetes.default        --- PASS: TestFunctional/parallel/SSHCmd (0.15s)            functional_test.go:537: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 ssh "echo hello"        --- SKIP: TestFunctional/parallel/TunnelCmd (0.01s)            fn_tunnel_cmd.go:47: password required to execute 'route', skipping testTunnel: exit status 1        --- PASS: TestFunctional/parallel/MySQL (12.36s)            functional_test.go:548: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 replace --force -f testdata/mysql.yaml            functional_test.go:555: (dbg) waiting for pods with labels "app=mysql" in namespace "default" ...            helpers.go:247: "mysql-5787d7b5fc-kqdrj" [90a614ba-f2d8-4318-92ae-6edd7463f14c] Pending            helpers.go:247: "mysql-5787d7b5fc-kqdrj" [90a614ba-f2d8-4318-92ae-6edd7463f14c] Pending / Ready:ContainersNotReady (containers with unready status: [mysql]) / ContainersReady:ContainersNotReady (containers with unready status: [mysql])            helpers.go:247: "mysql-5787d7b5fc-kqdrj" [90a614ba-f2d8-4318-92ae-6edd7463f14c] Running            functional_test.go:555: pod "app=mysql" failed to start: timed out waiting for the condition            helpers.go:292: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 get po -A --show-labels            helpers.go:298: (dbg) kubectl --context functional-20191104T082125.872658733-18873 get po -A --show-labels:                NAMESPACE     NAME                               READY   STATUS      RESTARTS   AGE    LABELS                default       busybox                            1/1     Running     0          5s     integration-test=busybox                default       busybox-mount                      0/1     Completed   0          3s     integration-test=busybox-mount                default       mysql-5787d7b5fc-kqdrj             1/1     Running     0          5s     app=mysql,pod-template-hash=5787d7b5fc                kube-system   coredns-5644d7b6d9-p9dkz           1/1     Running     0          104s   k8s-app=kube-dns,pod-template-hash=5644d7b6d9                kube-system   coredns-5644d7b6d9-wbttz           1/1     Running     0          104s   k8s-app=kube-dns,pod-template-hash=5644d7b6d9                kube-system   etcd-minikube                      1/1     Running     0          41s    component=etcd,tier=control-plane                kube-system   kube-addon-manager-minikube        1/1     Running     0          112s   component=kube-addon-manager,kubernetes.io/minikube-addons=addon-manager,version=v9.0.2                kube-system   kube-apiserver-minikube            1/1     Running     0          34s    component=kube-apiserver,tier=control-plane                kube-system   kube-controller-manager-minikube   1/1     Running     0          37s    component=kube-controller-manager,tier=control-plane                kube-system   kube-proxy-wc5fw                   1/1     Running     0          104s   controller-revision-hash=56ffd4ff47,k8s-app=kube-proxy,pod-template-generation=1                kube-system   kube-scheduler-minikube            1/1     Running     0          26s    component=kube-scheduler,tier=control-plane                kube-system   storage-provisioner                1/1     Running     0          103s   addonmanager.kubernetes.io/mode=Reconcile,integration-test=storage-provisioner            helpers.go:301: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 describe po mysql-5787d7b5fc-kqdrj -n default            helpers.go:305: (dbg) kubectl --context functional-20191104T082125.872658733-18873 describe po mysql-5787d7b5fc-kqdrj -n default:                Name:           mysql-5787d7b5fc-kqdrj                Namespace:      default                Priority:       0                Node:           minikube/192.168.39.24                Start Time:     Mon, 04 Nov 2019 08:24:52 +1100                Labels:         app=mysql                                pod-template-hash=5787d7b5fc                Annotations:    <none>                Status:         Running                IP:             172.17.0.5                Controlled By:  ReplicaSet/mysql-5787d7b5fc                Containers:                  mysql:                    Container ID:   docker://38da7fcc7cc8841b3233ef5fc800c8375602336625daf18456b13eafb40ac7d9                    Image:          mysql:5.6                    Image ID:       docker://sha256:b3983abaa3feff23bf58b849c7c2c0f84862e3a749a3181a0bdd6fcfa023f318                    Port:           3306/TCP                    Host Port:      0/TCP                    State:          Running                      Started:      Mon, 04 Nov 2019 08:24:54 +1100                    Ready:          True                    Restart Count:  0                    Environment:                      MYSQL_ROOT_PASSWORD:  password                    Mounts:                      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s7jh6 (ro)                Conditions:                  Type              Status                  Initialized       True                   Ready             True                   ContainersReady   True                   PodScheduled      True                 Volumes:                  default-token-s7jh6:                    Type:        Secret (a volume populated by a Secret)                    SecretName:  default-token-s7jh6                    Optional:    false                QoS Class:       BestEffort                Node-Selectors:  <none>                Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s                                 node.kubernetes.io/unreachable:NoExecute for 300s                Events:                  Type    Reason     Age        From               Message                  ----    ------     ----       ----               -------                  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned default/mysql-5787d7b5fc-kqdrj to minikube                  Normal  Pulled     4s         kubelet, minikube  Container image "mysql:5.6" already present on machine                  Normal  Created    4s         kubelet, minikube  Created container mysql                  Normal  Started    3s         kubelet, minikube  Started container mysql            helpers.go:308: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 logs mysql-5787d7b5fc-kqdrj -n default            helpers.go:312: (dbg) kubectl --context functional-20191104T082125.872658733-18873 logs mysql-5787d7b5fc-kqdrj -n default:                2019-11-03 21:24:54+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.46-1debian9 started.                2019-11-03 21:24:54+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'                2019-11-03 21:24:54+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.46-1debian9 started.                2019-11-03 21:24:54+00:00 [Note] [Entrypoint]: Initializing database files                2019-11-03 21:24:54 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).                2019-11-03 21:24:54 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.                2019-11-03 21:24:54 0 [Note] /usr/sbin/mysqld (mysqld 5.6.46) starting as process 49 ...                2019-11-03 21:24:54 49 [Note] InnoDB: Using atomics to ref count buffer pool pages                2019-11-03 21:24:54 49 [Note] InnoDB: The InnoDB memory heap is disabled                2019-11-03 21:24:54 49 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins                2019-11-03 21:24:54 49 [Note] InnoDB: Memory barrier is not used                2019-11-03 21:24:54 49 [Note] InnoDB: Compressed tables use zlib 1.2.11                2019-11-03 21:24:54 49 [Note] InnoDB: Using Linux native AIO                2019-11-03 21:24:54 49 [Note] InnoDB: Using CPU crc32 instructions                2019-11-03 21:24:54 49 [Note] InnoDB: Initializing buffer pool, size = 128.0M                2019-11-03 21:24:54 49 [Note] InnoDB: Completed initialization of buffer pool                2019-11-03 21:24:54 49 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!                2019-11-03 21:24:54 49 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB                2019-11-03 21:24:54 49 [Note] InnoDB: Database physically writes the file full: wait...                2019-11-03 21:24:54 49 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB                2019-11-03 21:24:54 49 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB                2019-11-03 21:24:55 49 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0                2019-11-03 21:24:55 49 [Warning] InnoDB: New log files created, LSN=45781                2019-11-03 21:24:55 49 [Note] InnoDB: Doublewrite buffer not found: creating new                2019-11-03 21:24:55 49 [Note] InnoDB: Doublewrite buffer created                2019-11-03 21:24:55 49 [Note] InnoDB: 128 rollback segment(s) are active.                2019-11-03 21:24:55 49 [Warning] InnoDB: Creating foreign key constraint system tables.                2019-11-03 21:24:55 49 [Note] InnoDB: Foreign key constraint system tables created                2019-11-03 21:24:55 49 [Note] InnoDB: Creating tablespace and datafile system tables.                2019-11-03 21:24:55 49 [Note] InnoDB: Tablespace and datafile system tables created.                2019-11-03 21:24:55 49 [Note] InnoDB: Waiting for purge to start                2019-11-03 21:24:55 49 [Note] InnoDB: 5.6.46 started; log sequence number 0                2019-11-03 21:24:55 49 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.                2019-11-03 21:24:55 49 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.                2019-11-03 21:24:56 49 [Note] Binlog end                2019-11-03 21:24:56 49 [Note] InnoDB: FTS optimize thread exiting.                2019-11-03 21:24:56 49 [Note] InnoDB: Starting shutdown...                2019-11-03 21:24:57 49 [Note] InnoDB: Shutdown completed; log sequence number 1625977                                                2019-11-03 21:24:57 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).                2019-11-03 21:24:57 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.                2019-11-03 21:24:57 0 [Note] /usr/sbin/mysqld (mysqld 5.6.46) starting as process 72 ...                2019-11-03 21:24:57 72 [Note] InnoDB: Using atomics to ref count buffer pool pages                2019-11-03 21:24:57 72 [Note] InnoDB: The InnoDB memory heap is disabled                2019-11-03 21:24:57 72 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins                2019-11-03 21:24:57 72 [Note] InnoDB: Memory barrier is not used                2019-11-03 21:24:57 72 [Note] InnoDB: Compressed tables use zlib 1.2.11                2019-11-03 21:24:57 72 [Note] InnoDB: Using Linux native AIO                2019-11-03 21:24:57 72 [Note] InnoDB: Using CPU crc32 instructions                2019-11-03 21:24:57 72 [Note] InnoDB: Initializing buffer pool, size = 128.0M                2019-11-03 21:24:57 72 [Note] InnoDB: Completed initialization of buffer pool                2019-11-03 21:24:57 72 [Note] InnoDB: Highest supported file format is Barracuda.                2019-11-03 21:24:57 72 [Note] InnoDB: 128 rollback segment(s) are active.                2019-11-03 21:24:57 72 [Note] InnoDB: Waiting for purge to start                2019-11-03 21:24:57 72 [Note] InnoDB: 5.6.46 started; log sequence number 1625977                2019-11-03 21:24:57 72 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.                2019-11-03 21:24:57 72 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.                2019-11-03 21:24:57 72 [Note] Binlog end                2019-11-03 21:24:57 72 [Note] InnoDB: FTS optimize thread exiting.                2019-11-03 21:24:57 72 [Note] InnoDB: Starting shutdown...            functional_test.go:555: (dbg) waiting for pods with labels "app=mysql" in namespace "default" ...            helpers.go:247: "mysql-5787d7b5fc-kqdrj" [90a614ba-f2d8-4318-92ae-6edd7463f14c] Running            functional_test.go:555: (dbg) pods app=mysql up and healthy within 5.005878835s            functional_test.go:559: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 exec mysql-5787d7b5fc-kqdrj -- mysql -ppassword -e "show databases;"        --- PASS: TestFunctional/parallel/AddonsCmd (0.12s)            functional_test.go:478: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 addons list            functional_test.go:492: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 addons list --format "{{.AddonName}}":"{{.AddonStatus}}"            functional_test.go:506: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 addons list -f "{{.AddonName}}":"{{.AddonStatus}}"            functional_test.go:520: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 addons list -o json        --- PASS: TestFunctional/parallel/ProfileCmd (0.05s)            functional_test.go:357: (dbg) Run:  out/minikube profile list            functional_test.go:377: (dbg) Run:  out/minikube profile list --output json        --- PASS: TestFunctional/parallel/PersistentVolumeClaim (7.75s)            fn_pvc.go:40: (dbg) waiting for pods with labels "integration-test=storage-provisioner" in namespace "kube-system" ...            helpers.go:247: "storage-provisioner" [6c9c2d17-8119-4170-9544-5a1cbdcc68cc] Running            fn_pvc.go:40: (dbg) pods integration-test=storage-provisioner up and healthy within 5.010901682s            fn_pvc.go:45: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 get storageclass -o=json            fn_pvc.go:65: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 apply -f testdata/pvc.yaml            fn_pvc.go:71: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 get pvc testpvc -o=json            fn_pvc.go:71: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 get pvc testpvc -o=json        --- PASS: TestFunctional/parallel/ServiceCmd (29.28s)            functional_test.go:401: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node            functional_test.go:405: (dbg) Run:  kubectl --context functional-20191104T082125.872658733-18873 expose deployment hello-node --type=NodePort --port=8080            functional_test.go:410: (dbg) waiting for pods with labels "app=hello-node" in namespace "default" ...            helpers.go:247: "hello-node-7676b5fb8d-nxjqs" [9b3adf91-a112-4dbc-9b19-9d8801af3b43] Pending / Ready:ContainersNotReady (containers with unready status: [hello-node]) / ContainersReady:ContainersNotReady (containers with unready status: [hello-node])            helpers.go:247: "hello-node-7676b5fb8d-nxjqs" [9b3adf91-a112-4dbc-9b19-9d8801af3b43] Running            functional_test.go:410: (dbg) pods app=hello-node up and healthy within 27.004810384s            functional_test.go:414: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 service list            functional_test.go:414: (dbg) Done: out/minikube -p functional-20191104T082125.872658733-18873 service list: (1.640625892s)            functional_test.go:423: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 service --namespace=default --https --url hello-node            functional_test.go:441: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 service hello-node --url --format={{.IP}}            functional_test.go:450: (dbg) Run:  out/minikube -p functional-20191104T082125.872658733-18873 service hello-node --url            functional_test.go:464: url: http://192.168.39.24:31849        --- PASS: TestFunctional/parallel/DashboardCmd (41.31s)            functional_test.go:236: (dbg) daemon: [../../out/minikube dashboard --url -p functional-20191104T082125.872658733-18873 --alsologtostderr -v=1]            functional_test.go:241: (dbg) stopping [../../out/minikube dashboard --url -p functional-20191104T082125.872658733-18873 --alsologtostderr -v=1] ...            helpers.go:387: unable to kill pid 21063: os: process already finished    helpers.go:167: (dbg) Run:  out/minikube delete -p functional-20191104T082125.872658733-18873    helpers.go:167: (dbg) Done: out/minikube delete -p functional-20191104T082125.872658733-18873: (1.601200457s)=== RUN   TestAddons/parallel=== RUN   TestAddons/parallel/Registry=== PAUSE TestAddons/parallel/Registry=== RUN   TestAddons/parallel/Ingress=== PAUSE TestAddons/parallel/Ingress=== CONT  TestAddons/parallel/Registry=== CONT  TestAddons/parallel/Ingress2019/11/04 08:27:50 [DEBUG] GET http://192.168.39.56:5000--- PASS: TestAddons (192.32s)    helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:24:52.30063752 +1100 AEDT m=+240.074648116    addons_test.go:46: (dbg) Run:  out/minikube start -p addons-20191104T082452.300667929-18873 --wait=false --memory=2600 --alsologtostderr -v=1 --addons=ingress --addons=registry     addons_test.go:46: (dbg) Done: out/minikube start -p addons-20191104T082452.300667929-18873 --wait=false --memory=2600 --alsologtostderr -v=1 --addons=ingress --addons=registry : (2m28.528135985s)    --- PASS: TestAddons/parallel (0.00s)        --- PASS: TestAddons/parallel/Ingress (29.68s)            addons_test.go:97: (dbg) waiting for pods with labels "app.kubernetes.io/name=nginx-ingress-controller" in namespace "kube-system" ...            helpers.go:247: "nginx-ingress-controller-6fc5bcc8c9-pvwsq" [03a32eb7-f4fd-43ad-89f9-73f9c9407f1b] Running / Ready:ContainersNotReady (containers with unready status: [nginx-ingress-controller]) / ContainersReady:ContainersNotReady (containers with unready status: [nginx-ingress-controller])            addons_test.go:97: (dbg) pods app.kubernetes.io/name=nginx-ingress-controller up and healthy within 5.005704761s            addons_test.go:101: (dbg) Run:  kubectl --context addons-20191104T082452.300667929-18873 replace --force -f testdata/nginx-ing.yaml            addons_test.go:101: (dbg) Done: kubectl --context addons-20191104T082452.300667929-18873 replace --force -f testdata/nginx-ing.yaml: (2.671133592s)            addons_test.go:105: (dbg) Run:  kubectl --context addons-20191104T082452.300667929-18873 replace --force -f testdata/nginx-pod-svc.yaml            addons_test.go:110: (dbg) waiting for pods with labels "run=nginx" in namespace "default" ...            helpers.go:247: "nginx" [21d6e955-3c60-4673-b019-4b25d0af8f9d] Pending / Ready:ContainersNotReady (containers with unready status: [nginx]) / ContainersReady:ContainersNotReady (containers with unready status: [nginx])            helpers.go:247: "nginx" [21d6e955-3c60-4673-b019-4b25d0af8f9d] Running            addons_test.go:110: (dbg) pods run=nginx up and healthy within 21.024403427s            addons_test.go:119: (dbg) Run:  out/minikube -p addons-20191104T082452.300667929-18873 ssh "curl http://127.0.0.1:80 -H 'Host: nginx.example.com'"            addons_test.go:136: (dbg) Run:  out/minikube -p addons-20191104T082452.300667929-18873 addons disable ingress --alsologtostderr -v=1        --- PASS: TestAddons/parallel/Registry (30.14s)            addons_test.go:152: registry stabilized in 17.21522ms            addons_test.go:154: (dbg) waiting for pods with labels "actual-registry=true" in namespace "kube-system" ...            helpers.go:247: "registry-qw4s5" [3e6cc7cb-d88f-4a0c-9a83-29c7fbd01596] Running            addons_test.go:154: (dbg) pods actual-registry=true up and healthy within 5.008192127s            addons_test.go:157: (dbg) waiting for pods with labels "registry-proxy=true" in namespace "kube-system" ...            helpers.go:247: "registry-proxy-dd8rs" [f07e58a3-2550-4bd1-bbbe-d0413fee18ab] Running            addons_test.go:157: (dbg) pods registry-proxy=true up and healthy within 5.006662151s            addons_test.go:162: (dbg) Run:  kubectl --context addons-20191104T082452.300667929-18873 delete po -l run=registry-test --now            addons_test.go:167: (dbg) Run:  kubectl --context addons-20191104T082452.300667929-18873 run --rm registry-test --restart=Never --image=busybox -it -- sh -c "wget --spider -S http://registry.kube-system.svc.cluster.local"            addons_test.go:167: (dbg) Done: kubectl --context addons-20191104T082452.300667929-18873 run --rm registry-test --restart=Never --image=busybox -it -- sh -c "wget --spider -S http://registry.kube-system.svc.cluster.local": (19.592473452s)            addons_test.go:177: (dbg) Run:  out/minikube -p addons-20191104T082452.300667929-18873 ip            addons_test.go:206: (dbg) Run:  out/minikube -p addons-20191104T082452.300667929-18873 addons disable registry --alsologtostderr -v=1    addons_test.go:70: (dbg) Run:  out/minikube stop -p addons-20191104T082452.300667929-18873    addons_test.go:70: (dbg) Done: out/minikube stop -p addons-20191104T082452.300667929-18873: (13.233220482s)    addons_test.go:74: (dbg) Run:  out/minikube addons enable dashboard -p addons-20191104T082452.300667929-18873    addons_test.go:78: (dbg) Run:  out/minikube addons disable dashboard -p addons-20191104T082452.300667929-18873    helpers.go:167: (dbg) Run:  out/minikube delete -p addons-20191104T082452.300667929-18873--- PASS: TestDockerFlags (235.66s)    helpers.go:358: Waiting for start slot at 2019-11-04 08:25:52.300626185 +1100 AEDT m=+300.074636780 (sleeping 59.999921632s)  ...    docker_test.go:42: (dbg) Run:  out/minikube start -p docker-flags-20191104T082552.300971973-18873 --wait=false --docker-env=FOO=BAR --docker-env=BAZ=BAT --docker-opt=debug --docker-opt=icc=true --alsologtostderr -v=5     docker_test.go:42: (dbg) Done: out/minikube start -p docker-flags-20191104T082552.300971973-18873 --wait=false --docker-env=FOO=BAR --docker-env=BAZ=BAT --docker-opt=debug --docker-opt=icc=true --alsologtostderr -v=5 : (2m54.076176539s)    docker_test.go:47: (dbg) Run:  out/minikube -p docker-flags-20191104T082552.300971973-18873 ssh "systemctl show docker --property=Environment --no-pager"    docker_test.go:58: (dbg) Run:  out/minikube -p docker-flags-20191104T082552.300971973-18873 ssh "systemctl show docker --property=ExecStart --no-pager"    helpers.go:167: (dbg) Run:  out/minikube delete -p docker-flags-20191104T082552.300971973-18873    helpers.go:167: (dbg) Done: out/minikube delete -p docker-flags-20191104T082552.300971973-18873: (1.132863792s)--- PASS: TestVersionUpgrade (369.17s)    helpers.go:358: Waiting for start slot at 2019-11-04 08:25:22.300626185 +1100 AEDT m=+270.074636780 (sleeping 29.999935731s)  ...    version_upgrade_test.go:70: (dbg) Run:  /tmp/minikube-release.878849777.exe start -p vupgrade-20191104T082522.300800849-18873 --kubernetes-version=v1.11.10 --alsologtostderr -v=1     version_upgrade_test.go:70: (dbg) Done: /tmp/minikube-release.878849777.exe start -p vupgrade-20191104T082522.300800849-18873 --kubernetes-version=v1.11.10 --alsologtostderr -v=1 : (2m58.197200416s)    version_upgrade_test.go:79: (dbg) Run:  /tmp/minikube-release.878849777.exe stop -p vupgrade-20191104T082522.300800849-18873    version_upgrade_test.go:79: (dbg) Done: /tmp/minikube-release.878849777.exe stop -p vupgrade-20191104T082522.300800849-18873: (14.155612976s)    version_upgrade_test.go:84: (dbg) Run:  /tmp/minikube-release.878849777.exe -p vupgrade-20191104T082522.300800849-18873 status --format={{.Host}}    version_upgrade_test.go:84: (dbg) Non-zero exit: /tmp/minikube-release.878849777.exe -p vupgrade-20191104T082522.300800849-18873 status --format={{.Host}}: exit status 1 (68.698765ms)        -- stdout --        Stopped        -- /stdout --    version_upgrade_test.go:86: status error: exit status 1 (may be ok)    helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:28:37.164720757 +1100 AEDT m=+464.938731345    version_upgrade_test.go:95: (dbg) Run:  out/minikube start -p vupgrade-20191104T082522.300800849-18873 --kubernetes-version=v1.16.2 --alsologtostderr -v=1     version_upgrade_test.go:95: (dbg) Done: out/minikube start -p vupgrade-20191104T082522.300800849-18873 --kubernetes-version=v1.16.2 --alsologtostderr -v=1 : (2m23.324335645s)    helpers.go:167: (dbg) Run:  out/minikube delete -p vupgrade-20191104T082522.300800849-18873--- PASS: TestStartStop (626.25s)    --- PASS: TestStartStop/group (0.00s)        --- PASS: TestStartStop/group/docker (416.69s)            helpers.go:358: Waiting for start slot at 2019-11-04 08:26:22.300626185 +1100 AEDT m=+330.074636780 (sleeping 1m29.999762906s)  ...            start_stop_delete_test.go:91: (dbg) Run:  out/minikube start -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3 --wait=true --cache-images=false --kubernetes-version=v1.11.10 --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --container-runtime=docker             start_stop_delete_test.go:91: (dbg) Done: out/minikube start -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3 --wait=true --cache-images=false --kubernetes-version=v1.11.10 --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --container-runtime=docker : (3m9.827465706s)            start_stop_delete_test.go:102: (dbg) Run:  kubectl --context docker-20191104T082622.300761702-18873 create -f testdata/busybox.yaml            start_stop_delete_test.go:102: (dbg) Done: kubectl --context docker-20191104T082622.300761702-18873 create -f testdata/busybox.yaml: (1.045195614s)            start_stop_delete_test.go:107: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [06dd0c10-fe81-11e9-a894-5c5fdd3720f5] Pending / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])            helpers.go:247: "busybox" [06dd0c10-fe81-11e9-a894-5c5fdd3720f5] Running            start_stop_delete_test.go:107: (dbg) pods integration-test=busybox up and healthy within 16.06885973s            start_stop_delete_test.go:113: (dbg) Run:  kubectl --context docker-20191104T082622.300761702-18873 exec busybox -- /bin/sh -c "ulimit -n"            start_stop_delete_test.go:130: (dbg) Run:  out/minikube stop -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3            start_stop_delete_test.go:130: (dbg) Done: out/minikube stop -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3: (17.353839074s)            start_stop_delete_test.go:135: (dbg) Run:  out/minikube status --format={{.Host}} -p docker-20191104T082622.300761702-18873            start_stop_delete_test.go:135: (dbg) Non-zero exit: out/minikube status --format={{.Host}} -p docker-20191104T082622.300761702-18873: exit status 1 (37.663869ms)                -- stdout --                Stopped                -- /stdout --            start_stop_delete_test.go:135: status error: exit status 1 (may be ok)            helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:30:07.029124735 +1100 AEDT m=+554.803135321            start_stop_delete_test.go:141: (dbg) Run:  out/minikube start -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3 --wait=true --cache-images=false --kubernetes-version=v1.11.10 --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --container-runtime=docker             start_stop_delete_test.go:141: (dbg) Done: out/minikube start -p docker-20191104T082622.300761702-18873 --alsologtostderr -v=3 --wait=true --cache-images=false --kubernetes-version=v1.11.10 --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --container-runtime=docker : (1m35.29120763s)            start_stop_delete_test.go:149: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [06dd0c10-fe81-11e9-a894-5c5fdd3720f5] Running            start_stop_delete_test.go:149: (dbg) pods integration-test=busybox up and healthy within 5.642024689s            start_stop_delete_test.go:153: (dbg) Run:  out/minikube status --format={{.Host}} -p docker-20191104T082622.300761702-18873            helpers.go:167: (dbg) Run:  out/minikube delete -p docker-20191104T082622.300761702-18873        --- PASS: TestStartStop/group/cni (467.93s)            helpers.go:358: Waiting for start slot at 2019-11-04 08:27:52.300626185 +1100 AEDT m=+420.074636780 (sleeping 2m59.999736017s)  ...            start_stop_delete_test.go:91: (dbg) Run:  out/minikube start -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3 --wait=true --feature-gates ServerSideApply=true --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubeadm.pod-network-cidr=192.168.111.111/16 --kubernetes-version=v1.16.2             start_stop_delete_test.go:91: (dbg) Done: out/minikube start -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3 --wait=true --feature-gates ServerSideApply=true --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubeadm.pod-network-cidr=192.168.111.111/16 --kubernetes-version=v1.16.2 : (3m50.111617403s)            start_stop_delete_test.go:99: WARNING: cni mode requires additional setup before pods can schedule :(            start_stop_delete_test.go:130: (dbg) Run:  out/minikube stop -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3            start_stop_delete_test.go:130: (dbg) Done: out/minikube stop -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3: (13.361886938s)            start_stop_delete_test.go:135: (dbg) Run:  out/minikube status --format={{.Host}} -p cni-20191104T082752.300771556-18873            start_stop_delete_test.go:135: (dbg) Non-zero exit: out/minikube status --format={{.Host}} -p cni-20191104T082752.300771556-18873: exit status 1 (40.269985ms)                -- stdout --                Stopped                -- /stdout --            start_stop_delete_test.go:135: status error: exit status 1 (may be ok)            helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:31:55.814766655 +1100 AEDT m=+663.588777252            start_stop_delete_test.go:141: (dbg) Run:  out/minikube start -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3 --wait=true --feature-gates ServerSideApply=true --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubeadm.pod-network-cidr=192.168.111.111/16 --kubernetes-version=v1.16.2             start_stop_delete_test.go:141: (dbg) Done: out/minikube start -p cni-20191104T082752.300771556-18873 --alsologtostderr -v=3 --wait=true --feature-gates ServerSideApply=true --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubeadm.pod-network-cidr=192.168.111.111/16 --kubernetes-version=v1.16.2 : (43.328528439s)            start_stop_delete_test.go:148: WARNING: cni mode requires additional setup before pods can schedule :(            start_stop_delete_test.go:153: (dbg) Run:  out/minikube status --format={{.Host}} -p cni-20191104T082752.300771556-18873            helpers.go:167: (dbg) Run:  out/minikube delete -p cni-20191104T082752.300771556-18873        --- PASS: TestStartStop/group/containerd (555.16s)            helpers.go:358: Waiting for start slot at 2019-11-04 08:26:52.300626185 +1100 AEDT m=+360.074636780 (sleeping 1m59.999750306s)  ...            start_stop_delete_test.go:91: (dbg) Run:  out/minikube start -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3 --wait=true --container-runtime=containerd --docker-opt containerd=/var/run/containerd/containerd.sock --apiserver-port=8444             start_stop_delete_test.go:91: (dbg) Done: out/minikube start -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3 --wait=true --container-runtime=containerd --docker-opt containerd=/var/run/containerd/containerd.sock --apiserver-port=8444 : (4m29.732924894s)            start_stop_delete_test.go:102: (dbg) Run:  kubectl --context containerd-20191104T082652.300780979-18873 create -f testdata/busybox.yaml            start_stop_delete_test.go:102: (dbg) Done: kubectl --context containerd-20191104T082652.300780979-18873 create -f testdata/busybox.yaml: (3.022589629s)            start_stop_delete_test.go:107: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [90846118-a891-4edd-86a1-4df14dbdacdd] Pending            helpers.go:247: "busybox" [90846118-a891-4edd-86a1-4df14dbdacdd] Pending / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])            helpers.go:247: "busybox" [90846118-a891-4edd-86a1-4df14dbdacdd] Running            start_stop_delete_test.go:107: (dbg) pods integration-test=busybox up and healthy within 11.883410366s            start_stop_delete_test.go:113: (dbg) Run:  kubectl --context containerd-20191104T082652.300780979-18873 exec busybox -- /bin/sh -c "ulimit -n"            start_stop_delete_test.go:130: (dbg) Run:  out/minikube stop -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3            start_stop_delete_test.go:130: (dbg) Done: out/minikube stop -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3: (1m31.837876123s)            start_stop_delete_test.go:135: (dbg) Run:  out/minikube status --format={{.Host}} -p containerd-20191104T082652.300780979-18873            start_stop_delete_test.go:135: (dbg) Non-zero exit: out/minikube status --format={{.Host}} -p containerd-20191104T082652.300780979-18873: exit status 1 (35.925354ms)                -- stdout --                Stopped                -- /stdout --            start_stop_delete_test.go:135: status error: exit status 1 (may be ok)            helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:33:09.026967426 +1100 AEDT m=+736.800978015            start_stop_delete_test.go:141: (dbg) Run:  out/minikube start -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3 --wait=true --container-runtime=containerd --docker-opt containerd=/var/run/containerd/containerd.sock --apiserver-port=8444             start_stop_delete_test.go:141: (dbg) Done: out/minikube start -p containerd-20191104T082652.300780979-18873 --alsologtostderr -v=3 --wait=true --container-runtime=containerd --docker-opt containerd=/var/run/containerd/containerd.sock --apiserver-port=8444 : (52.484285038s)            start_stop_delete_test.go:149: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [90846118-a891-4edd-86a1-4df14dbdacdd] Running / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])            helpers.go:247: "busybox" [90846118-a891-4edd-86a1-4df14dbdacdd] Running            start_stop_delete_test.go:149: (dbg) pods integration-test=busybox up and healthy within 5.008644281s            start_stop_delete_test.go:153: (dbg) Run:  out/minikube status --format={{.Host}} -p containerd-20191104T082652.300780979-18873            helpers.go:167: (dbg) Run:  out/minikube delete -p containerd-20191104T082652.300780979-18873        --- PASS: TestStartStop/group/crio (626.25s)            helpers.go:358: Waiting for start slot at 2019-11-04 08:27:22.300626185 +1100 AEDT m=+390.074636780 (sleeping 2m29.999745542s)  ...            start_stop_delete_test.go:91: (dbg) Run:  out/minikube start -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3 --wait=true --container-runtime=crio --disable-driver-mounts --extra-config=kubeadm.ignore-preflight-errors=SystemVerification             start_stop_delete_test.go:91: (dbg) Done: out/minikube start -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3 --wait=true --container-runtime=crio --disable-driver-mounts --extra-config=kubeadm.ignore-preflight-errors=SystemVerification : (4m52.989940404s)            start_stop_delete_test.go:102: (dbg) Run:  kubectl --context crio-20191104T082722.304206154-18873 create -f testdata/busybox.yaml            start_stop_delete_test.go:107: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [2660e105-871f-4b5e-8e06-306ee3ed6648] Pending            helpers.go:247: "busybox" [2660e105-871f-4b5e-8e06-306ee3ed6648] Pending / Ready:ContainersNotReady (containers with unready status: [busybox]) / ContainersReady:ContainersNotReady (containers with unready status: [busybox])            helpers.go:247: "busybox" [2660e105-871f-4b5e-8e06-306ee3ed6648] Running            start_stop_delete_test.go:107: (dbg) pods integration-test=busybox up and healthy within 8.510315758s            start_stop_delete_test.go:113: (dbg) Run:  kubectl --context crio-20191104T082722.304206154-18873 exec busybox -- /bin/sh -c "ulimit -n"            start_stop_delete_test.go:130: (dbg) Run:  out/minikube stop -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3            start_stop_delete_test.go:130: (dbg) Done: out/minikube stop -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3: (1m32.087370333s)            start_stop_delete_test.go:135: (dbg) Run:  out/minikube status --format={{.Host}} -p crio-20191104T082722.304206154-18873            start_stop_delete_test.go:135: (dbg) Non-zero exit: out/minikube status --format={{.Host}} -p crio-20191104T082722.304206154-18873: exit status 1 (35.519744ms)                -- stdout --                Stopped                -- /stdout --            start_stop_delete_test.go:135: status error: exit status 1 (may be ok)            helpers.go:361: No need to wait for start slot, it is already 2019-11-04 08:33:56.977436358 +1100 AEDT m=+784.751446950            start_stop_delete_test.go:141: (dbg) Run:  out/minikube start -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3 --wait=true --container-runtime=crio --disable-driver-mounts --extra-config=kubeadm.ignore-preflight-errors=SystemVerification             start_stop_delete_test.go:141: (dbg) Done: out/minikube start -p crio-20191104T082722.304206154-18873 --alsologtostderr -v=3 --wait=true --container-runtime=crio --disable-driver-mounts --extra-config=kubeadm.ignore-preflight-errors=SystemVerification : (1m15.390161848s)            start_stop_delete_test.go:149: (dbg) waiting for pods with labels "integration-test=busybox" in namespace "default" ...            helpers.go:247: "busybox" [2660e105-871f-4b5e-8e06-306ee3ed6648] Running            start_stop_delete_test.go:149: (dbg) pods integration-test=busybox up and healthy within 5.011162349s            start_stop_delete_test.go:153: (dbg) Run:  out/minikube status --format={{.Host}} -p crio-20191104T082722.304206154-18873            helpers.go:167: (dbg) Run:  out/minikube delete -p crio-20191104T082722.304206154-18873PASSok  k8s.io/minikube/test/integration866.327s

Kubernetes Brain Dump (ALWAYS WIP)

Sat, 30 Nov 2019 00:00:00 +0000

Linux Network Namespaces is one of the important feature in Linux Kernel that are leveraged by containers like docker and kubernetes. It is used in the CNI (Container Network Interface).
A Kubernetes cluster can be deployed on either physical or virtual machines.
Once you’ve created a Deployment, the Kubernetes master schedules mentioned application instances onto individual Nodes in the cluster.
Once the application instances are created, a Kubernetes Deployment Controller continuously monitors those instances. If the Node hosting an instance goes down or is deleted, the Deployment controller replaces the instance with an instance on another Node in the cluster. This provides a self-healing mechanism to address machine failure or maintenance.
Node - A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster.
Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master. The node should also have tools for handling container operations, such as Docker or rkt.
A Kubernetes cluster that handles production traffic should have a minimum of three nodes.
The nodes communicate with the master using the Kubernetes API. End users can also use the Kubernetes API directly to interact with the cluster.
Node Components run on every node, maintaining running pods and providing the Kubernetes runtime environment.
- kubelet - An agent that runs on each node in the cluster. It makes sure that containers are running in a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.
- kube-proxy - kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept. kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster. kube-proxy uses the operating system packet filtering layer if there is one and it’s available. Otherwise, kube-proxy forwards the traffic itself.
- Container Runtime - The container runtime is the software that is responsible for running containers.
- A node is a worker machine in Kubernetes, previously known as a minion. A node may be a VM or physical machine, depending on the cluster. Each node contains the services necessary to run pods and is managed by the master components.
- A node’s status contains the following information:
  - Addresses
  - Conditions
  - Capacity and Allocatable
  - Info

Node

Currently, there are three components that interact with the Kubernetes node interface: node controller, kubelet, and kubectl.
- Node Controller
  - The node controller is a Kubernetes master component which manages various aspects of nodes.
  - Assigning a CIDR block to the node when it is registered (if CIDR assignment is turned on).
  - Keeping the node controller’s internal list of nodes up to date with the cloud provider’s list of available machines. When running in a cloud environment, whenever a node is unhealthy, the node controller asks the cloud provider if the VM for that node is still available. If not, the node controller deletes the node from its list of nodes.
  - The node controller is responsible for updating the NodeReady condition of NodeStatus to ConditionUnknown when a node becomes unreachable (i.e. the node controller stops receiving heartbeats for some reason, for example due to the node being down), and then later evicting all the pods from the node (using graceful termination) if the node continues to be unreachable. (The default timeouts are 40s to start reporting ConditionUnknown and 5m after that to start evicting pods.) The node controller checks the state of each node every –node-monitor-period seconds.

Pod

Pod is something logical and not physical. It is just a logical wrapper for a set of containers with it’s resources. In terms of Docker constructs, a Pod is modelled as a group of Docker containers with shared namespaces and shared filesystem volumes.
Containers within a Pod share an IP address and port space, and can find each other via localhost. They can also communicate with each other using standard inter-process communications like SystemV semaphores or POSIX shared memory. Containers in different Pods have distinct IP addresses and can not communicate by IPC without special configuration. These containers usually communicate with each other via Pod IP addresses.
A Pod is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.
A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as docker or rkt), and some shared resources for those containers. Those resources include:
- Shared storage, as Volumes
- Networking, as a unique cluster IP address
- Information about how to run each container, such as the container image version or specific ports to use
- The containers in a Pod share an IP Address and port space, are always co-located and co-scheduled, and run in a shared context on the same Node.
A Pod always runs on a Node
- A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster.
- Each Node is managed by the Master.
- A Node can have multiple pods, and the Kubernetes master automatically handles scheduling the pods across the Nodes in the cluster.
- The Master’s automatic scheduling takes into account the available resources on each Node.
- Every Kubernetes Node runs at least:
  - Kubelet, a process responsible for communication between the Kubernetes Master and the Node; it manages the Pods and the containers running on a machine.
  - A container runtime (like Docker, rkt) responsible for pulling the container image from a registry, unpacking the container, and running the application.
  - Containers should only be scheduled together in a single Pod if they are tightly coupled and need to share resources such as disk.
The Kubernetes Master is a collection of three processes that run on a single node in your cluster, which is designated as the master node. Those processes are: kube-apiserver, kube-controller-manager and kube-scheduler.
- Each individual non-master node in your cluster runs two processes:
  - kubelet, which communicates with the Kubernetes Master.
  - kube-proxy, a network proxy which reflects Kubernetes networking services on each node.
Pod Lifecycle
- Container probes
  - A Probe is a diagnostic performed periodically by the kubelet on a Container. To perform a diagnostic, the kubelet calls a Handler implemented by the Container.
    - ExecAction: Executes a specified command inside the Container. The diagnostic is considered successful if the command exits with a status code of 0.
    - TCPSocketAction: Performs a TCP check against the Container’s IP address on a specified port. The diagnostic is considered successful if the port is open.
    - HTTPGetAction: The diagnostic is considered successful if the response has a status code greater than or equal to 200 and less than 400.
  - The kubelet can optionally perform and react to three kinds of probes on running Containers:
    - livenessProbe: Indicates whether the Container is running. If the liveness probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy. If a Container does not provide a liveness probe, the default state is Success.
    - readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod. The default state of readiness before the initial delay is Failure. If a Container does not provide a readiness probe, the default state is Success.
    - startupProbe: Indicates whether the application within the Container is started. All other probes are disabled if a startup probe is provided, until it succeeds. If the startup probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy.
- Use different kind of pods for different kind of usage:
  - Use a Job for Pods that are expected to terminate, for example, batch computations. Jobs are appropriate only for Pods with restartPolicy equal to OnFailure or Never.
  - Use a ReplicationController, ReplicaSet, or Deployment for Pods that are not expected to terminate, for example, web servers. ReplicationControllers are appropriate only for Pods with a restartPolicy of Always.
  - Use a DaemonSet for Pods that need to run one per machine, because they provide a machine-specific system service.
- Init Containers
  - A Pod can have one or more init containers, which are run before the app containers are started.
  - Init containers are exactly like regular containers, except:
    - Init containers always run to completion.
    - Each init container must complete successfully before the next one starts.
  - Kubernetes repeatedly restarts the Pod until the init container succeeds. However, if the Pod has a restartPolicy of Never, Kubernetes does not restart the Pod.
  - Init containers support all the fields and features of app containers, including resource limits, volumes, and security settings.
  - Init containers do not support readiness probes because they must run to completion before the Pod can be ready.
  - If you specify multiple init containers for a Pod, Kubelet runs each init container sequentially. Each init container must succeed before the next can run. When all of the init containers have run to completion, Kubelet initializes the application containers for the Pod and runs them as usual.
  - Example of defining init containers for a pod. The init containers will keep
```
apiVersion: v1kind: Podmetadata:  name: myapp-pod  labels:    app: myappspec:  containers:  - name: myapp-container    image: busybox:1.28    command: ['sh', '-c', 'echo The app is running! && sleep 3600']  initContainers:  - name: init-myservice    image: busybox:1.28    command: ['sh', '-c', "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"]  - name: init-mydb    image: busybox:1.28    command: ['sh', '-c', "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done"]---apiVersion: v1kind: Servicemetadata:  name: myservicespec:  ports:  - protocol: TCP    port: 80    targetPort: 9376---apiVersion: v1kind: Servicemetadata:  name: mydbspec:  ports:  - protocol: TCP    port: 80    targetPort: 9377 
```
- Pod Preset
  - Kubernetes provides an admission controller (PodPreset) which, when enabled, applies Pod Presets to incoming pod creation requests.
  - When a pod creation request occurs, the system does the following:
    - Retrieve all PodPresets available for use.
    - Check if the label selectors of any PodPreset matches the labels on the pod being created.
    - Attempt to merge the various resources defined by the PodPreset into the Pod being created.
    - On error, throw an event documenting the merge error on the pod, and create the pod without any injected resources from the PodPreset.
    - Annotate the resulting modified Pod spec, the annotation is of the form podpreset.admission.kubernetes.io/podpreset-: “”.

Controllers

ReplicaSet
- A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.
- How a ReplicaSet works
  - A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number of replicas indicating how many Pods it should be maintaining.
  - A ReplicaSet then fulfills its purpose by creating and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod template.
  - A ReplicaSet identifies new Pods to acquire by using its selector.
  - Example:
```
apiVersion: apps/v1kind: ReplicaSetmetadata:  name: frontend  labels:    app: guestbook    tier: frontendspec:  # modify replicas according to your case  replicas: 3  selector:    matchLabels:      tier: frontend  template:    metadata:      labels:    tier: frontend    spec:      containers:      - name: php-redis    image: gcr.io/google_samples/gb-frontend:v3
```
- Make sure when declaring Pod kind: the metadata.labels.tier does not match the one in the ReplicaSet selector.matchLabels.tier field. If it does match the Pod will be obtained as part of the ReplicaSet. Look under Non-Template Pod acquisitions in for more info.
ReplicationController
- A ReplicationController ensures that a specified number of pod replicas are running at any one time. In other words, a ReplicationController makes sure that a pod or a homogeneous set of pods is always up and available.
- A ReplicationController is similar to a process supervisor, but instead of supervising individual processes on a single node, the ReplicationController supervises multiple pods across multiple nodes.
- A simple case is to create one ReplicationController object to reliably run one instance of a Pod indefinitely. A more complex use case is to run several identical replicas of a replicated service, such as web servers.
- Example:
```
apiVersion: v1kind: ReplicationControllermetadata:  name: nginxspec:  replicas: 3  selector:    app: nginx  template:    metadata:      name: nginx      labels:    app: nginx    spec:      containers:      - name: nginx    image: nginx    ports:    - containerPort: 80
```
Deployments
- A Deployment provides declarative updates for Pods and ReplicaSets. The Deployment Controller changes the actual state to the desired state at a controlled rate.
- You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.
- Example:
```
apiVersion: apps/v1kind: Deploymentmetadata:  name: nginx-deployment  labels:    app: nginxspec:  replicas: 3  selector:    matchLabels:      app: nginx  template:    metadata:      labels:    app: nginx    spec:      containers:      - name: nginx    image: nginx:1.14.2    ports:    - containerPort: 80
```
- Following are the explanation of the meaning of the above example:
  - A Deployment named nginx-deployment is created, indicated by the .metadata.name field.
  - The Deployment creates three replicated Pods, indicated by the replicas field.
  - The selector field defines how the Deployment finds which Pods to manage. In this case, you simply select a label that is defined in the Pod template (app: nginx). However, more sophisticated selection rules are possible, as long as the Pod template itself satisfies the rule.
  - The template field contains the following sub-fields:
    - The Pods are labeled app: nginxusing the labels field.
    - The Pod template’s specification, or .template.spec field, indicates that the Pods run one container, nginx, which runs the nginx Docker Hub image at version 1.14.2.
    - Create one container and name it nginx using the name field.
- To see the Deployment rollout status, run
```
kubectl rollout status deployment.v1.apps/nginx-deployment.
```
- To see the ReplicaSet (rs) created by the Deployment, run
```
kubectl get rs
```
- Updating deployment:
  - Update the nginx Pods to use the nginx:1.16.1 image instead of the nginx:1.14.2 image.
```
kubectl --record deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1
```
    or simply use the following command
```
kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 --record
```
    Alternatively, you can edit the Deployment and change .spec.template.spec.containers[0].image from nginx:1.14.2 to nginx:1.16.1
```
kubectl edit deployment.v1.apps/nginx-deployment
```
- Get details of your Deployment:
```
kubectl describe deployments
```
- For example, suppose you create a Deployment to create 5 replicas of nginx:1.14.2, but then update the Deployment to create 5 replicas of nginx:1.16.1, when only 3 replicas of nginx:1.14.2 had been created. In that case, the Deployment immediately starts killing the 3 nginx:1.14.2 Pods that it had created, and starts creating nginx:1.16.1 Pods. It does not wait for the 5 replicas of nginx:1.14.2 to be created before changing course
- Rolling Back a Deployment
  - Suppose that you made a typo while updating the Deployment, by putting the image name as nginx:1.161 instead of nginx:1.16.1:
```
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true
```
  - The rollout gets stuck. You can verify it by checking the rollout status:
```
kubectl rollout status deployment.v1.apps/nginx-deployment
```
  - Rolling Back to a Previous Revision
```
 kubectl rollout undo deployment.v1.apps/nginx-deployment
```
- Scaling a Deployment
  - You can scale a Deployment by using the following command:
```
kubectl scale deployment.v1.apps/nginx-deployment --replicas=10
```
  - Assuming horizontal Pod autoscaling is enabled in your cluster, you can setup an autoscaler for your Deployment and choose the minimum and maximum number of Pods you want to run based on the CPU utilization of your existing Pods.
```
kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80
```
- Pausing and Resuming a Deployment
  - Pause by running the following command:
```
kubectl rollout pause deployment.v1.apps/nginx-deployment
```
  - You can make as many updates as you wish, for example, update the resources that will be used:
```
kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
```
  - Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates:
```
 kubectl rollout resume deployment.v1.apps/nginx-deployment
```
StatefulSets
- StatefulSet is the workload API object used to manage stateful applications. Manages the deployment and scaling of a set of Pods , and provides guarantees about the ordering and uniqueness of these Pods. Like a Deployment , a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.
- StatefulSets are valuable for applications that require one or more of the following.
  - Stable, unique network identifiers.
  - Stable, persistent storage.
  - Ordered, graceful deployment and scaling.
  - Ordered, automated rolling updates.
  Stable is synonymous with persistence across Pod (re)scheduling. If an application doesn’t require any stable identifiers or ordered deployment, deletion, or scaling, you should deploy your application using a workload object that provides a set of stateless replicas. Deployment or ReplicaSet may be better suited to your stateless needs
- Example:
```
apiVersion: v1kind: Servicemetadata:  name: nginx  labels:    app: nginxspec:  ports:  - port: 80    name: web  clusterIP: None  selector:    app: nginx---apiVersion: apps/v1kind: StatefulSetmetadata:  name: webspec:  selector:    matchLabels:      app: nginx # has to match .spec.template.metadata.labels  serviceName: "nginx"  replicas: 3 # by default is 1  template:    metadata:      labels:    app: nginx # has to match .spec.selector.matchLabels    spec:      terminationGracePeriodSeconds: 10      containers:      - name: nginx    image: k8s.gcr.io/nginx-slim:0.8    ports:    - containerPort: 80      name: web    volumeMounts:    - name: www      mountPath: /usr/share/nginx/html  volumeClaimTemplates:  - metadata:      name: www    spec:      accessModes: [ "ReadWriteOnce" ]      storageClassName: "my-storage-class"      resources:    requests:      storage: 1Gi
```
  Explanation about the above example:
  - A Headless Service, named nginx, is used to control the network domain.
  - The StatefulSet, named web, has a Spec that indicates that 3 replicas of the nginx container will be launched in unique Pods.
  - The volumeClaimTemplates will provide stable storage using PersistentVolumes provisioned by a PersistentVolume Provisioner
- Stable Network ID
  - Each Pod in a StatefulSet derives its hostname from the name of the StatefulSet and the ordinal of the Pod.
  - The pattern for the constructed hostname is $(statefulset name)-$(ordinal). The example above will create three Pods named web-0,web-1,web-2.
  - A StatefulSet can use a Headless Service to control the domain of its Pods. The domain managed by this Service takes the form: $(service name).$(namespace).svc.cluster.local, where “cluster.local” is the cluster domain.
  - As each Pod is created, it gets a matching DNS subdomain, taking the form: $(podname).$(governing service domain), where the governing service is defined by the serviceName field on the StatefulSet.
  - Here are some examples of choices for Cluster Domain, Service name, StatefulSet name, and how that affects the DNS names for the StatefulSet’s Pods.
- Stable Storage
  - In the nginx example above, each Pod will receive a single PersistentVolume with a StorageClass of my-storage-class and 1 Gib of provisioned storage.
  - If no StorageClass is specified, then the default StorageClass will be used. When a Pod is (re)scheduled onto a node, its volumeMounts mount the PersistentVolumes associated with its PersistentVolume Claims.
  - Note that, the PersistentVolumes associated with the Pods’ PersistentVolume Claims are not deleted when the Pods, or StatefulSet are deleted. This must be done manually.
- Pod Name Label
  - When the StatefulSet Controller creates a Pod, it adds a label, statefulset.kubernetes.io/pod-name, that is set to the name of the Pod. This label allows you to attach a Service to a specific Pod in the StatefulSet
DaemonSet
- A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.
- Some typical uses of a DaemonSet are:
  - running a cluster storage daemon, such as glusterd, ceph, on each node.
  - running a logs collection daemon on every node, such as fluentd or filebeat.
  - running a node monitoring daemon on every node, such as Prometheus Node Exporter, Flowmill, Sysdig Agent, collectd, Dynatrace OneAgent, AppDynamics Agent, Datadog agent, New Relic agent, Ganglia gmond, Instana Agent or Elastic Metricbeat.
- In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon. A more complex setup might use multiple DaemonSets for a single type of daemon, but with different flags and/or different memory and cpu requests for different hardware types.
- Example:
```
apiVersion: apps/v1kind: DaemonSetmetadata:name: fluentd-elasticsearchnamespace: kube-systemlabels:k8s-app: fluentd-loggingspec:selector:matchLabels:  name: fluentd-elasticsearchtemplate:metadata:  labels:name: fluentd-elasticsearchspec:  tolerations:  # this toleration is to have the daemonset runnable on master nodes  # remove it if your masters can't run pods  - key: node-role.kubernetes.io/mastereffect: NoSchedule  containers:  - name: fluentd-elasticsearchimage: quay.io/fluentd_elasticsearch/fluentd:v2.5.2resources:  limits:    memory: 200Mi  requests:    cpu: 100m    memory: 200MivolumeMounts:- name: varlog  mountPath: /var/log- name: varlibdockercontainers  mountPath: /var/lib/docker/containers  readOnly: true  terminationGracePeriodSeconds: 30  volumes:  - name: varloghostPath:  path: /var/log  - name: varlibdockercontainershostPath:  path: /var/lib/docker/containers
```
- How Daemon Pods are Scheduled
  - Scheduled by default scheduler
    - A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the node that a Pod runs on is selected by the Kubernetes scheduler.
    - However, DaemonSet pods are created and scheduled by the DaemonSet controller instead. That introduces the following issues:
      - Inconsistent Pod behavior: Normal Pods waiting to be scheduled are created and in Pending state, but DaemonSet pods are not created in Pending state. This is confusing to the user.
      - Pod preemption is handled by default scheduler. When preemption is enabled, the DaemonSet controller will make scheduling decisions without considering pod priority and preemption.
    - ScheduleDaemonSetPods allows you to schedule DaemonSets using the default scheduler instead of the DaemonSet controller, by adding the NodeAffinity term to the DaemonSet pods, instead of the .spec.nodeName term.
    - The default scheduler is then used to bind the pod to the target host. If node affinity of the DaemonSet pod already exists, it is replaced. The DaemonSet controller only performs these operations when creating or modifying DaemonSet pods, and no changes are made to the spec.template of the DaemonSet.
      nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - target-host-name
  - Taints and Tolerations

Services

A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. Services enable a loose coupling between dependent Pods. A Service is defined using YAML (preferred) or JSON, like all Kubernetes objects. The set of Pods targeted by a Service is usually determined by a LabelSelector (see below for why you might want a Service without including selector in the spec).
Services can be exposed in different ways by specifying a type in the ServiceSpec:
- ClusterIP (default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.
- NodePort - Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using :. Superset of ClusterIP.
- LoadBalancer - Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.
- ExternalName - Exposes the Service using an arbitrary name (specified by externalName in the spec) by returning a CNAME record with the name. No proxy is used. This type requires v1.7 or higher of kube-dns.
Services match a set of Pods using labels and selectors, a grouping primitive that allows logical operation on objects in Kubernetes. Labels are key/value pairs attached to objects and can be used in any number of ways:
- Designate objects for development, test, and production
- Embed version tags
- Classify an object using tags
Every node in a Kubernetes cluster runs a kube-proxy. kube-proxy is responsible for implementing a form of virtual IP for Services of type other than ExternalName
DNS
- For example, if you have a Service called “my-service” in a Kubernetes Namespace “my-ns”, the control plane and the DNS Service acting together create a DNS record for “my-service.my-ns”. Pods in the “my-ns” Namespace should be able to find it by simply doing a name lookup for my-service (“my-service.my-ns” would also work). Pods in other Namespaces must qualify the name as my-service.my-ns. These names will resolve to the cluster IP assigned for the Service.
- Kubernetes also supports DNS SRV (Service) records for named ports. If the “my-service.my-ns” Service has a port named “http” with the protocol set to TCP, you can do a DNS SRV query for _http._tcp.my-service.my-ns to discover the port number for “http”, as well as the IP address
Once you’ve set your desired state, the Kubernetes Control Plane makes the cluster’s current state match the desired state via the Pod Lifecycle Event Generator (PLEG). The Kubernetes Control Plane consists of a collection of processes running on your cluster:
- The various parts of the Kubernetes Control Plane, such as the Kubernetes Master and kubelet processes, govern how Kubernetes communicates with your cluster. The Control Plane maintains a record of all of the Kubernetes Objects in the system, and runs continuous control loops to manage those objects’ state. At any given time, the Control Plane’s control loops will respond to changes in the cluster and work to make the actual state of all the objects in the system match the desired state that you provided.
Master Components-Master components provide the cluster’s control plane. Master components make global decisions about the cluster (for example, scheduling), and they detect and respond to cluster events (for example, starting up a new pod when a deployment’s replicas field is unsatisfied). For simplicity, set up scripts typically start all master components on the same machine, and do not run user containers on this machine. See Building High-Availability Clusters for an example multi-master-VM setup.
- kube-apiserver - Component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane.
- etcd - Consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.
- kube-scheduler - Component on the master that watches newly created pods that have no node assigned, and selects a node for them to run on.
- kube-controller-manager - Component on the master that runs controllers.
- Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. These controllers include:
  - Node Controller: Responsible for noticing and responding when nodes go down.
  - Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
  - Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
  - Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.
- cloud-controller-manager - cloud-controller-manager runs controllers that interact with the underlying cloud providers. The cloud-controller-manager binary is an alpha feature introduced in Kubernetes release 1.6.
  - cloud-controller-manager runs cloud-provider-specific controller loops only. You must disable these controller loops in the kube-controller-manager. You can disable the controller loops by setting the –cloud-provider flag to external when starting the kube-controller-manager. cloud-controller-manager allows the cloud vendor’s code and the Kubernetes code to evolve independently of each other. In prior releases, the core Kubernetes code was dependent upon cloud-provider-specific code for functionality. In future releases, code specific to cloud vendors should be maintained by the cloud vendor themselves, and linked to cloud-controller-manager while running Kubernetes.
  - The following controllers have cloud provider dependencies:
    [x] Node Controller – For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
    [x] Route Controller – For setting up routes in the underlying cloud infrastructure
    [x] Service Controller - For creating, updating and deleting cloud provider load balancers
    [x] Volume Controller - For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes

Addons

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features. Because these are providing cluster-level features, namespaced resources for addons belong within the kube-system namespace.

DNS - While the other addons are not strictly required, all Kubernetes clusters should have cluster DNS, as many examples rely on it. Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services. Containers started by Kubernetes automatically include this DNS server in their DNS searches.
Web UI (Dashboard) - Dashboard is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage and troubleshoot applications running in the cluster, as well as the cluster itself.
Container Resource Monitoring - Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.
Cluster-level Logging - Cluster-level logging mechanism is responsible for saving container logs to a central log store with search/browsing interface.
The basic Kubernetes objects include:
- Pod
- Service
- Volume
- Namespace

kubeadm

Running ‘kubeadm init’ will run phases of initialization to setup Kubernetes cluster. The code that break down these steps reside inside app/cmd/phases/init
- preflight step reside inside app/preflight
- app/preflight/checks.go – contains lots of checking related to preflight (check port availability, check existence of directory, check app availability in PATH, etc)
- app/cmd/phases – these directory contains code that is tied to the different command line function in kubeadm (eg: phases/init –> has code that link to ‘kubeadm init’
  - /init – directory contains code for all the phases when executing ‘kubeadm init’
  - /join – directory contains code for the ‘kubeadm join’ command
  - /reset – directory contains code for the ‘kubeadm reset’ command
  - /upgrade – directory contains code for the ‘kubeadm upgrade’ command
- app/phases – contains code used as part of the kubeadm phases
- app/util/system
  - cgroup_validator – validator for cgroups
  - docker_validator – docker validator
  - kernel_validator – kernel validator
  - os_validator – OS validator (version)
  - package_validator – package manager validator

CSI (Container Storage Interface)

Interfacing K8 with storage implementation
- Node – A host where a workload (such as Pods in Kubernetes) will be running.
- Plugin - In the CSI world, this points to a service that exposes gRPC endpoints
Specification defines the boundary between the CO and the CSI plugin.
Plugin part actually is separated into two individual plugins.
- Node Plugin
  - The Node plugin is a gRPC server that needs to run on the Node where the volume will be provisioned. So suppose you have a Kubernetes cluster with three nodes where your Pod’s are scheduled, you would deploy this to all three nodes.
- Controller Plugin
  - The Controller plugin is a gRPC server that can run anywhere. In terms of a Kubernetes cluster, it can run on any node (even on the master node).These two entities can live in a single binary or you can separate them.

References

Experimenting with Kubernetes

Books

Programming Kubernetes - OReilly
Kubernetes Patterns - OReilly
Managing Kubernetes - OReilly

Minikube Development

Sat, 30 Nov 2019 00:00:00 +0000

This document goes through things that are helpful for development purposes

cache

This command is useful to use the local docker image in minikube. Docker images are stored in the local repo and using the cache command allow the image to be uploaded into the VM.

This is useful when doing local development and we want to use the image inside the VM

Inside minikube source code

Tue, 17 Sep 2019 00:00:00 +0000

NOTE: The source code discussed in this article is based on master branch

We are going to create a separate brand new GOPATH directory for minikube to demonstrate how to do the whole process fresh from scratch. On my local machine I’ve create a directory called minikubegopath with the following structure

.├── bin├── pkg└── src    └── k8s.io

Make sure are you inside the src/k8s.io directory when cloning minikube

git clone https://github.com/kubernetes/minikube

Open the minikube inside the IDE and set the GOPATH to the new GOPATH directory that have minikube, setup it as shown in the screenshot below

Select the the main.go file that reside in cmd/minikube directory and hit the run button. You will get an error output as shown below, this is because there are some files that need to be generated and this can be done using the make command

go: downloading github.com/mattn/go-isatty v0.0.8go: downloading github.com/pkg/profile v0.0.0-20161223203901-3a8809bd8a80go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b...............# k8s.io/minikube/pkg/minikube/translate../../pkg/minikube/translate/translate.go:77:12: undefined: AssetCompilation finished with exit code 2

Use the following make command to generate the files needed

make pkg/minikube/assets/assets.go pkg/minikube/translate/translations.go

You will see the following output

which go-bindata || GO111MODULE=off GOBIN=XXXXXXXX/minikubegopath/bin go get github.com/jteeuwen/go-bindata/.......... go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...gofmt -s -w pkg/minikube/assets/assets.go......GOOS="linux" GOARCH="amd64" go build -tags "container_image_ostree_stub containers_image_openpgp" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.4.0-beta.2 -X k8s.io/minikube/pkg/version.isoVersion=v1.4.0-beta.2 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="d1e468085d9af12e5d130a6cb3b2186a5db87a0e-dirty"" -a -o out/minikube-linux-amd64 k8s.io/minikube/cmd/minikube

Following are the generated files

pkg/minikube/assets/assets.go --> generate source code to allow loading of addons .yaml filespkg/minikube/translate/translations.go --> generate source code for translations data file

Makefile

Following table explains some of the available tasks:


pkg/minikube/assets/assets.go ==> generate source code needed to build minikube. Source code generated are used for loading of .yaml file
pkg/minikube/translate/translations.go ==> generate source code for translations data file
minikube-linux-amd64 ==> generate minikube for Linux 64-bit. Windows task minikube-windows-amd64.exe and MacOS minikube-darwin-amd64
out/e2e-linux-amd64 ==> compile end-to-end testing and minikube binary for testing purposes
minikube_iso ==> used internally to generate the .iso file. Better use the out/minikube.iso
iso-menuconfig ==> change buildroot configuration for minikube iso
linux-menuconfig ==> change kernel configuration for minikube iso
all ==> compile and generate minikube for the 3 platforms - Linux, Windows and MacOS
cross ==> build minikube for the 3 platforms - Linux, Windows and MacOS

Source structure

.├── cmd│   ├── drivers│   │   ├── hyperkit│   │   └── kvm│   ├── extract│   ├── gvisor│   ├── minikube│   │   └── cmd│   │       └── config│   └── storage-provisioner├── deploy│   ├── addons│   │   ├── dashboard│   │   ├── efk│   │   ├── freshpod│   │   ├── gpu│   │   ├── gvisor│   │   ├── heapster│   │   ├── ingress│   │   ├── logviewer│   │   ├── metrics-server│   │   ├── registry│   │   ├── registry-creds│   │   ├── storageclass│   │   ├── storage-provisioner│   │   └── storage-provisioner-gluster│   ├── gvisor│   ├── iso│   │   └── minikube-iso│   │       ├── board│   │       │   └── coreos│   │       │       └── minikube│   │       │           └── rootfs-overlay│   │       │               ├── etc│   │       │               │   ├── cni│   │       │               │   │   └── net.d│   │       │               │   ├── docker│   │       │               │   ├── ssh│   │       │               │   ├── systemd│   │       │               │   │   ├── network│   │       │               │   │   └── system│   │       │               │   │       ├── getty.target.wants│   │       │               │   │       └── systemd-timesyncd.service.d│   │       │               │   └── udev│   │       │               │       └── rules.d│   │       │               ├── usr│   │       │               │   └── bin│   │       │               └── var│   │       │                   ├── lib│   │       │                   │   └── boot2docker│   │       │                   ├── log│   │       │                   └── run│   │       │                       └── crio│   │       ├── configs│   │       └── package│   │           ├── automount│   │           ├── cni-bin│   │           ├── cni-plugins-bin│   │           ├── conmon-master│   │           ├── containerd-bin│   │           ├── crictl-bin│   │           ├── crio-bin│   │           ├── docker-bin│   │           ├── gluster│   │           ├── hyperv-daemons│   │           ├── podman│   │           ├── runc-master│   │           └── vbox-guest│   ├── minikube│   └── storage-provisioner├── docs│   └── contributors├── hack│   ├── boilerplate│   ├── help_text│   ├── jenkins│   │   └── installers│   ├── kubernetes_version│   ├── prow│   └── release_notes├── images│   └── logo├── installers│   ├── darwin│   │   └── brew-cask│   ├── linux│   │   ├── archlinux│   │   ├── archlinux-driver│   │   ├── deb│   │   │   ├── kvm2_deb_template│   │   │   │   └── DEBIAN│   │   │   └── minikube_deb_template│   │   │       └── DEBIAN│   │   ├── kvm│   │   └── rpm│   │       ├── kvm2_rpm_template│   │       └── minikube_rpm_template│   └── windows├── pkg│   ├── drivers│   │   ├── hyperkit│   │   ├── kvm│   │   └── none│   ├── gvisor│   ├── initflag│   ├── kapi│   ├── minikube│   │   ├── assets│   │   ├── bootstrapper│   │   │   └── kubeadm│   │   │       └── testdata│   │   │           ├── v1.11│   │   │           ├── v1.12│   │   │           ├── v1.13│   │   │           ├── v1.14│   │   │           ├── v1.15│   │   │           └── v1.16│   │   ├── cluster│   │   ├── command│   │   ├── config│   │   │   └── testdata│   │   ├── constants│   │   ├── cruntime│   │   ├── drivers│   │   │   ├── hyperkit│   │   │   ├── hyperv│   │   │   ├── kvm2│   │   │   ├── none│   │   │   ├── parallels│   │   │   ├── virtualbox│   │   │   ├── vmware│   │   │   └── vmwarefusion│   │   ├── exit│   │   ├── extract│   │   │   └── testdata│   │   ├── kubeconfig│   │   │   └── testdata│   │   │       └── kubeconfig│   │   ├── logs│   │   ├── machine│   │   ├── notify│   │   ├── out│   │   ├── problem│   │   ├── proxy│   │   ├── registry│   │   ├── service│   │   ├── sshutil│   │   ├── storageclass│   │   ├── tests│   │   ├── translate│   │   └── tunnel│   ├── provision│   ├── storage│   ├── util│   │   ├── lock│   │   └── retry│   └── version├── site│   ├── assets│   │   ├── icons│   │   └── scss│   ├── content│   │   └── en│   │       ├── blog│   │       │   ├── news│   │       │   └── releases│   │       ├── community│   │       └── docs│   │           ├── Concepts│   │           ├── Contributing│   │           ├── Examples│   │           ├── Overview│   │           ├── Reference│   │           │   ├── Commands│   │           │   ├── Configuration│   │           │   ├── Drivers│   │           │   │   └── includes│   │           │   └── Networking│   │           ├── Start│   │           │   └── includes│   │           ├── Tasks│   │           │   └── Registry│   │           └── Tutorials│   ├── layouts│   │   ├── partials│   │   │   └── hooks│   │   └── shortcodes│   ├── static│   │   ├── css│   │   └── js│   └── themes│       └── docsy├── test│   └── integration│       └── testdata│           ├── kvm2-driver-older-version│           └── kvm2-driver-without-version├── third_party│   └── go9p│       └── ufs└── translations

Following table are description of the different directories of minikube’s source code. Note: This is by no means a complete list of the source directory structure as master branch do change from time to time

Directory Name	Description
drivers	docker-machine plugin code (libvirt and hyperkit)
cmd/gvisor	Code to start gvisor and do all the necessary steps to restart the dependencies (containerd, etc)
minikube	minikube cli handling code
storage-provisioner	Main function to trigger the storage provisioner module
deploy	Contains files that are needed to build minikube inside docker
deploy/addons	Lots of minikube addons (*.yaml) inside this directory such as: gvisor → used to securely run pods with untrusted workloads, freshpods → restart container,etc
deploy/gvisor	contain Dockerfile to run gvisor-addon
deploy/iso	Contains documentation, packages and configuration for building minikube iso. Some of the packages that are included – automount,containerd and many more. It also contains configuration file for building buildroot. Pretty much anything related to building iso in inside this directory
deploy/minikube	Contains few json files related to release
deploy/storage-provisioner	Contain dockerfile to run storage-provisioner
hack/jenkins	Jenkins configuration file
hack/kubernetes_version	Tool to generate Minikube version number
hack/release_notes	Code to create release notes
installers/darwin	script to create Darwin installer
installers/linux	Linux – deb, archlinux, rpm, etc
installers/windows	Windows installers
pkg/drivers	Driver related code – kvm, hyperkit
site	Docsy template for generating documents as website
test	Integration test for Minikube
go9p	Go implementation of 9P2000 protocol (9P (or the Plan 9 Filesystem Protocol or Styx) is a network protocol developed for the Plan 9 from Bell Labs distributed operating system as the means of connecting the components of a Plan 9 system. Files are key objects in Plan 9. They represent windows, network connections, processes, and almost anything else available in the operating system. )
translations	Language translation at the moment CN and FR

Inside minikube ISO

Tue, 10 Sep 2019 00:00:00 +0000

Minikube uses a generated .iso file that is used to bootstrap the VM with it’s own kernel and rootfs. Once the VM has been initialized properly it will use the .iso to bootstrap the Linux kernel allowing all the different applications to be installed and run. This means that pretty much minikube is running on it’s own operating system and does not use any dependencies from the host system (except through the VM)

Building .iso file

Building the .iso file for minikube requires docker to be running on your local machine. Internally it uses docker container to build everything (kernel + rootfs) and package it into an .iso file. There are 2 separate make tasks that need to be triggered as shown below:

buildroot-image: $(ISO_BUILD_IMAGE) # convenient alias to build the docker container$(ISO_BUILD_IMAGE): deploy/iso/minikube-iso/Dockerfiledocker build $(ISO_DOCKER_EXTRA_ARGS) -t $@ -f $< $(dir $<)@echo ""@echo "$(@) successfully built"

out/minikube.iso: $(shell find "deploy/iso/minikube-iso" -type f)ifeq ($(IN_DOCKER),1)$(MAKE) minikube_isoelsedocker run --rm --workdir /mnt --volume $(CURDIR):/mnt $(ISO_DOCKER_EXTRA_ARGS) \--user $(shell id -u):$(shell id -g) --env HOME=/tmp --env IN_DOCKER=1 \$(ISO_BUILD_IMAGE) /usr/bin/make out/minikube.isoendif

This means there are 2 separate tasks that need to be executed

make buildroot-image 2>&1 | tee buildrootimage.txt && make out/minikube.iso   2>&1 | tee buildminikubeiso.txt

The output of the build process will be piped into buildrootimage.txt and buildminikubeiso.txt file to make it easy for troubleshooting purposes. Here is a snapshpot of how the output looks like from both the logs

buildrootimage.txt

docker build  -t gcr.io/k8s-minikube/buildroot-image -f deploy/iso/minikube-iso/Dockerfile deploy/iso/minikube-iso/Sending build context to Docker daemon  184.3kBStep 1/6 : FROM ubuntu:18.0418.04: Pulling from library/ubuntu35c102085707: Pulling fs layer251f5509d51d: Pulling fs layer..........8e829fe70a46: Pull complete6001e1789921: Verifying Checksum6001e1789921: Download complete6001e1789921: Pull completeDigest: sha256:d1d454df0f579c6be4d8161d227462d69e163a8ff9d20a847533989cf0c94d90Status: Downloaded newer image for ubuntu:18.04 ---> a2a15febcdf3Step 2/6 : RUN apt-get update && apt-get install -y apt dpkg apt-utils ca-certificates && apt-get upgrade -y && apt-get install -y build-essential git wget cpio python unzip bc gcc-multilib automake libtool gnupg2 p7zip-full locales rsync dumb-init golang-go libpcre3-dev && rm -rf /var/lib/apt/lists/* ---> Running in 3e3ff8ddac3eGet:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]Get:3 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [766 kB]Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]Get:6 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]..........Unpacking manpages-dev (4.15-1) ...Selecting previously unselected package p7zip.Preparing to unpack .../153-p7zip_16.02+dfsg-6_amd64.deb ...Unpacking p7zip (16.02+dfsg-6) .............Removing intermediate container 3e3ff8ddac3e ---> 89cfa69b7f86Step 3/6 : RUN localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 ---> Running in e68c807a6dbfRemoving intermediate container e68c807a6dbf ---> c69896a88da3Step 4/6 : ENV LANG en_US.utf8 ---> Running in 58df4bca1563Removing intermediate container 58df4bca1563 ---> c181b08523adStep 5/6 : ENTRYPOINT ["/usr/bin/dumb-init", "--"] ---> Running in 46918f491955Removing intermediate container 46918f491955 ---> 3e76d78225fcStep 6/6 : CMD ["/bin/bash"] ---> Running in 6cefbf569aadRemoving intermediate container 6cefbf569aad ---> 2217637c1629Successfully built 2217637c1629Successfully tagged gcr.io/k8s-minikube/buildroot-image:latestgcr.io/k8s-minikube/buildroot-image successfully built

buildminikubeiso.txt

docker run --rm --workdir /mnt --volume /home/nanik/Downloads/temp/packages/src/k8s.io/minikube:/mnt  \--user 1000:1000 --env HOME=/tmp --env IN_DOCKER=1 \gcr.io/k8s-minikube/buildroot-image /usr/bin/make out/minikube.iso/usr/bin/make minikube_isomake[1]: Entering directory '/mnt'echo v1.4.0-beta.0 > deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/VERSIONif [ ! -d ./out/buildroot ]; then \mkdir -p ./out; \git clone --depth=1 --branch=2018.05.3 https://github.com/buildroot/buildroot ./out/buildroot; \fi;Cloning into './out/buildroot'...Note: checking out '717309783755a883974896ea822675a7bf46da3a'...../usr/bin/make BR2_EXTERNAL=../../deploy/iso/minikube-iso minikube_defconfig -C ./out/buildroot................

Using Virtualbox

Just for fun let’s try to run the .iso file by ourself using the VirtualBox.

Start the VirtualBox UI from your local machine

Create and configure virtual machine using the same setting as shown in the screenshot

Make sure you specify the newly generated .iso file from your local build like the below screenshot

Once configured properly starts up the VM and you will see output similar to the following screenshot

When the kernel startup properly the screen will show the prompt as shown the following screenshot. Enter root as the username and you will be inside the bash environment

Even though you are able to run the .iso file the VM is still in it’s raw form as there are no Kubernetes app or configuration installed. So pretty much this is just an empty OS wihtout Kubernetes.

By going through through this exercise you now have a better idea the gaps that minikube try to fill in. Minikube fills in the gap by automating the whole process until the whole thing is up and running.

Dissecting minikube .iso file

Now that we have successfully generate and use the freshly build iso file, it’s time to dissect and take a look inside it. The iso file is a ISO9660 file which means it can be mounted to a local directory.

You can create any directory anywhere you like and mount the iso file, in my case I have created a directory /home/nanik/Downloads/playwithiso/mountiso and used the following command to mount the file

sudo mount -o loop ./minikube.iso /home/nanik/Downloads/playwithiso/mountiso/

After the mount command successfully executed you will see the following tree structure

├── boot│   ├── bzImage│   └── initrd├── boot.catalog└── isolinux    ├── isolinux.bin    ├── isolinux.cfg    └── ldlinux.c32

as can be see the iso file contain the rootfs (initrd) and kernel (bzImage) along with the the ISOLINUX bootloader files

Chroot

To further having fun with the iso file let’s use the rootfs to boot into it using chroot. Copy the initrd file from the mounted directory described on the previous section and use the following command to unzip it

bunzip2 ./initrd

the initrd will be unzipped to a file called initrd.out, use the following command to unpack the file

cpio -idv < initrd.out

the file will be unpacked to the following tree structure

├── bin -> usr/bin├── dev├── etc│   ├── cni│   ├── containerd│   ├── containers│   ├── crio│   ├── dbus-1│   ├── docker│   ├── glusterfs│   ├── init.d│   ├── iproute2│   ├── kernel│   ├── lvm│   ├── modules-load.d│   ├── pam.d│   ├── profile.d│   ├── ssh│   ├── ssl│   ├── sudoers.d│   ├── sysconfig│   ├── sysctl.d│   ├── systemd│   ├── tmpfiles.d│   ├── udev│   ├── vmware-tools│   ├── X11│   └── xdg├── home│   └── docker├── lib -> usr/lib├── lib64 -> lib├── media├── mnt├── opt│   └── cni├── proc├── root├── run│   ├── crio│   ├── dbus│   ├── gluster│   ├── nfs│   └── sudo├── sbin -> usr/sbin├── srv├── sys├── tmp├── usr│   ├── bin│   ├── etc│   ├── lib│   ├── lib64 -> lib│   ├── libexec│   ├── sbin│   └── share└── var    ├── empty    ├── lib    ├── log    └── run -> ../run

Make sure you are inside the extracted directory and use the following command to chroot into it

sudo chroot .

you will get bash prompt as follows

bash-4.4#

To come out from the bash just type exit.

Using local build .iso file

The following command specified minikube to use locally built iso file

minikube start --iso-url=file:///home/nanik/Downloads/temp/packages/src/k8s.io/minikube/out/minikube.iso

Below is the log output

😄  minikube v0.0.0-unset on Ubuntu xx.xx🔥  Creating virtualbox VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...Downloading /home/nanik/.minikube/cache/boot2docker.iso from file:///home/nanik/Downloads/temp/packages/src/k8s.io/minikube/out/minikube.iso...Creating VirtualBox VM...Creating SSH key...Starting the VM...Check network to re-create if needed...Waiting for an IP...Setting Docker configuration on the remote daemon...Waiting for SSH to be available...Detecting the provisioner...🐳  Preparing Kubernetes v1.16.0-beta.1 on Docker 18.09.8 ...🚜  Pulling images ...🚀  Launching Kubernetes ... ⌛  Waiting for: apiserver proxy etcd scheduler controller dns🏄  Done! kubectl is now configured to use "minikube"💡  For best results, install kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/

NOTE: notice the message Downloading /home/nanik/.minikube/cache/boot2docker.iso the process behind this message is actually using the specified minikube.iso and copying it to boot2docker.iso

Reference

Contributing minikube iso

Minikube Startup Process

Fri, 06 Sep 2019 00:00:00 +0000

Overview

Minikube is called cluster-in-a-box and the reason why it is called that is because it contains everything that you will need to run cluster on a local environment. The tool is good to get a head strat into the Kubernetes world.

Kubernetes as a project is a collection of tools that works together to deliver container clustering solution. The difference with minikube is that as a developer we don’t have to worry installing the different projects together as it’s neatly packaged into a simple file that can be run directly.

This article will outlined the different steps that minikube goes through until it starts up and running ready to be used for deploying application. This article assume you have successfully gone through the steps outlined in the Building and hacking minikube article.

NOTE: This article is using VirtualBox so there are some explanation which are VirtualBox specific

Startup Flow

Following are the startup process when minikube starts up:

Download .iso file if it is not available locally
Extract boot2docker.iso from the downloaded .iso file
Create on-the-fly certificate to be used for ssh purposes
Create VirtualBox VM file with specified configuration
Setup storage to mount boot2docker.iso file
Setup network related configuration (IP, DHCP, etc)
Setup SSH inside the VM
Startup VM
Setup /etc/hostname, /etc/hosts
Setup systemd relevant files to allow docker to start properly
Prepare Kubernetes and Docker
Download all the relevant Kubernetes files - kubelet, kubeadm, etc
Pulling docker images for the different packages needed for Kubernetes
Start up different services such as - etcd, scheduler, controller, apiserver, etc

All configuration, keys, iso image, etc that are used to prepare minikube are available inside $HOME/.minikube. Following is how the directory structure looks like

├── [4.0K]  addons├── [1.3K]  apiserver.crt├── [1.6K]  apiserver.key├── [4.0K]  cache│   ├── [4.0K]  images│   │   ├── [4.0K]  gcr.io│   │   │   └── [4.0K]  k8s-minikube│   │   │       └── [ 20M]  storage-provisioner_v1.8.1│   │   └── [4.0K]  k8s.gcr.io│   │       ├── [ 12M]  coredns_1.3.1│   │       ├── [ 73M]  etcd_3.3.10│   │       ├── [ 11M]  k8s-dns-dnsmasq-nanny-amd64_1.14.13│   │       ├── [ 14M]  k8s-dns-kube-dns-amd64_1.14.13│   │       ├── [ 12M]  k8s-dns-sidecar-amd64_1.14.13│   │       ├── [ 29M]  kube-addon-manager_v9.0│   │       ├── [ 47M]  kube-apiserver_v1.15.2│   │       ├── [ 46M]  kube-controller-manager_v1.15.2│   │       ├── [ 29M]  kube-proxy_v1.15.2│   │       ├── [ 43M]  kubernetes-dashboard-amd64_v1.10.1│   │       ├── [ 28M]  kube-scheduler_v1.15.2│   │       └── [312K]  pause_3.1│   ├── [4.0K]  iso│   │   └── [131M]  minikube-v1.3.0.iso│   └── [4.0K]  v1.15.2│       ├── [ 38M]  kubeadm│       └── [114M]  kubelet├── [1.0K]  ca.crt├── [1.6K]  ca.key├── [1.0K]  ca.pem├── [1.0K]  cert.pem├── [4.0K]  certs│   ├── [1.6K]  ca-key.pem│   ├── [1.0K]  ca.pem│   ├── [1.0K]  cert.pem│   └── [1.6K]  key.pem├── [1.1K]  client.crt├── [1.6K]  client.key├── [4.0K]  config├── [4.0K]  files├── [1.6K]  key.pem├── [  29]  last_update_check├── [4.0K]  logs├── [4.0K]  machines│   ├── [4.0K]  minikube│   │   ├── [131M]  boot2docker.iso│   │   ├── [2.7K]  config.json│   │   ├── [2.0G]  disk.vmdk│   │   ├── [1.6K]  id_rsa│   │   ├── [ 381]  id_rsa.pub│   │   └── [4.0K]  minikube│   │       ├── [4.0K]  Logs│   │       │   └── [ 80K]  VBox.log│   │       ├── [3.7K]  minikube.vbox│   │       └── [3.6K]  minikube.vbox-prev│   ├── [1.6K]  server-key.pem│   └── [1.1K]  server.pem├── [4.0K]  profiles│   └── [4.0K]  minikube│       └── [1.5K]  config.json├── [1.0K]  proxy-client-ca.crt├── [1.6K]  proxy-client-ca.key├── [1.1K]  proxy-client.crt└── [1.6K]  proxy-client.key

Inside VirtualBox

Once minikube is up and running we can ssh into the VM. Setup the configuration as shown in the screenshot below

You will get output that looks like the following and it will display a minikube shell for you to work inside the VM.

COMMAND: /usr/bin/VBoxManage showvminfo minikube --machinereadableSTDOUT:{name="minikube"groups="/"ostype="Linux 2.6 / 3.x / 4.x (64-bit)"UUID="xxxxxx"CfgFile="/home/nanik/.minikube/machines/minikube/minikube/minikube.vbox"SnapFldr="/home/nanik/.minikube/machines/minikube/minikube/Snapshots"LogFldr="/home/nanik/.minikube/machines/minikube/minikube/Logs"............}STDERR:{}Using SSH client type: native....                         _             _                        _         _ ( )           ( )             ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  /' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)$

Executing the following ps command

sudo ps -A

will show the different process that is currently running inside the VM. As can be seen there are a number of Kubernetes service that are running.

  PID TTY          TIME CMD    1 ?        00:00:19 systemd    2 ?        00:00:00 kthreadd    3 ?        00:00:01 kworker/0:0    4 ?        00:00:00 kworker/0:0H    6 ?        00:00:00 mm_percpu_wq   18 ?        00:00:00 netns   19 ?        00:00:00 kauditd  371 ?        00:00:00 oom_reaper  455 ?        00:00:00 writeback  456 ?        00:00:00 kcompactd0  458 ?        00:00:00 crypto  459 ?        00:00:00 kblockd  518 ?        00:00:00 ata_sff  535 ?        00:00:00 md..... 1223 ?        00:00:00 sshd 1225 ?        00:00:00 sshd 1226 pts/0    00:00:00 bash 1241 ?        00:00:00 ipv6_addrconf 1246 ?        00:00:00 ceph-msgr ..... 2014 tty1     00:00:00 getty 2018 ?        00:00:00 rpcbind 2028 ?        00:00:03 VBoxService 2033 ?        00:00:01 kworker/0:1H 2059 ?        00:00:00 lockd 2182 ?        00:00:00 sshd  ..... 2369 ?        00:04:42 dockerd 2376 ?        00:02:10 containerd 3644 ?        00:00:00 containerd-shim 3648 ?        00:00:00 containerd-shim 3656 ?        00:00:00 containerd-shim 3665 ?        00:00:00 containerd-shim 3719 ?        00:00:00 pause 3725 ?        00:00:00 pause 3732 ?        00:00:00 pause 3734 ?        00:00:00 pause 3865 ?        00:00:00 containerd-shim  ..... 3939 ?        00:07:46 kube-apiserver 3948 ?        00:00:00 bash 3970 ?        00:04:27 etcd 3975 ?        00:00:23 kube-scheduler 4321 ?        00:00:00 containerd-shim 4351 ?        00:00:00 containerd-shim 4361 ?        00:00:00 containerd-shim  ..... 4470 ?        00:00:00 containerd-shim 4498 ?        00:00:12 kube-proxy 4811 ?        00:00:00 containerd-shim 4829 ?        00:00:47 coredns 4864 ?        00:00:00 containerd-shim 4882 ?        00:00:00 pause 4906 ?        00:00:00 containerd-shim 4925 ?        00:00:07 storage-provisi 4996 ?        00:00:00 containerd-shim 5013 ?        00:00:51 coredns29243 ?        00:00:23 kubelet29441 ?        00:00:00 containerd-shim  .....29483 ?        00:00:00 containerd-shim29499 ?        00:00:10 kube-controller

Let’s query docker and see what are the images that are available inside the VM

$ docker image listREPOSITORY                                TAG                 IMAGE ID            CREATED             SIZEk8s.gcr.io/kube-scheduler                 v1.15.2             88fa9cb27bd2        4 weeks ago         81.1MBk8s.gcr.io/kube-controller-manager        v1.15.2             9f5df470155d        4 weeks ago         159MBk8s.gcr.io/kube-apiserver                 v1.15.2             34a53be6c9a7        4 weeks ago         207MBk8s.gcr.io/kube-proxy                     v1.15.2             167bbf6c9338        4 weeks ago         82.4MBk8s.gcr.io/kube-addon-manager             v9.0                119701e77cbc        7 months ago        83.1MBk8s.gcr.io/coredns                        1.3.1               eb516548c180        7 months ago        40.3MBk8s.gcr.io/kubernetes-dashboard-amd64     v1.10.1             f9aed6605b81        8 months ago        122MBk8s.gcr.io/etcd                           3.3.10              2c4adeb21b4f        9 months ago        258MBk8s.gcr.io/k8s-dns-sidecar-amd64          1.14.13             4b2e93f0133d        11 months ago       42.9MBk8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.13             55a3c5209c5e        11 months ago       51.2MBk8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.13             6dc8ef8287d3        11 months ago       41.4MBk8s.gcr.io/pause                          3.1                 da86e6ba6ca1        20 months ago       742kBgcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        22 months ago       80.8MB

Minikube cod uses VirtualBox’s CLI command called VBoxManage, which resides inside /usr/bin. Following log snippets show some of the call that are performed when minikube starts up

......COMMAND: /usr/bin/VBoxManage --version......COMMAND: /usr/bin/VBoxManage list hostonlyifs......Downloading /home/nanik/.minikube/cache/boot2docker.iso from file:///home/nanik/.minikube/cache/iso/minikube-v1.3.0.iso...Creating VirtualBox VM...Creating SSH key...Creating disk image...Creating 20000 MB hard disk image...Writing magic tar headerWriting SSH key tar headerCalling inner createDiskImage&{/usr/bin/VBoxManage [/usr/bin/VBoxManage convertfromraw stdin /home/nanik/.minikube/machines/minikube/disk.vmdk 20971520000 --format VMDK] [] ......}Starting commandCopying to stdinFilling zeroesClosing STDINWaiting on cmdCOMMAND: /usr/bin/VBoxManage createvm --basefolder /home/nanik/.minikube/machines/minikube --name minikube --register......COMMAND: /usr/bin/VBoxManage modifyvm minikube --firmware bios --bioslogofadein off --bioslogofadeout off --bioslogodisplaytime 0 --biosbootmenu disabled --ostype Linux26_64 --cpus 2 --memory 2000 --acpi on --ioapic on --rtcuseutc on --natdnshostresolver1 on --natdnsproxy1 off --cpuhotplug off --pae on --hpet on --hwvirtex on --nestedpaging on --largepages on --vtxvpid on --accelerate3d off --boot1 dvd......COMMAND: /usr/bin/VBoxManage modifyvm minikube --nic1 nat --nictype1 virtio --cableconnected1 on......COMMAND: /usr/bin/VBoxManage storagectl minikube --name SATA --add sata --hostiocache on......COMMAND: /usr/bin/VBoxManage storageattach minikube --storagectl SATA --port 0 --device 0 --type dvddrive --medium /home/nanik/.minikube/machines/minikube/boot2docker.iso......COMMAND: /usr/bin/VBoxManage storageattach minikube --storagectl SATA --port 1 --device 0 --type hdd --medium /home/nanik/.minikube/machines/minikube/disk.vmdk......COMMAND: /usr/bin/VBoxManage guestproperty set minikube /VirtualBox/GuestAdd/SharedFolders/MountPrefix /......COMMAND: /usr/bin/VBoxManage guestproperty set minikube /VirtualBox/GuestAdd/SharedFolders/MountDir /......setting up shareDir '/home' -> 'hosthome'COMMAND: /usr/bin/VBoxManage sharedfolder add minikube --name hosthome --hostpath /home --automount......COMMAND: /usr/bin/VBoxManage setextradata minikube VBoxInternal2/SharedFoldersEnableSymlinksCreate/hosthome 1......Starting the VM...COMMAND: /usr/bin/VBoxManage showvminfo minikube --machinereadable......Check network to re-create if needed...COMMAND: /usr/bin/VBoxManage list hostonlyifs......Searching for hostonly interface for IPv4: xxxxx and Mask: xxxxxFound: vboxnet0COMMAND: /usr/bin/VBoxManage list dhcpservers......Removing orphan DHCP servers...COMMAND: /usr/bin/VBoxManage list hostonlyifs......Adding/Modifying DHCP server "xxxxxxx" with address range "xxxx" - "yyyyy"...COMMAND: /usr/bin/VBoxManage list dhcpservers......

Building and hacking minikube

Thu, 05 Sep 2019 00:00:00 +0000

This document will outline the internals on minikube. It will outline how minikube works and what are the steps involved in running it inside an IDE. Having the ability to run inside the IDE will give the power of more better understanding by stepping through the code.

Minikube makes heavy use of VM and out of the box it supports VMs such as VirtualBox, Hyperkit, etc. We going to go through the VirtualBox setup in Linux environment for this article.

Prerequisite

Minikube has dependencies on packages that will need to be installed on your local dev machine. The best way to ensure that you have all the relevant packages installed is to run through the Quickstart steps on minikube tutorial website. Once you are able to run minikube on it’s own on your dev machine you should be able to run and debug it without any problem from inside the IDE.

Building minikube

This article will be using GoLand IDE as it provide a lot of features that makes development in Go very easy. You can download the IDE here and install the IDE using the instruction outline on the website.

The GOPATH used in this article has been set to /home/nanik/Downloads/temp/packages

Completing the IDE installation checkout minikube into your GOPATH directory. On my local machine the minikube is stored under $GOPATH/src/k8s.io/minikube directory, as shows on the following screenshot

We need to setup the vendor directory for minikube by executing the following command

GO111MODULE=on go mod vendor

You will see vendor/ directory under the minikube source code.

Before running minikube there are code that need to be generated by executing the make command. On your terminal execute the following command

make

The other step that need to be done is to set the correct version of the .iso file that will be used by minikube. This .iso file will be automatically downloaded when minikube starts up. The version need to be set inside the file version.go under pkg/version directory as shown below, set it to v1.3.0

Once the make build is complete the minikube code is not ready. The main function to execute minikube is inside the file main.go under the cmd/minikube directory

Following is the directory structure of minikube source code (the screenshot shows 2 level deep directory tree)

├── cmd│   ├── drivers│   ├── extract│   ├── gvisor│   ├── minikube│   └── storage-provisioner├── deploy│   ├── addons│   ├── gvisor│   ├── iso│   ├── minikube│   └── storage-provisioner├── docs│   └── contributors├── hack│   ├── boilerplate│   ├── help_text│   ├── jenkins│   ├── kubernetes_version│   ├── prow│   └── release_notes├── images│   └── logo├── installers│   ├── darwin│   ├── linux│   └── windows├── out├── pkg│   ├── drivers│   ├── gvisor│   ├── initflag│   ├── kapi│   ├── minikube│   ├── provision│   ├── storage│   ├── util│   └── version├── site│   ├── assets│   ├── content│   ├── layouts│   ├── resources│   ├── static│   └── themes├── test│   └── integration├── third_party│   └── go9p├── translations└── vendor    ├── cloud.google.com    ├── github.com    ├── golang.org    ├── google.golang.org    ├── go.opencensus.io    ├── gopkg.in    ├── k8s.io    └── sigs.k8s.io

Now we are ready to run the minikube. Use the parameter start as part of the program argument as shown in the screenshot

The parameter –v=7 used is to provide verbose logging to help in troubleshooting in case of issues. The log output will look somewhat like the following

😄  minikube v0.0.0-unset on ......💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.COMMAND: /usr/bin/VBoxManage showvminfo minikube --machinereadableSTDOUT:{name="minikube"groups="/"ostype="Linux 2.6 / 3.x / 4.x (64-bit)"UUID="....."CfgFile="/home/nanik/.minikube/machines/minikube/minikube/minikube.vbox"SnapFldr="/home/nanik/.minikube/machines/minikube/minikube/Snapshots"LogFldr="/home/nanik/.minikube/machines/minikube/minikube/Logs"hardwareuuid="....."memory=2000....................................END SSH🔄  Relaunching Kubernetes using kubeadm ... ⌛  Waiting for: apiserver proxy etcd scheduler controller dns🏄  Done! kubectl is now configured to use "minikube"💡  For best results, install kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/

Once minikube has been ran successfully the application will exit as the VM now run in the bacground complete will the relevant kubernetes components installed inside it. To see the background process use the following command to peek into minikube running process on your machine

ps -A | grep -i vb

you will see output similar to the following

..........28685 ?        00:00:51 VBoxXPCOMIPCD28690 ?        00:02:35 VBoxSVC28824 ?        15:53:35 VBoxHeadless28841 ?        00:00:06 VBoxNetDHCP

Now to confirm that all working well we need to execute command using minikube to check the cluster status. Use the following program argument to do this

You will get output that shows the following

host: Runningkubelet: Runningapiserver: Runningkubectl: Correctly Configured: pointing to minikube-vm at xx.xx.xx.xxs

Now you are ready to start playing around with minikube source code.

Deploying to Minikube(Work in progress)

Sun, 01 Sep 2019 00:00:00 +0000

We will deploy 2 different projects in this article. One is a simple single command line application and another is a web app that requires database. We will be using the skaffold project as it will make it easy to do continous deployment

Launching minikube

Deployment

deploying via IDE
deploying via CLI

Deploying hello-world

https://kubernetes.io/docs/setup/learning-environment/minikube/
- kubectl run hello-minikube –image=k8s.gcr.io/echoserver:1.10 –port=8080
- kubectl expose deployment hello-minikube –type=NodePort
- kubectl get pod
- minikube service hello-minikube –url

K8 Tools

skaffold - continues local development/deployment in k8 for golang
k8guard - example to show deploying different applications inside k8 ( https://k8guard.github.io/deploy/minikube/ )
ko - Build and deploy Go applications on Kubernetes ( https://github.com/google/ko )

Golang

Sun, 01 Sep 2019 00:00:00 +0000

Following are some of the thing that need to be remembered when programming in Golang. This information is taken from variety of sources.

Variables

For variable naming, omit words that are obvious given a variable’s type declaration or otherwise clear from the surrounding context. Following is not a good example
```
returnSlice := make([]string, len(src))
```
Variable names in Go should be short rather than long. This is especially true for local variables with limited scope. Prefer c to lineCount. Prefer i to sliceIndex.
The basic rule: the further from its declaration that a name is used, the more descriptive the name must be. For a method receiver, one or two letters is sufficient. Common variables such as loop indices and readers can be a single letter (i, r). More unusual things and global variables need more descriptive names.
For example, “URL” should appear as “URL” or “url” (as in “urlPony”, or “URLPony”), never as “Url”. ServeHTTP not ServeHttp.
For identifiers with multiple initialized “words”, use for example “xmlHTTPRequest” or “XMLHTTPRequest”.
Have a reasonable number of parameters and reasonably short variable names.
Having one doubled up field in the struct makes the test data difficult to scan. Following is not a good example :
```
...prefix, postfix string
```

Declare empty slices as

var t []string (declares a nil slice value)

and not

t := []string{} (is non-nil but zero-length)

Interfaces & Receivers

The following snippets shows how to do ‘force checking’ of contact between type interface and type struct.

type ValueConverter interface {ConvertValue(v interface{}) (Value, error)}type boolType struct{}var Bool boolTypevar _ ValueConverter = boolType{}  --> (a)func (boolType) String() string { return "Bool" }func (boolType) ConvertValue(src interface{}) (Value, error) {....}

the above provides a static (compile time) check that boolType satisfies the ValueConverter interface. The _ used as a name of the variable tells the compiler to effectively discard the RHS value, but to type-check it and evaluate it if it has any side effects, but the anonymous variable per se doesn’t take any process space.

It is a handy construct when developing and the method set of an interface and/or the methods implemented by a type are frequently changed. The construct serves as a guard against forgetting to match the method sets of a type and of an interface where the intent is to have them compatible. It effectively prevents to go install a broken (intermediate) version with such omission.

Another example (taken from https://github.com/Kong/kuma/tree/master/pkg/core/discovery) where the sink.go enforce the contract inside the interfaces.go

sink.go

    package discovery    import (        mesh_core "github.com/Kong/kuma/pkg/core/resources/apis/mesh"        core_model "github.com/Kong/kuma/pkg/core/resources/model"    )    var _ DiscoverySource = &DiscoverySink{}    var _ DiscoveryConsumer = &DiscoverySink{}    // DiscoverySink is both a source and a consumer of discovery information.    type DiscoverySink struct {        DataplaneConsumer DataplaneDiscoveryConsumer    }    func (s *DiscoverySink) AddConsumer(consumer DiscoveryConsumer) {        s.DataplaneConsumer = consumer    }    func (s *DiscoverySink) OnDataplaneUpdate(dataplane *mesh_core.DataplaneResource) error {        if s.DataplaneConsumer != nil {            return s.DataplaneConsumer.OnDataplaneUpdate(dataplane)        }        return nil    }    func (s *DiscoverySink) OnDataplaneDelete(key core_model.ResourceKey) error {        if s.DataplaneConsumer != nil {            return s.DataplaneConsumer.OnDataplaneDelete(key)        }        return nil    }

interfaces.go

    package discovery    import (        discovery_proto "github.com/Kong/kuma/api/discovery/v1alpha1"        mesh_core "github.com/Kong/kuma/pkg/core/resources/apis/mesh"        "github.com/Kong/kuma/pkg/core/resources/model"    )    // DiscoverySource is a source of discovery information, i.e. Services and Workloads.    type DiscoverySource interface {        AddConsumer(DiscoveryConsumer)    }    // ServiceInfo combines original proto object with auxiliary information.    type ServiceInfo struct {        Service *discovery_proto.Service    }    // WorkloadInfo combines original proto object with auxiliary information.    type WorkloadInfo struct {        Workload *discovery_proto.Workload        Desc     *WorkloadDescription    }    // WorkloadDescription represents auxiliary information about a Workload.    type WorkloadDescription struct {        Version   string        Endpoints []WorkloadEndpoint    }    type WorkloadEndpoint struct {        Address string        Port    uint32    }    // DiscoveryConsumer is a consumer of discovery information, i.e. Services and Workloads.    type DiscoveryConsumer interface {        DataplaneDiscoveryConsumer    }    type DataplaneDiscoveryConsumer interface {        OnDataplaneUpdate(*mesh_core.DataplaneResource) error        OnDataplaneDelete(model.ResourceKey) error    }

Go interfaces generally belong in the package that uses values of the interface type, not the package that implements those values. The implementing package should return concrete (usually pointer or struct) types: that way, new methods can be added to implementations without requiring extensive refactoring.
Design the API so that it can be tested using the public API of the real implementation.
Do not define interfaces before they are used: without a realistic example of usage, it is too difficult to see whether an interface is even necessary, let alone what methods it ought to contain.

Go interfaces generally belong in the package that uses values of the interface type, not the package that implements those values.

    package consumer  // consumer.go    type Thinger interface { Thing() bool }    func Foo(t Thinger) string { … }

    package consumer // consumer_test.go    type fakeThinger struct{ … }    func (t fakeThinger) Thing() bool { … }    …    if Foo(fakeThinger{…}) == "x" { … }

Instead of doing this

    // DO NOT DO IT!!!    package producer    type Thinger interface { Thing() bool }    type defaultThinger struct{ … }    func (t defaultThinger) Thing() bool { … }    func NewThinger() Thinger { return defaultThinger{ … } }

…better do this

      package producer    type Thinger struct{ … }    func (t Thinger) Thing() bool { … }    func NewThinger() Thinger { return Thinger{ … } }

In Go, the receiver of a method is just another parameter and therefore, should be named accordingly. The name need not be as descriptive as that of a method argument, as its role is obvious and serves no documentary purpose. It can be very short as it will appear on almost every line of every method of the type; familiarity admits brevity. Be consistent, too: if you call the receiver “c” in one method, don’t call it “cl” in another.
- If in doubt, use a pointer, but there are times when a value receiver makes sense, usually for reasons of efficiency, such as for small unchanging structs or values of basic type. Some useful guidelines:
  - NO POINTER for the following:
    - If the receiver is a map, func or chan, don’t use a pointer to them. If the receiver is a slice and the method doesn’t reslice or reallocate the slice, don’t use a pointer to it.
    - If the receiver is a small array or struct that is naturally a value type (for instance, something like the time.Time type), with no mutable fields and no pointers, or is just a simple basic type such as int or string, a value receiver makes sense. A value receiver can reduce the amount of garbage that can be generated; if a value is passed to a value method, an on-stack copy can be used instead of allocating on the heap. (The compiler tries to be smart about avoiding this allocation, but it can’t always succeed.) Don’t choose a value receiver type for this reason without profiling first.
  - USE POINTER for the following:
    - If the method needs to mutate the receiver, the receiver must be a pointer.
    - If the receiver is a struct that contains a sync.Mutex or similar synchronizing field, the receiver must be a pointer to avoid copying.
    - If the receiver is a large struct or array, a pointer receiver is more efficient. How large is large? Assume it’s equivalent to passing all its elements as arguments to the method. If that feels too large, it’s also too large for the receiver.
    - Can function or methods, either concurrently or when called from this method, be mutating the receiver? A value type creates a copy of the receiver when the method is invoked, so outside updates will not be applied to this receiver. If changes must be visible in the original receiver, the receiver must be a pointer.
    - If the receiver is a struct, array or slice and any of its elements is a pointer to something that might be mutating, prefer a pointer receiver, as it will make the intention more clear to the reader.
    - Finally, when in doubt, use a pointer receiver.
Compile-time checking
```
  // Compile-time check to assert this config matches requirements.var _ setup.KeyManagerProvider = (*Config)(nil)var _ setup.BlobStorageConfigProvider = (*Config)(nil)var _ setup.DBConfigProvider = (*Config)(nil)
```
the above is to check whether the Config object implement the function defined in KeyManagerProvider, BlobStorageConfigProvider and DBConfigProvider.
The (*Config)(nil) is to do the verification between the interface and struct WITHOUT allocating any memory. See below for more examples

Compile-time checking examples

  package maintype I interface{ M() }type T struct{}func (T) M() {}//func (*T) M() {} //var _ I = T{}: T does not implement I (M method has pointer receiver)func main() {  //avoids allocation of memory  var _ I = T{}       // Verify that T implements I.  var _ I = (*T)(nil) // Verify that *T implements I.  //allocation of memory  var _ I = &T{}      // Verify that &T implements I.  var _ I = new(T)    // Verify that new(T) implements I.}

Compile-time checking examples

  var (        _ blobref.StreamingFetcher = (*CachingFetcher)(nil)        _ blobref.SeekFetcher      = (*CachingFetcher)(nil)        _ blobref.StreamingFetcher = (*DiskCache)(nil)        _ blobref.SeekFetcher      = (*DiskCache)(nil))

Ensure compiler checks:

That CachingFether implements public functions of StreamingFetcher and SeekFetcher.
RHS portion uses a pointer constructor syntax with a nil parameter. What does this syntax mean in Go language ?

Type Casting for checking. The following is based on the example from github.com/google/exposure-notifications-server. The KeyManagerProvider is an interface as follows (found inside setup.go)
```
  type KeyManagerProvider interface {KeyManager() bool}
```
Config object is defined as follows (found inside config.go)
```
  type Config struct {Database          *database.Config......WorkerTimeout     time.Duration `envconfig:"WORKER_TIMEOUT" default:"5m"`...}func (c *Config) KeyManager() bool {return false}
```
the struct implement the KeyManager() function. The setup.Setup(..) function is defined as follows
```
  func Setup(ctx context.Context, config DBConfigProvider) (*serverenv.ServerEnv, Defer, error) {......}
```
it is called as follows from export/main.go
```
  var config export.Configenv, closer, err := setup.Setup(ctx, &config)
```
the following code inside Setup(..) will be analysed (found inside setup.go)
```
  ...if _, ok := config.(KeyManagerProvider); ok {...}...
```
is actually a way to check whether the passed config parameter implemented the function ‘KeyManager()’.

Testing

Tests should fail with helpful messages saying what was wrong, with what inputs, what was actually got, and what was expected.

  if got != tt.want {    t.Errorf("Foo(%q) = %d; want %d", tt.in, got, tt.want) // or Fatalf, if test can't test anything more past this point}

Comments

For comments use // instead of /* */
Start comment with the name of the function
All top-level, exported names should have doc comments, as should non-trivial unexported type or function declarations.
For readability, move the comment to appear on it’s own line before the variable. Following is NOT a good example
```
 ...OutPrefix   = "> " // OutPrefix notes output
```

Package comments, like all comments to be presented by godoc, must appear adjacent to the package clause, with no blank line.

 // Package math provides basic constants and mathematical functions.package math

 /*Package template implements data-driven templates for generating textualoutput such as HTML.....*/package template

For “package main” comments, other styles of comment are fine after the binary name. For example, for a package main in the directory seedgen you could write:

 // Binary seedgen ...package main// Command seedgen ...package main// Program seedgen ...package main// The seedgen command ...package main// The seedgen program ...package main// Seedgen ..package main

Functions

If a function returns two or three parameters of the same type, or if the meaning of a result isn’t clear from context, adding names may be useful in some contexts.
Don’t name result parameters just to avoid declaring a var inside the function; that trades off a minor implementation brevity at the cost of unnecessary API verbosity.
Naked returns are okay if the function is a handful of lines. Once it’s a medium sized function, be explicit with your return values. Corollary: it’s not worth it to name result parameters just because it enables you to use naked returns. Clarity of docs is always more important than saving a line or two in your function.
Don’t pass pointers as function arguments just to save a few bytes, however This advice does not apply to large structs, or even small structs that might grow.

Error Handling

Don’t use panic for normal error handling. Use error and multiple return values.
Error strings should not be capitalized (unless beginning with proper nouns or acronyms) or end with punctuation
Do not discard errors using _ variables. Handle the error, return it, or, in truly exceptional situations, panic.
Do not use return value such as in C using -1 to flag error. Function can return multiple result so use one of the returned result to flag if there is an error. Eg:
```
 value, ok := Lookup(key)if !ok  {    return fmt.Errorf("no value for %q", key)}return Parse(value)
```

This is recommended

 if err != nil {    // error handling    return // or continue, etc.}// normal code

instead of

 if err != nil {    // error handling} else {    // normal code}

Packages

Avoid renaming imports except to avoid a name collision; good package names should not require renaming. In the event of collision, prefer to rename the most local or project-specific import.
Packages that are imported only for their side effects (using the syntax import _ “pkg”) should only be imported in the main package of a program, or in tests that require them.
In this case, the test file cannot be in package foo because it uses bar/testutil, which imports foo. So we use the ‘import .’ form to let the file pretend to be part of package foo even though it is not. Except for this one case, do not use import . in your programs. It makes the programs much harder to read because it is unclear whether a name like Quux is a top-level identifier in the current package or in an imported package.
Avoid meaningless package names like util, common, misc, api, types, and interfaces

Concurrency

This article explain in simple ways on how the directional nature of the channel works. The article details the different direction that a channel can have.

    // SliceIterChan returns each element of a slice on a channel for concurrent    // consumption, closing the channel on completion    func SliceIterChan(s []int) <-chan int {        outChan := make(chan int)        go func() {            for i := range s {                outChan <- s[i]            }            close(outChan)        }()        return outChan    }

The article explains

Diving into the implementation, the function creates a bidirectional channel for its own use, and then all it needs to do to ensure that it has full control over writing to and closing the channel is to return it, whereupon it will be converted into a read-only channel automatically.

Following are excerpts from this and this
- Both the channel and the value expression are evaluated before communication begins.
- Communication blocks until the send can proceed.
- A send on an unbuffered channel can proceed if a receiver is ready.
- A send on a buffered channel can proceed if there is room in the buffer.
- A send on a closed channel proceeds by causing a run-time panic.
- Communication on nil channels can never proceed, a select with only nil channels and no default case blocks forever.

repeatFn := func(    done <-chan interface{},    fn func() interface{},) <-chan interface{} {    valueStream := make(chan interface{})    go func() {        defer close(valueStream)        for {            select {            case <-done:                return            case valueStream <- fn():            }        }    }()    return valueStream}

One of repeatFn(..) argument is to receive ‘done’ variable that the repeatFn(..) will use as a flag to exit from the functionThe value of the receive operation <-ch is the value received from the channel ch. The expression blocks until a value is available. Receiving from a nil channel blocks forever. A receive operation on a closed channel can always proceed immediately, yielding the element type’s zero value after any previously sent values have been received.

Following are questions to answer in order to have a better understanding how the whole multi-threading channel based works in Go. These are not an exhaustive list but a starting point that will help to stir in the right direction.
- How do we call goroutine inside a function ?
- When using goroutine inside a function how will the code flow ?
- How does the for(..) range loops works with channels ? how does it differ than normal for(..) counter loop
- How to use the select{ case : } operator with channels ?
- When closing channel is it closed AFTER operation is done ?

Send the channel that your function want to receive the data from

    func First(query string,  c chan Result, replicas ...Search){        collating := make(chan Result)        searchReplica := func(i int, collating chan Result) {             c <- replicas[i](query)         }        for i := range replicas {            go searchReplica(i,collating)        }    }

in the above example the First(..) method’s param is a channel parameter ‘c’, it uses this channel to send data that will be read by the caller function.

Use timeout as much as possible to avoid the app from ‘hang’ mode. Having timeout enable troubleshooting as function will automatically terminate after certain amount of time.

    timeout := time.After(800 * time.Millisecond)    for {        select {        ...        ...        case <-timeout:            fmt.Println("timed out")            return        }    }

If we close the channel and we try to read data (even AFTER reading the data) no exception thrown

func main(){    // size chosen to ensure it never gets filled up completely.    threadChannel := make(chan string,1)     go func() {        threadChannel <- "TESTING from function 1"        defer close(threadChannel)    }()    fmt.Println("Obtained  1" , <-threadChannel)    time.Sleep(1000 * time.Millisecond)    fmt.Println("----------------")    fmt.Println("Obtained  2" , <-threadChannel)}

If we DO NOT close the channel and try to read data (AFTER reading the data) deadlock exception will be thrown

func main(){    threadChannel := make(chan string,1) // size chosen to ensure it never gets filled up completely.    go func() {        threadChannel <- "TESTING from function 1"        //defer close(threadChannel)    }()    fmt.Println("Obtained  1" , <-threadChannel)    time.Sleep(1000 * time.Millisecond)    fmt.Println("----------------")    fmt.Println("Obtained  2" , <-threadChannel)}

ALWAYS close(..) the channel to stop the go func(..) running and resides in memory. Without closing the channel possibility of memory leak and/or deadlock is high
Mechanism should always be in place to make sure that the parent can instruct the child goroutine to cancel it’s operation.

Goroutine

When you spawn goroutines, make it clear when or whether - they exit.
Goroutines can leak by blocking on channel sends or receives: the garbage collector will not terminate a goroutine even if the channels it is blocked on are unreachable.
Even when goroutines do not leak, leaving them in-flight when they are no longer needed can cause other subtle and hard-to-diagnose problems.
Try to keep concurrent code simple enough that goroutine lifetimes are obvious. If that just isn’t feasible, document when and why the goroutines exit.
DO NOT use package math/rand to generate keys, instead, use crypto/rand’s Reader.
Every time running a goroutine(..) we must think how it will finish. What will be the trigger to return or exit from the gorotine
At any time we want to to send data into a channel make sure the channel variable is send as parameter to the called method

Select packages

Following is an example to use SelectCase in situation where there are multiple channels to process

package mainimport ("fmt""reflect")func produce(ch chan<- string, i int) {for j := 0; j < 5; j++ {ch <- fmt.Sprint(i*10 + j)}close(ch)}func main() {numChans := 4var chans = []chan string{}for i := 0; i < numChans; i++ {ch := make(chan string)chans = append(chans, ch)go produce(ch, i+1)}cases := make([]reflect.SelectCase, len(chans))for i, ch := range chans {cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}}remaining := len(cases)for remaining > 0 {chosen, value, ok := reflect.Select(cases)if !ok {// The chosen channel has been closed, so zero out the channel to disable the casecases[chosen].Chan = reflect.ValueOf(nil)remaining -= 1continue}fmt.Printf("Read from channel %#v and received %s\n", chans[chosen], value.String())}}

SelectCase can be used to receive or send data.

To use it to receive data use the SelectRecv direction

cases[i] = reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(ch)}

the Chan will point to the channel that will be used to receive the data.

To read the data use the following

chosen, value, ok := reflect.Select(cases)

the value variable will contain the data from the channel

References

* Best practises in Golang* Lesson learned from contributing minikube* Code Review Best Practises Golang* Go Advices

Minikube commands (Work in progress)

Sun, 01 Sep 2019 00:00:00 +0000

Following are the commands available for minikube.

Basic Commands:  start          Starts a local kubernetes cluster  status         Gets the status of a local kubernetes cluster  stop           Stops a running local kubernetes cluster  delete         Deletes a local kubernetes cluster  dashboard      Access the kubernetes dashboard running within the minikube clusterImages Commands:  docker-env     Sets up docker env variables; similar to '$(docker-machine env)'  cache          Add or delete an image from the local cache.Configuration and Management Commands:  addons         Modify minikube's kubernetes addons  config         Modify minikube config  profile        Profile gets or sets the current minikube profile  update-context Verify the IP address of the running cluster in kubeconfig.Networking and Connectivity Commands:  service        Gets the kubernetes URL(s) for the specified service in your local cluster  tunnel         tunnel makes services of type LoadBalancer accessible on localhostAdvanced Commands:  mount          Mounts the specified directory into minikube  ssh            Log into or run a command on a machine with SSH; similar to 'docker-machine ssh'  kubectl        Run kubectlTroubleshooting Commands:  ssh-key        Retrieve the ssh identity key path of the specified cluster  ip             Retrieves the IP address of the running cluster  logs           Gets the logs of the running instance, used for debugging minikube, not user code.  update-check   Print current and latest version number  version        Print the version of minikubeOther Commands:  completion     Outputs minikube shell completion for the given shell (bash or zsh)

The source code for the configurations outlined above can be found inside cmd/minikube/cmd directory. Noticed how the .go files are named the same as the configuration parameter which makes it easier for identifying the relevant code.

├── cache.go├── cache_list.go├── completion.go├── dashboard.go├── delete.go├── env.go├── env_test.go├── generate-docs.go├── ip.go├── kubectl.go├── logs.go├── mount.go├── root.go├── root_test.go├── service.go├── service_list.go├── ssh.go├── ssh-key.go├── start.go├── start_test.go├── status.go├── stop.go├── tunnel.go├── update-check.go├── update-context.go└── version.go

The main minikube entry point reside inside cmd/minikube directory in a file called main.go

package mainimport ("bytes""fmt""log""os""regexp"......    ......)    ......func main() {bridgeLogMessages()defer glog.Flush()if os.Getenv(minikubeEnableProfile) == "1" {defer profile.Start(profile.TraceProfile).Stop()}    ......cmd.Execute()}    ......    ......    ......

start

Command to trigger the minikube starting process. There are varieties of options that you can pass into minikube, most of the parameters are self described.

      --apiserver-ips=[]: A set of apiserver IP Addresses which are used in the generated certificate for kubernetes.  This can be used if you want to make the apiserver available from outside the machine      --apiserver-name='minikubeCA': The apiserver name which is used in the generated certificate for kubernetes.  This can be used if you want to make the apiserver available from outside the machine      --apiserver-names=[]: A set of apiserver names which are used in the generated certificate for kubernetes.  This can be used if you want to make the apiserver available from outside the machine      --apiserver-port=8443: The apiserver listening port      --cache-images=true: If true, cache docker images for the current bootstrapper and load them into the machine. Always false with --vm-driver=none.      --container-runtime='docker': The container runtime to be used (docker, crio, containerd).      --cpus=2: Number of CPUs allocated to the minikube VM.      --cri-socket='': The cri socket path to be used.      --disable-driver-mounts=false: Disables the filesystem mounts provided by the hypervisors      --disk-size='20000mb': Disk size allocated to the minikube VM (format: <number>[<unit>], where unit = b, k, m or g).      --dns-domain='cluster.local': The cluster dns domain name used in the kubernetes cluster      --dns-proxy=false: Enable proxy for NAT DNS requests (virtualbox driver only)      --docker-env=[]: Environment variables to pass to the Docker daemon. (format: key=value)      --docker-opt=[]: Specify arbitrary flags to pass to the Docker daemon. (format: key=value)      --download-only=false: If true, only download and cache files for later use - don't install or start anything.      --embed-certs=false: if true, will embed the certs in kubeconfig.      --enable-default-cni=false: Enable the default CNI plugin (/etc/cni/net.d/k8s.conf). Used in conjunction with "--network-plugin=cni".      --extra-config=: A set of key=value pairs that describe configuration that may be passed to different components.                The key should be '.' separated, and the first part before the dot is the component to apply the configuration to.                Valid components are: kubelet, kubeadm, apiserver, controller-manager, etcd, proxy, scheduler                Valid kubeadm parameters: ignore-preflight-errors, dry-run, kubeconfig, kubeconfig-dir, node-name, cri-socket, experimental-upload-certs, certificate-key, rootfs, pod-network-cidr      --feature-gates='': A set of key=value pairs that describe feature gates for alpha/experimental features.      --force=false: Force minikube to perform possibly dangerous operations      --host-dns-resolver=true: Enable host resolver for NAT DNS requests (virtualbox driver only)      --host-only-cidr='192.168.99.1/24': The CIDR to be used for the minikube VM (virtualbox driver only)      --hyperkit-vpnkit-sock='': Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac VPNKit connection, otherwise uses the specified VSock (hyperkit driver only)      --hyperkit-vsock-ports=[]: List of guest VSock ports that should be exposed as sockets on the host (hyperkit driver only)      --hyperv-virtual-switch='': The hyperv virtual switch name. Defaults to first found. (hyperv driver only)      --image-mirror-country='': Country code of the image mirror to be used. Leave empty to use the global one. For Chinese mainland users, set it to cn.      --image-repository='': Alternative image repository to pull docker images from. This can be used when you have limited access to gcr.io. Set it to "auto" to let minikube decide one for you. For Chinese mainland users, you may use local gcr.io mirrors such as registry.cn-hangzhou.aliyuncs.com/google_containers      --insecure-registry=[]: Insecure Docker registries to pass to the Docker daemon.  The default service CIDR range will automatically be added.      --iso-url='https://storage.googleapis.com/minikube/iso/minikube-v0.0.0-unset.iso': Location of the minikube iso.      --keep-context=false: This will keep the existing kubectl context and will create a minikube context.      --kubernetes-version='v1.16.0-rc.2': The kubernetes version that the minikube VM will use (ex: v1.2.3)      --kvm-gpu=false: Enable experimental NVIDIA GPU support in minikube      --kvm-hidden=false: Hide the hypervisor signature from the guest in minikube (kvm2 driver only)      --kvm-network='default': The KVM network name. (kvm2 driver only)      --kvm-qemu-uri='qemu:///system': The KVM QEMU connection URI. (kvm2 driver only)      --memory='2000mb': Amount of RAM allocated to the minikube VM (format: <number>[<unit>], where unit = b, k, m or g).      --mount=false: This will start the mount daemon and automatically mount files into minikube.      --mount-string='/home/nanik:/minikube-host': The argument to pass the minikube mount command on start.      --native-ssh=true: Use native Golang SSH client (default true). Set to 'false' to use the command line 'ssh' command when accessing the docker machine. Useful for the machine drivers when they will not start with 'Waiting for SSH'.      --network-plugin='': The name of the network plugin.      --nfs-share=[]: Local folders to share with Guest via NFS mounts (hyperkit driver only)      --nfs-shares-root='/nfsshares': Where to root the NFS Shares, defaults to /nfsshares (hyperkit driver only)      --no-vtx-check=false: Disable checking for the availability of hardware virtualization before the vm is started (virtualbox driver only)      --registry-mirror=[]: Registry mirrors to pass to the Docker daemon      --service-cluster-ip-range='10.96.0.0/12': The CIDR to be used for service cluster IPs.      --uuid='': Provide VM UUID to restore MAC address (hyperkit driver only)      --vm-driver='': Driver is one of: [virtualbox parallels vmwarefusion kvm2 vmware none] (defaults to virtualbox)      --wait=true: Wait until Kubernetes core services are healthy before exiting.      --wait-timeout=6m0s: max time to wait per Kubernetes core services to be healthy.

The start option source code detecting the different options pass resides inside cmd/minikube/cmd/start.go. Following tables outline some of the commands that require bit more explanation of it’s usage


–enable-default-cni – using this flag it will use the default cni config inside pkg/minikube/bootstrapper/kubeadm/default_cni.go which will be stored inside /etc/cni/net.d directory as k8s.conf

Enabling cache –cache-images=true when starting up minikube helps in deployment process by caching the images used inside the VM. Following is what local cache looks like. Most files are docker image files.

├── images│   ├── gcr.io│   │   └── k8s-minikube│   │       ├── storage-provisioner_v1.8.1│   │       └── storage-provisioner_v1.8.1.823687817.tmp│   └── k8s.gcr.io│       ├── coredns_1.3.1│       ├── coredns_1.6.2│       ├── etcd_3.3.10.801406343.tmp│       ├── etcd_3.3.15-0│       ├── k8s-dns-dnsmasq-nanny-amd64_1.14.13│       ├── k8s-dns-kube-dns-amd64_1.14.13│       ├── k8s-dns-kube-dns-amd64_1.14.13.681617407.tmp│       ├── k8s-dns-sidecar-amd64_1.14.13│       ├── kube-addon-manager_v9.0│       ├── kube-addon-manager_v9.0.682111654.tmp│       ├── kube-apiserver_v1.16.0│       ├── kube-apiserver_v1.16.0-rc.2.869485211.tmp│       ├── kube-controller-manager_v1.16.0│       ├── kube-controller-manager_v1.16.0-rc.2│       ├── kube-proxy_v1.16.0│       ├── kube-proxy_v1.16.0-rc.2│       ├── kubernetes-dashboard-amd64_v1.10.1│       ├── kubernetes-dashboard-amd64_v1.10.1.460043582.tmp│       ├── kube-scheduler_v1.16.0│       ├── kube-scheduler_v1.16.0-rc.2│       └── pause_3.1├── iso│   └── minikube-v1.4.0.iso└── v1.16.0    ├── kubeadm    └── kubelet

status

Gets the status of a local kubernetes cluster.        Exit status contains the status of minikube's VM, cluster and kubernetes encoded on it's bits in this order from rightto left.        Eg: 7 meaning: 1 (for minikube NOK) + 2 (for cluster NOK) + 4 (for kubernetes NOK)Options:      --format='host: {{.Host}}kubelet: {{.Kubelet}}apiserver: {{.APIServer}}kubectl: {{.Kubeconfig}}': Go template format string for the status output.  The format for Go templates can be found here:https://golang.org/pkg/text/template/For the list accessible variables for the template, see the struct values here:https://godoc.org/k8s.io/minikube/cmd/minikube/cmd#Status

cache

Add or delete an image from the local cache.Available Commands:  add         Add an image to local cache.  delete      Delete an image from the local cache.  list        List all available images from the local cache.

docker-env

This command will outputs the necessary environment variable that you need to set to point your local docker to minikube. Following is the output of the command when ran locally

export DOCKER_TLS_VERIFY="1"export DOCKER_HOST="tcp://192.168.99.127:2376"export DOCKER_CERT_PATH="/home/nanik/.minikube/certs"

To use the output settings use the following command

eval $(minikube docker-env)

Once the above command is ran everytime you run docker CLI it will communicate with the docker inside minikube.

addons

The addons command allow you to enable, disable, etc addons inside minikube

Available Commands:  configure   Configures the addon w/ADDON_NAME within minikube (example: minikube addons configure registry-creds). Fora list of available addons use: minikube addons list   disable     Disables the addon w/ADDON_NAME within minikube (example: minikube addons disable dashboard). For a listof available addons use: minikube addons list   enable      Enables the addon w/ADDON_NAME within minikube (example: minikube addons enable dashboard). For a list ofavailable addons use: minikube addons list   list        Lists all available minikube addons as well as their current statuses (enabled/disabled)  open        Opens the addon w/ADDON_NAME within minikube (example: minikube addons open dashboard). For a list ofavailable addons use: minikube addons list Options:      --format='- {{.AddonName}}: {{.AddonStatus}}': Go template format string for the addon list output.  The format for Go templates can be found here:https://golang.org/pkg/text/template/For the list of accessible variables for the template, see the struct values here:https://godoc.org/k8s.io/minikube/cmd/minikube/cmd/config#AddonListTemplate

config

The below configuration are relevant to the parameter that you specified when you use the ‘start’ command.

config modifies minikube config files using subcommands like "minikube config set vm-driver kvm"Configurable fields:  * vm-driver * container-runtime * feature-gates * v * cpus * disk-size * host-only-cidr * memory * log_dir * kubernetes-version * iso-url * WantUpdateNotification * ReminderWaitPeriodInHours * WantReportError * WantReportErrorPrompt * WantKubectlDownloadMsg * WantNoneDriverWarning * profile * bootstrapper * ShowDriverDeprecationNotification * ShowBootstrapperDeprecationNotification * dashboard * addon-manager * default-storageclass * heapster * efk * ingress * insecure-registry * registry * registry-creds * freshpod * default-storageclass * storage-provisioner * storage-provisioner-gluster * metrics-server * nvidia-driver-installer * nvidia-gpu-device-plugin * logviewer * gvisor * hyperv-virtual-switch * disable-driver-mounts * cache * embed-certs * native-sshAvailable Commands:  get         Gets the value of PROPERTY_NAME from the minikube config file  set         Sets an individual value in a minikube config file  unset       unsets an individual value in a minikube config file  view        Display values currently set in the minikube config file

service

Gets the kubernetes URL(s) for the specified service in your local cluster. In the case of multiple URLs they will beprinted one at a time.Available Commands:  list        Lists the URLs for the services in your local clusterOptions:      --https=false: Open the service URL with https instead of http      --interval=6: The initial time interval for each check that wait performs in seconds  -n, --namespace='default': The service namespace      --url=false: Display the kubernetes service URL in the CLI instead of opening it in the default browser      --wait=20: Amount of time to wait for a service in seconds

mount

Mounts the specified directory into minikube.Options:      --9p-version='9p2000.L': Specify the 9p version that the mount should use      --gid='docker': Default group id used for the mount      --ip='': Specify the ip that the mount should be setup on      --kill=false: Kill the mount process spawned by minikube start      --mode=493: File permissions used for the mount      --msize=262144: The number of bytes to use for 9p packet payload      --options=[]: Additional mount options, such as cache=fscache      --type='9p': Specify the mount filesystem type (supported types: 9p)

Minikube parameters

mapping of arguments to source code
references
- https://evalle.xyz/posts/configure-kube-apiserver-in-minikube/
- https://www.marcolancini.it/2019/blog-deploy-kubernetes-lab/

Playing with Docker SDK (Work in progress)

Sun, 01 Sep 2019 00:00:00 +0000

This article will walk through on how to use the docker SDK using Go. Docker provide a very rich SDK allowing developers to communicate with the docker daemon.

Intercepting Docker

Run the following to intercept

socat -v UNIX-LISTEN:/tmp/fake,fork UNIX-CONNECT:/var/run/docker.sock

Run the following to make sure docker CLI send command to our proxy in /tmp/fake

export DOCKER_HOST=unix:///tmp/fakedocker image list

The output will show as follows

> 2019/09/12 20:38:27.924464  length=81 from=0 to=80HEAD /_ping HTTP/1.1\rHost: docker\rUser-Agent: Docker-Client/xx.xx.1 (linux)\r\r< 2019/09/12 20:38:27.924817  length=287 from=0 to=286HTTP/1.1 200 OK\rApi-Version: 1.41\rCache-Control: no-cache, no-store, must-revalidate\rContent-Length: 0\rContent-Type: text/plain; charset=utf-8\rDocker-Experimental: false\rOstype: linux\rPragma: no-cache\rServer: Docker/library-import (linux)\rDate: Thu, 12 Sep 2019 10:38:27 GMT\r\r> 2019/09/12 20:38:27.925517  length=92 from=81 to=172GET /v1.40/images/json HTTP/1.1\rHost: docker\rUser-Agent: Docker-Client/xx.xx.1 (linux)\r\r< 2019/09/12 20:38:27.926859  length=889 from=287 to=1175HTTP/1.1 200 OK\rApi-Version: 1.41\rContent-Type: application/json\rDocker-Experimental: false\rOstype: linux\rServer: Docker/library-import (linux)\rDate: Thu, 12 Sep 2019 10:38:27 GMT\rContent-Length: 679\r\r[{"Containers":-1,"Created":1568087900,"Id":.......................

Mon, 01 Jan 0001 00:00:00 +0000

Report

Analysis based on the published post mortem report by Canva

The API Gateway cluster failure was caused by multiple factors, including:

A software deployment related to Canva’s editor, which contributed to system instability.
A locking issue, potentially affecting resource availability and request handling.
Network issues in Cloudflare, which further disrupted connectivity and compounded the failure.

User and Canva Editor

The Canva Editor is the main UI application that users interact with.
Single Page Application (SPA) (assumed built using TypeScript and/or JavaScript).
Symptons:
- The page appears to be continuously loading, seemingly waiting for assets to be downloaded.
- Users may keep waiting, expecting the application to load eventually.
- Users must have been impatience and refreshing the browser repeatedly.

Cloudflare (CF)

CF uses cache stream (Concurrent Streaming Acceleration) to optimize caching performance. The feature consolidate all user’s request requesting the same assets to be batch together and once assets are obtained from origin all the request will get the response.
Since request are not rejected (it’s put on hold by CF) so no request errors are logged so not alert are triggered.
Even as there are 270,000 requests waiting in CF queue for the assets there is only 1 connection from CF going to the origin server to get the assets, as the assets has not been cache. Once the assets are cached it is delivered to all of the 270,000 waiting request.

API Gateway

The 270,000 user devices have the latest code running. The number of request hitting the API gateway now triple peak load (1.5 million request/sec).
The running gateways were not able to process the requests successfully as newly introduced telemetry SDK code introduced a new bug that created lock contention.
The lock contention bug magnified the issue as requests are now failing because requests cannot be processed by the available running gateway.
Auto scaling should be able to solve the problem but it was not able to scale up quickly as the number of requests are overhelming the infrastructure.

Lock Contention

What is lock contention:

this refers to a scenario in concurrent programming where multiple threads or processes are contending for access to a shared resource, and there is a high likelihood of conflicts or contention events occurring.

In the context of the Telemtry SDK there is a shared data structure that probably were not intended for concurrent access but the code that usesthe data are accessing it concurrently creating a lock contention.

The post mortem highlight the part where the gateway is indeed doing it’s job in an event loop and code running in the event loop should notperform any blocking operation, but in this case it is performing blocking operation producing lock contention.

The API Gateways use an event loop model, where code running on event loop threads must not perform any blocking operations.

Key Takeaways

Understanding the SDK (even if it is an internally developed) limitation is crucial in using it correctly.
Designing concurrent application is hard so duty of care should be taken to perform proper testing.
Concurrent performance testing should be part of the release process. Having standard performance baseline enables tocompare how the application is performing and this can assist in picking up anomaly.

Mon, 01 Jan 0001 00:00:00 +0000

The following is a function type with function called ServeHTTP

type HandlerFunc func(ResponseWriter, *Request)func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {    f(w, r)}

having function declared as follow

func final(w http.ResponseWriter, r *http.Request) {    log.Println("Executing finalHandler")    w.Write([]byte("OK"))}

we can wrap the final(..) function as handler using the following

finalHandler := http.HandlerFunc(final)

After wrapping, the finalHandler will hace access to the ServerHTTP(..) functionwhich basically is calling the final(….) function

This is the way to wrap a function (final) that has the same signatureas the named wrapper function (ServeHTTP). The benefit of doing this is the abilityfor other code to just know the type function (ServeHTTP) and call it with the comfortof knowing that it will call the correct implementation function.

Another example:

package mainimport (    "fmt")type MyHandlerFunc  func(string, int)func (f MyHandlerFunc) FunctionOne(s string, i int){    fmt.Println("FunctionOne = ", s , " with i = ", i)    f(s,i)}func (f MyHandlerFunc) FunctionTwo(s string, i int) {    fmt.Println("FunctionTwo = ", s, " with i = ", i)    f(s,i)}func normalFunction(s string, i int) {    fmt.Println("normalFunction = ", s, " with i = ", i)}func main() {    finalHandler := MyHandlerFunc(normalFunction)    finalHandler.FunctionOne("fromFunctionOne",10)    finalHandler.FunctionTwo("fromFunctionTwo",10)}

Another example:

package mainimport (    "fmt"    "log"    "net/http")func middlewareOne(next http.Handler) http.Handler {    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {        log.Println("Executing middlewareOne")        next.ServeHTTP(w, r)        log.Println("Executing middlewareOne again")    })}func middlewareTwo(next http.Handler) http.Handler {    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {        log.Println("Executing middlewareTwo")        if r.URL.Path != "/" {            return        }        next.ServeHTTP(w, r)        log.Println("Executing middlewareTwo again")    })}func final(w http.ResponseWriter, r *http.Request) {    log.Println("Executing finalHandler")    w.Write([]byte("OK"))}func main() {    finalHandler := http.HandlerFunc(final)    http.Handle("/", middlewareOne(middlewareTwo(finalHandler)))    http.ListenAndServe(":3000", nil)}

Another example:

type tokenizer func() (token string, ok bool)func split(s, sep string) tokenizer {    tokens, last := strings.Split(s,sep),0    return func() (string, bool) {        if  len(tokens) == last {            return "", false        }        last = last+1        return tokens[last-1], true    }}

Interface using inside github.com/tdewolff/minify

minifier.go

Global variable declared as

var minifier = minify.New()

THe init() function do the following

func init() {    htmlMinify := &html.Minifier{        KeepWhitespace:   true,        KeepDocumentTags: true,    }    minifier.AddFunc("text/css", css.Minify)    minifier.Add("text/html", htmlMinify)    minifier.Add("text/x-template", htmlMinify)    minifier.AddFunc("text/javascript", js.Minify)    minifier.AddFunc("application/javascript", js.Minify)    minifier.AddFunc("application/x-javascript", js.Minify)    minifier.AddFunc("image/svg+xml", svg.Minify)    minifier.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)    minifier.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)}

the code inside init() add the relevant function for the different mime type:eg: for text/css it will call css.minify

The following code is the code that is called in the above .AddFunc(..)

func Minify(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {    return (&Minifier{}).Minify(m, w, r, params)}

The following code

(&Minifier{}).Minify(m, w, r, params)

means to call the .Minify(..) pointer receiver that is part of the Minifier struct type. Following isthe declaration of the pointer receiver

func (o *Minifier) Minify(m *minify.M, w io.Writer, r io.Reader, params map[string]string) error {    ....    isInline := params != nil && params["inline"] == "1"    ....    if err := c.minifyGrammar(); err != nil && err != io.EOF {        return err    }    return nil}

Reference reading:

Mon, 01 Jan 0001 00:00:00 +0000

date: “2019-11-30”title: “Service Mesh Brain Dump (ALWAYS WIP)”author : “Nanik Tolaram (nanikjava@gmail.com)”

Service Mesh

Resources
- Consul whitepaper that explains about service mesh.
A service mesh is a software-driven approach to routing and segmentation. The goal is to solve the networking and security challenges of operating microservices and cloud infrastructure. Service mesh solutions bring additional benefits such as failure handling, retries, and network observability. A service mesh is an abstract concept that solves multiple service networking challenges in an integrated way.
Following diagrams show the simplest diagram of what service mesh is all about

A service mesh has two key components
- Control Plane – The control plane holds the state of the system and plays a coordination role. It provides a centralized registry of where services are running, and the policies that restrict traffic. It must scale to handle tens of thousands of service instances, and efficiently update the data plane in real time.
- Data Plane – The data plane is distributed and responsible for transmission of data between different services. It must be high-performance and integrate with the control plane.
Following are some of the projects that perform the above 2 different planes
- Data planes – Linkerd, NGINX, HAProxy, Envoy, Traefik
- Control planes – Istio, Nelson, SmartStack
Ingress – term used for incoming request
Egress – term used for outgoing request
By default, proxies handle only intra-service mesh cluster traffic - between the source (upstream) and the destination (downstream) services. To expose a service which is part of service mesh to outside word, you have to enable ingress traffic. Similarly, if a service depends on an external service you may require enabling the egress traffic.
Due to distributed nature of service mesh, a control plane or a similar centralized management utility is desirable. This elps in the management of routing tables, service discovery, and load balancing pools.
API Gateway vs Service Mesh
- There is some overlap between API Gateway and Service mesh patterns i.e. request routing, composition, authentication, rate limiting, error handling, monitoring, etc. The primary focus of a service mesh is service-to-service communication (internal traffic), whereas an API Gateway offers a single entry point for external clients such as a mobile app or a web app used by end-user to access services (external traffic).
- Technically one can use API Gateway to facilitate internal service-to-service communication but not without introducing the latency and performance issues. As the composition of services can change over time and should be hidden from external clients - API Gateway makes sure that this complexity is hidden from the external client.
- It is important to note that a service mesh can use a diverse set of protocols (RPC/gRPC/REST), some of which might not be friendly to external clients.
- API Gateway can handle multiple types of protocols and if can support protocol translation if required. There is no reason why one can not use API Gateway in front of a service mesh.
Remember that even thought we are using container orchestration this will not take care of what microservices will need. Container orchestration does not provide what microservice application need that are specificly relevant for the application such as - finding other microservice application, app-to-app communication (gRPC), app-to-app security (session, token, etc). Following table outlined the difference

In service mesh architecture we need resiliency features such as circuit breaking, timeouts, deadlines, etc.
Don’t think service mesh as a technology, think of it as an abstract concept that bring services together and mesh them together as a single entity.

Consul

Following is a high level diagram showing how consul works as a control plane capacity

Istio

Resources
- The explanation in this article is extracted from here
- Indepth article about the traffic flow using envoy as sidecar
- An indepth article on creating your own sidecar proxy for app running in Kubernetes.
- Cloud patterns
- Using kiali.io with Istio presentation
Istio provides automatic injection of sidecar proxy in Kubernetes
Istio control plane consist of - Mixer, Pilot and Citadel.
In simple terms, sidecar injection is adding the configuration of additional containers to the pod template. The added containers needed for the Istio service mesh are:
- istio-init This init container is used to setup the iptables rules so that inbound/outbound traffic will go through the sidecar proxy. An init container is different than an app container in following ways:
  - It runs before an app container is started and it always runs to completion.
  - If there are many init containers, each should complete with success before the next container is started.

So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules. istio-proxy This is the actual sidecar proxy (based on Envoy).* This is done by labelling the namespace where you are deploying the app with istio-injection=enabled

kubectl get namespaces --show-labels

NAME           STATUS    AGE       LABELSdefault        Active    40d       <none>istio-dev      Active    19d       istio-injection=enabledistio-system   Active    24d       <none>kube-public    Active    40d       <none>kube-system    Active    40d       <none>

For automatic sidecar injection, Istio relies on Mutating Admission Webhook. Following is the instruction used to get the webhook info

kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml

apiVersion: admissionregistration.k8s.io/v1beta1kind: MutatingWebhookConfigurationmetadata:  annotations:    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"admissionregistration.k8s.io/v1beta1","kind":"MutatingWebhookConfiguration","metadata":{"annotations":{},"labels":{"app":"istio-sidecar-injector","chart":"sidecarInjectorWebhook-1.0.1","heritage":"Tiller","release":"istio-remote"},"name":"istio-sidecar-injector","namespace":""},"webhooks":[{"clientConfig":{"caBundle":"","service":{"name":"istio-sidecar-injector","namespace":"istio-system","path":"/inject"}},"failurePolicy":"Fail","name":"sidecar-injector.istio.io","namespaceSelector":{"matchLabels":{"istio-injection":"enabled"}},"rules":[{"apiGroups":[""],"apiVersions":["v1"],"operations":["CREATE"],"resources":["pods"]}]}]}  creationTimestamp: 2018-12-10T08:40:15Z  generation: 2  labels:    app: istio-sidecar-injector    chart: sidecarInjectorWebhook-1.0.1    heritage: Tiller    release: istio-remote  name: istio-sidecar-injector  .....webhooks:- clientConfig:    service:      name: istio-sidecar-injector      namespace: istio-system      path: /inject  name: sidecar-injector.istio.io  namespaceSelector:    matchLabels:      istio-injection: enabled  rules:  - apiGroups:    - ""    apiVersions:    - v1    operations:    - CREATE    resources:    - pods

Can see the webhook namespaceSelector label that is matched for sidecar injection with the label istio-injection: enabled.

The ‘app’ that is intercepting this request is as follows

kubectl get svc --namespace=istio-system | grep sidecar-injector

istio-sidecar-injector   ClusterIP   10.102.70.184   <none>        443/TCP             24d

Now that we are clear about how a sidecar container and an init container are injected into an application manifest, how does the sidecar proxy grab the inbound and outbound traffic to and from the container? We did briefly mention that it is done by setting up the iptable rules within the pod namespace, which in turn is done by the istio-init container. Now, it is time to verify what actually gets updated within the namespace.

Let’s get into the application pod namespace we deployed in the previous section and looked at the configured iptables. I am going to show an example using nsenter. Alternatively, you can enter the container in a privileged mode to see the same information. For folks without access to the nodes, using exec to get into the sidecar and running iptables is more practical.

Get the pid of the container that has the sidecar proxy running (normally this is our app container)

docker inspect b8de099d3510 --format '{{ .State.Pid }}'

enter into the container and execute the command

nsenter -t 4215 -n iptables -t nat -S

The output will be as follows

-P PREROUTING ACCEPT-P INPUT ACCEPT-P OUTPUT ACCEPT-P POSTROUTING ACCEPT-N ISTIO_INBOUND-N ISTIO_IN_REDIRECT-N ISTIO_OUTPUT-N ISTIO_REDIRECT-A PREROUTING -p tcp -j ISTIO_INBOUND-A OUTPUT -p tcp -j ISTIO_OUTPUT-A ISTIO_INBOUND -p tcp -m tcp --dport 80 -j ISTIO_IN_REDIRECT-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN-A ISTIO_OUTPUT -j ISTIO_REDIRECT-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

Internal diagrams

Following are resources that gives detail explanation on working with Istio
- https://openliberty.io/guides/istio-intro.html#deploying-version-2-of-the-system-microservice

Kuma

Kuma is an open source service mesh project that provides both control and data plane functionality. The interesting part about this project is the ability to use it for both Kubernetes and non-Kubernetes environment (bare metal and VM).

Following are the steps to run the sample app in kuma:

Run kuma-echo sample app
Run the kuma-cp run

Generate Dataplane data

echo "type: Dataplane                                                         mesh: defaultname: dp-echo-1networking:  inbound:  - interface: 127.0.0.1:10000:9000    tags:      service: echo" | ./kumactl apply -f -

Generate the data token using the following

./kumactl generate dataplane-token --dataplane=dp-echo-1 > /tmp/kuma-dp-echo-1

Run kuma-dp using the generated token as follows

./kuma-dp run   --name=dp-echo-1   --mesh=default   --cp-address=http://127.0.0.1:5681   --dataplane-token-file=/tmp/kuma-dp-echo-1

Debugging through kuma-dp following is the generated configuration used for envoy

dynamicResources:  adsConfig:    apiType: GRPC    grpcServices:    - envoyGrpc:        clusterName: ads_cluster  cdsConfig:    ads: {}  ldsConfig:    ads: {}node:  cluster: echo  id: default.dp-echo-1  metadata:    dataplaneTokenPath: /tmp/kuma-dp-echo-1staticResources:  clusters:  - connectTimeout: 1s    http2ProtocolOptions: {}    loadAssignment:      clusterName: ads_cluster      endpoints:      - lbEndpoints:        - endpoint:            address:              socketAddress:                address: localhost                portValue: 5678    name: ads_cluster    type: STRICT_DNS    upstreamConnectionOptions:      tcpKeepalive: {}  - connectTimeout: 1s    http2ProtocolOptions: {}    loadAssignment:      clusterName: access_log_sink      endpoints:      - lbEndpoints:        - endpoint:            address:              pipe:                path: /tmp/kuma-access-logs-dp-echo-1-default.sock    name: access_log_sink    type: STATIC    upstreamConnectionOptions:      tcpKeepalive: {}

and following is the configuration used to run envoy

 0 = {string} "-c" 1 = {string} "/tmp/kuma-dp-443537827/bootstrap.yaml" 2 = {string} "--drain-time-s" 3 = {string} "30" 4 = {string} "--disable-hot-restart"

tstorage (Time series embedded db)

Table of Contents

Project Info

Example

Internal Document

Go Concurrency Notes

Table of Contents

Restricting Writers

Scenario

Solution

In-Depth with Ollama

Table of Contents

Ollama

Architecture

llama.cpp or llama

ollama or llama.go

Source Layout

Building Source

Running Ollama

Packaging

Ollama Internals

Debugging

Ollama to Llama

Llama Endpoints

Slots

Model

GGUF

Concurrent map in Go

Why are map operations not defined to be atomic?

concurrent-map

ristretto

go-workers

Dissecting minict

pull

run

Hacking

gvisor

Building gvisor

Sentry the ‘Secret Sauce’

Internals

Execution

vDSO

Seccomp

Syscalls

Loading, Executing and Switching app/kernel

ptrace

Log

Other Notes

Resources

gvisor

Seccomp

ptrace

user mode Linux & Linux in general

go-rest-api project

References

Janos Web Utility

Samples

Internals of Container Networking

First look at the internals of containerd and runc

containerd

runc

libcontainer

Docker and lower level

Running container with runc

References

Concurrency Design Pattern in Go

Design Patterns

Pipeline

Tee-Channel

Fan-in-Fan-out channel

Context

Bridge Channel

Error Propagation

Concurrency Project

Blackhole

Deadline

Circuit Breaker

Batcher

Deploying k8guard (Work in progress)

Deploying k8guard