Jekyll2023-06-14T15:08:50+00:00https://mingliang.me/feed.xmlYou Will Never Walk AloneA programmer loves exploring the world!MingliangAdd a new disk to ubuntu in Virtualbox2023-01-12T16:58:53+00:002023-01-12T16:58:53+00:00https://mingliang.me/blog/add-a-new-disk-to-ubuntu-in-virtualbox<p>I have a Ubuntu box running in virutualbox as a local development environment. It’s very handy for exploring some new tools or working on some side projects. It has been working well for quite a long time until I recently noticed that the disk space is almost used up. So I decided to add one extra disk to it. That sounds really easy with VirtualBox, righ?</p>
<p>This is what I did:</p>
<ul>
<li>Shutdown the vm</li>
<li>Create a new disk in VirtualBox Manager and assign it to the Ubuntu vm.</li>
<li>Start the VM, but surprisingly that the newly added disk didn’t show up when I run <code class="language-plaintext highlighter-rouge">df -lh</code></li>
</ul>
<p>After some google search, I realized that I need to format the disk and mount it to Ubuntu file systems, so here is what needed:</p>
<ul>
<li>run <code class="language-plaintext highlighter-rouge">sudo fdisk -l</code>, to list all available disks, and the newly added disk should be in it. In my case, it’s <code class="language-plaintext highlighter-rouge">/dev/sdb</code></li>
<li>run <code class="language-plaintext highlighter-rouge">sudo fdisk /dev/sdb</code>, and follow the prompts to format the disk</li>
<li>run <code class="language-plaintext highlighter-rouge">sudo mkfs.ext4 /dev/sdb1</code> to format the disk to <code class="language-plaintext highlighter-rouge">ext4</code></li>
<li>run <code class="language-plaintext highlighter-rouge">sudo mkdir /disk2</code> as the mount point</li>
<li>edit <code class="language-plaintext highlighter-rouge">/etc/fstab</code> and add a line like: <code class="language-plaintext highlighter-rouge">dev/sdb1 /disk2 ext4 defaults 1 2</code></li>
<li>restart and the disk should be available to use.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li><a href="https://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html">Vmware Linux Guest Add a New Hard Disk Without Rebooting Guest</a></li>
<li><a href="https://kb.vmware.com/s/article/1003940">Creating a new virtual disk for an existing Linux virtual machine</a></li>
</ul>MingliangI have a Ubuntu box running in virutualbox as a local development environment. It’s very handy for exploring some new tools or working on some side projects. It has been working well for quite a long time until I recently noticed that the disk space is almost used up. So I decided to add one extra disk to it. That sounds really easy with VirtualBox, righ?GCP CLI Cheatsheet2022-11-12T18:09:02+00:002022-11-12T18:09:02+00:00https://mingliang.me/blog/gcp-cli-cheatsheet<h2 id="retrieve-project-owner-info">Retrieve project owner info</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># gcloud projects list - list projects accessible by the active account</span>
<span class="c"># Lists all active projects, where the active account has Owner, Editor,</span>
<span class="c"># Browser or Viewer permissions. Projects are listed in alphabetical order by</span>
<span class="c"># project name. Projects that have been deleted or are pending deletion are</span>
<span class="c"># not included.</span>
gcloud projects list <span class="nt">--sort-by</span><span class="o">=</span>projectId <span class="nt">--limit</span><span class="o">=</span>5
gcloud projects list <span class="nt">--filter</span><span class="o">=</span><span class="s1">'lifecycleState:DELETE_REQUESTED'</span>
<span class="c"># retrieve project owners</span>
gcloud projects get-iam-policy <span class="nv">$PROJECT_ID</span> <span class="nt">--flatten</span><span class="o">=</span>bindings <span class="nt">--filter</span><span class="o">=</span>bindings.role:roles/owner <span class="nt">--format</span><span class="o">=</span><span class="s1">'value(bindings.members)'</span>
<span class="c"># retrieve project creator</span>
gcloud logging <span class="nb">read</span> <span class="nt">--project</span> <span class="nv">$PROJECT_ID</span> <span class="nt">--order</span><span class="o">=</span>asc <span class="nt">--limit</span><span class="o">=</span>1 <span class="nt">--format</span><span class="o">=</span><span class="s1">'table(protoPayload.methodName, protoPayload.authenticationInfo.principalEmail)'</span>
</code></pre></div></div>MingliangRetrieve project owner infoRun vagrant inside a virtualbox vm2022-01-05T14:40:36+00:002022-01-05T14:40:36+00:00https://mingliang.me/blog/run-vagrant-inside-a-virtualbox-vm<p>It might be a question why do you need to run vagrant inside a vm. The reason I am trying to do this is because I have a home server which runs on Windows (currently on Windows 11). It’s a powerful server which has 64GB memory, so I decided to run some VMs on it, to play with some stuffs easily.</p>
<p>Basically what needs to be done is:</p>
<ul>
<li>Disable <code class="language-plaintext highlighter-rouge">hyper-v</code> in windows features</li>
<li>Enable VT-x in the VM settings (virtualbox)
<ul>
<li>In the VM settings, navigate to “System -> Processor” and toggle the “Enable Nested VT-x/AMD-V” check box.</li>
</ul>
</li>
</ul>MingliangIt might be a question why do you need to run vagrant inside a vm. The reason I am trying to do this is because I have a home server which runs on Windows (currently on Windows 11). It’s a powerful server which has 64GB memory, so I decided to run some VMs on it, to play with some stuffs easily.How Ansible run playbooks across mulitple hosts2021-12-10T03:18:33+00:002021-12-10T03:18:33+00:00https://mingliang.me/blog/how-ansible-run-playbooks-across-mulitple-hosts<p>This has been a question in my mind for a while, and it comes back recently because I have to run some slow playbooks in multiple hosts and I wish I could run them in parallel to speed up the whole process.</p>
<p>Here is what I find in Ansible doc - <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html">Controlling playbook execution: strategies and more</a></p>
<blockquote>
<p>By default, Ansible runs each task on all hosts affected by a play before starting the next task on any host, using 5 forks. If you want to change this default behavior, you can use a different strategy plugin, change the number of forks, or apply one of several keywords like serial.</p>
</blockquote>
<p>So basically if nothing is specified, Ansible will run each tasks in all hosts and will start the next task after the previous one finishes in all the hosts. And this is good enough in most cases.</p>
<h2 id="how-to-run-playbook-one-host-at-a-time">How to run playbook one host at a time?</h2>
<p>Sometimes you might want to one the playbook in one host at a time. This typically happens when you want to do a rolling update, where the update may take down the host for a while, which you don’t want to have multiple nodes down at the same time. To achieve this, you can use <code class="language-plaintext highlighter-rouge">serial: 1</code>, which will run one host a time. Or you can change to a comfortable number to run in parallel.</p>
<h2 id="difference-between-serial-and-forks">Difference between <code class="language-plaintext highlighter-rouge">serial</code> and <code class="language-plaintext highlighter-rouge">forks</code></h2>
<p>Basically <code class="language-plaintext highlighter-rouge">serial</code> controls the concurrency at the playbook level, while <code class="language-plaintext highlighter-rouge">forks</code> controls at task level. i.e. by default, ansible will run <code class="language-plaintext highlighter-rouge">5</code> concurrent tasks at the same time, and move to the next task once the 5 are done. However, if <code class="language-plaintext highlighter-rouge">serial: 2</code> is used, ansible will run 2 hosts for the playbook at the same time. With it, even <code class="language-plaintext highlighter-rouge">forks: 5</code> is configured, only two tasks will run.</p>
<p>If you want to go further, go through the article to see if other strategies work for you.</p>MingliangThis has been a question in my mind for a while, and it comes back recently because I have to run some slow playbooks in multiple hosts and I wish I could run them in parallel to speed up the whole process.Plex server file permission issue2021-12-08T20:01:50+00:002021-12-08T20:01:50+00:00https://mingliang.me/blog/home/plex-server-file-permission-issue<p>I have a Synology NAS at home and I have setup a plex server to serve some of the my media collections. It has been working pretty well in the past. However, I haven’t used it much since I have other online media services. And after I have upgraded the Synology OS to DSM7.0 for a while, I noticed that the plex media server started having issues to access files in some of the folders. After spending some time to dig out what has caused it. Here is my findings:</p>
<p>For plex, it has two user/group that controls the access to the files. One is the regular <code class="language-plaintext highlighter-rouge">plex</code> user which is more visible in the permission settings, which you usually won’t miss. And the other one is called <code class="language-plaintext highlighter-rouge">System internal user</code>, and the user name is <code class="language-plaintext highlighter-rouge">PlexMediaServer</code>. To access it, go to <code class="language-plaintext highlighter-rouge">Control panel -> Shared Folder -> Edit -> Permissions -> System Internal user (in the drop down) -> PlexMediaServer</code>. In order to have Plex work properly for the media folder, make sure both users are granted <code class="language-plaintext highlighter-rouge">Ready Only</code> access.</p>MingliangI have a Synology NAS at home and I have setup a plex server to serve some of the my media collections. It has been working pretty well in the past. However, I haven’t used it much since I have other online media services. And after I have upgraded the Synology OS to DSM7.0 for a while, I noticed that the plex media server started having issues to access files in some of the folders. After spending some time to dig out what has caused it. Here is my findings:Troubleshoot Java Runtime Errors2021-07-26T21:33:11+00:002021-07-26T21:33:11+00:00https://mingliang.me/blog/troubleshoot-java-runtime-errors<h2 id="javalangnosuchmethoderror"><code class="language-plaintext highlighter-rouge">java.lang.NoSuchMethodError</code></h2>
<p>This is a very common error when the runtime class doesn’t match with the complied time classes. It is usually caused by mismatched runtime dependencies. If it’s a maven/gradle based project, you can figure it out by building the dependency tree to understand which version is actually pulled in and used in runtime.</p>
<p>For gralde, the command is - <code class="language-plaintext highlighter-rouge">./gradlew dependencies</code></p>
<p>However, the dependency tree approach only works if all the dependencies are managed correctly, i.e. all external dependencies are declared via maven or gradle. If there is a bad library which encloses its dependent into its own jar, it will make the troubleshooting much harder. This is something really bad and every library should avoid doing it.</p>
<p>If you are sure everything looks good in the dependency tree but you still see runtime errors, you should check if any classes have multiple versions and could have caused the problem. To check it, you can do:</p>
<h3 id="using-the-verbose-parameter-to-show-the-detail-information-about-where-a-class-is-loaded">Using the <code class="language-plaintext highlighter-rouge">verbose</code> parameter to show the detail information about where a class is loaded:</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-verbose</span>:class <other args>
</code></pre></div></div>
<p>The output is like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>2.209s][info][class,load] org.springframework.security.authentication.AuthenticationTrustResolverImpl <span class="nb">source</span>: jar:file:/home/gary/java/spring/spring-boot-playground/build/libs/spring-boot-playground-0.0.1.jar!/BOOT-INF/lib/spring-security-core-5.2.8.RELEASE.jar!/
<span class="o">[</span>2.210s][info][class,load] org.springframework.security.core.CredentialsContainer <span class="nb">source</span>: jar:file:/home/gary/java/spring/spring-boot-playground/build/libs/spring-boot-playground-0.0.1.jar!/BOOT-INF/lib/spring-security-core-5.2.8.RELEASE.jar!/
</code></pre></div></div>
<h2 id="using-the-ide-feature-to-find-the-offending-classes">Using the IDE feature to find the offending classes</h2>
<p>Intellij does a great job helping you locate the class files in the project, if there are multiple versions loaded, you can see them in the search (using <code class="language-plaintext highlighter-rouge">Shift</code>+<code class="language-plaintext highlighter-rouge">Shift</code> to show the global search dialog.)</p>
<h2 id="how-to-read-the-nosuchmethoderror-error-message">How to read the <code class="language-plaintext highlighter-rouge">NoSuchMethodError</code> error message</h2>
<p>Here is a sample error message:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java.lang.NoSuchMethodError:
com.sun.tools.doclets.formats.html.SubWriterHolderWriter.printDocLinkForMenu(
ILcom/sun/javadoc/ClassDoc;Lcom/sun/javadoc/MemberDoc;
Ljava/lang/String;Z)Ljava/lang/String;
at com.sun.tools.doclets.formats.html.AbstractExecutableMemberWriter.writeSummaryLink(
AbstractExecutableMemberWriter.java:94)
</code></pre></div></div>
<p>It could be quite confusing to figure out what does <code class="language-plaintext highlighter-rouge">ILcom/sun/javadoc/ClassDoc;Lcom/sun/javadoc/MemberDoc;Ljava/lang/String;Z)</code> mean. To understand it, we need to understand how JVM represents the type of class, instance or variables. According to <a href="https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.3.2">JVM: Field Descriptors</a>, here is the table which shows the FiledType characters:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Character Type Interpretation
------------------------------------------
B byte signed byte
C char Unicode character
D double double-precision floating-point value
F float single-precision floating-point value
I int integer
J long long integer
L<classname>; reference an instance of class
S short signed short
Z boolean true or false
[ reference one array dimension
</code></pre></div></div>
<p>Based on the table, we can translate the parameters in the error message above to:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">I</code> -> <code class="language-plaintext highlighter-rouge">int</code></li>
<li><code class="language-plaintext highlighter-rouge">Lcom/sun/javadoc/ClassDoc;</code> -> <code class="language-plaintext highlighter-rouge">com.sun.javadoc.ClassDoc</code></li>
<li><code class="language-plaintext highlighter-rouge">Lcom/sun/javadoc/MemberDoc;</code> -> <code class="language-plaintext highlighter-rouge">com.sun.javadoc.MemberDoc</code></li>
<li><code class="language-plaintext highlighter-rouge">Ljava/lang/String;</code> -> <code class="language-plaintext highlighter-rouge">java.lang.String</code></li>
<li><code class="language-plaintext highlighter-rouge">Z</code> -> <code class="language-plaintext highlighter-rouge">boolean</code></li>
</ul>
<p>Or: <code class="language-plaintext highlighter-rouge">printDocLinkForMenu(int, ClassDoc, MemberDoc, String, boolean)</code></p>
<h2 id="reference">Reference:</h2>
<ul>
<li><a href="https://docs.oracle.com/en/java/javase/11/tools/java.html#GUID-3B1CE181-CD30-4178-9602-230B800D4FAE">Java command options</a></li>
<li><a href="https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.3.2">JVM Field Descriptors</a></li>
<li><a href="https://stackoverflow.com/a/2945933/958071">StackOverflow Answer - Interpreting NoSuchMethodError message</a></li>
</ul>Mingliangjava.lang.NoSuchMethodErrorMemo for Avro Schema2021-07-08T18:55:42+00:002021-07-08T18:55:42+00:00https://mingliang.me/blog/memo-for-avro-schema<ul>
<li><a href="https://avro.apache.org/docs/1.4.0/spec.html">Avro 1.4.0 Specification</a></li>
</ul>
<h2 id="avro-schema-constraints">Avro schema constraints</h2>
<h3 id="nullable-and-default-values">Nullable and default values</h3>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">{</span>
<span class="s">"name"</span><span class="o">:</span> <span class="s">"AvroMapEntity"</span><span class="o">,</span>
<span class="s">"fields"</span><span class="o">:</span> <span class="o">[</span>
<span class="o">{</span>
<span class="s">"name"</span><span class="o">:</span> <span class="s">"mapField"</span><span class="o">,</span>
<span class="s">"type"</span><span class="o">:</span> <span class="o">[</span> <span class="s">"null"</span><span class="o">,</span>
<span class="o">{</span>
<span class="s">"type"</span><span class="o">:</span> <span class="s">"map"</span><span class="o">,</span>
<span class="s">"values"</span><span class="o">:</span> <span class="s">"int"</span>
<span class="o">}</span>
<span class="o">],</span>
<span class="s">"default"</span><span class="o">:</span> <span class="kc">null</span>
<span class="o">}</span>
<span class="o">]</span>
<span class="o">}</span>
</code></pre></div></div>MingliangAvro 1.4.0 SpecificationSetup Gitbook on MacOS2020-09-07T17:52:23+00:002020-09-07T17:52:23+00:00https://mingliang.me/blog/setup-gitbook-on-macos<h2 id="install-gitbook-cli">Install gitbook CLI</h2>
<p>Somehow the <a href="https://github.com/GitbookIO/gitbook-cli">gitbook-cli</a> doesn’t support the latest <a href="https://www.npmjs.com">npm</a>. To properly install <a href="https://github.com/GitbookIO/gitbook-cli">gitbook-cli</a>, follow the steps below:</p>
<ol>
<li>Install <a href="https://github.com/nvm-sh/nvm">nvm</a></li>
<li>Use <code class="language-plaintext highlighter-rouge">nvm install 9</code>, which gives <code class="language-plaintext highlighter-rouge">npm</code> version <code class="language-plaintext highlighter-rouge">5.60</code></li>
<li>Install <code class="language-plaintext highlighter-rouge">gitbook-cli</code> using: <code class="language-plaintext highlighter-rouge">npm install -g gitbook-cli</code></li>
<li>Start enjoying <code class="language-plaintext highlighter-rouge">gitbook</code> CLI.
<ul>
<li><code class="language-plaintext highlighter-rouge">gitbook install</code></li>
<li><code class="language-plaintext highlighter-rouge">gitbook build</code></li>
<li><code class="language-plaintext highlighter-rouge">gitbook serve</code></li>
</ul>
</li>
</ol>
<h2 id="use-gitbook">Use Gitbook</h2>
<p>Follow this <a href="https://docs.gitbook.com/integrations/github/content-configuration">Content Configuration</a> to configure a githook repository.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/chusiang/how-to-build-the-gitbook-with-gitbook-cli">How to build gitbook with gitbook-cli</a></li>
</ul>MingliangInstall gitbook CLICreate custom operators for kubernetes2020-08-29T16:22:33+00:002020-08-29T16:22:33+00:00https://mingliang.me/blog/create-custom-operators-for-kubernetes<h2 id="create-custom-resources-for-kubernetes">Create custom resources for kubernetes</h2>
<h2 id="how-to-extend-kubernetes">How to extend kubernetes</h2>
<h3 id="extension-patterns">Extension patterns</h3>
<blockquote>
<p>There is a specific pattern for writing client programs that work well with Kubernetes called the Controller pattern. Controllers typically read an object’s .spec, possibly do things, and then update the object’s .status.
A controller is a client of Kubernetes. When Kubernetes is the client and calls out to a remote service, it is called a Webhook. The remote service is called a Webhook Backend. Like Controllers, Webhooks do add a point of failure.</p>
</blockquote>
<h3 id="the-controller-pattern">The controller pattern</h3>
<blockquote>
<p>The Controller Pattern
Whether you use CRDs or an aggregated API, the best way to implement the behavior of your custom resources is using the controller pattern, the same way Kubernetes controls its own resources. It is often described by three adjectives: declarative, asynchronous and level-based; most of its mechanics are implemented in Go in client-go/tools/cache and client-go/util/workqueue and documented in several blogs.</p>
</blockquote>
<blockquote>
<ol>
<li>The user declares the desired state of an object, e.g. kubectl apply -f mapping.yaml, where mapping.yaml contains the specification (Spec) of a Mapping object. The Kubernetes API responds as soon as the desired state has been stored in etcd, but before the user’s intent has been fulfilled—which is to create a dummy Service object, owned by the Mapping object, annotated according to its Spec.</li>
<li>Asynchronously, the controller watches for CRUD events on the Mapping and Service resources. For each event, the state of the corresponding object is cached, and its key goes to a work queue; the key is the namespace and name of the Mapping object, obtained directly, or via owner references for Service events. This logic is the work of listers and informers.</li>
<li>The work queue is backed by a queue and a set, so that if a key is added multiple times before it is processed, it is only processed once. The processing function gets the latest state of the Mapping and Service objects from cache, updates the Mapping’s Status (e.g. Service not yet created, or outdated annotation) and takes action to reconcile it according to the Spec (e.g. create the dummy Service, or update its annotation; other use cases could place calls to external services here). Thus, the control loop is level-based, because it reconciles the desired and observed states based on their latest observations, not on their historical changes.</li>
</ol>
</blockquote>
<h2 id="kubebuilder-vs-operator-sdk">Kubebuilder v.s. operator sdk</h2>
<p>Refer to: https://github.com/operator-framework/operator-sdk/issues/1758#issuecomment-517432349</p>
<blockquote>
<p>There’s not a huge difference between the Go projects that kubebuilder and operator-sdk scaffold. Both use controller-tools and controller-runtime and both scaffold substantially similar go package structures.</p>
</blockquote>
<blockquote>
<p>Where they differ is:</p>
<ul>
<li>Operator SDK also has support for Ansible and Helm operators, which make it easy to write operators without having to learn Go and if you already have experience with Ansible or Helm</li>
<li>Operator SDK includes integrations with the Operator Lifecycle Manager (OLM), which is a key component of the Operator Framework that is important to Day 2 cluster operations, like managing a live upgrade of your operator.</li>
<li>Operator SDK includes a scorecard subcommand that helps you understand if your operator follows best practices.</li>
<li>Operator SDK includes an e2e testing framework that simplifies testing your operator against an actual cluster.</li>
<li>Kubebuilder includes an envtest package that allows operator developers to run simple tests with a standalone etcd and apiserver.</li>
<li>Kubebuilder scaffolds a Makefile to assist users in operator tasks (build, test, run, code generation, etc.); Operator SDK is currently using built-in subcommands. Each has pros and cons. The SDK team will likely be migrating to a Makefile-based approach in the future.</li>
<li>Kubebuilder uses Kustomize to build deployment manifests; Operator SDK uses static files with placeholders.</li>
<li>Kubebuilder has recently improved its support for admission and CRD conversion webhooks, which has not yet made it into SDK.
The SDK and Kubebuilder teams work closely together, and we’re planning to increase our efforts to help the kubebuilder team maintain controller-tools and controller-runtime so that the entire community has access to the latest features and bug fixes.</li>
</ul>
</blockquote>
<h2 id="references">References</h2>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/">Extending your Kubernetes Cluster</a></li>
<li><a href="https://admiralty.io/blog/kubernetes-custom-resource-controller-and-operator-development-tools/">Kubernetes Custom Resource, Controller and Operator Development Tools</a></li>
<li>Refer to <a href="https://github.com/microsoft/azure-databricks-operator">Azure Databriks operator</a> for a real project implemented using <code class="language-plaintext highlighter-rouge">kubebuilder</code></li>
</ul>MingliangCreate custom resources for kubernetesCheatsheet for golang2020-08-16T17:51:48+00:002020-08-16T17:51:48+00:00https://mingliang.me/blog/cheatsheet-for-golang<h2 id="run-tests">Run tests</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>go <span class="nb">test</span> ./... <span class="c"># run all tests in current directory and all of its sub-directories</span>
go <span class="nb">test</span> ./a/... ./b/... <span class="c"># run tests for some particular directories</span>
go <span class="nb">test</span> ... <span class="c"># run all tests in your $GOPATH</span>
</code></pre></div></div>
<h2 id="specify-default-values-for-json-parsing">Specify default values for json parsing</h2>
<ul>
<li>https://stackoverflow.com/questions/30445479/how-to-specify-default-values-when-parsing-json-in-go</li>
</ul>
<h2 id="tools">Tools</h2>
<ul>
<li><a href="https://www.alexedwards.net/blog/an-overview-of-go-tooling">An Overview of Go’s Tooling</a></li>
</ul>MingliangRun tests