# README ## analytics [convert json perline to panads data frame](/analytics/convert-json-perline-to-panads-data-frame)\ [hierarchical data format](/analytics/hierarchical-data-format)\ [pandas format custom date in data frame](/analytics/pandas-format-custom-date-in-data-frame) ## bash [bash forloop](/bash/bash-forloop)\ [parameter expansion](/bash/parameter-expansion)\ [prompt confirmation in bash](/bash/prompt-confirmation-in-bash) ## db [disabling foreign key when importing dump](/db/disabling-foreign-key-when-importing-dump)\ [postgres add object to jsonb array from the same column](/db/postgres-add-object-to-jsonb-array-from-the-same-column)\ [postgres audit log trigger](/db/postgres-audit-log-trigger)\ [postgres naming trigger](/db/postgres-naming-trigger)\ [postgres rename enum](/db/postgres-rename-enum)\ [postgres reset sequence after import](/db/postgres-reset-sequence-after-import)\ [postgres skip table from restore](/db/postgres-skip-table-from-restore)\ [set a column with value from different table](/db/set-a-column-with-value-from-different-table) ## dgraph [dgraph docker compose whitelist ip](/dgraph/dgraph-docker-compose-whitelist-ip) ## docker [docker compose make sure postgres is ready before starting other service](/docker/docker-compose-make-sure-postgres-is-ready-before-starting-other-service) ## fish [edit last command in editor](/fish/edit-last-command-in-editor) ## git [checkout last branch](/git/checkout-last-branch)\ [different between two dots and three](/git/different-between-two-dots-and-three)\ [force fail commit on master](/git/force-fail-commit-on-master)\ [git finish helper script delete current branch and update master](/git/git-finish-helper-script-delete-current-branch-and-update-master)\ [git mergetool and diff with p4merge](/git/git-mergetool-and-diff-with-p4merge)\ [global gitignore file](/git/global-gitignore-file)\ [intellij as diff and mergetool](/git/intellij-as-diff-and-mergetool)\ [push only current branch](/git/push-only-current-branch)\ [reset email multiple commit](/git/reset-email-multiple-commit)\ [sign commit with pgp](/git/sign-commit-with-pgp)\ [worktree switching branch without stash](/git/worktree-switching-branch-without-stash) ## go [default math.rand.source is thread save while rand.new source is not](/go/default-math.rand.source-is-thread-save-while-rand.new-source-is-not)\ [deploying go to a vpn ubuntu server with github and ssl](/go/deploying-go-to-a-vpn-ubuntu-server-with-github-and-ssl) ## k8s [copy file to pod](/k8s/copy-file-to-pod)\ [getting cpu and memory usage for container](/k8s/getting-cpu-and-memory-usage-for-container)\ [how cert manager and letsencrypt challenge works](/k8s/how-cert-manager-and-letsencrypt-challenge-works)\ [kubernetes resource short name](/k8s/kubernetes-resource-short-name)\ [scale deployment](/k8s/scale-deployment)\ [view secret](/k8s/view-secret) ## linux [boot zfs root filesystem on grub](/linux/boot-zfs-root-filesystem-on-grub)\ [clearing up swap space](/linux/clearing-up-swap-space)\ [connect bluetooth device from cli](/linux/connect-bluetooth-device-from-cli)\ [free and vmstat command](/linux/free-and-vmstat-command)\ [get full argument from a process](/linux/get-full-argument-from-a-process)\ [merge multiple pdf into a single file](/linux/merge-multiple-pdf-into-a-single-file)\ [removing old kernel](/linux/removing-old-kernel)\ [symbolic vs hard link](/linux/symbolic-vs-hard-link)\ [zfs auto snapshot](/linux/zfs-auto-snapshot)\ [zfs external backup drive with snapshot and encryption](/linux/zfs-external-backup-drive-with-snapshot-and-encryption) ## net [dnssec](/net/dnssec)\ [html form submit to different action depending on properties](/net/html-form-submit-to-different-action-depending-on-properties)\ [ldap list users](/net/ldap-list-users) ## osx [checksum a file from a url](/osx/checksum-a-file-from-a-url)\ [cluster ssh in iterm2 with i2cssh](/osx/cluster-ssh-in-iterm2-with-i2cssh)\ [list open port](/osx/list-open-port)\ [manage clipboard easily with jumpcut](/osx/manage-clipboard-easily-with-jumpcut)\ [pipe output to clipboard with pbcopy and pbpaste](/osx/pipe-output-to-clipboard-with-pbcopy-and-pbpaste)\ [show hidden file](/osx/show-hidden-file)\ [sign application with self certificate](/osx/sign-application-with-self-certificate)\ [starting program on startup with login items](/osx/starting-program-on-startup-with-login-items) ## python [count frequency with lambda](/python/count-frequency-with-lambda)\ [dijkstra algorithm shortest path](/python/dijkstra-algorithm-shortest-path)\ [double slash arithmetic operator](/python/double-slash-arithmetic-operator)\ [min and max of dict values](/python/min-and-max-of-dict-values)\ [python3 match case](/python/python3-match-case)\ [reduce and opeator](/python/reduce-and-opeator) ## react [react named export vs default export](/react/react-named-export-vs-default-export)\ [react useeffect](/react/react-useeffect)\ [react.useref](/react/react.useref) ## unix [bulk renaming multiple file](/unix/bulk-renaming-multiple-file)\ [convert pdf to text using ocr](/unix/convert-pdf-to-text-using-ocr)\ [diff output of 2 command](/unix/diff-output-of-2-command)\ [encryption with gpg](/unix/encryption-with-gpg)\ [extend letsencrypt certificate with dns challenge](/unix/extend-letsencrypt-certificate-with-dns-challenge)\ [ffmpeg monitor and restart stream when it hung or stall](/unix/ffmpeg-monitor-and-restart-stream-when-it-hung-or-stall)\ [file size older than x days](/unix/file-size-older-than-x-days)\ [filtering json with jq](/unix/filtering-json-with-jq)\ [find lines that matches on 2 different sorted file](/unix/find-lines-that-matches-on-2-different-sorted-file)\ [find out what is using swap](/unix/find-out-what-is-using-swap)\ [fish environment variables from 1password](/unix/fish-environment-variables-from-1password)\ [fix gpg warning unsafe permissions on homedir](/unix/fix-gpg-warning-unsafe-permissions-on-homedir)\ [formatting or parse json in command line](/unix/formatting-or-parse-json-in-command-line)\ [get all line except n last one](/unix/get-all-line-except-n-last-one)\ [grep print only matched](/unix/grep-print-only-matched)\ [grep using input file as pattern to search other file](/unix/grep-using-input-file-as-pattern-to-search-other-file)\ [jq counting lenght of an array](/unix/jq-counting-lenght-of-an-array)\ [jq extracting properties to arrays from json row line](/unix/jq-extracting-properties-to-arrays-from-json-row-line)\ [keep n recent item in folder](/unix/keep-n-recent-item-in-folder)\ [open last command in the editor with fc](/unix/open-last-command-in-the-editor-with-fc)\ [parsing epoch timestamp to date](/unix/parsing-epoch-timestamp-to-date)\ [pbcopy alternative for copying to clipboard](/unix/pbcopy-alternative-for-copying-to-clipboard)\ [process pipe operator](/unix/process-pipe-operator)\ [record a web stream to youtube](/unix/record-a-web-stream-to-youtube)\ [regex for validating password](/unix/regex-for-validating-password)\ [rename tmux window](/unix/rename-tmux-window)\ [repeat content of text x time](/unix/repeat-content-of-text-x-time)\ [replacing last command and execute it](/unix/replacing-last-command-and-execute-it)\ [reusing last command argument](/unix/reusing-last-command-argument)\ [sending curl post with file](/unix/sending-curl-post-with-file)\ [send slack message from command line](/unix/send-slack-message-from-command-line)\ [sort file inline](/unix/sort-file-inline)\ [specify compression level in tar gzip](/unix/specify-compression-level-in-tar-gzip)\ [zsh ctrl p same behavior as up arrow](/unix/zsh-ctrl-p-same-behavior-as-up-arrow) ## vim [paste yanked text on command buffer](/vim/paste-yanked-text-on-command-buffer) ## web [this article is published to dev to with github action](/web/this-article-is-published-to-dev-to-with-github-action) # analytics # convert json perline to panads data frame Sometimes we have a file that contains a json object per line. For example log file in json format ``` {foo: 1, bar: 2} {foo: 3, bar: 4} ``` if we want to read this in python pandas we need to convert it to ``` [ {foo: 1, bar: 2}, {foo: 3, bar: 4} ] ``` easy way to do this is with `jq --slurp` ``` $ cat file.json | jq --slurp . > one_array.json ``` then you can read it in python pandas (notebook) like this ```python sf = pd.read_json('one_array.json') ``` # hierarchical data format {% embed url="" %} Is a file format that allow you to store tree like file system inside one file. python library to interact with the file: UI viewer # pandas format custom date in data frame say you have a data frame like this ``` [ {foo: 1, created_date:"2016-06-04 03:00:01.727"}, ... ] ``` the timestamp doesn't recognize immediately. To make this field as timestamp and as index we could do ```python # assuming df contains the data frame df['timestamp'] = pd.to_datetime(be.timestamp, format="%Y-%m-%d %H:%M:%S.%f") df.set_index(df['timestamp'], inplace=True) ``` # bash # bash forloop Example of forloop syntax ```bash for i in 1 2 3 4 5 do echo "i=$i" done ``` with range ```bash for i in {1..5} do echo "i=$i" done ``` cutomize increment ```bash for i in {0..10..2} do echo "i=$i" done ``` # parameter expansion **getting only the file name without extension**\ \*\*\*\*`${FILENAME%.*}` **getting only the extension**\ \*\*\*\*`${FILENAME##*.}` **remove first folder (until / )**\ \*\*\*\*`${FILENAME#*/}` **getting filename**\ \*\*\*\*`${PATHNAME##*/}` Much more on # prompt confirmation in bash From ```bash read -p "Are you sure? " -n 1 -r echo # (optional) move to a new line if [[ ! $REPLY =~ ^[Yy]$ ]] then exit 1 fi ``` # db # disabling foreign key when importing dump After dumping the whole database with `pgdump` and try to execute the file, I found a lot of constrain being violated. This could be because the order of insertion does not match with schema foreign key. To fix that, add this following statement to the beginning of the import file ```sql SET session_replication_role = replica; ``` # postgres add object to jsonb array from the same column We want to create a new column where it's a combination for several column as a jsonb object. ```sql ALTER TABLE items ADD new_arrays_col jsonb DEFAULT '[]'; UPDATE items i1 SET new_arrays_col = new_arrays_col || JSON_BUILD_OBJECT( 'col1', col1, 'col1', col2 )::jsonb ``` the `new_arrays || ...` is kind of like appending to exisitng array with an object. `JSON_BUILD_OBJECT` is a postgres function to create a JSON object from the parameters # postgres audit log trigger This triggers will log before and after fields for a table in JSON format. ```sql CREATE TABLE changelog ( id serial NOT NULL, ts timestamp DEFAULT NOW(), table_name text, operation text, new_val json, old_val json ); CREATE INDEX changelog_ts_table_name_index ON changelog (ts DESC, table_name ASC); CREATE OR REPLACE FUNCTION logchange() RETURNS trigger AS $$ BEGIN IF TG_OP = 'INSERT' THEN INSERT INTO changelog (table_name, operation, new_val) VALUES (TG_RELNAME, TG_OP, ROW_TO_JSON(NEW)); RETURN NEW; ELSIF TG_OP = 'UPDATE' THEN INSERT INTO changelog (table_name, operation, new_val, old_val) VALUES (TG_RELNAME, TG_OP, ROW_TO_JSON(NEW), ROW_TO_JSON(OLD)); RETURN NEW; ELSIF TG_OP = 'DELETE' THEN INSERT INTO changelog (table_name, operation, old_val) VALUES (TG_RELNAME, TG_OP, ROW_TO_JSON(OLD)); RETURN OLD; END IF; END; $$ LANGUAGE 'plpgsql' SECURITY DEFINER; CREATE TRIGGER users_changelog_biudt BEFORE INSERT OR UPDATE OR DELETE ON users FOR EACH ROW EXECUTE PROCEDURE logchange(); ``` # postgres naming trigger Taken From Pattern `[TBALE_NAME]_[FUNCITON NAME]_[WHEN][1 or more WHY][t]` When: * b - Before * a - After * i - Instead of WHY: * i - Insert * u - Update * d - Delete * t - Truncate t = 'trigger' eg: `my_table_update_ts_biut` means it's a trigger on my\_table, triggers update\_ts before insert & update Reasoning: Just by looking at the trigger name, you know which table, and when it is triggered example ``` -- function to update updated_at column during insert or update CREATE OR REPLACE FUNCTION update_timestamp() RETURNS trigger AS $$ BEGIN NEW.updated_at := NOW(); RETURN NEW; END; $$ LANGUAGE 'plpgsql'; CREATE TRIGGER users_update_timestamp_biut BEFORE INSERT OR UPDATE ON users FOR EACH ROW EXECUTE PROCEDURE update_timestamp(); ``` # postgres rename enum To rename an enum ``` ALTER TYPE name RENAME TO 'new_enum_name' ``` To rename a value (label) ``` ALTER TYPE name RENAME VALUE 'existing_enum_value' TO 'new_enum_value' ``` # postgres reset sequence after import Sometimes you get error after importing pg\_dump data caused by constrain error ``` ERROR: duplicate key value violates unique constraint "transactions_pkey" (SQLSTATE 23505) ``` This can be caused by sequence that is not updated. Some symptoms include that if you retry multiple time, it works. That is due to every error increments the sequence. To reset the value to the last id: (replace the `table_name` with your table) ```sql ERROR: duplicate key value violates unique constraint "transactions_pkey" (SQLSTATE 23505) ``` # postgres skip table from restore Restore done in 3 steps: 1. Generate task list ``` pg_restore -l dump-path/ > task ``` 2. Edit the generated `task file` and remove the step that we wanted to ignore. Example, if we want to skip the restoring `sometable`, we remove this line from the file. ``` 4749; 0 106924 TABLE DATA public sometable someuser ``` 3. Restore using the task file ``` pg_restore -d somedatabase -L ./task dump-path/ ``` # set a column with value from different table Say that you have Table A: * username * email And Table B: * item\_id * username You want to add email to `table B` where the value taken from `table A`. After adding a column on Table B, this query can be used: ```sql UPDATE tableB SET email = tableA.Email FROM tableA WHERE tableB.username = tableA.username AND tableB.email IS DISTINCT FROM tableA.email; -- optional, see below ``` The final `WHERE` clause prevents updates that wouldn't change anything - which is practically always a good idea (almost full cost but no gain, exotic exceptions apply). If both old and new value are guaranteed to be `NOT NULL`, simplify to: ``` AND tableB.email <> tableA.email ``` Taken from: # dgraph # dgraph docker compose whitelist ip When using file. I found an error ``` curl -X POST localhost:8080/admin/schema --data-binary '@schema.graphql' {"errors":[{"message":"resolving updateGQLSchema failed because unauthorized ip address: 172.23.0.1 (Locations: [{Line: 3, Column: 4}])","extensions":{"code":"Error"}}]}⏎ ``` Apparently I need to whitelist my host ip address. `--whitelist` on alpha. ``` dgraph alpha --my=alpha:7080 --zero=zero:5080 --whitelist="0.0.0.0/0" ``` This is the full docker-compose file ``` # This Docker Compose file can be used to quickly bootup Dgraph Zero # and Alpha in different Docker containers. # It mounts /tmp/data on the host machine to /dgraph within the # container. You can change /tmp/data to a more appropriate location. # Run `docker-compose up` to start Dgraph. version: "3.2" services: zero: image: dgraph/dgraph:latest volumes: - /tmp/data:/dgraph ports: - 5080:5080 - 6080:6080 restart: on-failure command: dgraph zero --my=zero:5080 alpha: image: dgraph/dgraph:latest volumes: - /tmp/data:/dgraph ports: - 8080:8080 - 9080:9080 restart: on-failure command: dgraph alpha --my=alpha:7080 --zero=zero:5080 --whitelist="0.0.0.0/0" ratel: image: dgraph/dgraph:latest ports: - 8000:8000 command: dgraph-ratel ``` # docker # docker compose make sure postgres is ready before starting other service Sometimes we want to start a service in docker compose after some dependency is ready. In this case, we want to wait for PostgreSQL. The key here is `healthcheck` on the PostgreSQL ```yaml healthcheck: test: [ "CMD-SHELL", "pg_isready -U postgres" ] interval: 10s timeout: 5s retries: 5 ``` and `depends_on` on the service ```yaml depends_on: postgres: condition: service_healthy ``` Full docker-compose.yaml ```yaml version: '3.8' services: postgres: image: postgres:11.6-alpine environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: 12345 POSTGRES_DB: todo ports: - 5432:5432 healthcheck: test: [ "CMD-SHELL", "pg_isready -U postgres" ] interval: 10s timeout: 5s retries: 5 backend: build: backend environment: DB_HOST: postgres ports: - 5000:5000 depends_on: postgres: condition: service_healthy volumes: postgres_volume: ``` # fish # edit last command in editor Sometimes it's easier to edit your last command in text editor (e.g. vim) than fixing the command line in cli. while bash has fc, `fish` has built in commands. go to your previous command with up arrow (or ctrl-p), and press alt-e make sure `$EDITOR` environment is set. # git # checkout last branch Sometimes you are checking out mutltiple different git branches and want to navigate back to previous branch. You can use `git checkout -` to do it example : ```bash $ git checkout ay/convert-errors-lib Updating files: 100% (2480/2480), done. Switched to branch 'ay/convert-errors-lib' $ git checkout - Updating files: 100% (2480/2480), done. Switched to branch 'master' Your branch is up to date with 'origin/master'. ``` If you want to checkout 2 branches a go. you can do ```bash $ git checkout @{-2} ``` Basically `git checkout -` is equql to `git checkout @{-1}` If you want to list all of the checked out branch (history) you can use this script ```bash $ for i in {1..10}; do echo -n "[$i] "; git rev-parse --symbolic-full-name @{-$i}; done ``` I added this as a git command on my zsh. have a look at [this commit](https://github.com/yulrizka/oh-my-zsh/commit/2b730e3bf2777f83e6a215da4a03e51e67145ff3) # different between two dots and three In git the range notation behave differently between `log` and `diff` ``` A---B---C topic / D---E---F---G master ``` ## Git log ### two dots (..) `git log topic..master` : in *master* but not in *topic* (G, F) `git log master..topic` : in *topic* but not *master* (C, B, A) ### three dots (...) `git log topic...master` : in *master* and in *topic* but \*\*not both \*\* (G, F, C, B, A) `git log master...topic` : the same (G, F, C, B, A) ## Git diff ### two dots (..) `git diff topic..master` : what **is** and what **not** in *master* compared to *topic* (D, E, -A, -B, -C, +F, +G) `git diff master..topic` : what **is** and what **not** in *topic* compared to *master* (D, E, -F, -G, +A, +B, +C) ### three dots (...) `git diff topic...master` : in *master* but **not in \_topic**\_\*\* \*\* (D, E, +F, +G) `git diff master...topic` : in *topic* but **not in \_master**\_\*\* \*\* (D, E, +A, +B, +C) # force fail commit on master Most of the time, you want to fail if you mistakenly commit to master. This is if you normally work with feature branch. This script can make git commit fail if the branch is master add to `.git/hooks/pre-commit` ``` current_branch=$(git symbolic-ref HEAD | sed -e 's,.*/\(.*\),\1,') if [ "$current_branch" == "master" ] then echo "Should not commit on master branch. Use 'git commit -n' to override" exit 1 fi ``` or you can check out [pre-commit](https://github.com/pre-commit/pre-commit) a framework to manage your git commit scripts # git finish helper script delete current branch and update master When I finished working on feature branch. I can do ```bash (BRANCH) $ git finish cleaning up BRANCH Sure (y/n) ? y Switched to branch 'master' Your branch is up to date with 'origin/master'. Deleted branch BRANCH (was 5a868ef). Pruning origin ... Successfully rebased and updated refs/heads/master. (master) $ ``` Put this script to `/usr/local/bin/git-finish` ```bash #!/bin/bash branch_name=$(git branch | grep \* | cut -d ' ' -f2) echo "cleaning up $branch_name" echo -n "Sure (y/n) ? " read input < /dev/tty if [ "$input" == "y" ]; then git checkout master git branch -D $branch_name git remote prune origin git fetch origin git pull --rebase --autostash fi ``` # git mergetool and diff with p4merge Some time you want visually merge conflict in git. To use p4merge ## git mergetool ### Linux ``` $ git config --global merge.tool p4mergetool $ git config --global mergetool.p4mergetool.cmd \ $ "/opt/p4v/bin/p4merge \$PWD/\$BASE \$PWD/\$REMOTE \$PWD/\$LOCAL \$PWD/\$MERGED" $ git config --global mergetool.p4mergetool.trustExitCode false $ git config --global mergetool.keepBackup false ``` ### Mac ``` $ git config --global merge.tool p4mergetool $ git config --global mergetool.p4mergetool.cmd \ "/Applications/p4merge.app/Contents/Resources/launchp4merge \$PWD/\$BASE \$PWD/\$REMOTE \$PWD/\$LOCAL \$PWD/\$MERGED" $ git config --global mergetool.p4mergetool.trustExitCode false $ git config --global mergetool.keepBackup false ``` ## Set as diff tool ### Linux $ git config --global merge.tool p4mergetool $ git config --global mergetool.p4mergetool.cmd "/opt/p4v/bin/p4merge $LOCAL $REMOTE" ### Mac $ git config --global diff.tool p4mergetool $ git config --global difftool.p4mergetool.cmd "/Applications/p4merge.app/Contents/Resources/launchp4merge $LOCAL $REMOTE" # global gitignore file To create a global git ignore that apply across all git repository ``` $ echo `.DS_Store` > ~/.gitignore $ git config --global core.excludesfile ~/.gitignore ``` # intellij as diff and mergetool ## Setup idea script First, you need to set up `idea` script by following the instruction [here](https://www.jetbrains.com/help/idea/working-with-the-ide-features-from-command-line.html#toolbox) If you are using mac, make sure you add `-W` on the open script as follows (see [this issue](https://youtrack.jetbrains.com/issue/IDEA-253240)) ``` open -Wna "IntelliJ IDEA.app" --args "$@" ``` ## setup git difftool & mergetool Taken from open `.git/config` (or your global `gitconfig` file) ```bash [merge] tool = intellij [mergetool "intellij"] cmd = idea merge $(cd $(dirname "$LOCAL") && pwd)/$(basename "$LOCAL") $(cd $(dirname "$REMOTE") && pwd)/$(basename "$REMOTE") $(cd $(dirname "$BASE") && pwd)/$(basename "$BASE") $(cd $(dirname "$MERGED") && pwd)/$(basename "$MERGED") trustExitCode = true [diff] tool = intellij [difftool "intellij"] cmd = idea diff $(cd $(dirname "$LOCAL") && pwd)/$(basename "$LOCAL") $(cd $(dirname "$REMOTE") && pwd)/$(basename "$REMOTE") ``` Explanation about LOCAL, REMOTE, BASE from [stackoverflow](https://stackoverflow.com/questions/20381677/in-a-git-merge-conflict-what-are-the-backup-base-local-and-remote-files-that) * foo.LOCAL: the "ours" side of the conflict - ie, your branch (HEAD) that will contain the results of the merge * foo.REMOTE: the "theirs" side of the conflict - the branch you are merging into HEAD * foo.BASE: the common ancestor. useful for feeding into a three-way merge tool * foo.BACKUP: the contents of file before invoking the merge tool, will be kept on the filesystem if mergetool.keepBackup = true. `idea merge` [syntax](https://www.jetbrains.com/help/idea/command-line-merge-tool.html) ``` idea merge [] ``` # push only current branch Git has multiple strategy whey you do `git push` Before git 2.0, it was `matching` meaning that it will push all the local branch that has the same name as remote. I don't like this because most of the time i just want to push my current branch and avoid mistakenly local branches that is not ready to be push yet. After 2.0 it changes to `simple` it will push the current branch to upstream with added safety so that it will refuse to push if the upstream branch name is different from the local one. This is ok but when you created a new branch, it will ask you to explicitly set the upstream branch first. My workflow is my local branch is going to be always the same name as the remote, and if not exist it should be created automatically. this is why i like the `current` strategy. To change this use ``` git config --global push.default current ``` from the git documentation push.default Defines the action git push should take if no refspec is given (whether from the command-line, config, or elsewhere). Different values are well-suited for specific workflows; for instance, in a purely central workflow (i.e. the fetch source is equal to the push destination), upstream is probably what you want. Possible values are: * **nothing** - do not push anything (error out) unless a refspec is given. This is primarily meant for people who want to avoid mistakes by always being explicit. * **current** - push the current branch to update a branch with the same name on the receiving end. Works in both central and non-central workflows. * **upstream** - push the current branch back to the branch whose changes are usually integrated into the current branch (which is called @{upstream}). This mode only makes sense if you are pushing to the same repository you would normally pull from (i.e. central workflow). * **tracking** - This is a deprecated synonym for upstream. * **simple** - in centralized workflow, work like upstream with an added safety to refuse to push if the upstream branch’s name is different from the local one. When pushing to a remote that is different from the remote you normally pull from, work as current. This is the safest option and is suited for beginners. This mode has become the default in Git 2.0. * **matching** - push all branches having the same name on both ends. This makes the repository you are pushing to remember the set of branches that will be pushed out (e.g. if you always push maint and master there and no other branches, the repository you push to will have these two branches, and your local maint and master will be pushed there). To use this mode effectively, you have to make sure all the branches you would push out are ready to be pushed out before running git push, as the whole point of this mode is to allow you to push all of the branches in one go. If you usually finish work on only one branch and push out the result, while other branches are unfinished, this mode is not for you. Also this mode is not suitable for pushing into a shared central repository, as other people may add new branches there, or update the tip of existing branches outside your control. This used to be the default, but not since Git 2.0 (simple is the new default). # reset email multiple commit I started not to use my real email for git commit. Instead i uses the one that is provided by [github](https://help.github.com/articles/about-commit-email-addresses/) After enabling **Block command line pushes that expose my email** on [Block command line pushes that expose my email](https://github.com/settings/emails), it will reject all email that exposes public email adress. So in order to rewrite git history and change the author email, I found this [github help](https://help.github.com/articles/changing-author-info/) ```bash #!/bin/sh git filter-branch --env-filter ' OLD_EMAIL="your-old-email@example.com" CORRECT_NAME="Your Correct Name" CORRECT_EMAIL="your-correct-email@example.com" if [ "$GIT_COMMITTER_EMAIL" = "$OLD_EMAIL" ] then export GIT_COMMITTER_NAME="$CORRECT_NAME" export GIT_COMMITTER_EMAIL="$CORRECT_EMAIL" fi if [ "$GIT_AUTHOR_EMAIL" = "$OLD_EMAIL" ] then export GIT_AUTHOR_NAME="$CORRECT_NAME" export GIT_AUTHOR_EMAIL="$CORRECT_EMAIL" fi ' --tag-name-filter cat -- --branches --tags ``` Another neat way (after you change your email with git config), you can also do this: ```bash $ git rebase -i -p --exec 'git commit --amend --reset-author --no-edit' ``` # sign commit with pgp We can sign our commit with PGP to get verification that it was committed by us. On github we would also get a green "Verified" check mark in our commit. Make sure you setup A PGP key on your local machine. See [this article](/unix/encryption-with-gpg) to set it up Make sure you have a key that matched the email that you used for the commit. (eg: `username@users.github.com` add `-S` flag when you want to sign the commit ``` $ git commit -S -m "your commit message" # Creates a signed commit ``` or add enable it by default `Tell git which key to use` ``` git config --global user.signingkey ABC123 ``` ``` git config --global commit.gpgsign true ``` If you are having issue, `GIT_TRACE=1` with your git comment ``` GIT_TRACE=1 git commit -S -m "your commit message" ``` # worktree switching branch without stash Some times you are working on some feature but then you need to check or fix something on some other branch. The normal way to do it is that we do: 1. git stash 2. switch to other branch 3. commit changes 4. swith to previous branch 5. git stash pop This can be complicated especially you have multiple stash. You can't easily understand which stash belongs where. After version 2.5.2 there is new command called `git worktree` Basically it can create new directory which checkout different branch example ``` $ git worktree add ../branch-temp some-branch $ cd ../branch-temp // hack // finish $ cd ../original-path $ rm -rf ../branch-temp // cleanup $ git worktree prune ``` more detail about the command are available at # go # default math.rand.source is thread save while rand.new source is not ## Default math rand.Source is thread save while rand.New is not Default pacakge rand uses Source that is thread safe with default seed `1` we can use the pacakge's method to have thread safe random number generation ```go package main import ( "fmt" "math/rand" "time" ) func main() { seed := time.Now().UnixNano() rand.Seed(seed) fmt.Println(rand.Int63()) } ``` But the `rand.NewSource()` offers thread unsafe implementation. One of the reason to choose the unsafe implementation to avoid synchronization, especially when you have only single go routine. ```go package main import ( "fmt" "math/rand" "time" ) func main() { seed := time.Now().UnixNano() r := rand.New(rand.NewSource(seed)) fmt.Println(r.Int63()) } ``` # deploying go to a vpn ubuntu server with github and ssl Deploy a go application to a small server with SSL support and Github as CI/CD Assumptions: * root repository contain `` main.go with the package name `aocweb` `` * we are using small VPS with ubuntu 20.04 * app will be deployed to `/home/web/aocweb` * App runs HTTP on PORT 8080 ### Setup user and home some VPS have root as default user, if not already we can create a user. In this example `web` ``` adduser web usermod -aG sudo web ``` ### Install NGINX & Letsencrypt ``` sudo apt-get update sudo apt-get install nginx sudo ufw allow 'Nginx HTTP' sudo apt-get install certbot python3-certbot-nginx sudo certbot --nginx ``` Generate certificate ``` sudo certbot --nginx certonly ``` to test auto renewal ``` sudo certbot renew --dry-run ``` Add `/etc/nginx/sites-enabled/`example`.com` ``` server { listen 80; server_name example.com; return 301 https://$server_name$request_uri; } server { server_name example.com; listen 443 ssl; ssl on; ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; root /var/www/html; location / { proxy_pass http://127.0.0.1:8080; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } location /ws { proxy_buffering off; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr;proxy_pass http://127.0.0.1:8080/ws; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } } ``` ### Install systemd service if the application requires environment we can put it on `/home/web/aocweb/.env` ``` ENV=prod CLIENT_ID=abcdef ``` create `/etc/systemd/system/aocweb.service` ``` [Unit] Description= AocWeb app After=network.target After=mysql.service [Service] User=web Group=www-data EnvironmentFile=/home/web/aocweb/.env WorkingDirectory=/home/web/aocweb ExecStart=/home/web/aocweb/aocweb web [Install] WantedBy=multi-user.target ``` ``` sudo systemctl daemon-reload ``` To make sure we can run systemctl restart with sudo without password. Create sudo configuration file (make sure to use `visudo` so you don't accidentally locked your self. ``` visudo -f /etc/sudoers.d/systemctl ``` With content ``` web ALL=NOPASSWD: /usr/bin/systemctl restart aocweb web ALL=NOPASSWD: /usr/bin/systemctl status aocweb ``` ### Deploy with Github Action #### Generate SSH key for deployment ``` ssh-keygen -t ed25519 -C "user@email.com" ``` example with that command I created 2 files on `~/.ssh/` : `aocweb` & `aocweb.pub` #### add generated public key to server Copy the content of `aocweb.pub` to `/home/web/.ssh/authorized_key.` This allows login with ssh private key. Test that you can login to the server with that key (check `ssh -v` output) #### Add secrets {% embed url="" %} create these secrets on the github repository * HOST * KEY * USERNAME `KEY` is the content of `aocweb.pub` ssh key #### Github workflow create `.github/workflows/release.yaml` ``` name: Release on: push: branches: - release jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: install go uses: actions/setup-go@v2 with: go-version: '1.17.3' - name: build run: go build . - name: send binary uses: appleboy/scp-action@master with: host: ${{ secrets.HOST }} username: ${{ secrets.USERNAME }} key: ${{ secrets.KEY }} source: "aocweb" target: "/home/web/aocweb" - name: restart uses: appleboy/ssh-action@master with: host: ${{ secrets.HOST }} username: ${{ secrets.USERNAME }} key: ${{ secrets.KEY }} script: sudo systemctl restart aocweb && sudo systemctl status aocweb ``` # k8s # copy file to pod Copy File from local to remote ``` kubectl cp ./local-file pod-name:remote-file ``` Copy file between pod ``` kubectl cp pod1:file-name pod2:file-name ``` # getting cpu and memory usage for container ``` $ kubectl top po --containers POD NAME CPU(cores) MEMORY(bytes) 1-cpu-pod ubuntu 2m 73Mi ... ``` # how cert manager and letsencrypt challenge works If one creates an ingress will TLS and cert-manager is available on the cluster, it will try to generate an SSL certificate. In order for letsencrypt server (ACME server) verify that we actually control the domain, they will verify using some kind of token challenges There are 2 type of the resolver ### DNS resolver ACME server will check that `_acme_challenge.yourdomain.com` contains TXT record with a token (token is generated by cert-manager by communicating with the ACME server) When cert-manager is setup using DNS-01. It should have ability to create a new DNS record. For example using [Azure DNS](https://cert-manager.io/docs/configuration/acme/dns01/azuredns/). Once a ssl certificate was requested, it will create that TXT record. and once letsencrypt verify it will store the certificate on a secret HTTP resolver This type of resolver checks a token by providing an HTTP endpoint. with HTTP-01 challenge, cert-manager will create an ingress that and a pod that serve the token for example ``` ❯ kubectl describe ingress cm-acme-http-solver-sc8p9 Name: cm-acme-http-solver-sc8p9 Namespace: foo Address: 1.222.333.444 Default backend: default-http-backend:80 Rules: Host Path Backends ---- ---- -------- accord.shell.com /.well-known/acme-challenge/3AEp9usn1kG261SOfOkJYHuVzga6VS0aD14PiUjPtK8 cm-acme-http-solver-gvmqg:8089 (1.22.33.44:8089) Annotations: nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0 ``` this should serve `http://yourdomain.com//.well-known/acme-challenge/3AEp9usn1kG261SOfOkJYHuVzga6VS0aD14PiUjPtK8` ACME server will call this endpoint and check that the response match the token. # kubernetes resource short name type less with the resource short name ``` kubectl get ingress kubectl get ing ``` Most common usage | deployments | deploy | | ----------- | ------ | | ingresses | ing | | pods | po | | replicasets | rs | Full commands | certificates | cert,certs | | -------------------------- | ---------- | | certificatesigningrequests | csr | | certificiaterequests | cr,crs | | componentstatuses | cs | | configmaps | cm | | cronjobs | cj | | customresourcedefinitions | crd,crds | | daemonsets | ds | | endpoints | ep | | events | ev | | horizontalpodautoscalers | hpa | | limitranges | limits | | namespaces | ns | | networkpolicies | netpol | | nodes | no | | persistentvolumeclaims | pvc | | persistentvolumes | pv | | podsecuritypolicies | psp | | priorityclasses | pc | | replicasets | rs | | replicationcontrollers | rc | | resourcequotas | quota | | scheduledscalers | ss | | serviceaccounts | sa | | services | svc | | statefulsets | sts | | storageclasses | sc | # scale deployment ``` $ kubectl scale deployment --replicas=1 deployment_name ``` # view secret List secrets ``` $ kubecetl get secrets ``` View particular secret ``` $ kubectl describe secret foo Name: foo Namespace: foonamespac Labels: Annotations: Type: Opaque Data ==== DB_PASSWORD: 25 bytes ``` to view the content of a secret ``` kubectl get secret foo -o jsonpath="{.data.DB_PASSWORD}" | base64 --decode someSecurePassword ``` # linux # boot zfs root filesystem on grub from [here](https://askubuntu.com/questions/1225751/recovering-a-borked-grub-with-a-ubuntu-19-10-zfs-root-filesystem) For some reason, grub was broken. I need to be able to boot my linux from GRUB shell ``` set root=(hd0,gptXXX) linux /BOOT/ubuntu_YYY/@/vmlinuz root=ZFS=rpool/ROOT/ubuntu_YYY boot=zfs initrd /BOOT/ubuntu_YYY/@/initrd.img boot ``` the `gptXXX` i need to search around for a file until I found the `/BOOT` directory # clearing up swap space *WARNING: make sure the memory is enough to put the data back from swap, else system will start killing processes* To clear up swap space (put the data back into memory) we can do ``` # swapoff -a && swapon -a ``` Because this is quite slow process, it's a good idea to run this inside screen session. # connect bluetooth device from cli We use bluetoothctl tool which is command-line to BlueZ ``` $ sudo bluetoothctl ``` Enable authentication agent ``` [bluetooth]# agent on ``` Run the scan process ``` [bluetooth]# scan on ``` Pair with the device ``` [bluetooth]# pair 00:25:56:D1:36:6B ``` Connect to the device ``` [bluetooth]# connect 00:25:56:D1:36:6B ``` more information # free and vmstat command from: ## free ``` $ free -h total used free shared buff/cache available Mem: 23Gi 14Gi 575Mi 3,3Gi 12Gi 8,8Gi Swap: 8,0Gi 6,6Gi 1,4Gi ``` Free parses /proc/meminfo and prints totals for physical memory and swap, along with kernel buffers and cache. Use -h for human-readable units, -s 1 to refresh every second, and -c N to stop after N samples which is handy to get a trend when doing something in parallel. For example, free -s 60 -c 1440 gives a 24-hour CSV-friendly record without installing extra monitoring daemons. Free memory refers to RAM that is entirely unoccupied. It isn’t being used by any process or for caching. Available memory, on the other hand, represents an estimate of how much memory can be used by new or running processes without resorting to swap. It includes free memory plus parts of the cache and buffers that the system can reclaim quickly if needed. It is not a problem to have a low free memory, **available memory is usually what to be concerned about.** ### Spotting problems with free * Rapidly shrinking available combined with rising swap used indicates real memory pressure. * Large swap-in/out spikes point to thrashing workloads or runaway memory consumers. ## vmstat ``` $ vmstat 1 3 procs -----------memory---------- r b swpd free buff cache 2 0 7102404 1392528 36 12335148 0 0 7102404 1392560 36 12335188 0 0 7102404 1373640 36 12349928 ---swap-- -----io---- si so bi bo 8 21 130 724 0 0 0 0 0 0 8 48 -system-- -------cpu------- in cs us sy id wa st gu 2851 19 15 7 77 0 0 0 5779 7246 14 10 77 0 0 0 5141 6525 12 9 79 0 0 0 ``` ``` Procs r: The number of runnable processes (running or waiting for run time). b: The number of processes blocked waiting for I/O to complete. Memory These are affected by the --unit option. swpd: the amount of swap memory used. free: the amount of idle memory. buff: the amount of memory used as buffers. cache: the amount of memory used as cache. inact: the amount of inactive memory. (-a option) active: the amount of active memory. (-a option) Swap These are affected by the --unit option. si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). IO bi: Kibibyte received from a block device (KiB/s). bo: Kibibyte sent to a block device (KiB/s). System in: The number of interrupts per second, including the clock. cs: The number of context switches per second. CPU These are percentages of total CPU time. us: Time spent running non-kernel code. (user time, including nice time) sy: Time spent running kernel code. (system time) id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time. wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle. st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown. gu: Time spent running KVM guest code (guest time, including guest nice). ``` ### Catching a memory leak Run vmstat 500 in one terminal while your suspect application runs in another. If free keeps falling and si/so climb over successive samples, physical RAM is being exhausted and the kernel starts swapping, which is classic leak behavior. ### Finding I/O saturation When wa (CPU wait) and bo (blocks out) soar while r remains modest, the CPU is idle but stuck waiting for the disk. Consider adding faster storage or tuning I/O scheduler parameters. ### Detecting CPU over-commit A sustained r that is double the number of logical cores with low wa and plenty of free means CPU is the bottleneck, not memory or I/O. Use top or htop to locate the busiest processes, or scale out workloads accordingly. # get full argument from a process We can find it out by using `ps`. For example ``` $ ps aux | grep cassandra ... cassand+ 24615 40.5 41.9 12064088 5033964 ? SLl 14:28 5:15 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4096M -Xmx4096M -Xmn1024M -XX:+PreserveFramePointer -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=2 -XX:MaxTenuringThreshold=15 -XX:CMSInitiatingOccupancyFraction=25 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:+UnlockDiagnosticVMOptions -XX:-UseBiasedLocking -XX:+UseGCTaskAffinity -XX:+BindGCTaskThreadsToCPUs -XX:ConcGCThreads=16 -XX:ParallelGCThreads=16 -XX:ParGCCardsPerStrideChunk=4096 -XX:+ParallelRefProcEnabled -XX:CMSMaxAbortablePrecleanTime=60000 -XX:CMSWaitDuration=30000 -XX:+AlwaysPreTouch -XX:+UseTLAB -XX:+ResizeTLAB -Dcassandra.max_local_pause_in_ms=40000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -XX:+PrintAdaptiveSizePolicy -XX:+PrintSafepointStatistics -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -Xloggc:/var/log/cassandra/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/usr/lib/jolokia/jolokia-jvm.jar=port=8089,host=0.0.0.0 -Dcassandra.metricsReporterConfigFile=/etc/cassandra/graphite.yaml -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=/var/lib/cassandra -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jna-4.4.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-graphite-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache- ... ``` The output is truncated. This is the same if you cat `/proc/[pid]/cmdline` According to [http://praveen.kumar.in/2010/02/24/getting-untruncated-command-line-options-passed-to-a-solaris-process/](https://github.com/yulrizka/til/tree/5515cab32fe44d17cce33a885ae6b10f0220c2bc/linux/this/README.md) (at least for solaris) the kernel only keep up until certain lenght. some tool can read into memory of the program to get the ARGS value. for linux we can use `ps eww -p ` ``` root@cass001:/proc/5367# ps eww -p 24615 PID TTY STAT TIME COMMAND 24615 ? SLl 10:00 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4096M -Xmx4096M -Xmn1024M -XX:+PreserveFramePointer -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=2 -XX:MaxTenuringThreshold=15 -XX:CMSInitiatingOccupancyFraction=25 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:+UnlockDiagnosticVMOptions -XX:-UseBiasedLocking -XX:+UseGCTaskAffinity -XX:+BindGCTaskThreadsToCPUs -XX:ConcGCThreads=16 -XX:ParallelGCThreads=16 -XX:ParGCCardsPerStrideChunk=4096 -XX:+ParallelRefProcEnabled -XX:CMSMaxAbortablePrecleanTime=60000 -XX:CMSWaitDuration=30000 -XX:+AlwaysPreTouch -XX:+UseTLAB -XX:+ResizeTLAB -Dcassandra.max_local_pause_in_ms=40000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:PrintFLSStatistics=1 -XX:+PrintAdaptiveSizePolicy -XX:+PrintSafepointStatistics -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -Xloggc:/var/log/cassandra/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/usr/lib/jolokia/jolokia-jvm.jar=port=8089,host=0.0.0.0 -Dcassandra.metricsReporterConfigFile=/etc/cassandra/graphite.yaml -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=/var/lib/cassandra -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jna-4.4.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-graphite-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache- TERM=xterm-256color SHELL=/bin/bash USER=root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: SUDO_USER=root SUDO_UID=0 MALLOC_ARENA_MAX=4 USERNAME=root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin MAIL=/var/mail/root PWD=/ LANG=en_US.UTF-8 HOME=/var/lib/cassandra SUDO_COMMAND=/etc/init.d/cassandra restart SHLVL=1 LOGNAME=root SUDO_GID=0 ``` # merge multiple pdf into a single file ``` pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf ``` pdfunit is part of # removing old kernel When upgrading ubuntu, sometimes it failed because of insufficient disk on boot. This could be caused by too many kernel installed on the machine. Command to remove old kernel from [here](http://askubuntu.com/questions/2793/how-do-i-remove-old-kernel-versions-to-clean-up-the-boot-menu) ``` dpkg --list | grep linux-image | awk '{ print $2 }' | sort -V | sed -n '/'`uname -r`'/q;p' | xargs sudo apt-get -y purge ``` dpkg --list | grep linux-image | awk '{ print $2 }' | sort -V | sed -n '/'`uname -r`'/q;p' | xargs sudo apt-get -y purgenation (remember, `|` uses the output of the previous command as the input to the next) * `dpkg --list` lists all installed packages * `grep linux-image` looks for the installed linux images * `awk '{ print $2 }'` just outputs the 2nd column (which is the package name) * `sort -V` puts the items in order by version number * ``sed -n '/'`uname -r`'/q;p'`` prints the lines before the current kernel * `xargs sudo apt-get -y purge` purges the found kernels Unwinding the `sed` invocation: * `-n` tells `sed` to be quiet * `` `uname -r` `` outputs the current installed kernel release - we include it in backticks so that the output is includes as part of the command (you might also see this as `$(uname -r)` * `/something/q` says stop when you match 'something' (in this case, something is output of `uname -r`) - the `/` surround a regular expression * `p` is print * the `;` is the command separtor, so `/something/q;p` says quit when you match something, else print altogether, ``sed -n '/'`uname -r`'/q;p'`` is print the lines until it matches with the current kernel name. # symbolic vs hard link from > Underneath the file system files are represented by inodes (or is it multiple inodes not sure) > > A file in the file system is basically a link to an inode. A hard link then just creates another file with a link to the same underlying inode. > > When you delete a file it removes one link to the underlying inode. The inode is only deleted (or deletable/over-writable) when all links to the inode have been deleted. > > A symbolic link is a link to another name in the file system. > > Once a hard link has been made the link is to the inode. deleting renaming or moving the original file will not affect the hard link as it links to the underlying inode. Any changes to the data on the inode is reflected in all files that refer to that inode. > > Note: Hard links are only valid within the same File System. Symbolic links can span file systems as they are simply the name of another file. # zfs auto snapshot Having ZFS snapshot saved me once when I acidentally `rm -f /home/myuser`. There is a simple package that easily make auto snapshot. ``` $ sudo apt install zfs-auto-snapshot ``` This will install the script and the cron job associated with it. By default it will backup all filesystem and volumes. You can disable the filesystems completely by ``` $ sudo zfs set com.sun:auto-snapshot=false tank/data-set-name ``` or only for specific interval ``` sudo zfs set com.sun:auto-snapshot=true tank/data-set-name sudo zfs set com.sun:auto-snapshot:monthly=false tank/data-set-name sudo zfs set com.sun:auto-snapshot:weekly=false tank/data-set-name sudo zfs set com.sun:auto-snapshot:daily=true tank/data-set-name sudo zfs set com.sun:auto-snapshot:hourly=false tank/data-set-name sudo zfs set com.sun:auto-snapshot:frequent=false tank/data-set-name ``` # zfs external backup drive with snapshot and encryption [main source](http://ryan.himmelwright.net/post/zfs-backups-to-luks-external/) ## Get device id ``` $ ls /dev/disk/by-id -alh ... lrwxrwxrwx 1 root root 10 okt 24 06:06 ata-WDC_WD10EZEX-08M2NA0_WD-WMC3F1471486-part4 -> ../../sda4 ... ``` For example, I'm going to use `/dev/disk/by-id/ata-WDC_WD10EZEX-08M2NA0_WD-WMC3F1471486-part4` ## Setup disk encryption with LUKS setup [LuKS](http://ryan.himmelwright.net/post/zfs-backups-to-luks-external/) ``` $ sudo apt install cryptsetup $ cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --iter-time 10000 --use-random -y /dev/disk/by-id/ata-WDC_WD10EZEX-08M2NA0_WD-WMC3F1471486-part4 ``` * `--cipher` encryption algorithm * `--key-size` encryption key size * `--iter-time` Number of millisecond to spend P8KDF passphrase processing * `--use-random` use /dev/random * `-y` verify passphrase Disk device can now be opened. ``` $ sudo cryptsetup luksOpen /dev/disk/by-id/ata-WDC_WD10EZEX-08M2NA0_WD-WMC3F1471486-part4 luks1 ``` ## Create new zfs pool ``` $ sudo zpool create ext-backup /dev/mapper/luks1 ``` ## Create initial snapshot Example i have data set `tank/ROOT/home` ``` $ sudo zfs snapshot tank/ROOT/home@ext-backup ``` ## Send the backup ``` $ sudo zfs send tank/ROOT/home@ext-backup | pv | sudo zfs recv ext-backup/home ``` Or with incremental-backup ``` $ sudo zfs snapshot tank/ROOT/home@ext-backup20180101 $ sudo zfs send -R -i tank/ROOT/home@ext-backup tank/ROOT/home@ext-backup20180101 | sudo zfs recv ext-backup/hom ``` A good idea is to set the external drive to be read-only ``` $ sudo zfs set readonly=on ext-backup ``` ## Safely close and remove external drive After finished sending the snapshot, close the disk and export the pool ``` $ sudo zpool export ext-backup $ sudo cryptsetup luksClose ext-backup ``` Reference: # net # dnssec > DNSSEC is a technology that was developed to, among other things, protect against such attacks by digitally 'signing' data so you can be assured it is valid. However, in order to eliminate the vulnerability from the Internet, it must be deployed at each step in the lookup from root zone to final domain name (e.g., [www.icann.org](http://www.icann.org)). Signing the root (deploying DNSSEC on the root zone) is a necessary step in this overall processii. Importantly it does not encrypt data. It just attests to the validity of the address of the site you visit. It's basically backward compatible addition to the protocol where DNS server can verify the authenticity of the record using asymetric key certificate There is also as concept of KSK (Key Signing key) a long term key and ZSK (Zone Signing Key) a short term key. They rotate the ZSK using KSK so that it's harder to bruteforce to key # html form submit to different action depending on properties Sometimes you want the form to submit to different URL based on which submit button you click. This is already supported by HTML with `formaction` attribute example ```


``` More information on the attirbute # ldap list users On ldap connected machine ``` $ genet group groupname groupname*:2141:usera,userb,userc,userd ``` # osx # checksum a file from a url ``` $ wget -qO- https://github.com/yulrizka/osx-push-to-talk/archive/v0.1.5.tar.gz | sha256sum ``` * q : quiet, disable output log * O : output to a file * `-` : stdout flag is the same as `-q -O /dev/stdout` # cluster ssh in iterm2 with i2cssh I'm a fan of cluster ssh. It allows you to control multiple ssh session with one keyboard. I tried [csshx](https://github.com/brockgr/csshx) which uses the default osx terminal, but I love iterm2. But today I found [i2cssh](https://github.com/wouterdebie/i2cssh) which does that for iterm2 One of the feature is broadcast where you can send command to multiple window. It's basically the same with csshx but their approach is to create another extra window that receive an input and send it to all other windows. I like i2cssh approach better with shortcut. # list open port ``` $ sudo netstat -atp tcp| grep -i listen ``` or ``` $ sudo lsof -i -P | grep -i listen ``` # manage clipboard easily with jumpcut [Jumpcut](http://jumpcut.sourceforge.net/) is clipboard manager that allows you to select your copy history. Let say you copy fil the contents of A, B, C, D. After you copy file D you want to paste the file B. Usually you have to go back to copy the content of file, but now you don't have too. # pipe output to clipboard with pbcopy and pbpaste Using terminal and needs to copy the result of command to clipboard ? use pbcopy ```bash $ cat somefile.txt | pbcopy ``` Or the other way around, paste from clipboard and pipe it to other program ```bash $ pbpaste | grep foo ``` # show hidden file Tho show hidden file (file with `x`): **Mojave**: In finder `CMD + Shift + .` to toggle it **Before Mojave**: Show ``` defaults write com.apple.finder AppleShowAllFiles YES; killall Finder /System/Library/CoreServices/Finder.app ``` Hide ``` defaults write com.apple.finder AppleShowAllFiles NO; killall Finder /System/Library/CoreServices/Finder.app ``` # sign application with self certificate When compiling binary that open port, (depending on your settings) it will ask to allow if the process can make the connection. We can sign our binary with certificate to avoid this. From While RedYeti's link is useful, just to save a few clicks for others let me recap how to generate a code-signing cert and to use it for code (re-)signing: ``` 1. Create your own code signing cert: In Keychain Access, Keychain Access > Certificate Assistant > Create a certificate. This launches the Certificate Assistant: Name: Enter some arbitrary string here that you can remember. Avoid spaces otherwise you'll need to escape the cert's name when using codesign from the command line. Identity type: Self Signed Root Certificate Type: Code Signing Check the box "Let me override defaults", this is quite important Serial number: 1 (OK as long as the cert name/serial no. combination is unique) Validity Period: 3650 (gives you 10 years) Email, Name, etc. fill out as you wish. Key pair info: set to RSA, 2048 bits. Does not really matter IMHO. From "Key usage extension" up to "Subject Alternate Name Extension": accept the defaults. Location: login keychain. Once it is created, set to "Always trust" in the Login keychain. 2. Re-signing an app: codesign -f -s /path/to/app --deep 3. Verify that it worked: codesign -dvvvv /path/to/app Enjoy! ``` # starting program on startup with login items I need to start [jumpcut](https://github.com/yulrizka/til/blob/master/osx/manage%20clipboard%20easily%20with%20jumpcut.md) during startup because I have to start it manually now. We can add one or more program to start when we are logged in. Use system preference > Users & Groups. Select your username and there should be *Login items* tabs where you can add more or disable programs. # python # count frequency with lambda `count` function can be used in map to count a frequency Also to use `*` operator in exchange for `list()` ``` >>> a='1122345' >>> list(map(a.count, '1234567')) [2, 2, 1, 1, 1, 0, 0] list(map(a.count, (1,2,3,4,6,7))) [2, 2, 1, 1, 0, 0] a=(1,1,2,2,3,4,5) [*map(a.count, (1,2,3,4,5,6,7))] ``` ### Using Collections.Count ``` >>> a='1122345' >>> import collections >>> collections.Counter(a) Counter({'1': 2, '2': 2, '3': 1, '4': 1, '5': 1}) ``` # dijkstra algorithm shortest path # double slash arithmetic operator `//` give the integer division ``` >>> 9/4 2.25 >>> 9//4 >>> 9/5 1.8 >>> 9//5 1 ``` # min and max of dict values ``` >>> vals={"a": 2, "b":3, "c": 4} >>> min(vals, key=lambda x: vals[x]) 'a' ``` # python3 match case since python 3.10, it support switch and case. it even can check on type and value ``` def check(v): match v: case "1", "2": print('both 1 2 is string') case "1", 2: print('1 is string 2 is int') case a, b: print(f'a={a}, b={b}') case a: print(f'a=a') check(("1", "2")) # both 1 2 is string check(("1", 2)) # 1 is string 2 is int check((1, None)) # a=1, b=None check(1) # a=a ``` # reduce and opeator Reduce is operation that reducing an array into a single value. For example multiplying all element in the array ``` a=[1,2,3,4,5,6] from functools import reduce from operator import mul a=[1,2,3] reduce(mul, a) # 6 # or with lambda reduce(lambda x, y: x*y, a) # 6 ``` # react # react named export vs default export taken from Default export only export one item (object, function,...) ```javascript // import import MyDefaultComponent from "./MyDefaultExport"; // export const MyComponent = () => {} export default MyComponent; ``` Named export can have multiple name and can be imported selectively ```javascript // module "my-module.js" function cube(x) { return x * x * x; } const foo = Math.PI + Math.SQRT2; var graph = { options: { color:'white', thickness:'2px' }, draw: function() { console.log('From graph draw function'); } } export { cube, foo, graph }; // to use it in some other file import { cube, foo, graph } from './my-module.js'; graph.options = { color:'blue', thickness:'3px' }; graph.draw(); console.log(cube(3)); // 27 console.log(foo); // 4.555806215962888 ``` # react useeffect In react [life-cycle](https://reactjs.org/docs/react-component.html#the-component-lifecycle) there are sets of method that being called when a component is being created and inserted to the DOM. Example when creating a component for the first time. * [**`constructor()`**](https://reactjs.org/docs/react-component.html#constructor) * [`static getDerivedStateFromProps()`](https://reactjs.org/docs/react-component.html#static-getderivedstatefromprops) * [**`render()`**](https://reactjs.org/docs/react-component.html#render) * [**`componentDidMount()`**](https://reactjs.org/docs/react-component.html#componentdidmount) `ComponentDidMount` is called later when the componen is added to the DOM. Usually this function is used to run additional code after react has updated the DOM. (Eg, fetching external data, manual updating DOM mutation). This function has "side-effect" When using a Function Component, react has similar method called `useEffect` ```jsx import React, { useState, useEffect } from 'react'; function Example() { const [count, setCount] = useState(0); // Similar to componentDidMount and componentDidUpdate: useEffect(() => { // Update the document title using the browser API document.title = `You clicked ${count} times`; }); return (

You clicked {count} times

); } ``` This example shows function without cleanup method. To create a effect that willbe cleanup after Unmount & every render, ```javascript function FriendStatus(props) { const [isOnline, setIsOnline] = useState(null); useEffect(() => { function handleStatusChange(status) { setIsOnline(status.isOnline); } ChatAPI.subscribeToFriendStatus(props.friend.id, handleStatusChange); // Specify how to clean up after this effect: return function cleanup() { ChatAPI.unsubscribeFromFriendStatus(props.friend.id, handleStatusChange); }; }); if (isOnline === null) { return 'Loading...'; } return isOnline ? 'Online' : 'Offline'; } ``` ### Only run when some data change Running `useEffect` function for every render might not be good for performance. A second argument to the `useEffect` can be pass to only run the function when the listed data changed ```jsx useEffect(() => { document.title = `You clicked ${count} times`; }, [count]); // Only re-run the effect if count changes ``` # react.useref `useRef` returns a mutable ref object whose `.current` property is initialized to the passed argument (`initialValue`). The returned object will persist for the full lifetime of the component. ```jsx function TextInputWithFocusButton() { const inputEl = useRef(null); const onButtonClick = () => { // `current` points to the mounted text input element inputEl.current.focus(); }; return ( <> ); } ``` Keep in mind that `useRef` *doesn’t* notify you when its content changes. Mutating the `.current` property doesn’t cause a re-render. If you want to run some code when React attaches or detaches a ref to a DOM node, you may want to use a [callback ref](https://reactjs.org/docs/hooks-faq.html#how-can-i-measure-a-dom-node) instead. # unix # bulk renaming multiple file There are couple of way to rename multiple files. From example we have files ``` image0001.png image0002.png image0003.png ... ``` And we would like to rename it to ``` 0001.png 0002.png 0003.png ... ``` ## with for loop this works with linux and mac without installing anything ```bash for f in *.png; do mv "$f" "${f#image}"; done ``` ## With ZSH This is my favorite way since I'm using [ZSH](http://www.zsh.org/) and [oh-my-zsh](https://github.com/robbyrussell/oh-my-zsh). Check it now, it's awesome ```bash $ autoload zmv zmv 'image(*.png)' '$1' ``` or let zsh automatically define $1, $2 etc ```bash zmv -w 'image*.png' '$1.png' ``` ## Using your editor for more complex operation, sometimes it easier to rename the file on your editor. there is `qmv` from renameutils ``` $ qmv *.png ``` it will open editor with 2 columns, you can edit the right size rename that file # convert pdf to text using ocr make sure you have imagemagick and tesseract are installed ``` $ sudo apt install imagemagick tesseract-ocr ``` It's a 2 step process: 1. Convert PDF to .tiff using `convert` from imagemagick ``` $ convert -density 300 input.pdf -depth 8 output.tiff ``` 2. convert .tiff to text using `tesseract` generate `out.txt` ``` $ tesseract output.tiff out ``` # diff output of 2 command for example, I want to compare output of 2 different curl comand ``` $ diff <(curl -s http://host.com/a) <(curl -s http://anotherhost.com/a) ``` # encryption with gpg ### Generate key ```bash $ gpg --gen-key ``` ### List keys ```bash $ gpg --list-keys ``` ### Encrypt Data With a passprhare ```bash $ gpg -ca -o output.txt.gpg input.txt ``` with a certificate and from STDOUT ```bash $ gpg -ea ``` or if you know the name already (can be key, name or email) ```bash $ gpg -ea -r "Ahmy" ``` ### Decrypt Data ```bash $ gpg -d file.txt.gpg ``` ### Export Public key ```bash $ gpg --export -a "name" > public.key ``` `-a` is to create armored ascii output. Private Key ```bash $ gpg --export-secret-key -a "name" > private.key ``` ### Import Public key ```bash $ gpg --export-secret-key -a "name" > private.key ``` Private Key ```bash $ gpg --allow-secret-key-import --import private.key ``` ### Deleting Public Key ```bash $ gpg --delete-key "Real Name" ``` Private Key ```bash $ gpg --delete-secret-key "Real Name" ``` ### Extend the expiration ``` gpg --edit-key "name" ``` Select the key to extend
``` gpg> key 1 gpg> expire ``` ### Send key to a key server ``` gpg --keyserver keyserver.ubuntu.com --send-key "name" ``` # extend letsencrypt certificate with dns challenge Assuming the domain name is `yulrizka.com` and the certificate was created with ```bash $ certbot certonly --standalone --preferred-challenges dns -d yulrizka.com ``` To extend this certificate with DNS challenge ```bash # certbot -d yulrizka.com --manual --preferred-challenges dns certonly ``` You will be asked to add TXT record of `_acme-challenge.yulrizka.com`. After updating that DNS record, wait couple of minutes and proceed. # ffmpeg monitor and restart stream when it hung or stall You are using ffmpeg to stream to youtube but after a while, it hung or stuck. You want to restart the streaming command when this happens. First run the `ffmpeg` command on a terminal in a while loop (this example is for fish terminal) ``` while true ffmpeg -re -i https://roundrobin3.videostreamingwowza.com/visdeurbel2/visdeurbel2.stream/playlist.m3u8 -c:v copy -c:a aac -ar 44100 -ab 128k -ac 2 -strict -2 -flags +global_header -bsf:a aac_adtstoasc -bufsize 3000k -f flv "rtmp://a.rtmp.youtube.com/live2/abcd-efgh-ijkl" > youtube.log sleep 5 end ``` This will run the stream and output it to a `log.txt` run this script on separate terminal to monitor and restart ffmpeg process ``` #!/bin/sh #sleep 10 while true do frameA=$(grep "videostreamingwowza" youtube.log | tail -n1 | sed -n 's/.*\(roundrobin3.videostreamingwowza.com.*\.ts.*\)/\1/p') echo "A: $frameA" sleep 30 frameB=$(grep "videostreamingwowza" youtube.log | tail -n1 | sed -n 's/.*\(roundrobin3.videostreamingwowza.com.*\.ts.*\)/\1/p') echo "B: $frameB" if [ "$frameA" = "$frameB" ] then echo "Stream has hung" printf "%s - Stream has hung\n" "$(date)" >> stream.log pkill ffmpeg echo "killed ffmpeg..." fi sleep 2 done ``` Basically it take some line from the log, and do it again after 30 seconds. If the line stays the same, it assume that the stream stuck and kill the `ffmpeg` process # file size older than x days from to list all file that modified more than more than X days, we can use ``` $ find . -mtime +[number of days] ``` if you use `-` sign than it became file is modified since X days Combine it with akw, we can get total file size ``` $ find . -mtime +180 -exec du -ks {} \; | cut -f1 | awk '{total=total+$1}END{print total/1024}' ``` # filtering json with jq [jq](https://stedolan.github.io/jq/) is a fantastic command line tools to parse and filter json in command line. It works for linux, osx and maybe windows. example, to format json ``` $ json='[{"genre":"deep house"}, {"genre": "progressive house"}, {"genre": "dubstep"}]' $ echo $json [{"genre":"deep house"}, {"genre": "progressive house"}, {"genre": "dubstep"}] $ echo $json | jq . [ { "genre": "deep house" }, { "genre": "progressive house" }, { "genre": "dubstep" } ] ``` To output a particular field ``` $ echo $json | jq '.[].genre' "deep house" "progressive house" "dubstep" ``` To filter based on a key ``` $ echo "$json" | jq -c '.[] | select(.genre | contains("house"))' {"genre":"deep house"} {"genre":"progressive house"} ``` more example could be found on [here](https://github.com/stedolan/jq/wiki/Cookbook) # find lines that matches on 2 different sorted file Supposed we have 2 sorted files: file1 ``` 1 2 3 4 5 ``` file2 ``` 3 4 5 6 7 ``` `comm` is a utility for finding lines that appear (or not) between 2 files. The normal output are 3 column. 1. Lines that is unique on file 1 2. Lines that is unique on file 2 3. Lines that are appear on both ### Examples ```bash $ comm file1 file2 1 2 3 4 5 6 7 ``` #### Find lines that on both file only ```bash $ comm -12 file1 file2 3 4 5 ``` `-12` supresses (do not include) colunm 1 & 2 #### Fine line that is unique on first file ```bash $ comm -23 file1 file2 1 2 ``` #### For file that is not uniq or sorted If file is not sorted, you can use the `<(..)` operator ```bash $ comm <(sort -u file1) <(sort -u file2) 1 2 3 4 5 6 7 ``` # find out what is using swap from ```bash #!/bin/bash # Get current swap usage for all running processes # Erik Ljungstrom 27/05/2011 SUM=0 OVERALL=0 for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do PID=`echo $DIR | cut -d / -f 3` PROGNAME=`ps -p $PID -o comm --no-headers` for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'` do let SUM=$SUM+$SWAP done echo "PID=$PID - Swap used: $SUM - ($PROGNAME )" let OVERALL=$OVERALL+$SUM SUM=0 done echo "Overall swap used: $OVERALL" ``` # fish environment variables from 1password A common task is to load environment variables from secret. With password manager like 1password, you can store it as notes and use the CLI to load it to your shell env Install 1password CLI create a secure notes with content like (eg: with the name "db-dev") ``` DB_USER=USER1 DB_PASSWORD=PASSWORD ``` login 1password ``` eval (op signin my) ``` load the note ```bash export (op get item "db-dev" | jq -r .details.notesPlain) ``` # fix gpg warning unsafe permissions on homedir When you copy gpg configuration form other machine, you might get warning as follow ``` gpg: WARNING: unsafe permissions on homedir '/home/user/.gnupg' ``` To fix this, make sure that The `.gnupg` directory has the correct permission. ``` chown -R $(whoami) ~/.gnupg/ ``` And files & folder inside has the correct permisson ``` find ~/.gnupg -type f -exec chmod 600 {} \; # Set 600 for files find ~/.gnupg -type d -exec chmod 700 {} \; # Set 700 for directories ``` # formatting or parse json in command line To easily format json or command line there [jq](https://stedolan.github.io/jq/) example ``` $ echo '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}' | jq . { "user": "stedolan", "titles": [ "JQ Primer", "More JQ" ] } ``` or just print specific value ``` $ echo '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}' | jq .titles[0] "JQ Primer" ``` # get all line except n last one with `head -n ` we can get the output except n lines example ``` cat a.txt | head -n -1 ``` will gives as the content of a.txt except the last one Let say we have file 1 2 3 4 5, and with grep we want to only get 1 2 3. we can achive this with ls and grep ``` $ ls | grep 4 -B 100000000 | head -n -1 ``` the -B show 100000000 lines before matched line, and the head -n -1 removes the matched # grep print only matched you can use the `-o` flag to only print matched value. Example ``` $ echo "Hello fellow unix fan" | grep --color -o nix nix $ echo "Hello fellow unix fan" | grep --color -o nux $ echo $? 1 ``` # grep using input file as pattern to search other file Sometimes we need to use another file which contains a multiple line that we want to use as input pattern example: You have `a.txt` ``` a b c d e ``` And you want to search `a`, `c`, `e`. This case you can create `input.txt` ``` a c e ``` and use this command ``` grep -f -F input.txt a.txt ``` If the input txt is a list of *regex* you can remove the `-F` option. ``` -F, --fixed-strings Interpret PATTERNS as fixed strings, not regular expressions. ``` # jq counting lenght of an array ``` echo '[{"username":"user1"},{"username":"user2"}]' | jq '. | length' ``` # jq extracting properties to arrays from json row line For example if we have a file that contains JSON object like ``` {"a": 1, "b":{"c": 11} {"a": 2, "b":{"c": 22}} {"a": 3, "b":{"c": 33}} ``` And you want to construct something like ``` ["1", "11"] ["2", "22"] ["3", "33"] ``` You can do that with jq ``` $ cat filename.json | jq -c '[.a, .b.c]' ``` this is particularly usefull to grep something from JSON log --- [Next Page](/llms-full.txt/1)