Recently I was planning to make a small contribution to project DefinitelyTyped/DefinitelyTyped. It is a huge repository that hosts and provides the type declaration files for thousands of packages on npm, with a structure like the following
types/├── 11ty__eleventy-img├── 1line-aa├── 3box├── ...and more of them...
Each sub-folder under the types/ directory corresponds to a specific npm package.
The package I am interested in is marked, whose type declaration files reside in types/marked/ directory. Towards this end, cloning the whole repository and doing a full checkout is considered worthless, since my disk space and network traffic would be eaten up by a bunch of irrelevant files.
Both Dart and Go support decentralized package distribution. One is able to directly adopt an existing git repository as dependency, easing the effort of distributing packages.
Sometimes we might expect more fine-grained control on what to pull from a git repository. For example, to lock a package’s version, we would specify a particular tag, commit or branch name to pull from. Or if it’s a mono-repo, we would choose a sub-directory from the repository root. This post summarizes how to achieve these purposes in both languages.
I came across a piece of interesting vulnerable script from a post on V2EX 1) on V2EX. A bash function named __curl inside the file retrieves contents over HTTP as a simple alternative for command curl or wget, in scenarios where no such utilities available.
The function makes use of certain less known features of Linux and the Bash language.
The first one is communicating over TCP through files. Linux employs a design philosophy of “everything are files”. One could find some special device files in directory /dev, through which we can manipulate the underlying devices. Specifically, manipulating a TCP socket connecting ${HOST}:${PORT} could be achieved by accessing device file /dev/tcp/${HOST}/${PORT}. Since HTTP is a text-based protocol over TCP, working with it is no more difficult than reading / writing a text file. Line exec 3<>$FILENAME opens file $FILENAME under read-write mode and binds it to descriptor 3. The next line then manually composes an HTTP payload and writes out to &3, which is in fact requesting the URL http ://${HOST}:${PORT}. By reading the same file descriptor, we retrieve the response content from the service. The trick serves as a primitive workaround for retrieving contents from web.
Another one is parameter substitution in Bash. The expression ${var//PATTERN/REPL} substitutes all occurrences of PATTERN in var into REPL. If REPL omitted, the matched substrings will be deleted. For example, in this script, ${1//// } would replace all slashes / into white spaces in variable $1.
Say we have a matrix A of shape N x M , which can be viewed as a collection of N vectors of shape 1 x M . The code below gives us the correlation matrix of A :
A_corr = np.corrcoef(A) # shape: (N, N)
To visualize it, just use plt.matshow(A_corr) .
If N is so large that the figure could not provide a clear insight, we might alternatively use histograms like this:
defcorr_matrix_to_array(corr_mat): N = corr_mat.shape[0]return np.array([corr_mat[i][j] for i inrange(1, N) for j inrange(i + 1, N)])plt.hist(corr_matrix_to_array(A_corr), bins=np.linspace(-1, 1, N_bins))
