Friday, 23 September 2022

More fun with pip

Again with the Python and pip fun, this time on my Mac, where commands such as: 

pip3 list

and: -

pip3 install --upgrade pip

were failing with: -

WARNING: There was an error checking the latest version of pip.

Long story short, the error was actually more obvious than I'd realised ....

In essence, it was actually telling me what was going wrong ...

Looking in indexes:.......

where the location was an internal repository for **ONE** single Python module, which would never have the things I was trying to install e.g. pip itself

I updated the config: -

vi ~/.pip/pip.conf

and commented out the aberrant index ... 

And now we're off to the races ...

Installing pylint on Linux - there's more than one way ...

 I'm running some Travis builds of a project which includes Python modules, and wanted to manually run the pylint linting tool across my GitHub repo, to compare/contrast what Travis was telling me.

I'm running Ubuntu 20.04.5 LTS so just went ahead and installed via the Aptitude package manager: -

apt-get update && apt-get install -y pylint

which seemed fine.

Then I noticed that the results differed - when I checked Travis, I was using pylint 2.15.2 whereas, on Ubuntu, I was way behind the curve: -

pylint --version

pylint 2.4.4
astroid 2.3.3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]

My mistake ? Using Aptitude ...

I uninstalled pylint 

apt-get remove -y pylint

and installed it using the Package Installer (for) Python, aka pip

I did notice that the installation location was different: -

- apt-get install ->>> /usr/bin/pylint

- pip  ->>> /usr/local/bin/pylint

so had to restart my shell to pick up the new version: -

pylint --version

pylint 2.15.3
astroid 2.12.10
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]

which is better ( part of my planned change will be to have the Travis job use 2.15.3 as well )

Right, onwards for some linting .....

Saturday, 10 September 2022

Been a while - back with some wget fun

Whilst digging into some Kubernetes testing, validating a fix that I'd made to my cluster, I was deploying both Nginx and Busybox pods.

The Nginx pods were deployed across all my Compute Nodes, to provide a web server tier, and I was using Busybox, again across all the Compute Nodes, to validate that Nginx connectivity was clean and green.

Having obtained the individual IPs of each of the Nginx pods- I'm using Calico Node as my Container Network Interface (CNI) layer - I wanted to hit each Nginx pod from each of the Busybox pods, in turn.

Now, ordinarily, I'd use a command such as: -


to retrieve the sample HTML page that Nginx presents.

However, Busybox is a very cut-down Linux-like environment and, therefore, it doesn't include the curl command.

Thankfully, Busybox does include the wget command, so I was able to use that in a similar manner: -

wget -q --output-document -

which did the trick i.e. dumped the content of nginx.html to the console ( stdout ).

The key parameters are as follows: -

-q == run wget in quiet mode, only returning the required web page, rather than the normal debug

--output-document == tells wget what and where to return the output

- == tells --output-document to write to stdout rather than a specific file/document

Nice !

Friday, 22 July 2022

Grokking grep

A colleague was tinkering with grep and, thanks to him, I discovered a bit more about the trusty little utility.

I had not really explored the -e switch: -

     -e pattern, --regexp=pattern

             Specify a pattern used during the search of the input: an input line is selected if it matches any of the specified patterns.  This option is

             most useful when multiple -e options are used to specify multiple patterns, or when a pattern begins with a dash (‘-’).

but he pointed out that that switch can have problems when parsing, say, a list of strings that are similar but different.

As an example, I have a set of files: -

-rw-r--r--   1 hayd  wheel    0 22 Jul 17:34 dave
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:34 dave_1
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:36 dave_2
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:36 dave_3

If I want to e.g. query a directory for all files containing the word 'dave' I could do this: -

ls -al | grep dave

or, to be more specific: -

ls -al | grep -e Dave

both of which return: -

-rw-r--r--   1 hayd  wheel    0 22 Jul 17:34 dave
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:34 dave_1
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:36 dave_2
-rw-r--r--   1 hayd  wheel    0 22 Jul 17:36 dave_3

However, if I only want to return the file that's named 'dave' and ignore the rest, I'm somewhat stymied.

However, grep -w comes to the rescue: -

     -w, --word-regexp
             The expression is searched for as a word (as if surrounded by ‘[[:<:]]’ and ‘[[:>:]]’; see re_format(7)).  This option has no effect if -x is
             also specified.

so I can run: -

ls -al | grep -w dave

and just get this: -

-rw-r--r--   1 hayd  wheel    0 22 Jul 17:34 Dave

Isn't that nifty ?

Wednesday, 20 July 2022

To start, press any key .... hey, where's the [Any] key ?

So I was writing some scripts for a demo that I delivered earlier today...

One script will run on my Mac, using the Zsh shell, the other will run on an Ubuntu box, using Bash.

In both cases, I wanted to pause the script and wait for user input before proceeding.

On Ubuntu, I did this: -

read -p "Press [Enter] to continue"

Sadly, however, on macOS 12.4 via zsh, that didn't work ...

read: -p: no coprocess

Thankfully, there's a better / different way: -

read "?Press [Enter] to continue"

which is nice

Today I Learned - Shorter redirection, what's not to like ?

So I write a lot of scrips for Bash and Zsh, and often make use of redirection where, for example, I generate a variable by running another command within parentheses.

As a specific example, I'd query my Floating IP and persist it to a text file for future use: -

ic is floating-ips --output JSON | jq -r '.[].address' > ~/floating_ip.txt

Later on, I'd look to use that Floating IP for, for example, a SSH connection, as follows: -

export floating_ip=$(cat floating_ip.txt)

ssh root@$floating_ip

Authorized uses only. All activity may be monitored and reported.

root@'s password:

However, one of my colleague's did point out that I could reduce the number of characters typed when setting the floating_ip variable: -

export floating_ip=$(< floating_ip.txt)

See, that saves a whole two characters ....

Mind you, I can also reduce things further via: -

ssh root@$(cat floating_ip.txt)

I love shelling !

Friday, 29 April 2022

TIL - read-only variables in Linux

 A co-worker was seeing an exception: -

 line 8: TMOUT: readonly variable

when trying to SCP a file from a remote Linux box.

I did some digging and found a RedHat article: -

Why does it prompt "bash: TMOUT: readonly variable" when sudo'ing or ssh'ing to the system? 

that said, in part: -

The TMOUT variable is usually defined read-only to avoid users from unsetting or modifying its value. Due to this it's not possible to set it twice.

I reproduced the problem on my Ubuntu 18.04 box: -

-bash: TMOUT: readonly variable

with a pair of scripts: -

The first script: -

cat /etc/profile.d/

readonly TMOUT=500; export TMOUT

sets TMOUT as readonly

The second script: -

cat /etc/profile.d/

TMOUT=600; export TMOUT

then tries to override it

which I validated: -

fgrep -R TMOUT /etc/profile.d/

/etc/profile.d/ TMOUT=500; export TMOUT
/etc/profile.d/; export TMOUT

I left my colleague to dig into /etc etc. and see what was going on, but TIL about read-only variables 

More fun with pip

Again with the Python and pip  fun, this time on my Mac, where commands such as:  pip3 list and: - pip3 install --upgrade pip were failing w...